Internal documents obtained by the American-Statesman show that last year, even as government overseers were taking the VA to task for failures in other high-profile IT projects, VA officials worried that the department’s $543 million contract with Hewlett-Packard Enterprise Services to implement a real-time locating system, or RTLS, was careening off the rails.
The system, which consists of tagging and wirelessly tracking everything from catheters to hospital beds, has been hailed as a way to potentially save millions of dollars in lost or misplaced equipment.
The VA has also vowed the project would prevent death and disease from unsterilized equipment, a persistent problem at the VA. The contract was awarded in 2012, soon after colonoscopies with dirty equipment led to a rash of hepatitis and HIV infection at veterans hospitals in Florida and Georgia.
But the contract has been beset by a host of problems, including failed operational tests, questions over the reliability of equipment tags and fundamental concerns over whether the department’s WiFi can support the system, according to thousands of pages of emails, reports and documents obtained by the American-Statesman using the Freedom of Information Act.
The previously undisclosed setbacks with the tracking system underscore the department’s larger information technology challenges as it embarks on its most ambitious IT project to date: replacing its decades-old VistA medical record system. At an estimated cost of up to $16 billion, it represents the first major IT test of the department’s new leadership under the Trump administration.
The disclosures also come four months after the U.S. Government Accountability Office labeled information technology a “high risk area” in the VA, and new House Veterans Affairs Committee chairman U.S. Rep. Phil Roe, R-Tennessee, announced an inspector will examine the troubled IT department.
In a statement to the American-Statesman, Roe called the RTLS problems “very concerning.” The issues “fit a 20-year pattern of VA struggling to manage its supply chain,” Roe said.
The tracking system should have gone live in VA facilities across the nation by June 2017. Central Texas VA officials have said work here stalled about 18 months ago due to contract issues. Officials say they have since fixed many of the problems and plan to launch the project nationwide by June 2018.
The failure to implement the project in a timely manner has already contributed to at least one troubling situation. Following up on a complaint, VA internal investigators visited the Washington, D.C. Medical Center in March, where they found the hospital had no functioning inventory system and was unable to ensure its equipment was properly sterilized.
That led to a series of supply-related crises: in recent months the hospital has run out of bloodlines for dialysis patients, clip appliers to close blood vessels during surgery, and supplies as basic as alcohol pads and wound dressings. The hospital has been forced to cancel prostate biopsies and in at least one case, a surgeon used expired surgical equipment.
In all, the medical center had $154 million in supplies that were not being inventoried. The audit also found that employees were shunning another troubled IT program, called Catamaran, that manages supplies through predictive analytics. The program, the result of a $275 million contract with Houston-based Shipcom Wireless, was terminated last year, according to documents obtained by the Pittsburgh Tribune-Review, which reported that employees complained of frequent crashes that made it difficult to order supplies.
“The Medical Center placed patients at unnecessary risk by failing to ensure that appropriate medical supplies and equipment were available to providers when needed,” investigators wrote.
The D.C. facility may not be an outlier. “These are the same supply chain problems we are seeing at other facilities,” Roe told the Statesman.
When it was awarded in 2012, the RTLS contract represented an opportunity for the VA to shed its reputation as an old-school tech dinosaur and chart a path toward what leaders were calling a “transformation initiative.”
The VA was in dire need of some tech transformation.
The same year the contract was awarded, the VA’s inspector general warned that despite calls for digitalization, thousands of pages of paper benefit files were stacked so high in VA offices that they were compromising the structural integrity of buildings.
A year earlier, the VA’s nine-year, $127 million effort to modernize its outpatient scheduling software was declared a bust. In 2013, the VA and Department of Defense abandoned their effort to merge medical records, after spending $564 million. Today, the VA spends more than half of its $4.3 billion IT budget on maintaining increasingly archaic legacy systems.
The RTLS, on the other hand, was a foray into “The Internet of Things,” a cutting-edge technology with a track record of making factories, warehouses and hospitals more efficient. The Hospital Corporation of America credits its RTLS with reducing time spent looking for equipment by 75 percent.
Driven in part by growth in medical facilities, the RTLS industry is expected to expand five times over between 2015 and 2022, to more than $8 billion.
Industry experts predicted the massive VA contract would be a boon to the nascent RTLS field. “It’s pretty rare that something this big comes along to jump-start a whole industry,” said Marcus Ruark, former vice president of RTLS subcontractor Intelligent InSites, Inc., in 2013.
The system doesn’t just help busy emergency rooms quickly find a needed piece of equipment like an IV pump or ensure it has enough beds available when patients arrive. Tags can also measure temperature, air pressure and humidity, so hospitals can remotely monitor refrigerators that hold temperature-sensitive medications and tissue and get quick alerts on water leaks in critical areas.
“RTLS technology is as transformative to hospital operations as Uber is to personal transportation,” wrote Ari Naim, CEO of RTLS developer CenTrak last year.
Critically, they can track equipment through the sterilization process, preventing an issue that continues to bedevil the department. As recently as October, nearly 600 veterans in Wisconsin were placed at risk of Hepatitis and HIV after it was discovered a VA dentist failed to correctly sterilize his equipment.
At St. David’s Medical Center in Austin, officials say their RTLS system has transformed operations since it was implemented in 2010. “Once you get the backbone of the system in place you can start growing it and adding devices,” said Marty Martinez, facilities engineering operations manager.
In addition to monitoring equipment, the hospital uses RTLS to monitor blood and tissue in its freezers and fridges and the system can send out alerts if it detects a problem.
“It’s increased our reliability,” said facility management director Mike Ridgway. “The question is, What did we prevent? How many times did we catch a water leak immediately or a fridge going out? It’s just helped us become much more efficient.”
Documents suggest the development of a nationwide tracking system at the VA has been besieged by problems since work began.
Six months after the contract was awarded in December 2012, the VA complained that tags were too wide for smaller pieces of equipment and “could be knocked off upon contact.”
In September 2013, the VA declared Hewlett-Packard’s underlying data architecture “unacceptable.”
Then, just as the Government Accountability Office was releasing its postmortem of the failed scheduling software and record merging project, the VA began large-scale testing of its RTLS tracking system.
The March 2015 test was an overwhelming failure: the system could only accurately track equipment in 40 percent of cases (a percentage disputed by Hewlett-Packard). In the majority of cases, the system could not pinpoint location and even misidentified the floor equipment was stored on.
The dismal performance was “endangering performance of the RTLS contract,” VA officials at the department’s Technology Acquisition Center in Austin warned the firm, threatening to cancel the contract if Hewlett-Packard didn’t make quick fixes.
Hewlett-Packard in turn blamed the VA’s inadequate wireless Internet infrastructure for the problems.
The company told the VA that 137 WiFi access points were non-operational and blamed the VA for cutting corners in trying to implement a cutting edge RTLS system while using faulty WiFi.
The root problem for the failures lay in “performance errors that it appears VA has failed to take responsibility to correct,” Hewlett Packard wrote, adding that the company “should not be held accountable for the VA’s WiFi limitations.”
By mid-2016, frustrations were boiling over. When a VA supervisor refused to let HP access its cloud-based back-up systems, another VA employee wondered why in an email chain. “I was also surprised,” a third VA employee wrote in an email. “But I read it as, I am 100% fed up with these nitwits and I’m not going to give on anything anymore.”
The VA later issued a stop work order and re-negotiated new or modified contracts with Hewlett-Packard. The VA downplayed the issues mentioned in internal documents, telling the Statesman that the stop work order “allowed the VA to refocus efforts toward delivering capabilities in an incremental approach beginning with those that were more mature” and “provided both parties the opportunity to finalize the path forward and realign the schedule to achieve those goals.”
“In a technology project of the scope and magnitude of RTLS, it’s common for those involved to periodically reassess the program and realign the approach in order to achieve the desired outcome,” VA officials said.
Hewlett Packard Enterprise Services, which has since become DXC Technology, gave a similar explanation. “With any large-scale project, priorities and available capabilities evolve over time and often the prudent action is to take a step back to give both parties an opportunity to re-align before moving forward,” a company spokesman said in a statement. “We are aggressively working toward the full RTLS implementation.”
After a new round of testing, the plan is to release “the fully integrated solution” by June 2018. The VA says a host of issues have been resolved, including equipment tags and the WiFi limitations and the system has been installed in a dozen facilities.
But just a week after the VA and Hewlett-Packard resolved the stop work order, concerns over testing continued, documents show.
On September 15, an outside consultant expressed concern over testing of of sterilization procedures, pointing out that Hewlett-Packard was conducting less “rigorous” testing than the VA had advocated.
“HPE would fail according to VA’s definition,” the engineer wrote. “But we already bought into the way they did this…because we thought we couldn’t win that fight, so we are probably obliged to accept it now, too.”
Roe has made IT contracts a key part of the House Committee on Veterans Affairs’ oversight of the department and tech challenges were the subject of the committee’s first hearing after the new Congress was seated in January.
In June, VA Secretary David Shulkin announced the VA would purchase on off-the-shelf system to replace its decades-old, homegrown VistA health records system. The VA, he said, would use the same vendor that is building the Department of Defense’s new records system.
The decision would allow the two departments to more easily sync patient records and was an acknowledgement that the VA’s attempts to build or fix systems in-house weren’t working.
Some lawmakers grumbled that in announcing it would purchase from the same vendor used by the Department of Defense — Missouri-based Cerner Corp. — the department was losing the pricing leverage of a bidding process. While the Defense Department is paying $4.3 billion for its system, the much larger VA’s price tag could come out to four times that amount.
But as the experience on the RTLS project shows, hiring a commercial vendor to build a new system does not ensure smooth sailing.
The chairman has called this year and next “pivotal” for the VA’s IT department, noting that it is trying to fix problems after years of failure. “Now,” he said during a February committee meeting, “is the last chance to get them right.”
©2017 Austin American-Statesman, Texas Distributed by Tribune Content Agency, LLC.