The company has also agreed to implement recommended improvements contained in a third-party audit of the system crash, which started because of an unknown electrical fault in a storage area network at the state data center. The restoration of corrupted data was then delayed due to human error and a lack of industry best practices, the assessment found.
Announced by Gov. Bob McDonnell on Thursday, May 19, the package includes $1.9 million in fines, which Northrop Grumman will pay back to the state in “monthly invoice credits” during the next two years. Sixteen of the 26 agencies affected by the outage said they incurred a financial cost because of the computer failure; state officials said the fine fully covered those losses.
Northrop Grumman has also agreed to invest nearly $2.1 million to improve database backup, storage systems, system monitoring, database monitoring, network monitoring and capacity management. The company will also spend another $750,000 on “snap/clone" duplication technology to protect the state’s data.
"I am satisfied that Northrop Grumman has been held accountable and that the commonwealth has been made whole," said Virginia Secretary of Technology Jim Duffey, who negotiated the settlement. "This compensation package will benefit all agencies impacted by the outage and enhance the state's information technology infrastructure."
The computer failure was a high-profile embarrassment for the IT partnership between Northrop Grumman and the Virginia Information Technologies Agency. The prolonged outage in late August 2010 prevented many agencies, including the Department of Motor Vehicles, from processing paperwork.
The $2.3 billion outsourcing deal, which made Northrop Grumman the primary manager and provisioner of Virginia state government’s IT, has been troubled the past few years, as the state and the vendor have struggled to improve service levels and control costs.
Since McDonnell took over the governorship last year, he has enacted a series of reforms and brought in state lawmaker Sam Nixon to serve as state CIO. In April 2010, McDonnell agreed to add $105 million to the contract’s value and extended the partnership with Northrop Grumman through 2019, but stipulated more costly penalties if there were service problems.
The governor said the financial agreement with Northrop Grumman announced Thursday brought closure to the incident.
