Photo: Sam Nixon, CIO, Virginia/Photo by David Kidd
Virginia CIO Sam Nixon's efforts to improve the state's troubled IT outsourcing contract with Northrop Grumman hit a snag this week when a memory card in a storage area network failed Wednesday, knocking the computer systems of two dozen state agencies out of service.
Officials at the Virginia Information Technologies Agency (VITA) said that 75 percent of servers attached to the storage system had been turned back on and were operational as of Friday afternoon, and that engineers were testing those servers and their applications. The state hopes to have all systems fully operational by Monday, a VITA spokesperson told Government Technology.
But it's still too soon to tell if any of the state's data will be permanently lost as result of the hardware failure, the official said. Nixon told a local newspaper in a story published Friday that experts had found no data was lost except for what was inputted into the system at the moment it failed.
Virginia officials also did not yet know if the outage was a violation of service-level agreements that the state negotiated this year with Northrop Grumman in a revision of the original, 10-year $2.3 billion agreement signed by the state and vendor in 2005. The revisions included increased monetary penalties to Northrop Grumman for failure to perform on "clearly identified and measured performance standards."
Complicating the picture is that EMC is the storage vendor. According to state officials, EMC worked overnight Thursday to restore and replace the affected storage area network, which failed Wednesday afternoon and included failure of the redundancy. The outage occurred in one of VITA's data centers in Virginia suburb Richmond.
Some state agencies' computers remained affected Friday, including at the Virginia Department of Motor Vehicles, which can't process drivers' licenses at customer service offices. For security reasons, VITA is not naming the other state agencies affected by the computer failure.
Data backup had been one of VITA's concerns. Last year then-state CIO George Coulter lamented that the state's computer systems didn't have redundancy backup. Nixon, a state delegate who had a hand in crafting legislation that created VITA, began working on the issue after he was named CIO in March by newly elected Gov. Bob McDonnell. Northrop Grumman and VITA jointly announced in June that they had successfully completed a disaster recovery test that restored "all critical systems in 24 hours and other less critical systems within 48 hours."
A legislative audit in fall 2009 slammed the VITA-Northrop Grumman agreement's performance and prompted criticism from lawmakers. Since then, the two parties have been working together to improve service.