Work Continues on 'Unprecedented' Computer Outage in Virginia

Thirteen percent of Virginia's file servers failed last week, affecting 27 state agencies.

by / August 30, 2010

The Virginia Information Technologies Agency (VITA) and partner Northrop Grumman were restoring data Monday, Aug. 30, as work continued to fix a faulty storage area network that has impacted more than two dozen Virginia agencies.

More details emerged about the outage, which occurred Wednesday, Aug. 25, leaving the Department of Motor Vehicles (DMV) unable to process driver's' licenses, and other state services impacted, such as the delivery of unemployment benefits. Virginia Secretary of Technology Jim Duffey said in a statement Monday that 13 percent of Virginia's file servers failed last week, affecting 27 state agencies.

The storage area network that failed was manufactured by EMC. The faulty hardware has been fixed, with service restored to all agencies except the DMV, the Department of Taxation and the State Board of Elections -- agencies with the state's largest and most complex databases. Other agencies continue to experience minor problems.

"According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented," Duffey said in a statement Monday. "The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred in more than 1 billion hours of run time. A root cause analysis of the failure is currently being conducted."

Restoring the data on the affected servers will be a time-consuming process, Duffey said.

Meanwhile, the Richmond Times-Dispatch reported last week that agencies' Web portals were down and the DMV's 74 service centers couldn't process drivers' licenses. The outage was caused by 228 malfunctioning servers, the newspaper reported.

This computer shutdown is a new black mark for the VITA-Northrop Grumman partnership, which began in 2005. A legislative audit last fall slammed the agreement''s performance and prompted criticism from lawmakers. Since then, the two parties have been working together to improve service.