Northrop Grumman Vows to Find Cause of Virginia Server Meltdown as Fix Nears

Virginia Gov. Bob McDonnell orders DMV locations to extend hours in order to process backlog of driver's license applications.

by / September 1, 2010

Virginia officials said Wednesday, Sept. 1, that the state continues to make progress on restoring data and applications for the Department of Motor Vehicles and two other agencies that are still hindered by a storage area network outage that occurred a week ago.

The Virginia Information Technologies Agency (VITA) and vendor partner Northrop Grumman have assigned staff to test the integrity of the affected data, officials said. The incident, which knocked out 485 of the state government's 4,800 servers, initially cut service to 27 state agencies. Service to all but three of those agencies was restored within three days, according to an announcement Tuesday from Sam Abbate, vice president of the VITA program for Northrop Grumman.

But the DMV is still affected, incapable of processing drivers' licenses at customer service centers because its databases remain down. Gov. Bob McDonnell on Wednesday, Sept. 1, ordered DMV locations to extend their business hours in order to process the backlog of driver's license applications. Officials expect the DMV to resume normal operations on Thursday.

Abbate vowed Tuesday that Northrop Grumman would "learn everything we can from this incident. We will conduct a root-cause analysis, carefully analyze and review the findings, develop lessons learned and make necessary changes.

"We cannot afford to let any vulnerability in the infrastructure go unresolved," Abbate added.

Earlier this week Virginia Secretary of Technology Jim Duffey called the storage area network malfunction "unprecedented," citing uptime data on the model of EMC server that was involved in the outage, which occurred in a VITA data center. Officials also said a failover wasn't triggered because too few servers were involved.

McDonnell has called for an independent analysis of the incident.

The computer shutdown is the latest hiccup for the 10-year, $2.3 billion VITA-Northrop Grumman contract, which went into effect in 2005. Over the past year, the VITA-Northrop-Grumman outsourcing model for Virginia's IT service and management has been subject to criticism from the inside and outside, including from some lawmakers who have questioned the effectiveness and long-term viability of the arrangement.

An audit last year slammed the pact for poor service and insufficient cost control. In response, last spring the newly elected McDonnell restructured VITA and named Sam Nixon as state CIO. Nixon, a former Virginia delegate who had a hand in VITA's creation, this year negotiated revisions to the contract with Northrop Grumman.