Second Time's a Charm

San Diego County applies hard-earned lessons in outsourcing IT to selecting a new outsourcing partner.

by / June 2, 2006 0
San Diego County will be among the first in the nation to switch IT outsourcing partners. Over the next year, Northrop Grumman will incrementally replace Computer Sciences Corp., (CSC) as the county's IT provider.

The county's executive staff took a risk that almost failed when the county started outsourcing its IT in 1999. The partnership ultimately succeeded, and county leaders applied hard-earned wisdom from the seven-year experience to negotiations with Northrop Grumman. The county promises its freshly minted agreement is bulletproof against the types of misunderstandings that almost sank its first attempt at outsourcing IT.

Other local governments hoping to successfully outsource the first time can gain insight from San Diego County's lessons learned the hard way, such as how not to write an outsourcing contract and how not to handle publicity.

Pre-Emptive Lessons
County executives knew that to pre-empt potential battles with Northrop Grumman, they would need a specific, unequivocally written contract to prevent squabbling over interpretations of responsibilities. Michael Moore, CIO of San Diego County, said the county purposely left many provisions vague in the CSC contract thinking the vagueness would create flexiblility in unanticipated circumstances.

Ironically, this vagueness actually caused those unanticipated problems.

For example, the county and CSC disputed interpretations of a provision concerning PC refreshes. The provision merely said CSC would replace PCs every 36 months. The county assumed CSC would conduct a staggered deployment, refreshing 230 PCs every month for three years, until every employee had a new computer.

CSC, however, wanted to do a single deployment for the entire county every 36 months, said Moore, which would have caused a lengthy countywide disruption.
"If you wait till the 36th month, you can't do it. You can't change 12,000 PCs in a month. CSC wanted to wait because PCs cost a lot of money, and that's capital they have to expend," Moore said.

"At month 36," he later added, "they came in and said, 'We're ready to refresh,' and the county said, 'Well, we can't put 12,000 PCs in at once,' and they said, 'Well, then it's not our fault.'"

On its end, the county failed to create a new image for the new PCs -- an important part of the refreshing process -- and soon, there were talks of waiving the refresh requirement altogether.

"CSC wanted to go ahead and put the new PCs out there without the image on them, but the county wouldn't let them," Moore said. "Now you're arguing about who is at fault for not doing the refresh."

The county and CSC also eventually engaged in monetary disagreements. A provision, which stated CSC would provide extra bandwidth on demand, failed to specify who would cover the cost. Each party believed the provision placed financial responsibility on the other.

These skirmishes and nearly 50 others like them prompted the county to send CSC a letter of default in 2002.

"We were just at wit's end, and both sides decided that the agreement was going to fall apart," Moore said.

However, both wanted to avoid a lengthy court battle, especially CSC, which had signed a no-suspension-of-services contract.

"Even when we were arguing about this, they couldn't stop, and the county had withheld money," Moore said, adding that this explains why CSC didn't want to agree to a no-suspension-of-services clause again.

Both sides came to an out-of-court agreement in July 2002 that allowed San Diego County to withhold $15 million annually from CSC's service charges and set up behavioral "milestones" for the company to reach before receiving any money.

The county's milestone authority enforced a staggered PC replacement cycle and numerous other procedures fought over before the negotiation. CSC and the county settled the roughly 50 major disagreements and haven't had problems since, said Moore, adding that the wording for those procedures was made crystal clear in the Northrop Grumman contract.

Drawing the Line
The county's list of non-negotiable terms and conditions was the deciding factor in choosing Northrop Grumman over two other bidders, CSC and IBM. Moore said the winning bidder had to accept the county's no-suspension-of-services clause, which ensures the company won't stop output for any reason.

"The fundamental concept of outsourcing in the public sector [is] that no matter what happens, you can't stop work," Moore said. "We can be arguing about payment, we can be arguing about scope, we can be arguing about a lot of things, but in the meantime you still have to deliver services. We're a public agency, and we can't have the IT provider saying, 'Well, you know, we're arguing about a payment on a bill, so we're not going to provide any service.'"

Clauses relating to "disentanglement" in the event the county changed providers at the end of the contract were also heavily weighed. The county demanded perpetual rights to anything the bidder might use in delivering services.

"If a company is bringing some proprietary software in, they have to agree, when we sign the contract, that they're going to turn over that proprietary stuff to us at the end of the deal so we can give it to the next team," Moore said.

He said the winning bidder also had to accept the county's lawsuit liability cap -- another giant risk for the bidder.

"If we get sued over an IT issue, the cap on the liability is $100 million per incident," Moore said. "If it's a security breach, [meaning] if somebody's financial data or health data is breached by the outsourcing provider, that cap goes up to $130 million."

Moore said some conditions actually remove the cap all together.

"In other words, you're betting the company," Moore said.

San Diego County had six of these "fundamental" terms and conditions, and Northrop Grumman was the only provider that agreed to all of them. Moore said CSC's refusal of the no-suspension-of-services clause, in particular, lost it the contract.

"In the current contract, they are already signed up to that provision. So it was interesting that they actually were going backward from what they had already agreed to in 1999," Moore said, adding that it's an onerous contract, so if the county withholds payment, for whatever reason, the provider doesn't get paid. "And they have to continue to provide service -- that's a very difficult position to be in as a company.
They didn't like having that provision in the current contract, so they wanted it loosened up for the new contract."

Opening the Closet
Luckily for Northrop Grumman, San Diego County is already outfitted with updated technology. The company will avoid the stress of refurbishing massive amounts of infrastructure, unlike CSC in 1999. Walter Ekard, chief administrative officer of San Diego County, emphatically warns any government attempting to outsource IT to make sure its outsourcing partner does a thorough quality analysis of its technological infrastructure.

"I don't know that the vendor initially realized how bad our infrastructure was here," Ekard said.

He said system failures increased after outsourcing because often, when CSC tried to fix or replace a piece of technology, another unanticipated malfunction existed in the same area.

"It's like opening a closet that you knew was there, but you never really looked in it, because you didn't think it was important," Ekard said. "You open it and find all kinds of things you didn't expect."

The county's drastic technological disparities between agencies hurt service for those that started with adequate technology when outsourcing began. CSC was so busy updating agencies with no technology that it was slow to serve those already equipped with it.

"Our probation department, when we outsourced, was still on three-by-five cards, and yet we had other departments with state-of-the-art stuff," Ekard said.

He said systems in the Assessor's Office malfunctioned constantly, a dire problem for citizens trying to record documents with legal and financial implications.

"The public would come in and record a document, and they couldn't record it," Ekard said. "Employees would get frustrated."

"We already had labor fueling employee discord over the issue," he added, "so it became easy, frankly, for employees who were used to doing things a certain way -- and probably rightly so -- to complain about the fact that, 'Hey, you hyped this contract as a way to improve how we provide services to the public, and frankly, things are worse than before.'"

From Public to Private
Time is the only indicator as to how many San Diego CSC employees will transfer to Northrop Grumman. The percentage will be large because the company will need their expertise in making a smooth transition.

Governments wanting to outsource for the first time can expect hostile county employees when transitioning them from the government's payroll to the private sector's, according to Moore.

"The employee resistance is dramatic," Moore said. "Employees don't want to be outsourced. They go to work for a county because they want to work for a county."

San Diego County negotiated for CSC to offer all county IT employees 7 percent raises. Ekard said he thought the transition gave employees an opportunity for upward mobility, and his staff presented it as such to the employees.

"We encouraged the notion that, 'Those of you who transition are venturing into a world where your opportunities are going to be unlimited,'" Ekard said, later adding, "We were attempting to get employees enthusiastic about the fact they were going to work for companies that would recognize their value far more than government did."

Ekard said roughly 70 percent of the county's IT workers moved to CSC.

Diminishing Politics
Northrop Grumman certainly didn't have to combat an angry barrage of union protesters. That county battle had been fought several years prior. However, the politics of outsourcing are diminishing because of the high volume of public-sector employees rapidly approaching retirement, said James Krouse, acting director of INPUT's public-sector market analysis.

"These people are voluntarily leaving the work force. That hits the unions whether [counties] outsource this function or not," Krouse said.

Regardless of the politics, he said, local governments are increasingly willing to outsource because studies repeatedly forecast employee shortages.

"Necessity breeds reasonableness," Krouse said, adding that outsourcing provides an opportunity for counties to make long overdue hardware upgrades. "The hardware that is 25 or 30 years old really can't be Scotch taped together anymore."

Moore said the resolve of San Diego politicians was crucial in making the venture successful, and wobbly politicians caused other local governments to fail at IT outsourcing in the past.

"The unions don't want it to happen, so they put a lot of pressure on the elected officials," he said.

Labor unions waged an expensive media campaign against the project, which the county wasn't financially equipped to counter. But the unions weren't the only ones making noise. County employees and the public in general flooded elected officials' offices with complaints.

The politicians and county executive staff remained aligned throughout the turmoil, however.

Krouse said Virginia is successfully sheltering a recently approved outsourcing initiative from politics by placing it in the hands of a "technology steering committee" made up of legislators, the CIO, CFO and others. Multiple officials representing different government interests will be included in the decision-making process, meaning its success won't be hindered by the volatility of left versus right political squabbling.

It would be difficult for a politician to kill an outsourcing contract if a diverse pack of government officials was already behind it, Krouse said. More state and local governments will likely imitate this decision-making model.

Krouse said he never agreed with the popular opinion that Virginia's outsourcing project wouldn't survive the state's recent election.

"The decision was made across this technology committee, which functionally, to me, removes the reds and the blues from the decision-making model," Krouse said. "You're talking about different folks who are sitting at the table, making the go decision, so why would a change of administration necessarily do anything more than give time for the new staff members to acclimate to the steering structure? The decision was made in a group."

The Right Lawyers
Ekard said he would be remiss if he didn't emphasize that any government wanting to outsource must hire lawyers and consultants specializing in IT outsourcing -- something governments loathe because of the added costs, he said.

San Diego County hired the law firm Gordon and Glickson and consulting firm Gartner for negotiations with both Northrop Grumman and CSC.

"Having world-class attorneys at the table with you negotiating with counterparts that obviously were not scrimping on what they paid their lawyers -- it was vital," Ekard said. "The value of that proved itself repeatedly during the course of the first contract because as we got into the disputes, our contract was written tightly enough that in the end, we basically won. As we went into the second round, we again engaged the same lawyers, and we had learned from some of the things, legally, from the first contract, [that] we needed to do or could do better to make our situation even more bulletproof."
Andy Opsahl

Andy Opsahl is a former staff writer and features editor for Government Technology magazine.