Articles

The Cloud Builders: Ann Arbor and Washtenaw County, Mich.

Ann Arbor, Mich., CIO Dan Rainey details the many benefits the city and Washtenaw County have seen from sharing data centers and systems.

by / January 9, 2012
The Michigan Theater in Ann Arbor. Photo courtesy of "Andypiper" / Flickr CC

In 2005, CIO Dan Rainey, newly on board in Ann Arbor, Mich.,  met Washtenaw County, Mich., CIO David Behen, who last year became Michigan’s state CIO. “During our first meeting,” said Rainey, “one of the things Dave brought to my attention was that we’re a block apart in location, we both run data centers, we both have IT staff — why aren’t we doing more together?”

Later, Washtenaw County’s administrative building was flooded. Officials begin looking at building a new data center. “And that was the opportunity,” said Rainey. In spring 2009 the city and county co-located their data centers. “Not long after that we built a new justice center,” said Rainey, “and as part of that new building, there was a space for including a new data center — raised floor, air conditioners, [uninterruptible power supply]: The things you really need to have in a modern data center. That’s when we moved everything together into this new data center.”

The data center is now physically managed by the city, but the county has full access to it. The arrangement has opened additional opportunities for shared services.

“We share a storage area network,” said Rainey. “We share an enterprise backup environment, Internet connectivity, enterprise content management system and GIS environment. We’re still operating as separate departments, but we’ve partnered on a lot of initiatives where it made sense. We put together an interagency agreement that spells that out. And when we wrote this agreement, we purposely used terms like ‘participant,’ so it’s open to any local unit in the county that wants to participate. And they’re not making a commitment to anything beyond, ‘We’re part of the conversation.’”

Subscribers and Providers

There are subscribers and providers, said Rainey. “The county is the provider on the SAN, and the city is the subscriber. The county leases it and we pay them for our half of it every year.” So the city might make a three-year commitment. But Rainey said jurisdictions should make sure they can do that type of leasing model before attempting it. For certain types of investments, like a storage area network or a large backup environment, you need to make sure you have both organizations agreeing to the financial commitment beforehand, he said. If one of the stakeholders refuses to appropriate funds, that could be a problem.

“If I want to be a provider of a service,” said Rainey, “I will approach the other participants and say ‘Hey, we have this new system we think we want to make available to other interested local units. Who’s interested?’ Then when they sign on as subscribers, we’ll talk about the terms around that specific service: the length of it, the cost for it, service levels around availability, who’s responsible for what — how you get out of it if you want to get out of it.

“The No. 1 rule,” said Rainey, “is to have an exit plan. You need an understanding of how you get into an agreement and how you get out of it. What are those expectations?

“Right now we have three agencies that are part of this interagency agreement: the city of Ann Arbor, Washtenaw County, and the Ann Arbor Transportation Authority. So the three IT directors of those agencies meet monthly and go over how the services are working out. Are there issues or new needs? And what are we all working on? So we share our high-level plans too. If I think I might want to buy an intrusion detection service, I might ask, ‘I’m looking at this, is there anybody that would like to look at the same thing?’”

The partners are also coordinating on larger issues, like a Windows 7 rollout, application virtualization and more.

Return on Investment

“Together, the arrangement saved a lot of money," said Rainey. “Washtenaw County was looking to build a new data center, and that was a $2 million capital project that they didn’t have budgeted anywhere. So that cost was avoided. And then, on the storage area networks there’s a cost just to bring in a platform and incremental cost to add more storage to it, so it saved us a good $300,000 just on SAN and the backup platform because we collaborated instead of [Ann Arbor] and the county buying their own.

“We have informal agreements to provide backups to each other and the equipment’s all in one data center. So where we each have a single resource that has expertise, we’ll learn each other’s systems — like in our backup environment. We each administer our own backups, but the county person can go on vacation now because [the city’s] staff will watch. People would go on vacation, but they’d get called and hunted down; now that doesn’t happen, so it’s provided some relief.”

Another opportunity occurred with content management.  “We partnered with Hyland Software, which makes OnBase,” said Rainey, “an enterprise content management system. The county had that — they purchased OnBase probably in 2007 — and the city decided it was going to use that same product.” So Ann Arbor used the county’s same RFP, the same integrator, and worked with Hyland. The company licensed the city and gave the city essentially free access to the county’s infrastructure. All Ann Arbor had to pay for was additional seats that were procured. “That saved me at least a half million dollars in the last 18 months,” Rainey said. “Because if you’re going to do an enterprise content management RFP, that takes three or four months to get in place, then you have to put it on the street. Then you’re going to be looking at OnBase and EMC and IBM for months, then you have to figure out who you’re going to go with, then you have put that in front of the City Council.” And that’s all before you install it, which amounts to a long project.”

Instead, Rainey was given a one-page agreement. ‘You’re now an additional user on the county system, sign here,’ the document said. “We went to deployment right away. With these collaborations, you can find vendors that will support that [simplification]. Don’t be afraid to ask the vendors to help out. We’ve been OnBase customers now for going on three years and we’ve gone very far, very fast because we didn’t have to spend time building the infrastructure.

“One of the direct benefits to both organizations,” said Rainey, “is we’ve been able to do some other business consolidations because the technology group has already worked together, and worked through a lot of the bottlenecks and barriers. So when the Police Department and the Sheriff’s Department wanted to put up a consolidated dispatch, technology wasn’t the issue. The phones worked, the network worked, so the first thing they did was to co-locate the county and city dispatch.”

Suggestions

Rainey has some suggestions for avoiding problems. “In the business of infrastructure consolidation, these are systems that must run 24 hours a day, seven days a week. They’re supporting dispatch and other public safety issues, so when you start these partnerships there’s usually concern that somebody won’t hold up their end of the deal. And that’s largely because they don’t know each other.” So you have to get people together and give them a chance to become acquainted.

Rainey says that the city IT shop isn’t unionized and the county shop is. “So where labor conflicts might arise, you need to understand if there are issues there and take care of those things before you get the people together.”

Rainey said that governance is managed internally. While a situation can be escalated to the next level of managers and then to the CEOs of the organizations if necessary, the interagency agreement handles most issues that arise. Key to success is that all organizations must benefit from the agreement.

“We haven’t got to the point of having co-located staff,” he said. “It’s hard to do that partnership at the managers’ level without having the staff in the same physical location. It's difficult for a manager to have two staffs in two separate physical locations.”

Since the consolidation, said Rainey, there have been three leadership changes in the county IT director position, but the consolidation has continued because of the good working relationship between city and county. “I’ve gotten to know the administrator, their commissioners, and they’ve gotten to know us too. And that’s why it’s important to have the higher-level support. And having the interagency agreement in place helped out too, as a basis for the continuity.”

It’s all about the people, said Rainey. The technology isn’t the major challenge. In several respects the consolidation had some advantages at the outset. “We were lucky. The county was one block away. The city owns the streets, and the infrastructure is in the streets, so it was really easy for us to move fiber.  In different organizations – where they have telco infrastructure, for example — it may be more difficult.

Next week: The city of El Paso and El Paso County, Texas, share their experiences working together.

Wayne Hanson

Wayne E. Hanson served as a writer and editor with e.Republic from 1989 to 2013, having worked for several business units including Government Technology magazine, the Center for Digital Government, Governing, and Digital Communities. Hanson was a juror from 1999 to 2004 with the Stockholm Challenge and Global Junior Challenge competitions in information technology and education.