Standing in the Spotlight

Reprinted from the Fall 2004 issue of

The word "CompStat" signifies a turning point in public-sector performance. In 1996, then-New York City Police Commissioner William Bratton launched a radical new way of combating crime. Out went the heavy reliance on 911 calls, which locked police officers into a reactive role. In came an aggressive policing strategy that focused on quality-of-life crimes coupled with technology that tracked arrests and crime statistics by week, day and hour.

For the first time, precinct captains were held accountable for driving down crime in their jurisdictions. Those who couldn't meet the new performance measures were removed from their commands. While civil libertarians objected to what they saw as abusive tactics, there was no denying CompStat's role in helping reduce New York crime. By 2002, the city's crime rates dropped to levels not seen since the 1960s. Today, variations of CompStat can be found around the country and the world. It is considered by many to be one of the best examples of how performance measurement works in the public sector.

"CompStat has been so successful for performance measurement," said Steve Kelman, professor of public management at Harvard University's John F. Kennedy School of Government. "It has really put some fire under the management practice." According to Kelman, performance measurement is now taught to all master's students at the Kennedy School.

Helping drive CompStat's success is, of course, technology. Databases massage the information collected by police. Geographic information systems track crimes based on time, location and other factors. Modeling tools help police commanders answer "what-if" questions concerning changes in manpower, patrols and other assets.

The story of IT and performance measurement could end there, with CIOs playing an important but secondary role to the police commissioner, labor secretary or social service director who must manage performance in today's outcome-based, value-driven public sector. CIOs, however, are also under the gun to show results.

The reasons are somewhat paradoxical, according to Amy Santenello, senior research analyst at Meta Group. "IT is increasing in importance for the public sector, but technology has disappeared as a priority among policy leaders," she said. "To demonstrate the value of IT, CIOs are using performance measurement to show they can deliver results and improve services."

At the federal level, the President's Management Agenda made performance measurement a priority, according to Thom Rubel, vice president of government strategies at Meta Group. At the state and local level, the fiscal crisis forced CIOs to be more accountable and show results. Despite these pressures, use of performance measurement continues to be spotty in the public-sector IT community, according to Rubel.

Where it is used, the terminology often differs. For example, some CIOs make performance metrics part of their overall strategic plan. For others, it's a requirement for building an enterprise architecture. Still others see it as a tool for making the business case for a new infrastructure, application or service.


But Is It Shareholder Value?
Mention performance measures, and what you often hear is something very specific, tangible or numeric, according to Genie N.L. Stowers, professor of public administration at San Francisco State University. "Number," "percent," "ratio," "incidence," "proportion" or similar words are often used, according to Stowers' report, Measuring the Performance of E-Government.

The private sector primarily uses profit to measure performance along with other nonfinancial measures. By comparison, the public sector has struggled to come up with a viable and meaningful way to measure performance, but Stowers identified many --inputs, outputs, activity measures, efficiency, and productivity measures or outcomes. For CIOs, inputs might mean application development and hardware costs; outputs could be system response time












Mixed Reviews






or the number of electronic permits processed if it's an e-government application. Adoption rates and cost savings are examples of outcomes. CIOs can use technology to help measure performance in their departments. Software firm SAS combines performance management techniques with software tools to help government agencies measure results. The key, according to Brian Rowland, performance management strategist for SAS, U.S. Public Sector, is to take a life-cycle approach to the problem. Start with information, then integrate it. "Integration is crucial, especially for high-level executives who have to look beyond the information silos that exist in government," he said. The next stage is to put the information into context so it can be viewed and understood. This is where tools, such as analytic and business intelligence software -- often called dashboards -- come in handy. "The dashboards can do the performance measures for the executives," Rowland explained. Finally there's collaboration. "This is when you push the results out and share them internally or externally," he added. What's missing from all these measures is one for value, say experts. "As a CIO, you really want performance measures for the value the [computer] system is creating," said Kelman. "What good is a system with 100 percent availability if it's not helping the organization do its work better?" That problem may be solved, now that performance measurement has moved to the forefront of key public-sector IT issues. The consulting firm Accenture recently looked at the issue and created a Public Sector Value Model. The private sector can measure shareholder value, which is pretty consistent no matter what industry you are in, according to Stephen Rohleder, Accenture's chief executive for its Government Operating Group. Public-sector value, which looks at cost-effectiveness as well as outcomes, varies depending on the service delivered, whether it involves a welfare worker, a sanitation engineer or a network administrator. Rholeder said the company has identified seven characteristics that measure high value performance. But before IT departments start measuring value, they must include performance measurement in the goals and objectives set by government, and that isn't easy. For example, Stowers looked at strategic plans from the 50 states and the District of Columbia, and found 67 percent don't contain identifiable performance measures, a key element of almost any type of strategic plan. "At the federal level, there is much more consistency in the development of performance measures, due to the influence of the U.S. Office of Management and Budget," wrote Stowers. But consistency is lacking in how agencies apply performance measurement, especially regarding IT management. In a report released in March, the General Accounting Office (GAO) gave a mixed review on the use of performance measures and strategic planning by federal agencies. "Agencies had IT strategic plans and goals, but they were not always linked to specific performance measures that were tracked," according to the report. "Without enterprise performance measures tracked against actual results, agencies lack critical information about whether their overall IT activities are achieving expected goals." The GAO recommends government IT managers use an information technology investment management (ITIM) framework, based on best practices, for evaluating IT performance. "The framework identifies and organizes critical processes for selecting, controlling and evaluating IT investments into a framework of increasingly mature stages," according to the GAO. That means agencies can follow a prescribed course for managing IT investments, using performance measurement to evaluate results at key junctures of the project. One federal agency that used performance measurement with great success is the U.S. Department of Labor. In 2003, it received a "B" grade for its efforts on computer security. Only three agencies scored higher, while the overall grade for security was a "D," according to the Subcommittee on Technology, Information Policy, Intergovernmental Relations and the Census.












Accountability Versus Performance












The federal agency was also recognized for the success of its GovBenefits.gov service and deployment of a common e-mail system that incorporated three different platforms and 20,000 accounts. The department used the e-mail project as a model for several other enterprise initiatives, including electronic procurement, an enterprise property tracking system and identity management via public key infrastructure. The Labor Department's IT office uses a review board to ensure technology projects stay on track and perform to specification, according to Department of Labor CIO Patrick Pizzella. "In fact, we track everything internally, using the scorecard method to measure progress," he said, noting that the department has used performance measurement for its IT projects since 2001 with considerable payoff. State CIOs are also setting up performance measures based on governors' agendas, establishing investment management plans -- similar to those laid out by the GAO -- that determine a return on investment, and tackling performance measurement across collaborative IT projects. Michigan CIO Teresa Takai is doing all of these and developing a governance model for performance measurement. "Right now, our governance is largely based around tracking individual projects we are working on for an agency. We have a monthly dashboard for tracking projects of a certain criteria, size, visibility and so on," she said. The new governance model -- currently in draft stage -- will expand to include metrics for ROI prior to a project launch, require a project sponsor and criteria for determining whether to track a project within an agency or at the enterprise level. "This governance model will span across all of the state agencies," she said. "It's going to give us an opportunity not only to measure results, but to know where we may need to redeploy resources enterprisewide. If there's a problem, this process will escalate the issue to the Cabinet level or the Governor's Office where it can get resolved." Like other public-sector CIOs, Takai struggles to measure performance on projects where the value delivered is not quantifiable. For instance, changes to a Medicaid computer system might speed benefit delivery and get benefits to more families, but how should citizen value and cost-effectiveness be measured? Another hurdle is accountability. Harvard's Kelman said employee resistance to performance management is a key stumbling block to its implementation and success. "People in government have the impression that performance measurement is an accountability tool. They don't see it dynamically as a way to improve performance," he said. Takai has run into such resistance, particularly with operational metrics. "People are very concerned that they are going to be evaluated on those metrics," she explained. The problem is particularly acute when performance measurement involves IT projects that span more than one agency or group. "If one of those groups is a bottleneck, the other groups don't want to be judged because somebody else didn't do their job," she added. The bottleneck problem leads to another issue, namely collaboration and metrics. As Takai has found, performance measurement in IT is difficult to execute when the project involves more than one agency. The solution, said Meta Group's Santenello, is proper governance among CIOs, something Takai and others are in the early stages of developing. Lack of fresh data on performance can also hinder the measurement process, experts say. "You can't do this once a year; it has to be part of the daily environment," said Kelman. "That's how it's done in the private sector." Despite these hurdles and the complexities they add, public-sector CIOs should embrace performance measurement because it will bolster their effectiveness as leaders. Without it, they can't deliver results to the government enterprise, cautioned Meta Group's Rubel. Kelman concurred. "Performance measurement is a new way of thinking for CIOs, and it ties the CIO to program success. It makes the CIO less of a technologist and more of a strategic partner." And that's important in today's changing, results-oriented public sector.