One federal agency CIO plans to study IT performance the way baseball general managers scrutinize hits and errors.
Not many CIOs would cite Oakland A's General Manager Billy Beane as their role model. But one of Bill Kirkendale's favorite books is Michael Lewis' Moneyball, which describes how Beane, with limited financial resources, analyzed reams of player performance data to optimize his small-market baseball team's chances of making the playoffs.
Kirkendale, CIO of the Court Services and Offender Supervision Agency (CSOSA) in Washington, D.C., sees the data warehouse and business intelligence project under way in his organization in the same light. "We have constrained resources, too," he said. "I wish we had more money and more staff, but we have to do the best with what we've got. Just like the Oakland A's, we are using the data that's available to us to do better than people using gut judgments."
An independent executive branch agency with an annual budget of $190 million, CSOSA has approximately 1,000 employees monitoring more than 20,000 offenders on probation, parole or supervised release in the nation's capital. The agency also monitors pretrial defendants.
Under the leadership of its director, Paul Quander Jr., CSOSA developed a top-notch case management system and automated risk assessment tool. So when Kirkdendale was appointed CIO in December 2005, he knew he was joining an agency keen on using technology to fine-tune its performance.
A former IT security consultant to federal agencies, Kirkendale also came to lead CSOSA's 40-person team at the Office of Information Technology just as the agency was beginning a three-year effort to build an enterprise data warehouse and business intelligence portal. Quander's objective was to make CSOSA a research-driven organization whose IT tools would allow it to use objective criteria to evaluate the agency's performance.
Although he bears ultimate responsibility for the project's success, Kirkendale also has a few luxuries that many CIOs do not. For one, he has in Quander an executive he considers a visionary. "He is technically sophisticated and can ask smart questions about the project just like a skilled IT person," Kirkendale said of his boss.
Finding a Single Version of the Truth
Perhaps more importantly, leading the project is a group of researchers who understand data measurement. In 2002, CSOSA created an Office of Research and Evaluation (ORE) to provide trend information on offenders to community supervision officers and their supervisors. Led by Calvin Johnson, who has a Ph.D. in criminology, the office also includes social scientists and statisticians.
"We have our hands on a lot of data naturally," Johnson said, "and spend a lot of time thinking about how to disseminate it."
Because the agency used data integration and business intelligence tools from several vendors, including Business Objects, Microsoft and SAS, the agency ran into problems with data quality and consistency of measuring individual and team performance.
Employees felt tugged in different directions because their immediate supervisor might use one set of measures to assess their work while the agency's executive team used a completely different set of standards.
"We had to get to a single version of the truth," Kirkendale explained.
After consultation with a systems integrator and the Gartner Group, in early 2005 CSOSA executives standardized on the SAS 9 Intelligence Platform, including data integration, business intelligence and analytics tools. The agency developed a tool called Supervision and Management Automated Record Tracking (SMART)-STAT, modeled after a program called CompStat used by the New York City Police Department. Building on data from the SMART case management system, the portal allows executives to drill down within each department to assess whether it's attaining performance objectives.
The decision to proceed with the project had occurred in the interim between Kirkendale's predecessor's retirement and his hiring. "That was a little unsettling," Kirkendale admits, but added that he soon realized the project's potential. He also understood that ORE's statisticians and analysts had to guide the implementation. "They were the ones longing for this capability," he said. "Plus, Calvin is a very bright guy. I recognized that right away."
Knowing When to Take the Back Seat
One challenge of Kirkendale's job is getting internal business customers to lead projects like this. "He has to give them the latitude to guide the project and let it grow, and know that it is consistent with his enterprise architecture," Johnson said.
Indeed, Johnson believes that in the best cases of business-IT alignment, a CIO has to know when to take a back seat. "It's almost necessary that IT take a secondary role in this," he said, "because IT often doesn't understand the business pain points. You have to have the subject matter experts in the business community involved."
ORE's statisticians, social scientists and criminologists can sit down together and ask how to measure the effectiveness of an intervention, where the data will come from, and how to establish baselines and benchmarks.
Without that expertise on staff, many organizations end up deferring to the vendor, Johnson added. "A lot of business intelligence projects fail," he noted. "Many are not even getting 10 percent of the results they thought they would. One reason is that they don't have subject matter experts and business analysts available. Vendors and business intelligence consultants can get data to the surface through a portal. But you can't surrender that decision-making about what and how you're going to measure to the vendor."
With leadership from ORE, implementation is being handled by a combination of SAS Services consultants and Kirkendale's IT team.
From Kirkendale's perspective, there were times when Johnson's team and outside vendors digressed beyond the project's scope and he had to rein them in. For instance, he said, talk about capacity planning led to consideration of switching to a UNIX platform. "I had to say, 'Wait a minute. We are a Microsoft shop and we don't need to add more complexity.'"
Kirkendale said a key role for him is communications and managing user expectations about the project. "There were a lot of promises made to a lot of parties about what this would deliver," he said, "so people want to see tangible results."
In September 2006, the agency rolled out an interim solution that delivered some of the reports executives had been promised. "We wanted to show we could deliver on the pain points that had been discussed," Kirkendale said, adding that the solution also created a beta environment that allowed users to give feedback.
Kirkendale and Johnson say the business intelligence software will let the agency do predictive modeling about which interventions and treatments work best to cut down on recidivism among offenders.
"As an agency we went from being indisciplined to being reactive," Kirkendale said. "Now we're more mature and recognize the need to be proactive."
Here's an example of how SMART-STAT works: CSOSA has branch offices located throughout the nation's capital. Within each branch office are teams made up of community service officers. The SMART-STAT portal is being used to assess how each branch, team and office is performing. One measure is how well they're keeping offenders on the street while ensuring the public's safety. For instance, if keeping 70 percent of an officer's caseload on the street is the benchmark, the portal will allow the branch chief to see when officers are below that mark and start looking for reasons why, Johnson said. Have they done all they can to give that person a reasonable chance to stay off the street? For instance, if the person tests positive for drugs a few times, did the officer consider treatment and contacting the offender's collateral contacts to seek help before sending the person back to prison? "The portal has all that information in it," he said.
Transformative and Transparent
Johnson said the focus on service and performance levels can be transformative for CSOSA employees. "Our employees aren't just going to a job and punching in and out," he said. They're starting to see themselves as providing services for constituent groups, such as the courts, the offenders and their families, as well as residents and visitors. "We have to see what we do as providing services that have an impact on public safety in the city," he stressed. "And our constituents expect that from us. The warehouse allows us to capture and report out how well we are doing. We have already captured 20 service areas. And for every conceptualized service area, we are finding the right data to measure it by."
Another main asset of the data warehouse is its transparency. "We wanted to get internally where there's no debate about the statistics used to make an argument," Kirkendale said. "Also, constituents can sit down with CSOSA executives and go over statistics on how many visits its officers are doing. And with supervisors gaining access to much more data about employee success rates, it may be uncomfortable for some people," he added.
But it's not just front-line employees whose performance is being scrutinized. Managers may recognize their own mistakes. Perhaps they should have balanced employees' caseloads differently. Or the problem could be in human resources or in IT. "If the network was down or there was a problem with an application, we are going to be able to see it," Kirkendale said, "and I will have to answer for that."
But Kirkendale is convinced that CSOSA is developing an IT tool that will help employees agencywide prioritize. "This is a CIO's dream," he said. "This is what business alignment means."
David Raths is a Philadelphia-based writer focused on information technology.