Program evaluation can provide roadmaps, not destinations. Reformers need to be wary of unrealistic expectations.
Every government employee is also a taxpayer, and so has ample incentive to be a good steward of the resources he or she oversees. Despite that, most government programs have not been evaluated for effectiveness, and three-fourths of those that do get evaluated barely make a difference, according to Ron Haskins of the Brookings Institution.
Fortunately, we're seeing a growing number of initiatives to evaluate programs. Unfortunately, elected officials may place unrealistic expectations on what these tools can accomplish.
One of those tools that does seem to be making a difference is Results First, an initiative of the Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation. Designed by the Washington State Institute for Public Policy, Results First provides assistance and knowledge at no cost to states whose governors and legislatures agree to provide resources and share data. Seventeen states now work with Results First, and collectively they have redirected more than $40 million in spending from less-effective programs to more-effective ones.
Although the number of affected programs and dollars is currently small, it is good to see any money being redirected to better programs. Without data, as Results First Director Gary van Landingham notes, all we have are stories about every program's version of "Little Timmy," the dead-end kid who is now a Rhodes Scholar and American Idol finalist entirely because of that program.
Shared data on programs can help managers and legislators alike, but data can provide answers only to the questions we ask. It cannot "lay the groundwork for a strong evidence-based approach that will identify the State government's vision, mission, philosophy, priority goals and benchmarks," as proposed legislation in one state suggested.
Vision and philosophy have to come first. GPS, after all, is only helpful if you know your destination. Otherwise, as the Cheshire Cat told Alice, "it doesn't matter which way you go." Too often, however, those of us in government look for data to provide the destination.
I learned that lesson the hard way when trying to take a data-driven approach to government reform in North Carolina. We developed hypotheses to test with a team of consultants. We then worked with other state-government employees to track down related research. That research generally proved ambivalent at best. Prison programs have the potential to reduce recidivism, yes, but we could not determine which programs. Converting failing schools to charter schools could be promising, but the number of schools to attempt that has been limited. Conditional cash grants have shown great promise in developing countries, but have been tried in only two American cities, where the behavioral changes are more difficult to measure.
We set aside these potentially transformative reforms because they would need to start small and would need significant support from leadership to reach the needed scale. Instead, we focused on improving the internal functions of state government, recommending, for instance, allowing agencies to keep a portion of their budget savings as an incentive to save more; improving core financial systems so that agencies could better manage their budgets; and re-allocating facility-maintenance money to consolidate space and provide better working environments.
If we could lay a solid foundation now, we reasoned, the policymakers would have more data to see what works, measure improvements with new programs, and choose among alternatives.
When legislators first heard our proposals, they expressed disappointment that we did not set out bold visions. They have since acknowledged that their hopes for policy-change recommendations had ignored the execution challenges we would have faced. With a narrower vision, the legislature has kept many of our reforms alive in budget negotiations, and executive agencies are actively planning for implementation.
Broader changes are more likely to succeed when built on these foundational reforms and informed by better data. First, though, political leaders need to agree on a vision for state government -- and for the state as a whole.
This article was originally published on Governing.