A focal point requirement of the 2002 No Child Left Behind Act (NCLB), federal legislation aimed at improving U.S. primary and secondary schools performance, was to implement accountability systems that analyze student and educator data, and report those results to the U.S. Department of Education. These reporting systems were heralded as an effective way to help state departments of education collect statistics to assess teacher proficiency and student progress.
"It's really important to be able to follow individual students from grade to grade, school to school, district to district and see how they are doing over time," said Jim Hull, a policy analyst at the National School Boards Association (NSBA). "We haven't been able to do that before."
In addition, educational data systems offer the advantage of getting assessment data back to educators faster than before. "You had the old-fashioned assessment test that's taken in April and no one gets results until October or November. It's not a useful timeframe; those kids have moved on," said Ann Flynn, director of education technology programs at the NSBA.
While there is little doubt that collecting more specific data - and publishing the results more quickly - is beneficial to educators, many states have struggled with how to best implement data systems. Limited funding, institutional resistance to change, and schools' use of various student information systems have been impediments.
New Mexico officials, however, believe they have solved some of those issues with the state's Student Teacher Accountability Reporting System (STARS).
Longitudinal Student Data
STARS is a statewide, "longitudinal" educational information system that collects data from students through all grade levels, starting in kindergarten and continuing through the 12th grade. Although NCLB doesn't require longitudinal systems, states such as Florida have shown that having that kind of long-term data can be a useful measurement when assessing how well a school, district or state is meeting educational benchmarks for schools and individual students. Florida has been electronically collecting its longitudinal student data since the 1980s, allowing the state to make decisions based on comprehensive, accurate and timely data about its schools.
The STARS system collects and aggregates a variety of student data: demographics and achievement information, exam scores on state- and federally mandated tests, districts' financial information, and teacher licensing data. "At a minimum, the system collects information on students, teachers, staff, programs and schools," said Philip Benowitz of Deloitte Consulting, the engagement director for the STARS project. "But there's no limit to what the system could collect. We're still in the early stages of understanding what makes sense and what's really valuable."
Moreover, the system standardizes data so it can be reported to the federal government as required by NCLB.
But Benowitz asserts that STARS has more value than just for NCLB compliance. New Mexico can provide data to the school districts for their own analyses and use. "The intent and the spirit is to put the data in the hands of educators and analysts who can make a difference in student achievement - the classroom teacher, the principal, the state educational analyst," he said. "People who can help to improve the curriculum and student achievement."
Overcoming Interoperability Issues
New Mexico CIO Roy Soto said it was a challenge to determine the best way to collect and consolidate data. "New Mexico is no different from any other state. We have 89 school districts, all collecting data in a different form and fashion."
Unlike many other states, New Mexico had been collecting student-level data since 1997 with the STARS predecessor, the Accountability Data System (ADS). But ADS had maintenance and system integrity issues. Before making critical implementation decisions on a new system, the state conducted several legislative audits. After careful consideration of the results, the state chose
a data warehouse solution and put out an RFP to find a vendor.
"We basically took the audits, with specific emphasis on what needed to be fixed, and put it into our request for proposal," said Robert Piro, CIO of the New Mexico Public Education Department (PED). "Deloitte Consulting presented us with a solution based on eScholar and Cognos."
With eScholar, an educational data collection and analysis tool, and Cognos business intelligence software, Deloitte created a commercial-off-the-shelf system that allows school districts to collect data as they've always done.
"In New Mexico, there are a dozen or more student information system vendors that have systems in place in one or more of the 89 school districts. The last thing we wanted to do is mandate that they all use the same system," Benowitz said. With the STARS solution, school districts can continue using their existing systems and produce a flat extract data file that can be uploaded to the data warehouse automatically.
New Mexico implemented the system in nine months. "We started the prototype in December of 2005, and then did a pilot with 11 districts in spring 2006," Piro said. "We're now in our second year of data collection."
One of New Mexico's biggest hurdles was change management. Since the system was implemented in less than one year, there was pushback at the district level from some educators.
"When you have so many different entities that are basically independent, doing things a certain way, it's hard," Soto said. "Some people saw it as, 'Here comes Big Brother.'"
Although school districts could keep their internal systems, the move to STARS required a redefinition of processes for what kind of data to collect and when to collect it. This caused some consternation from districts that already had workflows in place.
Daryl Landavazo, New Mexico's STARS IT project manager, said the districts have been collecting student-level data for some time. "So the assumption was, 'We're using data; we know how to report, and we know what we're collecting,'" he said. "But that's not always the case."
To combat resistance, the STARS team marketed a proof-of-concept system to both the school districts and the Legislature. "We showed them the proof of concept, and said,