Looking for Answers

Computerized grading helps Indiana cut cost of statewide essay tests.

by / September 2, 2004
The No Child Left Behind Act meant far-reaching changes for American schools. The act focuses on improving accountability and academic achievement, but put particular emphasis on assessment. Schools not only have to conduct assessments more often, they must demonstrate efforts to boost achievement in areas where assessments show students are failing.

But conducting statewide assessments is expensive and time-consuming -- especially when the tests include essays or short-answer questions. Although computers easily grade multiple-choice questions, essay assessments require human evaluations. This year, for example, Illinois paid $6.5 million to grade 1.2 million student writing samples, according to the Illinois State Board of Education. While federal funds are available for new tests, states must provide significant resources as well. As the number of assessments grows, states are looking to curb costs.

The Indiana Department of Education may have a solution. This year, Indiana was the first state to use computers to conduct student writing assessments, transfer those assessments electronically to a grading facility, have them graded via computer and sent electronically back to the schools. The state expects this new method to save taxpayer money, reduce testing time, speed up test results and give schools more time to help struggling students catch up.

"We've been concerned for years about the costs of state tests and the speed at which the tests get graded," said Stan Jones, Indiana's commissioner of higher education. "Right now, it takes about two months to get results back because they have to be graded by people. We've had essay tests as part of our state testing program for eight years, and we think it's an important part. Therefore we've been searching for a way to both cover costs and get our test results back more quickly."

Buying Time
Traditionally statewide essay assessments are graded by temporary employees who spend an average of three minutes on each one, according to Educational Testing Service (ETS). The cost of paying those employees can easily run into the millions, and the delay in obtaining results means less time to get struggling students into tutoring or remedial programs.

"The time involved is critical because these are end-of-course exams," said Jones. "If it's going to take two months to get results back, you're talking about giving the test much earlier in the year. That means less time to ensure students know the material and less time to help them if they don't."

Indiana uses ETS e-rater technology, a form of artificial intelligence designed to mimic human readers, to analyze two state-recommended Core 40 courses -- English/Language Arts and Mathematics -- for grades eight through 12. ETS and subcontractor Achievement Data Inc. provided test design and development, field test design and administration, online assessment system design, computer-based testing procedures, field test implementation, development and implementation of automated scoring procedures, score reporting, data analyses, and operational forms construction and delivery.

Approximately 60,000 Indiana students took part in the computerized assessment last spring, according to Jones. The state gave schools a choice of trying computerized testing or sticking with the traditional paper-and-pencil tests graded by people. According to Jones, 80 percent of the schools chose the online exams.

Schools that conducted online assessments received scores within three days, and the average cost per test dropped from about $30 to $15. "The primary cost saving is the human labor of having people grade the tests," said Jones. "But there are also administrative cost savings because you don't have to ship tests out and have them shipped back and reassembled."

Managing the Skeptics
Some critics of evaluating essays via computer charge that computers cannot effectively evaluate the markedly variable craft of writing.

To dispel skepticism, ETS conducted a two-year pilot program in which student essays simultaneously were graded by a computer and trained readers. The pilot found the e-rater generated grades virtually identical to those of the readers. "The technology is very good at analyzing the same things humans are trained to analyze within student essays," said Richard Swartz, ETS's executive director of technology products and services. "It gives you almost exactly the same results without the time and money requirements."

Because e-rater is designed to mimic human evaluators, it does not necessarily improve on a human evaluator's performance, nor was it designed to.

"If you're comparing it to a good high-school English teacher, there's no question the teacher can grade it better," said Jones. "But on state tests, the people hired are comparing students to each other, so it's more of a sorting process. So while this isn't a good alternative compared to a high-school English teacher, it is a good alternative for testing."

Other states are waiting to see the results of Indiana's pilot. Swartz predicts more states will adopt computer grading for essay tests, as long as Indiana's experience remains positive. "As computers continue to become less expensive and more powerful, there will be more of this testing," he said. "It's going to be the wave of the future."

In June, the Tennessee Department of Education said it intends for all public-school students to take their mandatory exams on computer by 2010. The state will try it in 10 districts during the next school year and expand the practice if things go well.

Tackling Technicals
Computerized testing requires state schools to reach a certain level of technical proficiency, and some still have a long way to go. States must ensure all school districts have the platform to support testing, enough computers for all students to take the tests and security adequate to protect systems from being compromised. Indiana has one of the highest computer-to-student ratios in the country -- with one instructional computer for every three students -- but many other states are struggling. Some already tried computerized assessments only to abandon the plan because they couldn't obtain computers for all students.

Once a state gets its schools properly equipped, however, computerized testing requires very little from the teacher's perspective. "We have training sessions for the coordinators in each school for the administration of it, but for teachers and students, it's ready to go," said Swartz.

So far, Indiana is pleased with the pilot's results. "It's working well," said Jones. "Teachers like that they get the scores back quickly, and students like taking the test online. We've been surprised that the schools have so readily adapted to this. We anticipated there would be more skepticism. Clearly that skepticism is out there, but it's more than outweighed by the benefits."

Jones said Indiana may look next at computerizing biology tests. The state also is working with universities, hoping that exams conducted at the 11th- and 12th-grade levels eventually can double as college placement exams and be electronically transferred to colleges the students plan to attend.

For now, the state simply hopes to reduce testing costs and translate faster test results into improved student achievement. "Clearly in terms of state testing, time and money are important," said Jones. "But the other thing is that when teachers have to wait too long to get results back, the results become significantly less important. If they can get results back in three days, those results become part of their teaching and they can adjust their strategy to help students."
Justine Brown Contributing Writer