Like many of their peers, software developers at the Texas Education Agency (TEA) previously started a project by documenting detailed requirements. Only when they finished that labor-intensive phase did they start writing code. And only when they finished the code did they show the application to the customer.
Customers weren't always thrilled. "By the time the product was being implemented, the requirements were stale and had already changed," said Martha Reesing, program manager of core technology in TEA's IT services division.
Business needs evolve -- that's a fact of life. And it's a big reason why some government IT shops are forsaking traditional software-development models. Instead, they're adopting a range of approaches known collectively as agile development.
Agile development has no hard-and-fast definition. Numerous methodologies -- the best known are Scrum and Extreme Programming -- live under the agile umbrella. But experts point to core principles that apply to agile models. Specifically agile development:
- Stresses collaboration and communication. Team members, customers and other stakeholders work together.
- Is highly iterative. Teams work in short development cycles -- perhaps two to four weeks -- and deliver a functional, tested software module at the end of each cycle.
- Is self-organizing. Agile uses formal processes but allows teams to govern themselves as much as possible.
- Emphasizes customer value. When customer requirements change, that's a normal part of the process, not a nuisance. Success is measured in value delivered, not in days ahead of schedule or dollars under budget.
- Encourages continuous improvement. After each development cycle, team members reflect on what went right and wrong, and propose improvements.
Agile development contrasts sharply with the development model known as "waterfall." Under that traditional approach, developers finish one project phase completely before moving to the next, cascading from requirements to design, implementation, verification and maintenance.
Waterfall development doesn't work well because humans can't predict every function and feature they will need in an application, said Scott Ambler, worldwide practice leader of agile development with the IBM Software Group. When customers see how software is shaping up, they refine their ideas. "If we go against human nature, we actually increase the risk," he said.
Since laws, business conditions and users' ideas are bound to change, some people decided they should build change into the development process, said Paul Clanton, CIO of Douglas County, Colo. "They concluded that we need a lot more conversation," he said. "We needed to shorten the process and deliver something in [the customer's] hand in a much shorter period than in the stereotypical 18-month project, where you deliver something that's no longer needed."
It's difficult to get a handle on how widely agile is used, said Elizabeth Zucker Barnett, principal analyst with EZ Insight, a consultancy in Bedford Hills, N.Y. In part, that's because people define agile in many different ways. Also, some agile teams operate under the radar. "It's hard to find a Fortune 1000 company that's not doing some agile work, but it's hard to understand how much is going on within any one company," she said.
Photo: Elizabeth Zucker Barnett, principal analyst EZ Insight
Productivity, Quality, Satisfaction
Organizations that go agile say the approach makes them more productive, improves software quality and makes stakeholders happier, according to Ambler, who conducts an annual survey on the subject for Dr. Dobb's Journal, which is now part of InformationWeek. Agile doesn't necessarily offer cost savings, but the upside far outweighs the downside, he said. "The decision to at least experiment with agile is blindingly clear."
Jim Highsmith, a consultant based in Flagstaff, Ariz., who directs the agile project management advisory service for the Cutter Consortium, agrees. "Accelerating time to market and enhancing responsiveness to customers are really important and are benefits of agile," he said. Also, agile promotes frequent testing -- particularly automated testing -- which yields better software, he said.
Photo: Jim Highsmith, consultant, Cutter Consortium
Last year, TEA and Douglas County started migrating to the agile development process known as Scrum. Based on a term in the sport of rugby, Scrum divides the development project into a series of short "sprints." Each sprint produces a module of usable software.
A Scrum includes four kinds of meetings, said Diane Wells, a TEA project manager. "We call them 'ceremonies' because developers don't like the word 'meeting.'" There's a sprint planning ceremony at the start; a 15-minute daily Scrum, attended by all team members and stakeholders; a sprint review for showing the software to the customer; and a retrospective devoted to improving the process.
TEA's developers became interested in Scrum because they wanted to use department resources more efficiently, complete projects faster and make customers happier. Agile development promotes those goals by keeping customers involved. "You're continually going back to your customer, saying, 'Let's make sure we're working on your hottest priorities and those priorities have not changed since the last time we met,'" said Connie Fannon, another TEA project manager.
Douglas County first experimented with Scrum in 2006, when developers used the method to create a sex offender management system for the sheriff's office. The county started a full-scale migration to Scrum last year, when Clanton, a Scrum proponent, came on board as CIO.
Clanton and his team are still determining how best to measure a project's success, perhaps by counting the number of features they deliver. But the CIO said he's pleased with the changes the agile method has wrought. "We're getting more projects done than we ever have," he said. "Everyone is busy all the time, and by the time they're done with the four-week period, they're exhausted. And they're smiling."
Agile Team, Inflexible World
One situation where agile might not work is when the larger organizational culture favors traditional processes and top-down management. "They like planning. They're not very amenable to change. They're not very collaborative," Highsmith said of executives who don't feel comfortable with agile methodologies.
In such an environment, agile teams might have to continue producing traditional-style reports to meet established reporting and auditing requirements, Barnett said. "Many times, at least early on, you'll see some kind of shadow organization that will translate what the agile team is doing into the language that the business demands." But if software teams conduct some successful agile pilots, management might eventually accept new metrics for measuring a project's success, she said.
At TEA, developers formed an agile task force to try to rewrite the agency's required processes and procedures so they would better match the agile work style, Reesing said.
Advocates also may have to overcome misconceptions about agile programming. "Some who are very steeped in waterfall development may think of Scrum or agile as some sort of Green Berets program, where the developers are just going on and doing things without any consideration for design work," said David Butler, TEA program manager of emerging technologies. But like any good methodology, Scrum has processes to ensure that its results are repeatable and documented, he said. "It's not just a shoot-from-the-hip sort of program."
Photo: Scott Ambler, worldwide practice leader of agile development, IBM Software Group
Since public-sector software developers answer to a range of people -- elected officials, the public, the press, end-users and upper management -- government IT departments might need to work especially hard to make the case for a methodology that doesn't gauge success against schedules and budgets. The good news, Ambler said, is that agile delivers. "It has a higher success rate, higher productivity, higher quality and much more opportunity for feedback." Every iteration brings tangible results. "If the elected officials and the other stakeholders were actually interested in successful software development," he said, "they would demand agile."