Proving Its Mettle

Open source software has come a long way, baby.

by / April 27, 2005 0
Having done yeoman's work on the infrastructure side of enterprise networks over the last few years, open source software now merits consideration for mission-critical applications.

Computer aficionados have long written their own clever little applications to solve problems they've run into on own their personal machines. Linus Torvalds developed the most famous piece of open source software -- the Linux kernel -- in 1991 because he wanted an operating system that used computer hardware resources more efficiently. After some tinkering, he devised a program he felt comfortable sharing with other people, and he published it on the Internet -- inviting comments and refinements.

But open source software isn't just about clever little applications anymore. In recent years, open source software has gained respect in mainstream computing. The February 2005 Netcraft Web Server Survey discovered, for example, that more than 68 percent of Web sites use Apache Web server software. That's a lot of traffic being handled by open source software.

Part of open source software's success comes from the community of people behind an application -- an army of gardeners quite willing to spend countless hours pruning, tending and nurturing the code. In contrast to commercial software, no one "owns" open source software. Everybody can use it and make refinements or improvements to any piece of it. Often, communities of programmers collaborate remotely to create an ad-hoc group or foundation to work on advancements to the application.

For many, there's no profit motive. Participants know full well the applications are freely available to anyone wishing to download them. But today, big boys like Novell, IBM, HP and Red Hat have found a way to profit from selling enterprise-level support to private-sector and government customers.

Open source software is catching on in government because it can cost-effectively integrate incompatible systems, free agencies from onerous software licensing requirements and create more flexibility through adherence to open standards.

Still, open source software's relative newness in the public sector has people uncertain of how to approach issues tied to using it, such as sacrificing their right to ownership, securing reliable support, and perhaps profiting by selling applications developed in-house.

And it hasn't completely won the trust of most public-sector CIOs or IT managers when it comes to mission-critical applications. It's one thing if a Web server goes down. It's quite another if, say, a county application that processes human services benefits locks up and refuses to do what it's supposed to do.


Proving Ground
The rap on open source is that it can't be trusted, and relying on it for important back-office functionality is premature. Mississippi, however, made Linux the core of a new mobile data infrastructure, the Automated System Project (ASP), which will link police, fire and emergency medical personnel from three counties. The ASP will launch in Hancock, Harrison and Jackson counties, but the hope is that the ASP will go statewide and perhaps serve as a model for a national program.

The ASP got a lot of press in mid-2004 on Web sites covering open source and Linux, with the general consensus being that the project is a significant milestone for Linux in government.

"Linux is poised to prove itself as a stable platform for government use," said Chris Alley, ASP chief architect. "Typically government IT managers are risk averse and would choose to go with a commercial operating system, rather than an open source platform. The commercial sector has embraced it over the last 10 years, and now it's poised to move into the government market."

Alley said he's received inquiries from several states about the ASP with the common theme of how Mississippi got agencies at federal, state and local levels to share information.

"It's such a huge problem to solve that it would be cost-prohibitive to try to do it with commercial software," he said. "The money we saved using open source software has allowed us to do more. We've been able to spend more time and effort with the integration of the various databases."

Alley said his team, using open source software components running on Linux, took two formerly isolated systems -- a records management system purchased from a vendor and an in-house developed jail management system -- and got them talking to each other. The open source tools didn't come with the hefty price tag associated with commercial integration products.

"The tools required -- enterprise application integration tools -- are very expensive," he explained. "They're difficult to use. You need advanced types of IT resources that can install, configure and make them run in your environment. Typically the large companies -- IBM, Microsoft and Sun -- provide those things, and it's been historically cost-prohibitive for government to effectively use those tools."


Repeating Results
The goal is to invite more agencies in Mississippi to join the ASP, and this summer is the target for rolling out the mobile data infrastructure, said ASP Director Maj. Julian Allen, though funding issues will prevent a full rollout to all participating agencies.

"We'll be able to roll it out, but not 100 percent in every agency," Allen said. "We may only do partial rollouts -- only a certain number of vehicles in each agency -- until we get all the funding completed over the next three years."

Allen said when he approached U.S. Sens. Trent Lott and Thad Cochran with the business case for using homeland security grant money to fund the ASP, he told the senators that open architecture and open source software would play an integral role in the ASP.

"I don't want to be tied to proprietary rights on anything because that costs money," Allen said. "That's why we bought open source products. Primarily what I'm looking to control there are the darn maintenance and support contracts afterward. If you go proprietary, you have no say in that. You take it or leave it. When I own it, I can go out and bid that on the open market. I can turn around and hire anybody, competitively bid it out, to get a competitive maintenance contract."

In addition to the original three counties, Forrest and Marion counties are lined up to join the system, Allen said. Marion County is the first agency to pay its own way to join the ASP, while the other counties relied on a portion of the grant money to fund their entrance into the ASP.

"Marion County is going to take advantage of our enterprise software license," he explained. "That license allows any public safety agency in Mississippi, any parish or any county that borders Mississippi, to come on board our data centers and use our computer-aided dispatch, records management and fire management software applications. We don't have to pay another nickel for it -- it's unlimited end-users."

The federal grant money bought the infrastructure, he said, and when that grant money runs out, small public safety agencies need only pay a small fee to share the cost of maintaining and supporting infrastructure and software. The more agencies that join the ASP, the lower the costs are for everybody.

"We're making this an affordable, state-of-the-art system that small, rural communities can afford to participate in," Allen said. "One, they don't have to buy software. Two, the maintenance and support contracts are peanuts [compared] to what they're paying today for maintenance and support on proprietary products."


At the Crossroads
Open source software is moving up the software stack, said Brad Westpfahl, director of IBM's Government Industry Programs.

"Over the course of time, I expect to see more application software available, and I can give you some solid reasons why I think government will be the leader in that," he said.

One key is governments' "collegiality," he explained, because public-sector IT professionals have no qualms about sharing their ideas with each other -- unlike the private sector.

"For government practitioners, the way they build their professional credentials is to demonstrate to their peers the value they've brought to their government by doing new, creative and innovative things," Westpfahl said.

Another factor is that the functions government units carry out across the country -- or even across the globe -- bear a strong resemblance no matter the location.

"There's a ready marketplace for the reuse of software because the business processes performed are so consistent," he said, and entities like the Government Open Code Collaborative will become the vehicles governments use to share open source applications they've developed.

A third reason government will likely lead the pack when it comes to open source, said Westpfahl, is that the wide variety of software applications in government will spur the public sector to adopt a diverse range of open source applications. The problem, he said, is that government needs can't always be met with appropriate and affordable commercial products.

"Yet somewhere in government, somebody has written a tool that does a more than adequate job of every one of those functions, and it's going to be those tools that become the application code base distributed through open source licensing through these new collaborations we see emerging around the world," he said. "You combine those three things, and I think you've got the makings of government being the leading industry for the use of open source licensing and development techniques in the application space."


Unresolved Issues
Opening homegrown government applications to other jurisdictions under a public license could be beneficial to governments that have similar computing needs, but there's no denying the appeal of raising money by selling a clever application developed by your IT staff.

Jefferson County, Colo., is on the horns of that dilemma, said David Gallaher, the county's IT development director.

"We've got a very large application that a lot of folks would be very interested in getting their hands on," Gallaher said. "We've spent probably a couple million dollars building it. It's a very robust, very nice application, and we haven't released it yet to anybody on any terms."

He said part of the hesitation stems from uncertainty that the application is ready for prime time, but at the root of the county's caution exists a harsh political reality.

"The issue is one we're having huge debates on internally," he said. "If we release it, should we not look at trying to get our taxpayers back some of their investment? That's a lot of money to shell out to build an app. We don't claim to have the answer here.

"It won't be a decision made by the technical folks," Gallaher added. "It's going to be a political decision -- that's the hard truth of the matter. If we go to the commissioners, we'll try to explain it both ways and say, 'What do you want us to do?'"

He said the county has two primary options: release the application as open source software via the GNU General Public License (GPL) or sell the application to a company that would package, sell and support it.

Releasing the application under the GPL creates its own set of problems for the county.

"This is a very large piece of software," Gallaher said. "If we put it out there as a project, we're going to have to sink considerable money into just managing it as a project. The folks managing those large, open source efforts spend some serious time. It's not just a hobby. If we're spending that kind of time, what is the benefit to us?"

Besides the time investment, he said, if Jefferson County releases its application under the GPL, what happens if changes are made to the application and they don't do the county any good?

"If we open it up under the GNU license, do we lose control of this thing?" he wondered. "This is not a product in its inception. This is a product that's in production. Does that make it different?"

Selling the application outright to a commercial entity is the most attractive option because the county makes money and doesn't have to worry about the application anymore, he said, because it's somebody else's responsibility.

Finding a third party partner to "manage" the application would probably be the simplest option for the county, he said, but that comes with a risk too.

"We've done that in the past, and that can be a very dangerous thing," Gallaher said. "People will say, 'We'll manage it, but we want you to guarantee that you'll spend x hours fixing anything we find wrong with it.' We went down that path once before, and it turned into a disaster.

"What was supposed to have been 100 hours per year turned into several thousand hours per year, and we ended up trying to support all the other counties that bought our product," he continued. "You can fall into that. People say, 'Oh, I'd never let that happen.' Well, try it and find out. My predecessor did, and that's why he's my predecessor."


Safety Net
Concerns over support have slowed government adoption of open source software, but those fears may be easing as major vendors strengthen their open source offerings.

Chicago is fine-tuning a Linux-based vehicle registration system using HP servers running the Red Hat Linux operating system and an Oracle database that will replace an aging mainframe system.

The registration system issues more than 1 million vehicle stickers annually, said Steve Philbrick, first deputy CIO of Chicago, and is set to go into production for this year's renewal cycle. Stickers go on sale in June, so the system was set to begin generating notices in late April.

"The registration system generates several million dollars in revenue," Philbrick said. "Government is real big into mission-critical when it comes to revenue generation."

Philbrick said Chicago reduced maintenance costs by replacing hardware based on proprietary UNIX software with an open source platform.

"Those proprietary UNIX boxes have huge maintenance and support price tags," he said. "The real savings we're getting is being able to buy a $10,000 commodity server, drop it in place and run Linux on it. Before, we'd be spending $100,000 for a UNIX box. Our maintenance on the $10,000 box is a few hundred dollars a year. Our maintenance on the $100,000 box is tens of thousands of dollars per year."

The key to further government adoption of either open source operating systems or applications is support, he said, especially from companies like Red Hat or Novell.

Open source software communities are an option for support, but the problem with turning to those groups is the ad hoc nature of the support. It's not that the groups aren't composed of skilled and knowledgeable people, but a government agency can't pick up the phone during business hours and reach somebody pronto.

Another problem is that it's labor intensive for governments to physically track the Linux kernel's progress or monitor bug fix Web sites for new developments in a particular open source solution.
from several states about the ASP with the common theme of how Mississippi got agencies at federal, state and local levels to share information.

"It's such a huge problem to solve that it would be cost-prohibitive to try to do it with commercial software," he said. "The money we saved using open source software has allowed us to do more. We've been able to spend more time and effort with the integration of the various databases."

Alley said his team, using open source software components running on Linux, took two formerly isolated systems -- a records management system purchased from a vendor and an in-house developed jail management system -- and got them talking to each other. The open source tools didn't come with the hefty price tag associated with commercial integration products.

"The tools required -- enterprise application integration tools -- are very expensive," he explained. "They're difficult to use. You need advanced types of IT resources that can install, configure and make them run in your environment. Typically the large companies -- IBM, Microsoft and Sun -- provide those things, and it's been historically cost-prohibitive for government to effectively use those tools."


Repeating Results
The goal is to invite more agencies in Mississippi to join the ASP, and this summer is the target for rolling out the mobile data infrastructure, said ASP Director Maj. Julian Allen, though funding issues will prevent a full rollout to all participating agencies.

"We'll be able to roll it out, but not 100 percent in every agency," Allen said. "We may only do partial rollouts -- only a certain number of vehicles in each agency -- until we get all the funding completed over the next three years."

Allen said when he approached U.S. Sens. Trent Lott and Thad Cochran with the business case for using homeland security grant money to fund the ASP, he told the senators that open architecture and open source software would play an integral role in the ASP.

"I don't want to be tied to proprietary rights on anything because that costs money," Allen said. "That's why we bought open source products. Primarily what I'm looking to control there are the darn maintenance and support contracts afterward. If you go proprietary, you have no say in that. You take it or leave it. When I own it, I can go out and bid that on the open market. I can turn around and hire anybody, competitively bid it out, to get a competitive maintenance contract."

In addition to the original three counties, Forrest and Marion counties are lined up to join the system, Allen said. Marion County is the first agency to pay its own way to join the ASP, while the other counties relied on a portion of the grant money to fund their entrance into the ASP.

"Marion County is going to take advantage of our enterprise software license," he explained. "That license allows any public safety agency in Mississippi, any parish or any county that borders Mississippi, to come on board our data centers and use our computer-aided dispatch, records management and fire management software applications. We don't have to pay another nickel for it -- it's unlimited end-users."

The federal grant money bought the infrastructure, he said, and when that grant money runs out, small public safety agencies need only pay a small fee to share the cost of maintaining and supporting infrastructure and software. The more agencies that join the ASP, the lower the costs are for everybody.

"We're making this an affordable, state-of-the-art system that small, rural communities can afford to participate in," Allen said. "One, they don't have to buy software. Two, the maintenance and support contracts are peanuts [compared] to what they're paying today for maintenance and support on proprietary products."


At the Crossroads
Open source software is moving up the software stack, said Brad Westpfahl, director of IBM's Government Industry Programs.

"Over the course of time, I expect to see more application software available, and I can give you some solid reasons why I think government will be the leader in that," he said.

One key is governments' "collegiality," he explained, because public-sector IT professionals have no qualms about sharing their ideas with each other -- unlike the private sector.

"For government practitioners, the way they build their professional credentials is to demonstrate to their peers the value they've brought to their government by doing new, creative and innovative things," Westpfahl said.

Another factor is that the functions government units carry out across the country -- or even across the globe -- bear a strong resemblance no matter the location.

"There's a ready marketplace for the reuse of software because the business processes performed are so consistent," he said, and entities like the Government Open Code Collaborative will become the vehicles governments use to share open source applications they've developed.

A third reason government will likely lead the pack when it comes to open source, said Westpfahl, is that the wide variety of software applications in government will spur the public sector to adopt a diverse range of open source applications. The problem, he said, is that government needs can't always be met with appropriate and affordable commercial products.

"Yet somewhere in government, somebody has written a tool that does a more than adequate job of every one of those functions, and it's going to be those tools that become the application code base distributed through open source licensing through these new collaborations we see emerging around the world," he said. "You combine those three things, and I think you've got the makings of government being the leading industry for the use of open source licensing and development techniques in the application space."


Unresolved Issues
Opening homegrown government applications to other jurisdictions under a public license could be beneficial to governments that have similar computing needs, but there's no denying the appeal of raising money by selling a clever application developed by your IT staff.

Jefferson County, Colo., is on the horns of that dilemma, said David Gallaher, the county's IT development director.

"We've got a very large application that a lot of folks would be very interested in getting their hands on," Gallaher said. "We've spent probably a couple million dollars building it. It's a very robust, very nice application, and we haven't released it yet to anybody on any terms."

He said part of the hesitation stems from uncertainty that the application is ready for prime time, but at the root of the county's caution exists a harsh political reality.

"The issue is one we're having huge debates on internally," he said. "If we release it, should we not look at trying to get our taxpayers back some of their investment? That's a lot of money to shell out to build an app. We don't claim to have the answer here.

"It won't be a decision made by the technical folks," Gallaher added. "It's going to be a political decision -- that's the hard truth of the matter. If we go to the commissioners, we'll try to explain it both ways and say, 'What do you want us to do?'"

He said the county has two primary options: release the application as open source software via the GNU General Public License (GPL) or sell the application to a company that would package, sell and support it.

Releasing the application under the GPL creates its own set of problems for the county.

"This is a very large piece of software," Gallaher said. "If we put it out there as a project, we're going to have to sink considerable money into just managing it as a project. The folks managing those large, open source efforts spend some serious time. It's not just a hobby. If we're spending that kind of time, what is the benefit to us?"

Besides the time investment, he said, if Jefferson County releases its application under the GPL, what happens if changes are made to the application and they don't do the county any good?

"If we open it up under the GNU license, do we lose control of this thing?" he wondered. "This is not a product in its inception. This is a product that's in production. Does that make it different?"

Selling the application outright to a commercial entity is the most attractive option because the county makes money and doesn't have to worry about the application anymore, he said, because it's somebody else's responsibility.

Finding a third party partner to "manage" the application would probably be the simplest option for the county, he said, but that comes with a risk too.

"We've done that in the past, and that can be a very dangerous thing," Gallaher said. "People will say, 'We'll manage it, but we want you to guarantee that you'll spend x hours fixing anything we find wrong with it.' We went down that path once before, and it turned into a disaster.

"What was supposed to have been 100 hours per year turned into several thousand hours per year, and we ended up trying to support all the other counties that bought our product," he continued. "You can fall into that. People say, 'Oh, I'd never let that happen.' Well, try it and find out. My predecessor did, and that's why he's my predecessor."


Safety Net
Concerns over support have slowed government adoption of open source software, but those fears may be easing as major vendors strengthen their open source offerings.

Chicago is fine-tuning a Linux-based vehicle registration system using HP servers running the Red Hat Linux operating system and an Oracle database that will replace an aging mainframe system.

The registration system issues more than 1 million vehicle stickers annually, said Steve Philbrick, first deputy CIO of Chicago, and is set to go into production for this year's renewal cycle. Stickers go on sale in June, so the system was set to begin generating notices in late April.

"The registration system generates several million dollars in revenue," Philbrick said. "Government is real big into mission-critical when it comes to revenue generation."

Philbrick said Chicago reduced maintenance costs by replacing hardware based on proprietary UNIX software with an open source platform.

"Those proprietary UNIX boxes have huge maintenance and support price tags," he said. "The real savings we're getting is being able to buy a $10,000 commodity server, drop it in place and run Linux on it. Before, we'd be spending $100,000 for a UNIX box. Our maintenance on the $10,000 box is a few hundred dollars a year. Our maintenance on the $100,000 box is tens of thousands of dollars per year."

The key to further government adoption of either open source operating systems or applications is support, he said, especially from companies like Red Hat or Novell.

Open source software communities are an option for support, but the problem with turning to those groups is the ad hoc nature of the support. It's not that the groups aren't composed of skilled and knowledgeable people, but a government agency can't pick up the phone during business hours and reach somebody pronto.

Another problem is that it's labor intensive for governments to physically track the Linux kernel's progress or monitor bug fix Web sites for new developments in a particular open source solution.

The advantage of support from a corporate entity is that somebody can be held accountable for support-related issues.

"If you're using a standard release of Linux, you need somebody who's actually giving you stable releases of code, patches they've actually tested to make sure they work before they hand them to you, so you have some degree of confidence that it's going to work when you apply it on your system, and that you're not applying something that's going to undo something you've already got out there," he said.


Stepping Up Support
When well known companies enter the open source market, they bring more than big marketing budgets; they can field armies of support techs to handle customer support issues.

"It's one of the top issues," said Morgan Spencer, government industry marketing manager of Novell. "Years ago, before companies like Novell made a commitment to Linux, governments had that concern. It was a very real concern. Governments obviously have internal resources. But what if those employees encountered something they couldn't solve on their own? There's a lot of comfort in knowing you can pick up the phone and reach us."

As an example of the importance of support to government, he cited one local government customer's migration to SUSE Linux from another Linux distribution over fears of lack of support.

The budget pressures shaping the public sector's decision to explore this relatively new software environment have created a new hardware and software market -- a market that looks to have some staying power. Novell -- which bought SUSE Linux last year -- IBM, HP and Red Hat all cater to government's increasing interest in open source software.

Novell took a menu approach to its open source support offerings, with six levels of support to choose from, said Bruce Lowry, the company's director of public relations. The levels range from telephone support to dedicated staff at a customer's site.

"Our approach has always been to treat it like a regular Novell technology," Lowry said. "We've taken a lot of the uncertainty out of the support process, the upgrades, the patching and all these sorts of things. We've made it comfortable. We've made it so that support for open source products is a familiar beast to them because it looks like what they've received in the past."

Still, supporting open source presents some unusual challenges for any company wanting to make a profit from a technology that's widely available and free.

"What's different is that the actual underlying technology isn't something Novell owns," Lowry said. "You can get that for free, if you want. You can go download any Linux distribution, including ours, at no cost. But we find that most larger customers are willing to pay for maintenance and upgrade protection -- the bug fixes, improvements and security patches -- that keep the platform up to speed."


Bona Fides
Although Chicago's Linux-based vehicle registration system is just one system for one city agency, Philbrick said improvements in open source applications are prompting the city to contemplate the next step: running open source applications on open source platforms for enterprise-level purposes.

"We're considering it here," he said, citing responses to RFPs that include open source applications. "We're not doing non-mission-critical stuff at this point, but it's using a MySQL back end for feeding information to a key Web presence we've got out there. We're probably going to end up approving [those responses] and say, 'Let's go ahead and try it.'

"Our Oracle database analysts are watching it closely," he continued. "There are a couple hurdles they [open source applications] need to get over to run enterprise-level applications, and I think they're strongly pushing on getting there."

Philbrick compared MySQL now to where Linux, as an operating system, was in its maturation two years ago. Linux is being deployed in enterprise systems now, and applications like MySQL should take root in enterprise systems in another two years. At this point, open source usually plays a supporting role to enterprise applications, he said. The possibility of open source applications displacing commercial applications higher in the stack depends on several factors.

Chicago relies on customized commercial off-the-shelf software, he said, and not a lot of in-house, custom development. While at the recent LinuxWorld conference, Philbrick said he saw many examples of enterprises with IT departments handling customer development in-house that make heavy use of open source applications in the software stack.

"Our adoption of Linux, MySQL or any of those things is going to be based on where our application vendors are going," he said. "As they become more mainstream, and as the permitting and licensing software we use has a MySQL option instead of just an Oracle or a SQL server option, we'll be looking at that as an upgrade path."
Shane Peterson Associate Editor