New York City’s HHS-Connect is a project designed to improve information sharing among disparate city health and human services agencies. When the initiative signed a three-year unlimited licensing agreement with Oracle in 2009 for the company’s WebLogic Enterprise Edition and WebCenter Enterprise Edition software, the move caused IT leaders in other mayoral health and human services agencies to rethink their middle-tier software plans.

Availability of the new agreement led Anna Stern, assistant  deputy commissioner for new initiatives at the NYC Human Resources Administration (HRA), on a nine-month odyssey to better understand how the software products her team needed are bundled, how maintenance costs would be allocated at the end of the licensing agreement, and how Oracle’s software de-support policies worked. What she learned led her to investigate alternative hardware/software combinations to avoid a dramatic — and unfunded — increase in maintenance costs.

“All of these agencies could now obtain WebLogic Enterprise Edition and WebCenter Enterprise Edition for free. I thought, nobody gives anything away for free,” she recalled. “My antenna went up.”

Stern isn’t alone in struggling to understand the implications of a new software licensing agreement. Amy Konary, a research director of software pricing and licensing for IDC, said two overwhelming problems persist: First, licensing agreements are long and difficult to understand. “They are written in legalese and are often 100 pages long,” Konary said. The second complaint she often hears from IT leaders is about the way software is sold, with a perpetual right to software and then an ongoing fee per user, or sometimes based on the number and speed of hardware processors. “Buyers say that doesn’t always match up with the way the software is used,” she said, “and that they are buying more than they need.”

In a 2009 IDC survey, 76 percent of enterprises said there’s a level of software underuse or “shelfware” in their organization. And although 61 percent of IT managers believe tracking software usage is important, most said they don’t have the appropriate tools to do so.

Besides ensuring that they aren’t paying for more licenses than they need, IT leaders also must look beyond the upfront costs of new software. As Stern investigated, she found that despite the substantial discounts that HHS-Connect had negotiated, migration costs would be significant. “It may be a free product, but because we have a PL/SQL Oracle Portal deployment, it could cost us $600,000 in incremental consulting fees to implement, and I have to reconfigure my team to devote two full-time resources to the middle tier and retrain key staff in Java,” she said.

One of Stern’s chief responsibilities involves a large enterprise data warehouse for the HRA, a multibillion-dollar social services agency that provides assistance to New Yorkers who require a mix of Medicaid, food stamps, public assistance and related services.

“We are satisfied with Oracle Application Server and Oracle Portal, but Oracle classifies its products as either supported or strategic,” she said. When Oracle acquired BEA Systems, its WebLogic products became strategic, while the software used by the HRA data warehouse became supported, which means it would be phased out over eight to 10 years. The same thing happened when Oracle acquired Siebel Analytics; Oracle Discoverer became a supported product.

“Even if you are happy with your current solution, having a product moved into the supported group means that you need to start thinking about migrating,” she added.

Since her team would be given two strategic product suites for free, Stern realized it made sense to migrate, but that would require rewriting the entire middle tier of the HRA’s successful warehouse. Moving from Oracle Discoverer to the strategic Business Intelligence Enterprise Edition (OBIEE) would again mean considerable migration costs and an increase in hundreds of thousands of dollars per year in maintenance costs.

Stern pulled information from the Oracle Fusion Middleware licensing manual to create Visio flowcharts to help herself and her colleagues better understand their options. She also dug into maintenance and de-support policies and found some interesting things. “They entice you with deep discounts — deeper than those on the GSA schedule — but I have come to believe you should ignore those discounts,” she said. “Let’s say I buy 100 database partitioning options, but in three years I can change the configuration and want to de-support 50. Oracle will reset the cost basis for maintenance to the E-Business list price, so de-supporting the licenses may actually increase maintenance costs.”

Stern noted that the licensing price reset information isn’t in the contract’s terms and conditions, but rather in the technical support policies.

Oracle declined to comment for this story.

Oracle has two types of software licensing: Named user and processor-based. Stern plans to deploy a mix for development, test and production, to keep costs down. Processor-based licensing fees are higher for systems with multicore central processing units (CPU), like the data warehouse’s Sun Enterprise 25000 servers. Stern’s team started looking at hardware changes that might save them money. “Ultimately we realized that we could move our production middle tier to a small Sun server,” she said. “We are downsizing from eight dual-core CPUs to a server with two single cores. When we complete all three migrations, including OBIEE, we think that replatforming the middle tier will generate maintenance cost avoidance of $350,000 per year.”

With the processor-based licensing that some database and business intelligence software companies use, agencies must realize that hardware upgrades can increase software costs dramatically, IDC’s Konary said. “Customers areseeing vendors change policies and cost structures on maintenance and how they charge for licensing over the life of the software,” she said. “There is no silver bullet on how to address this, but you have to realize as an organization that if you adopt virtualization or do a major server consolidation, you have to look closely at software costs as part of the total cost of ownership package.”

The Low-Hanging Fruit

So what’s a CIO to do? Consultants and analysts have several suggestions. One is to get a better handle on your current licensing usage. Products from companies like Flexera Software and Sassafras Software can help agencies gather information about what software is installed in their computing environment and what usage rights they have for those products. “That is the lowest-hanging fruit, looking at usage,” said Scott Lemm, who oversees the asset management team for TechTeam Global Inc.

He has used Sassafras’ K2 software to help public-sector customers track software assets for almost 10 years. “One organization I worked with licensed 1,800 copies of a piece of software, but no more than five were being used at any one time,” he said. “That is information to go back to the vendor with when negotiating.”

There are literally hundreds of ways to license software and some vendors change licensing terms regularly, Lemm added. “We try to help customers find that sweet spot, but it is constantly moving.”

Steve Schmidt, vice president of product management at Flexera Software, said his company’s FlexNet Manager Suite can provide business intelligence about an agency’s operations. He said at a basic level, asset management can help agencies identify software installed, purchase orders and contracts. But at a more advanced level, license optimization involves monitoring contracts and compliance so that you are prepared for maintenance renewals, license audits and reviews. “That allows you to minimize license consumption and software spending, and gain insights for vendor and application consolidation,” Schmidt said.

CIOs also can seek to build stronger internal software asset management teams to track licensing usage and options to negotiate with vendors. “Most government organizations will claim to have software asset management teams, but from my experience I can count on a few fingers the ones that do it well,” said Michael Swanson, president of Minneapolis-based consulting firm ISAM Inc. “What they call an asset management team is really just the procurement people who understand contracts and vendors, but not the functional features of the software or the functionally equivalent software that could save them money.”

Swanson said it’s a mistake to be too focused on discounts. “CIOs need to ask what is the lowest cost they can pay for the functionality they need,” he said. Two state governments he credits with improving their software asset management are California and Louisiana. “They do not sign any licensing agreements until they have done a thorough financial review,” he said. For instance, in its data center consolidation, California has studied the optimal way to configure its mainframe and distributed computing environments to lower the cost structure.

California has learned to gather data about its software usage and find functional product redundancies, said Patti Malensky, IT acquisitions supervisor in the California Department of General Services (DGS). “We collect data on the percentage of usage of each product,” she said. “Some products were loaded onto a system and then forgotten about. We have actually eliminated 30 percent of products due to lack of use.”

If software is infrequently used, that can be taken up in negotiations with vendors, she added, but more often the DGS can negotiate with the users. “Sixty percent of the time they are not willing to pay the full cost of the software when that is the option presented to them,” Malensky said. “We are sort of forcing them into the 21st century, but we don’t leave them hanging. There is always a functionally equivalent option.”

To IT leaders in other states, Malensky recommends developing a strong internal software asset management team and working with consulting firms that specialize in the area. “You have to do your homework on usage of hardware and software,” she added. “Some people may overtrust the vendor and think the vendor is working on their behalf, but that is not always the case.”

Making changes, however, can be an uphill battle politically, Swanson said. “If you have 700 software products in state government, there are staffers and a vendor behind each one telling you why you need all of them.”

Enterprise licensing agreements are still popular because CIOs are focused on the discounts, and there’s a sense that there is less management complexity overhead, IDC’s Konary said. Enterprise licensing agreements do have their place, she said, “but many customers end up frustrated with the largest software vendors because they are charged more and more in maintenance costs and feel that they are really hung out to dry. There is a real backlash and that will probably continue.”

Defense Department Makes Licensing Changes

The U.S. Department of Defense (DoD) has gradually grown more sophisticated about software licensing and using its size to negotiate better deals with software vendors.

A decade ago, the DoD’s IT operations were highly decentralized. The scale was appealing because IT groups could be responsive to users, but there was a lack of interoperability and an inability to leverage buying power, explained David Wennergren, deputy CIO and deputy assistant secretary of defense for information management, integration and technology for the DoD. “So we started some fundamental shifts to gain efficiencies. Part of that involved software licensing,” he said.

Launched in 1998, the DoD Enterprise Software Initiative (ESI) was created to negotiate licensing agreements across the department’s agencies. Since then, it has negotiated more than 75 enterprise software agreements with more than 50 software publishers and has been credited with more than $3 billion in cost avoidance for the DoD.

“The first step was to recognize that if we are Oracle customers buying 50,000 seats, we should aggregate that demand and know how many we are buying,” Wennergren said. “The next step is to move to enterprise licensing agreements to cover all 800,000 people who might use the product.” The ESI also works to identify the best vendors in a market, such as data encryption, and to negotiate licensing terms and conditions agreements with them for all DoD users.

Wennergren said software-licensing schemes must change with the way government agencies are using the technology. “In the old days, you used to buy AOL by the hour. Now you pay a cable provider a flat rate per month,” he explained. “The business model shifted.” The same thing is happening with software, he said. The data isn’t being used in isolation in a stand-alone computer. There could be data in a DoD enterprise resource planning system that people in the U.S. Department of Homeland Security need access to, but they aren’t covered under current licensing schemes.

“The licensing has to evolve to meet the government’s needs,” Wennergren said. “And as we migrate services to the cloud, we will have to work closely with software partners to rethink how licensing is done.”