About three months ago, most of the supermarket chains in the Boston area dispensed with clip-out discount coupons. Now, advertising circulars carry only announcements of the week's specials and "clipless" coupons -- shoppers get the sales price when they present their store discount cards to be scanned at checkout.
The chains promoted the change as a time saver for customers, but the biggest beneficiary is the chain itself. When the cards are presented -- whether or not customers purchase any sale items -- all the information about the purchase is captured. For chains with well-organized, funded and forward-looking IT departments, the information ends up in a massive data warehouse that feeds departmental data marts. Those are used to analyze the effectiveness of that week's circular, handle inventory and stocking predictions and measure the effectiveness of in-store promotions, among many other things.
Before the introduction of the card, transaction data could only be used for inventory and stocking purposes and analyzing very general sales and marketing trends. With the customer-specific data provided by the cards, sales data became more useful.
In government, the allure is similar, but the problem more complicated. DMV computers, for example, may only be able to "talk amongst themselves." Assessor's Office computers speak only "Assessor-ese," etc.
Furthermore, it isn't just a matter of getting systems to talk to one another. Legal and other issues restrict which systems are allowed to talk and what they are allowed to say -- personal tax return information, for example, can't end up in databases accessible to politicians, and police investigation records must stay within the police department.
Despite these problems and restrictions, governments at all levels need access to coordinated views of data. Data warehouses can provide that coordinated view, but more than that, they can help governments "grow their business" by helping identify and provide services to a wider range of constituents. But to make this work, state information managers need to deal with the threshold issues of standards and data definitions.
Massachusetts started its data warehouse initiative in January 1994 with the intention of providing a central financial information repository for internal use by state employees.
"It does contain a tremendous amount of data from the accounting system as well as the payroll cost system, and also much of the state budget information," said Joe Marinilli, information warehouse manager, Executive Office of Administration and Finance. "These are all individual mainframe systems we're pulling the data from. In the near future, human resource information will also be available."
A key factor in making the system work was putting the data warehouse definitions themselves online. "In the early days, we were keeping information about what was in the data warehouse and which source system they came from in hard copy, and it was getting into volumes," said Marinilli. "So we created -- in the warehouse itself, in table format -- two tables which are data definition tables. One contains information on all the fields in the data warehouse and which table they belong to. The second one lists which tables can be found in the data warehouse, what can be found in them and which mainframe they came from."
These data definitions are easy to query and available for read-only access to users. Because they are in standard table format and are more manageable in size than the data tables in the data warehouse -- 2,000 plus records compared to 15 million in some data tables -- the data definitions are usually used in beginning training classes. The state has also included "value added" information with the data definitions.
"Something we've done that's a bit different in the commonwealth," said Marinilli, "is to include algorithms for terms that are found in the accounting system. A user can look at these algorithms to