|
Videos
October 25, 2011 By Randall Jackson
Relational databases have been a powerful legacy technology over the last few decades, and a venerable one at that. From their inception by forward-thinking technologists in the '70's, relational databases have grown to worldwide de facto dominance in the IT world, forming the backbone of today's government data infrastructure. However, the 21st century information landscape is rapidly changing, and the traditional approach of always solving the data challenge with a relational database is no longer feasible. We are in the new information age, one comprised of unstructured data sources that are essentially impossible to model from a relational perspective. Yet government IT staffs continue to try and solve these challenges by applying relational database technology, incurring huge expense to develop and maintain such systems (if they perform at all) in an era of “do more with less” IT budgets.
Former Federal CIO Vivek Kundra recently said, “this notion of thinking about data in a structured, relational database is dead, some of the most valuable information is going to live in video, blogs, and audio, and it is going to be unstructured inherently." Modern, 21st century tools have evolved to tackle unstructured information, yet a huge majority of federal organizations continue to try and use relational databases to solve modern information challenges. The question is this - what is keeping us from realizing the full potential of our data, structured and unstructured, which is (among other things) a vital necessity to national security?
Oracle and other legacy relational database systems are so pervasive in government today that when a new federal contract is awarded, these systems are unilaterally brought in as well – often ordered alongside the carpet, furniture, and wall paint. Largely, it's a result of inertia and the mistaken belief that because these systems have been around for so long, and so many organizations use them, they must be the right solution. And since many government agencies have enterprise licenses with these relational database vendors, it appears they are getting something for 'free', when in fact the frequent misapplication of relational technology to the wrong data challenge is costing far more in development and operations/maintenance costs. Fully two-thirds of any system lifecycle cost is in operations and maintenance. If you were to build the Golden Gate Bridge with wood versus steel, you can imagine the constant manpower and material costs of maintaining that bridge over time. The same principle applies to building IT systems. Using the wrong 'tools and materials' will bankrupt your IT budget by driving O&M costs sky high, putting serious pressure on budget for innovation and new development.
Relationertia – a Measurable Tax on National Productivity
This de facto implementation of legacy relational databases across government organizations, or "Relationertia", has the effect of imposing a measureable tax on national productivity.
These systems are no longer equipped to deal with the type of information governments must leverage. Today, more than 80 percent of information is unstructured – that is, data that doesn't fit into the traditional columns and rows of a relational database. Given the hardware, power, and people costs required to scale a database to deal with this type of information, it's no wonder that government database budgets have reached billions annually.
There are newer, more modern approaches that would allow government organizations to break free of this Relationertia tax and put their budgets back on track, while at the same time unlocking new innovations. For example, there is a wave of next-generation databases that are specifically designed to manage and leverage unstructured data in the modern era. And these systems can be up and running in weeks, cutting costs by up to 95 percent. At a time when recent budget battles have resulted in the Office of Management and Budget (OMB) asking federal agencies to cut 10 percent out of their FY13 budgets and to identify projects that demonstrate efficiencies, public agencies need to invest in technologies that allow them to fulfill their mission and spend the right dollars to save money.
Unstructured Information – Get Control now or Pay the Price
The growing amount of unstructured data that government organizations must manage is at a critical stage. And the government is not spared from this explosion of unstructured information – the massive volume of mission-critical government information has serious implications on everything from the budget to national security. This means selecting and implementing the right technology is more important than ever.
Outdated yet entrenched relational databases can sometimes be made to work for the 80 percent of data that now qualifies as unstructured, but they are so inefficient that projects can quickly escalate into the $100,000,000 range. In many cases, organizations will be forced to keep adding zeros as projects take years to complete or are simply abandoned because the customer tried to use the wrong tool for the job – more wasted money that is in effect a tax on all of us and has a measureable impact on the GDP.
Breaking Free of the Tax on Productivity
As an alternative to the costly proposition of relational databases, unstructured databases can cut implementation times from years to months, IT staffs from dozens to as little as one person, hardware requirements by up to ninety-five percent, maintenance costs by seventy percent, and total budget from an astounding nine or ten figures to just fractions of those numbers.
Organizations as diverse as the FAA, Department of State, Library of Congress, The Department of Agriculture, the Department of Transportation, the National Archives, and the US Army have implemented unstructured databases in a variety of functions, including search and discovery, open source intelligence, metadata catalogs, dynamic information delivery, and analytics, among others. These new vendors are also making implementation and ongoing maintenance much more efficient by providing an information centric architecture that makes it easy to develop new applications, mix and match content in new ways, and enable new delivery methods.
Government organizations suffering from Relationertia and the public that feels that impact in what amounts to a federal tax on productivity have two huge challenges: Exact benefits from the insight that this enormous mass of unstructured data could bring – from the preservation of historical documents to new insights on the war on terror – and doing so under ever-tightening budgets.
Enter the new wave of databases for unstructured data, which are ready to help organizations face these new challenges head on – a welcome option in a time of national debt and the continued budget crisis.
Randall Jackson is the Vice President of Public Sector at MarkLogic.
You may use or reference this story with attribution and a link to
http://www.govtech.com/budget-finance/Time-to-Pull-the-Plug-on-Relational-Databases-.html
Collaborative Justice: Transforming Criminal Justice Services Through Unified Collaboration
Cloud-Based Services Accelerate Public Sector Adoption of Video Collaboration
I find your conclusion to be somewhat bizarre. Do you really believe that the primary cause of bloated and inefficient technology implementations is the relational database?
Scott, I don't think he said the relational database is the primary cause. While there are surely many human factors to creating more efficient technology implementations, sometimes a fundamental re-evaluation of underlying technology platforms can make a big difference. I've been a long-time fan of relational databases, in fact, I'm passionate about the power they offer on structured data. But I got even more excited about search engines and the power they offer, particularly on unstructured data, and spent the last five years leading over 50 Enterprise Search projects. When I encountered MarkLogic I was astounded at the elegant marriage of search technology with the power of a transactional database. We were very excited to partner with MarkLogic. Since then I've talked with plenty who can't believe there is one solution that provides the best of both worlds, while reducing development effort. All I can say is I hope MarkLogic is around for a long time, because it's hard for me to get excited about building serious applications on any other database.
This sounds very interesting. While I certainly love my realtional databases, I run acrros limitations to them every day. One of the data elements I collect in my system is an image, which I cannot query against in a traditional relational database. The author, however, is a little vague about what exactly these new databases are. I wish the he had provided specfic examples of the products he is talking about and the data models that he is suggesting will replace relational database architetures. It sounds like the suggested solution (if the data is truly unstructured) would employ algorithms that would discover relationships or similarities among the data, like a search engine. In this case, the definition of data relationships is removed from the database model, and pushed to the analyst, who could look for any relationships s/he wished. Anyway, sounds exciting, can anybody out there give me specific examples where this concept has been implemented? I would love to see it in action.
Mr. Jackson states that "unstructured databases can cut implementation times from years to months, IT staffs from dozens to as little as one person, hardware requirements by up to ninety-five percent, maintenance costs by seventy percent, and total budget from an astounding nine or ten figures to just fractions of those numbers." I can recall when similar claims were made for the then-new relational databases.
This article doesn't give very much information on anything except why relational databases are bad. If you're going to slam the old technology, please describe the new technology in better detail. Are you selling this new technology or reporting on it?
Three phrases come to mind: 1. Massive Oversimplification; 2. Vested Interest; 3. Sales Pitch
I tend to agree with the comments that seem to say Mr. Jackson's article has a lot missing and seems to offer an unfair criticism of relational databases. I did a search for [“unstructured database” software] expecting to see the first hits would tell me all about solutions that “can be up and running in weeks, cutting costs by up to 95 percent” but I found really nothing, except MarkLogic and Diamond Data Search. MarkLogic includes tools to bring structure, like a metadata repository, and a common repository. Relational databases are another tool to help bring structure, and clearly they can be improved upon. MarkLogic and Diamond also address tools to find (search) and understand data, like information intelligence and pattern-recognition algorithms. It seems to me the Internet is one example of an unstructured database, and clearly there is a growing amount of unstructured data. I see search and understand technologies as keys, but I am not certain that promoting greater and greater amounts of unstructured data and trashing all relational data is the right path forward, rather than instead using the lessons of providing more structure with data standards we learned in developing relational databases with the new tools to better find and understand data. Imagine if all banks decided to go it alone and make data more and more unstructured, so to do an income tax would take a different adapter to access and understand the data from every bank to generate our Form 1040’s.
This is a marketing message. Mr. Jackson is the Vice President of Marklogic. His company sells a database. http://www.marklogic.com/
Sure it's marketing, but Oracle probably spends millions ensuring that you don't look elsewhere. However, it's interesting he doesn't talk about the challenges of storing these same data types (unstructured) data in document management systems. I'd consider Marklogic an alternative to Documentum as much as an alternative to Oracle. Or perhaps to both? I do find an overwhelming bias to utilize the status quo in government agencies. When what you know is a hammer, everything looks like a nail.
Other have noticed that Randall is a salesman for MarkLogic. Otherwise, I would have assumed that he was a newbie who didn't know anything about the need for structured data in most organizations. The need for storage and retrieval of unstructured data is not universal, and needs to be dealt with appropriately, not by dumping relational databases. Another thing he mentioned was Oracle. No wonder costs escalate. Oracle's purpose in life is squeezing all of your cash into their pockets. Most relational database needs can be met by MySql or PostgreSQL for free (and can be managed by any DBA worth his pay), or by SQL Server for Microsoft slaves. There are others as well. There are also solutions (some free) that marry structured relational and relatively unstructured data. And, please don't make us go back to the time-consuming difficulty of programming to hierarchical databases such as IMS. Pay no attention to the man behind the curtain.
I would not throw away the proven technology that relational databases provide, namely a two value logical solution. The defined data is either there or it is not. Unstructured data is more likely three valued logic, with a yes, no, or maybe. While this is tremendously valuable for more creative uses of data, governments still need to depend on good old absolute certainties. Fingerprints, DNA, and Social Security numbers fit that paradigm. However, when we are looking for new solutions to problems, we need all the maybes we can get. So there is a place for both.
I agree with you. It's very easy to over simplify software development. After all, you really only need to add an if statement to fix a problem.
Not sure of govtech's purpose, but I really grow tired of tech websites masquerading editorials and marketing materials as news. It even seems like the author got some of his buddies, or co-workers, to laud the "virtues" of MarkLogic in the comments section.