IE 11 Not Supported

For optimal browsing, we recommend Chrome, Firefox or Safari browsers.

Handling Big Data in the Military

The journey to make use of big data is being undertaken by civilian organizations, law enforcement agencies and military alike.

The big data movement is changing the way people work. And just like state, local and federal organizations, the military is trying to get smarter, faster and more flexible with its data.

Dr. Eric Little, vice president and chief scientist for a company making big data software for the military -- a company appropriately named Modus Operandi -- explained to Government Technology that human experts who are capable of analyzing big data, and have the skills and experience to correctly interpret many different types of complex data structures, are hard to come by, which is why big data tools are so valuable today.

“An intelligence analyst in the world of the military services is normally a very young person with limited life experiences, limited technology skills,” Little said. “They receive maybe a month and a half, two months of training, and then they’re oftentimes put in situations where they need to perform their duties.”

And for such military branches as the Air Force, Army, Navy and Marines, performing analysis on big data can be an overwhelming task no matter how much experience an analyst has.

“Sometimes you’re talking about spreadsheets with a couple hundred columns or tens of thousands of rows. It’s very difficult to traverse those things,” Little said. “Now imagine if you have hundreds of thousands of those spreadsheets” and you’re trying to correlate everything manually. It’s almost impossible for a single person to do so.

Intelligence Gathering 101

Collecting, correlating and making sense of big data into a single, cohesive image is called creating a common operating picture (COP). Another problem with creating a useful COP, Little pointed out, is that oftentimes the most valuable information isn’t found anywhere in military intelligence files -- it’s in the expert’s head. The expert’s experience, knowledge and intuition often play a key role in how data is filtered and interpreted to create a COP.

With big data tools, however, it's possible to put the expert’s knowledge into the software so everyone can use it, Little said.

The tools provided by companies like Modus Operandi essentially take big data, infuse it with expert knowledge, and create a common framework so that someone with just a couple months of experience can navigate the data and easily identify patterns that might not otherwise be identified.

Traditional intelligence gathering in the field includes teams splitting up, gathering information, returning to base and writing reports -- and then the different teams may or may not learn of what the other teams discovered, and they may or may not gain access to that intelligence within a couple days. Considering the technology that's available, that, Little said, is a cumbersome way of sharing intelligence, Little said.

Big Data and the Semantic Wiki

With big data tools, teams separated in the field can work from a COP that is updated in near-real time.

The Marine Corps, for instance, is now testing Modus Operandi’s BLADE technology for use in the field. The product generates a semantic wiki from the COP that is built from the original data inputs. If, for instance, a soldier discovers some information about a person of interest who happened to have been involved in a bombing, he can update the COP with that new piece of information. Additionally, the wiki pages for that person and that bombing event would both be updated, as would any other relevant pages. Other soldiers in the field also would have access to that new information in near-real time. There’s no waiting for reports to come in and connecting dots to figure things out – it’s all in the system and can be found in the wiki pages.

A semantic wiki is just one output that can be generated from a COP. Once the data has been built into an underlying graph structure, that data can be traversed with complex queries to find different kinds of patterns and output different types of visualizations. For instance, a user interested in spatial data could submit a query that generates a map from the underlying common graph. A user primarily interested in dates could generate a timeline, and so on.

“This allows the war fighters much better situational awareness," Little said, "and it gives them a much broader understanding of things that could be related to the specific items that they’re looking at in their data that they otherwise might not have found or seen."

And this type of capability is immensely useful to workers in many other fields, Little said, which is why bio-informatics and emergency and disaster response are among the industries the company is now looking at expanding into.

Though Modus Operandi isn't currently looking at expanding into law enforcement, Little said other organizations are experimenting with such advanced data tools, too. The U.S. Military Academy at West Point is developing software called the Organizational, Relationship and Contact Analyzer (ORCA) that attempts to make sense of big data; it is being piloted by an undisclosed civilian police department in the U.S.

John P. Sullivan, a lieutenant with the Los Angeles County Sheriff’s Department, told Government Technology that such tools don’t just allow police access to more data, but they also allow agencies to work smarter and more efficiently.

Colin wrote for Government Technology and Emergency Management from 2010 through most of 2016.