-Robert Beckhusen, via Wired.com/DangerRoom
The military has a data problem. More specifically, it has a too-much-data problem. Analysts have to sort through massive amounts of information collected by orbiting surveillance drones and satellites, or finding the data trails left behind by spies inside defense networks. Sorting through all this data is also necessary for making unmanned vehicles more autonomous.
Bring on the White House’s new “big data” research initiative. Announced this morning, the plan aims to invest “more than $200 million” in six government agencies to develop systems to “extract knowledge and insights from large and complex collections of digital data,” according to a White House statement (.pdf). That means anything too large for normal software to handle, meaning data sets of at least dozens of terabytes, at minimum. The biggest beneficiary of all this could be the Department of Defense.
The Pentagon already spends hundreds of millions annually on “big data”-esque problems. The initiative announced today could add to that kitty up to $60 million per year for new research projects. That includes a $25 million yearly sum for a new Darpa data mining program called XDATA, which is broadly defined as a tool to analyze large amounts of meta-data and “unstructured” data like message traffic. (In comparison, the Department of Energy is receiving only $25 million in funding for a new data mining institute and the National Science Foundation is being granted $13.4 million.)
Where is all the rest of that defense research going? Several places, and a lot of it for helping drones crunch the massive amounts of information pulled from sensors.
“The Department of Defense if placing a big bet on big data.” Zachary Lemnios, the Pentagon’s research and engineering chief, told reporters on Thursday. “We are within sight of a new generation of systems that understand and interpret the real world with computer speed, computer precision and human agility. These systems will not only be central to helping our commanders and analysts make sense of the huge volumes of data our military sensors collect, they will also support multiple missions.
Some of these systems, like Darpa’s Mind’s Eye, seeks to develop “visual intelligence” in aerial sensors, which would give military computers the ability to connect visual data with pre-written cues. Effectively, that could mean giving drones the tools to spot enemy soldiers automatically. Other programs likely to benefit include the Insight program, which can help drones spot potential threats on the battlefield.
That information is “growing rapidly in both volume and complexity,” Darpa acting director Ken Gabriel said. “From scraps of paper to hard drives, to overhead imagery and intercepts — the data collected is often imperfect, incomplete and heterogeneous. This trend is further accelerated by the proliferation of various digital devices and the internet. All of which is used by our adversaries to operate and hide in this data terrain. The sheer volume of information itself is creating a background clutter.”
Clutter so thick, even a quarter-billion dollars in investments may not be enough to cut through.