October 2, 2015 | Written by: John Palfreyman
Categorized: Defence & Intelligence
Defence organisations are increasingly challenged by data. Sensors become more complex and the Internet of Things together with social media brings an ever-increasing volume, variety and veracity of data, which must be processed at speed to derive the insights needed to maintain information superiority.
We describe data curation as integrating data, enriching it by aligning with other data, reclassifying the results and projecting this forward.
An example of curation in action is the analysis of spatio-temporal data shown in Figure 1 where four independent data sources are fused into a unified searchable big data store of pre-processed physical data. This is then made available on the cloud for industry specific analyses, supporting complex queries such as “find areas with a specific soil type, surface temperatures between 5 & 15 degrees Centigrade with easy access to transport hubs”.
Figure 1 – Analysing Spatio-temporal Data
One could argue that defence & intelligence organisations have been doing this for some time for example in tactical picture generation, track correlation and target identification – so what’s changed?
Now all industries have a common goal to make the most of data, so commercial research & development is focused on the curation challenge. These investments can be directly leveraged by the military in making the most use of the available data.
Intelligence analysts often spend some 80% of their time finding and organizing the data needed to complete their request for intelligence. This will get worse as ever increasingly capable sensors – such as NATO’s Allied Ground Surveillance system shown in Figure 2 come on line. But this time can be minimized by effective data curation.
Figure 2 – NATO Allied Ground Surveillance (AGS) [Credits: NAGSMA.NATO.INT]
Curation can make the most of unreliable data sources, such as social media transactions. As an ever-increasing proportion of available data is of low veracity, military organisations cannot afford to ignore this if they want to maintain the quality of their intelligence product. Effective curation – through correlation with highly reliable data sources (a passport database for example) – will enable organisations to make the most use of this growing & freely available resource.
New technologies are emerging that are ideally suited to spotting patterns in massive amounts of data. Our work on Brain Inspired Computing (see this previous blog post) brings massive scale, low power, real time neural networks to assist with the curation challenge.
I’m very interested in a discussion on this topic. Please leave me a comment, send me an e-mail or get in touch!