Industry Insights

Generating insight sooner

Governments are increasingly embracing DevOps, both its cultural change as well as process automation in software delivery.  Cloud delivery models and practices have transformed the consumption of IT, enabling services to be introduced and changed quickly in response to business need.  Contrast this with equipping business users with insight.   All too often in my experience, people struggle to break away from the comfort of storing data before it is queried.  We are failing to empower business users and analysts with speed of response in what data can tell them.

Benefits of streaming analytics

Businesses can benefit from undertaking analytics on up to date data more than they realise.  I have seen continuous monitoring of incoming data offer value in these ways:

  • Identifying new business opportunities by enabling actions from perishable data. Early examples include outbound marketing, first in telcos.  A customer service or marketing opportunity is lost if you cannot personalize a targeted action on a customer who calls in before that call has finished.  Intervening in the moment has wider applications in preventing fraud, and increasingly with the Internet of Things.
  • Situational awareness with a continuously current view of what is happening. Again telcos were early adopters monitoring networks and quality of service, but there are further applications in traffic monitoring, acoustic monitoring, preventative maintenance and social buzz.
  • Using in line analytics to focus on the outcome including the effectiveness of avoiding bringing data to rest through aggregation, summarization, filtering and notifications.  There is a scale up requirement for such in line processing with increasing analysis of machine data and the rise of the Internet of Things.  But there is also a need to scale down and be highly optimized in the use of compute resources in small systems and devices, and network bandwidth.
  • Closing feedback mechanisms under user and analyst control. Business users are increasingly empowered to take action quickly themselves in response to changing operating circumstances, but such action is taken blindly without capability to measure the effect of the change.  This compromises an organisation’s ability to optimize use of its resources and its operational.
Acting on all your data in real time

Acting on all your data in real time

Continuous analysis of incoming data offers opportunities to reduce cost and increase operational success.  More data is being generated in more forms, and so a technology platform for such analysis needs to be engineered for machine scale.  Essential characteristics include processing efficiency, low latency, scale out, developer productivity, and perhaps less obviously state management.

The importance of managing state

Most analytics on incoming data cannot derive the insight that businesses seek simply by ingesting data feeds: they need to manage state.  Put simply, most algorithms need to hold some previous data to be able to process the next data.

It follows that recent developments in serverless computing, whilst offering scale out, are inadequate for continuous analytics.  Custom code can be added to ingest data, analyse it and manage state.  However, this is sub-optimal because it attempts to apply technology engineered to meet one set of design points for a different purpose.  To illustrate the importance of state in continuous analytics, here are three examples:

  • Predicting collisions of objects moving on a map – state is the velocity of each object.
  • Aggregates – state is the values over a moving period of time.
  • Deviation from an expected norm – state is the significance of the deviation.

Why store data?

Continuously executing analytics on incoming data enables insight to be pushed to users as the organisation has sufficient data to derive it.  The traditional approach of storing data first inevitably means that query results retrieved are historic.  However, storing data is necessary for the following activities:

  • Algorithm development, e.g. data exploration, pattern analysis, machine learning, test data.
  • Knowledge exploitation, e.g. augmenting and informing human activity, enrichment, historical reference.
  • Investigative analysis, e.g. forensics, audit, point in time financial and compliance reporting.

Analyse data as it arrives

Clearly organisations will continue to need to store data. But rather than accept that as the default they should consider the value in generating insight by analysing data as it arrives.  Areas that can benefit public sector include fraud detection and prevention, situational awareness in military and city operations including public safety, alerting applications, anomaly and deviation detection, and cyber security.  Why wait?

Client Technical Leader for UK Public Sector; Executive IT Specialist

More Industry Insights stories

From Just Another AI Pilot to Scaled Production: The Missing Links to Convert Ideas to Economic Value for Fortune 500 Companies

While 81% of Fortune 500 CEOs have recently cited artificial intelligence and machine learning as either very important or extremely important to their company’s future, up from just 54% in 2016*; many well-intended AI pilots continue to struggle with scaling and securing C-Suite business outcomes. In this Q&A, HBS Professor Prithwiraj Choudhury explains two crucial […]

Continue reading

Humanizing the tolling industry!

Have you noticed that tolling solutions are not evolving? Once a tolling solution is put into an operation, it is hardly improved during the following years. Governments have been transforming from eGovernment via Digital Government to Cognitive Government. But tolling solutions? Even a new mobile app supporting a tolling operation would make an industry news […]

Continue reading

Fear Factor: AI for Cybersecurity?

We asked federal IT managers how they felt about using AI technology to boost their agencies’ cybersecurity. And while the Federal AI IQ Test results revealed that the vast majority agreed that cybersecurity presents the biggest opportunity for AI in the federal government, only 20% of them are “very comfortable” implementing it today. So what […]

Continue reading