Getting started with Streaming Analytics and the Internet of Things

Share this post:

The Internet of Things (IoT) service and the Streaming Analytics service in Bluemix have two things in common: Both were created for real-time, and both are extremely scalable.  So it’s a natural fit to use the two services together in an Bluemix application, with the IoT service serving as an aggregator of device information, and the Streaming Analytics service performing real-time analytics on streams of device data.

With the announcement that Streaming Analytics is generally available, this article describes a typical use case and explains how to exchange messages between the two services over the MQTT protocol.

Overview of the most common use case

In the most common scenario, you have a set of devices producing information in real-time that you need to analyze and two services. These services are:

IoT Service: This service will be used to collect the data from the devices:

  • Devices registered with your IoT service will publish MQTT messages to your IoT service
  • The messages will contain information about the devices that you want to analyze

Streaming Analytics Service: This service will be used to analyze the device information

  • Your Streaming Analytics application will subscribe to the device messages published to your IoT service
  • The device messages will be analyzed according to the logic you’ve defined in your Streaming Analytics application

Because of the low latency of the IoT and Streaming Analytics technologies, the analysis of the device data happens in real-time, and both technologies are capable of scaling to support analysis of information from millions of individual devices. There are a number of variations possible on this base scenario, but regardless of the exact scenario, the information above should help you get started using the two services together.

Application components and how they interact

In an end-to-end Bluemix application, the diagram below shows the typical components and entities that will be a part of the solution:

Bluemix application composed of the Node.js runtime, the Internet of Things service, and the Streaming Analytics service.
Bluemix application composed of Node.js runtime, the IoT service, and Streaming Analytics service.

In addition to the two services discussed in this post, the figure above shows two additional components: i) The physical devices supplying the data to be analyzed, and ii) a Bluemix application written using the Node.js runtime. The overall application flow is described below:

  • The devices have been registered with the IoT service and they publish data to IoT via MQTT messages.
  • A Streams application deployed to the Streaming Analytics service subscribes to the device information in IoT.  Data flows as MQTT messages into the Streaming Analytics service.
  • The Streams application performs real-time analysis of the device data stream.  The analysis performed depends upon the logic and analytics used in the Streams application.  Examples of analysis might include anomaly detection, failure prediction, aggregation, etc.
  • The Streams applications publishes its results back to IoT as MQTT messages.  These messages could be thought of as notifications, alerts, events, etc.  Notice that the results MQTT stream is not as thick as the input MQTT stream because the results of analysis are normally not as voluminous as the data they were derived from.
  • The Bluemix (Node.js) application can subscribe to the results that were published back to IoT and take the appropriate actions, according to the needs of the solution.

Hands-on with the Streaming Analytics Integration Samples

This section describes how you can get an instance of the Internet of Things service and an instance of the Streaming Analytics service to exchange MQTT messages with one another.  Once you’ve established this integration, you should be able to build upon this example to create a Bluemix application that meets your specific requirements.

Create and configure your Internet of Things instance

  1. Log in to Bluemix. In the dashboard, click USE SERVICES OR APIS.  Choose Internet of Things from the service catalog.  Choose the selected plan of Free and click CREATE to create the service.
  2. A welcome page will be displayed.  Click the Launch dashboard button.  The Internet of Things dashboard will be displayed.
  3. The IoT dashboard displays an Organization ID near the top of the page.  Copy your organization ID and save it for later.
  4. Next, from the IoT dashboard, select the Access and then the API Keys tab.  Then click the Generate API Key button.  Copy both the API Key and the Authentication Token values and save them for later.

Create your Streaming Analytics instance

  1. From the Bluemix dashboard, click USE SERVICES OR APIS.  Choose Streaming Analytics from the catalog.  Choose the default plan and click CREATE to create the service.
  2. The Streaming Analytics dashboard will appear.  Use the Launch button to launch the Streaming Analytics console.

Download and submit the sample Streams application

  1. Download the zip file from the Streams Integration Samples project in Bluemix DevOps Services.  This zip file contains the pre-built Streams application you will deploy, as well as the source code for the application.
  2. Extract the zip to your local file system.
  3. Go to the Streaming Analytics console that you launched earlier, and select Submit Job.
  4. A Submit Job view will appear.  Click Browse and navigate to the root of the zip file that you extracted, then into the InternetOfThings directory.  Select the Main.sab file.  Then click Next in the Submit Job dialog.
  5. On the next view, you are prompted to fill in submission time parameters required by the application.  Fill in the values for Authentication_Token, Org_ID, and API_Key that you saved in the earlier steps.
  6. Press Submit, and the sample Streams application will be deployed to your Streaming Analytics instance.  Use the Streaming Analytics console to verify that the job is submitted and running.

Three views of the Streaming Analytics Console from steps 3 through 6 above (click to enlarge)

If you were able to successfully complete the instructions listed above, you should now have a running Streaming Analytics application.  In addition, your application is:

  • publishing MQTT messages to your Internet of Things instance
  • subscribing to those same messages as they appear in your IoT instance, and ingesting them back into the Streams application

You can use the Streaming Analytics console to verify that messages are flowing out to IoT and back into the Streams application.  Sending a message out of a Streams application and ingesting that same message back into the same Streams application is not a typical pattern, but it does serve as a good example of the mechanics of how Streaming Analytics and IoT can exchange messages with one another.

Additional Information

See Introduction to Bluemix Streaming Analytics for a more detailed walkthrough of the Streaming Analytics service.

More stories
May 1, 2019

Two Tutorials: Plan, Create, and Update Deployment Environments with Terraform

Multiple environments are pretty common in a project when building a solution. They support the different phases of the development cycle and the slight differences between the environments, like capacity, networking, credentials, and log verbosity. These two tutorials will show you how to manage the environments with Terraform.

Continue reading

April 29, 2019

Transforming Customer Experiences with AI Services (Part 1)

This is an experience from a recent customer engagement on transcribing customer conversations using IBM Watson AI services.

Continue reading

April 26, 2019

Analyze Logs and Monitor the Health of a Kubernetes Application with LogDNA and Sysdig

This post is an excerpt from a tutorial that shows how the IBM Log Analysis with LogDNA service can be used to configure and access logs of a Kubernetes application that is deployed on IBM Cloud.

Continue reading