How-tos

Getting started with Streaming Analytics and the Internet of Things

Share this post:

The Internet of Things (IoT) service and the Streaming Analytics service in Bluemix have two things in common: Both were created for real-time, and both are extremely scalable.  So it’s a natural fit to use the two services together in an Bluemix application, with the IoT service serving as an aggregator of device information, and the Streaming Analytics service performing real-time analytics on streams of device data.

With the announcement that Streaming Analytics is generally available, this article describes a typical use case and explains how to exchange messages between the two services over the MQTT protocol.

Overview of the most common use case

In the most common scenario, you have a set of devices producing information in real-time that you need to analyze and two services. These services are:

IoT Service: This service will be used to collect the data from the devices:

  • Devices registered with your IoT service will publish MQTT messages to your IoT service
  • The messages will contain information about the devices that you want to analyze

Streaming Analytics Service: This service will be used to analyze the device information

  • Your Streaming Analytics application will subscribe to the device messages published to your IoT service
  • The device messages will be analyzed according to the logic you’ve defined in your Streaming Analytics application

Because of the low latency of the IoT and Streaming Analytics technologies, the analysis of the device data happens in real-time, and both technologies are capable of scaling to support analysis of information from millions of individual devices. There are a number of variations possible on this base scenario, but regardless of the exact scenario, the information above should help you get started using the two services together.

Application components and how they interact

In an end-to-end Bluemix application, the diagram below shows the typical components and entities that will be a part of the solution:

Bluemix application composed of the Node.js runtime, the Internet of Things service, and the Streaming Analytics service.
Bluemix application composed of Node.js runtime, the IoT service, and Streaming Analytics service.

In addition to the two services discussed in this post, the figure above shows two additional components: i) The physical devices supplying the data to be analyzed, and ii) a Bluemix application written using the Node.js runtime. The overall application flow is described below:

  • The devices have been registered with the IoT service and they publish data to IoT via MQTT messages.
  • A Streams application deployed to the Streaming Analytics service subscribes to the device information in IoT.  Data flows as MQTT messages into the Streaming Analytics service.
  • The Streams application performs real-time analysis of the device data stream.  The analysis performed depends upon the logic and analytics used in the Streams application.  Examples of analysis might include anomaly detection, failure prediction, aggregation, etc.
  • The Streams applications publishes its results back to IoT as MQTT messages.  These messages could be thought of as notifications, alerts, events, etc.  Notice that the results MQTT stream is not as thick as the input MQTT stream because the results of analysis are normally not as voluminous as the data they were derived from.
  • The Bluemix (Node.js) application can subscribe to the results that were published back to IoT and take the appropriate actions, according to the needs of the solution.

Hands-on with the Streaming Analytics Integration Samples

This section describes how you can get an instance of the Internet of Things service and an instance of the Streaming Analytics service to exchange MQTT messages with one another.  Once you’ve established this integration, you should be able to build upon this example to create a Bluemix application that meets your specific requirements.

Create and configure your Internet of Things instance

  1. Log in to Bluemix. In the dashboard, click USE SERVICES OR APIS.  Choose Internet of Things from the service catalog.  Choose the selected plan of Free and click CREATE to create the service.
  2. A welcome page will be displayed.  Click the Launch dashboard button.  The Internet of Things dashboard will be displayed.
  3. The IoT dashboard displays an Organization ID near the top of the page.  Copy your organization ID and save it for later.
  4. Next, from the IoT dashboard, select the Access and then the API Keys tab.  Then click the Generate API Key button.  Copy both the API Key and the Authentication Token values and save them for later.

Create your Streaming Analytics instance

  1. From the Bluemix dashboard, click USE SERVICES OR APIS.  Choose Streaming Analytics from the catalog.  Choose the default plan and click CREATE to create the service.
  2. The Streaming Analytics dashboard will appear.  Use the Launch button to launch the Streaming Analytics console.

Download and submit the sample Streams application

  1. Download the zip file from the Streams Integration Samples project in Bluemix DevOps Services.  This zip file contains the pre-built Streams application you will deploy, as well as the source code for the application.
  2. Extract the zip to your local file system.
  3. Go to the Streaming Analytics console that you launched earlier, and select Submit Job.
  4. A Submit Job view will appear.  Click Browse and navigate to the root of the zip file that you extracted, then into the InternetOfThings directory.  Select the Main.sab file.  Then click Next in the Submit Job dialog.
  5. On the next view, you are prompted to fill in submission time parameters required by the application.  Fill in the values for Authentication_Token, Org_ID, and API_Key that you saved in the earlier steps.
  6. Press Submit, and the sample Streams application will be deployed to your Streaming Analytics instance.  Use the Streaming Analytics console to verify that the job is submitted and running.


Three views of the Streaming Analytics Console from steps 3 through 6 above (click to enlarge)

If you were able to successfully complete the instructions listed above, you should now have a running Streaming Analytics application.  In addition, your application is:

  • publishing MQTT messages to your Internet of Things instance
  • subscribing to those same messages as they appear in your IoT instance, and ingesting them back into the Streams application

You can use the Streaming Analytics console to verify that messages are flowing out to IoT and back into the Streams application.  Sending a message out of a Streams application and ingesting that same message back into the same Streams application is not a typical pattern, but it does serve as a good example of the mechanics of how Streaming Analytics and IoT can exchange messages with one another.

Additional Information

See Introduction to Bluemix Streaming Analytics for a more detailed walkthrough of the Streaming Analytics service.

Add Comment
3 Comments

Leave a Reply

Your email address will not be published.Required fields are marked *


Bryan

How much Stream creation and editing is possible without Streams Studio? I don’t see any visual mechanisms on BlueMix to edit existing streams.

Thanks!

Reply

    MikeBranson

    Bryan – You can still use Streams Studio to develop your Streams Application for Bluemix. The development model is to develop and test your application locally, and then deploy it to the cloud.

    Most of our Bluemix users download the Streams Quickstart edition. This enables you to have your own (complete) Streams environment to develop and test in. You can run Quickstart in a VM on your laptop or you can install it directly to run on Linux.

    See https://www.ng.bluemix.net/docs/services/StreamingAnalytics/c_getting_app_bluemix.html#t_deploytocloud for more information about the development model and links to the Quickstart edition.

    Reply

      BryanMiller

      Many thanks Mike. I have downloaded the QSE image and in deep learning mode 😉

      Bryan

      Reply
More How-tos Stories

Microservices on Bluemix: A multi-compute approach using Cloud Foundry and OpenWhisk

One of the benefits of microservices is that each of the applications that compose your solution can be built using whatever language or framework makes sense for that particular application. Learn here how we take Microservices implementations one step further.

Continue reading

Deploy a Drupal Application on IBM BlueMix

Drupal is a free and open-source content management framework written in PHP. It is used as a back-end framework for at least 2.1% of all websites worldwide ranging from personal blogs to corporate, political, and government sites including whitehouse.gov and data.gov.uk. It is also used for knowledge management and business collaborationIn order to have Drupal work on the IBM BlueMix Platform, do the following...

Continue reading

Securing Docker containers with Vulnerability Advisor in Bluemix

Vulnerability Advisor (VA) is a component in the IBM Bluemix Container Service. It's a security management tool that can be used use to identify and manage vulnerabilities and configuration best practices for Docker images and Docker instances.

Continue reading