Data Analytics

Streaming Analytics metrics using the Streams REST API

Streaming Analytics Do you want to gain a deeper understanding of your Streams apps running either on-premises or in the Streaming Analytics Bluemix service? The Streams web console enables interactive Streams monitoring. However, the Streams REST API provides powerful programmatic access to detailed metrics for Streams instances, jobs, PEs, and more. The REST API is available wherever you can access the Streams web console.

Note that the Streams REST API deals with the underlying Streams infrastructure and is different than the Streaming Analytics REST API. To learn about Streaming Analytics REST API instead, see Streaming Analytics service instance REST API reference.

This is an introduction to the Streams REST API for both Streams on-premise and Streaming Analytics in the cloud.

Prerequisites

You need either a Streams installation or Streaming Analytics Bluemix service to follow along:

Getting the Streams instances REST URI

We start with the instances REST call and dig into other REST calls from there. We derive the URI from the Streams web console URI.

Open the Streams console:

  • With on-premise Streams, click the Streams Console icon usually found on the Desktop of an application host.
  • With Streams in the cloud, navigate to your Streaming Analytics service in the Bluemix dashboard and click Launch.

The browser window navigates to a URI similar to https://[host]/streams/domain/console[#anchor]. The [host] part might include a port number, and the [#anchor] part might or might not be present. Regardless, change this URI to https://[host]/streams/rest/instances.

Open the base URI in a browser and authenticate:

  • For on-premise Streams, enter your Streams username and password. The default for Quick Start Edition is `streamsadmin` / `passw0rd`.
  • For Streaming Analytics, get the username and password from the service credentials tab of the Streaming Analytics dashboard.

Exploring the Streams REST API

From this JavaScript Object Notation (JSON), we get some general instance information, and we can dig further into the REST API by following the URIs within.

Tip: To format the JSON nicely, find and install a JSON formatter browser extension.

API call to /streams/rest/instances

<code>{
  total: 1,
  instances: [
      {
         creationUser: "streamsdomainowner",
         creationTime: 1462734010505,
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef024d-4843-8741-cd1627a71cec/operators",
         startTime: 1464717735906,
         activeViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeviews",
         activeVersion: {
            buildVersion: "20151029154444",
            productVersion: "4.1.0.0",
            minimumOSVersion: "Red Hat Enterprise Linux Server release 6.1 (Santiago)",
            editionName: "IBM InfoSphere Streams",
            minimumOSPatchVersion: "1",
            fullProductVersion: "4.1.0.0",
            minimumOSBaseVersion: "6",
            productName: "IBM InfoSphere Streams",
            architecture: "x86_64"
         },
         id: "955d9eef-024d-4843-8741-cd1627a71cec",
         views: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/views",
         pes: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes",
         resourceAllocations: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations",
         owner: "streamsdomainowner",
         restid: "955d9eef-024d-4843-8741-cd1627a71cec",
         exportedStreams: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/exportedstreams",
         startedBy: "streamsdomainowner",
         hosts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts",
         jobs: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs",
         importedStreams: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/importedstreams",
         health: "healthy",
         activeServices: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeservices",
         configuredViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/configuredviews",
         domain: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/domains/standard1",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         peConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections",
         operatorConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operatorconnections",
         resourceType: "instance",
         status: "running"
      }
   ],
   resourceType: "instanceList"
}</code>

Jobs information using the Streams REST API

To get information about jobs in an instance, simply navigate to the URI given under the jobs key in the instances JSON. In the above JSON, that is https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs. Note that if no jobs are currently running, the API call result will have total: 0 and no job details will be shown. In this case, you can submit a job from the StreamsSPLSamples repository.

API call to /streams/rest/instances/<instanceid>/jobs

<code>{
   total: 1,
   jobs: [
      {
         instance: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         submitParameters: [],
         dataPath: "/home/streamsadmin/workspace/Counter/data",
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operators?job.restid=0",
         outputPath: "Distributed",
         activeViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeviews?job.restid=0",
         id: "0",
         applicationName: "sample::Counter",
         views: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/views?job.restid=0",
         pes: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes?job.restid=0",
         resourceAllocations: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations?job.restid=0",
         applicationVersion: "1.0.0",
         restid: "0",
         startedBy: "f452348f-c47f-48d7-b734-b41f2307c680",
         hosts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts?job.restid=0",
         health: "healthy",
         jobGroup: "default",
         adlFile: "Distributed/sample.Counter.adl",
         submitTime: 1464725273000,
         applicationPath: "toolkits/Counter",
         checkpointPath: "/home/streamsadmin/workspace/Counter/data/ckpt",
         domain: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/domains/standard1",
         name: "sample::Counter_0",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs/0",
         peConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections?job.restid=0",
         operatorConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operatorconnections?job.restid=0",
         resourceType: "job",
         status: "running",
         applicationScope: "Default"
      }
   ],
   resourceType: "jobList"
}</code>

This JSON provides details such as the job name, status, health, and more. It also gives URIs we can use to dig in even further. We’ll follow one of these URIs to see the job’s PE information.

PE information using the Streams REST API

Pick one of the jobs from the jobs JSON array and navigate to its pes URI.

API call to /streams/rest/instances/<instanceid>/pes?job.restid=<jobid>

<code>{
   total: 1,
   pes: [
      {
         instance: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         restartable: true,
         relocatable: true,
         resourceTags: [],
         resourceAllocation: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations/10.121.213.173",
         indexWithinJob: 0,
         statusReason: "none",
         processId: "20317",
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operators?pe.restid=0",
         requiredConnections: "connected",
         outputPorts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/outputports",
         host: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts/10.121.213.173",
         id: "0",
         osCapabilities: [],
         connections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections?pe.restid=0",
         launchCount: 1,
         restid: "0",
         currentWorkingPath: "/home/streamsuser/.streams/var/Streams.sab_DLZ-standard1-955d9eef-024d-4843-8741-cd1627a71cec/b44fdc99-7229-4544-9c1f-cd5d2c5f9397/currentWorkingDir/0",
         health: "healthy",
         tracingLevel: "error",
         optionalConnections: "connected",
         pendingTracingLevel: null,
         domain: "https://streams-console-c6d1.ng.bl you want to gain a deeper understanding of your Streams apps running either on-premises or in the Streaming Analytics Bluemix service? Whereas the Streams wuemix.net/streams/rest/domains/standard1",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0",
         inputPorts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/inputports",
         metrics: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/metrics",
         job: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs/0",
         resourceType: "pe",
         status: "running"
      }
   ],
   resourceType: "peList"
}</code>

This gives details about the PEs in a particular job.

Conclusion

We have just scratched the surface of the Streams REST API. Dig into the other URIs inside the JSON to get even more details. It is also easy to access REST APIs programmatically, even from Streams itself. With programmatic access, you can proactively monitor your Streams instance, build your own custom dashboard, or store the instance state for later analysis.

  • Programmatically monitoring the REST API allows you to quickly diagnose and resolve Streams state anomalies. It also provides reassurance when everything is working correctly.
  • To easily access metrics not provided by the Streams web console, you can make a custom dashboard view from the Streams REST API metrics.
  • Persisting snapshots of the Streams REST API over time provides historical insight that you can use to pinpoint unexpected Streams state changes.
  • Using the REST API, you can now monitor your Streams instances both on-premise and in the cloud using the same REST interface.

Share this post:

Share on LinkedIn

Add Comment
No Comments

Leave a Reply

Your email address will not be published.Required fields are marked *

More Data Analytics Stories

Water Conservation Starter Kit demonstrates Streaming Analytics and Apache Quarks

Since our demonstration of this application in a Google+ Hangout, we have been asked numerous times about how others can try out the Water Conservation Application. The Water Conservation Application makes a great starter kit, as it demonstrates the reference architecture for Apache Quarks and how it can be used with the Streaming Analytics service. It also shows how one can easily write and deploy an IoT application using multiple services from Bluemix.

World of Watson: Bluemix sessions that you don’t want to miss

World of Watson kicks off on the 24th of October! Whether you’re going to be there in person or if you’re going to be joining the conference through the IBM app, we’ll be covering everything that you need to know about Bluemix and the conference right here. This post will whet your appetite for some of the sessions that are available to conference attendees and are special interest to Bluemix developers.

Mobile Analytics for Bluemix End of Beta

Mobile Analytics for Bluemix service became generally available (GA) on January 11, 2017, and the Beta service is being retired. If you have Beta instances of Mobile Analytics you should convert the to GA instances as soon as possible.