How-tos

Streaming Analytics metrics using the Streams REST API

Share this post:

Streaming Analytics Do you want to gain a deeper understanding of your Streams apps running either on-premises or in the Streaming Analytics Bluemix service? The Streams web console enables interactive Streams monitoring. However, the Streams REST API provides powerful programmatic access to detailed metrics for Streams instances, jobs, PEs, and more. The REST API is available wherever you can access the Streams web console.

Note that the Streams REST API deals with the underlying Streams infrastructure and is different than the Streaming Analytics REST API. To learn about Streaming Analytics REST API instead, see Streaming Analytics service instance REST API reference.

This is an introduction to the Streams REST API for both Streams on-premise and Streaming Analytics in the cloud.

Prerequisites

You need either a Streams installation or Streaming Analytics Bluemix service to follow along:

Getting the Streams instances REST URI

We start with the instances REST call and dig into other REST calls from there. We derive the URI from the Streams web console URI.

Open the Streams console:

  • With on-premise Streams, click the Streams Console icon usually found on the Desktop of an application host.
  • With Streams in the cloud, navigate to your Streaming Analytics service in the Bluemix dashboard and click Launch.

The browser window navigates to a URI similar to https://[host]/streams/domain/console[#anchor]. The [host] part might include a port number, and the [#anchor] part might or might not be present. Regardless, change this URI to https://[host]/streams/rest/instances.

Open the base URI in a browser and authenticate:

  • For on-premise Streams, enter your Streams username and password. The default for Quick Start Edition is `streamsadmin` / `passw0rd`.
  • For Streaming Analytics, get the username and password from the service credentials tab of the Streaming Analytics dashboard.

Exploring the Streams REST API

From this JavaScript Object Notation (JSON), we get some general instance information, and we can dig further into the REST API by following the URIs within.

Tip: To format the JSON nicely, find and install a JSON formatter browser extension.

API call to /streams/rest/instances

<code>{
  total: 1,
  instances: [
      {
         creationUser: "streamsdomainowner",
         creationTime: 1462734010505,
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef024d-4843-8741-cd1627a71cec/operators",
         startTime: 1464717735906,
         activeViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeviews",
         activeVersion: {
            buildVersion: "20151029154444",
            productVersion: "4.1.0.0",
            minimumOSVersion: "Red Hat Enterprise Linux Server release 6.1 (Santiago)",
            editionName: "IBM InfoSphere Streams",
            minimumOSPatchVersion: "1",
            fullProductVersion: "4.1.0.0",
            minimumOSBaseVersion: "6",
            productName: "IBM InfoSphere Streams",
            architecture: "x86_64"
         },
         id: "955d9eef-024d-4843-8741-cd1627a71cec",
         views: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/views",
         pes: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes",
         resourceAllocations: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations",
         owner: "streamsdomainowner",
         restid: "955d9eef-024d-4843-8741-cd1627a71cec",
         exportedStreams: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/exportedstreams",
         startedBy: "streamsdomainowner",
         hosts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts",
         jobs: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs",
         importedStreams: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/importedstreams",
         health: "healthy",
         activeServices: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeservices",
         configuredViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/configuredviews",
         domain: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/domains/standard1",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         peConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections",
         operatorConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operatorconnections",
         resourceType: "instance",
         status: "running"
      }
   ],
   resourceType: "instanceList"
}</code>

Jobs information using the Streams REST API

To get information about jobs in an instance, simply navigate to the URI given under the jobs key in the instances JSON. In the above JSON, that is https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs. Note that if no jobs are currently running, the API call result will have total: 0 and no job details will be shown. In this case, you can submit a job from the StreamsSPLSamples repository.

API call to /streams/rest/instances/<instanceid>/jobs

<code>{
   total: 1,
   jobs: [
      {
         instance: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         submitParameters: [],
         dataPath: "/home/streamsadmin/workspace/Counter/data",
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operators?job.restid=0",
         outputPath: "Distributed",
         activeViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeviews?job.restid=0",
         id: "0",
         applicationName: "sample::Counter",
         views: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/views?job.restid=0",
         pes: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes?job.restid=0",
         resourceAllocations: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations?job.restid=0",
         applicationVersion: "1.0.0",
         restid: "0",
         startedBy: "f452348f-c47f-48d7-b734-b41f2307c680",
         hosts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts?job.restid=0",
         health: "healthy",
         jobGroup: "default",
         adlFile: "Distributed/sample.Counter.adl",
         submitTime: 1464725273000,
         applicationPath: "toolkits/Counter",
         checkpointPath: "/home/streamsadmin/workspace/Counter/data/ckpt",
         domain: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/domains/standard1",
         name: "sample::Counter_0",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs/0",
         peConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections?job.restid=0",
         operatorConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operatorconnections?job.restid=0",
         resourceType: "job",
         status: "running",
         applicationScope: "Default"
      }
   ],
   resourceType: "jobList"
}</code>

This JSON provides details such as the job name, status, health, and more. It also gives URIs we can use to dig in even further. We’ll follow one of these URIs to see the job’s PE information.

PE information using the Streams REST API

Pick one of the jobs from the jobs JSON array and navigate to its pes URI.

API call to /streams/rest/instances/<instanceid>/pes?job.restid=<jobid>

<code>{
   total: 1,
   pes: [
      {
         instance: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         restartable: true,
         relocatable: true,
         resourceTags: [],
         resourceAllocation: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations/10.121.213.173",
         indexWithinJob: 0,
         statusReason: "none",
         processId: "20317",
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operators?pe.restid=0",
         requiredConnections: "connected",
         outputPorts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/outputports",
         host: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts/10.121.213.173",
         id: "0",
         osCapabilities: [],
         connections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections?pe.restid=0",
         launchCount: 1,
         restid: "0",
         currentWorkingPath: "/home/streamsuser/.streams/var/Streams.sab_DLZ-standard1-955d9eef-024d-4843-8741-cd1627a71cec/b44fdc99-7229-4544-9c1f-cd5d2c5f9397/currentWorkingDir/0",
         health: "healthy",
         tracingLevel: "error",
         optionalConnections: "connected",
         pendingTracingLevel: null,
         domain: "https://streams-console-c6d1.ng.bl you want to gain a deeper understanding of your Streams apps running either on-premises or in the Streaming Analytics Bluemix service? Whereas the Streams wuemix.net/streams/rest/domains/standard1",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0",
         inputPorts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/inputports",
         metrics: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/metrics",
         job: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs/0",
         resourceType: "pe",
         status: "running"
      }
   ],
   resourceType: "peList"
}</code>

This gives details about the PEs in a particular job.

Conclusion

We have just scratched the surface of the Streams REST API. Dig into the other URIs inside the JSON to get even more details. It is also easy to access REST APIs programmatically, even from Streams itself. With programmatic access, you can proactively monitor your Streams instance, build your own custom dashboard, or store the instance state for later analysis.

  • Programmatically monitoring the REST API allows you to quickly diagnose and resolve Streams state anomalies. It also provides reassurance when everything is working correctly.
  • To easily access metrics not provided by the Streams web console, you can make a custom dashboard view from the Streams REST API metrics.
  • Persisting snapshots of the Streams REST API over time provides historical insight that you can use to pinpoint unexpected Streams state changes.
  • Using the REST API, you can now monitor your Streams instances both on-premise and in the cloud using the same REST interface.

Add Comment
No Comments

Leave a Reply

Your email address will not be published.Required fields are marked *

More Data Analytics Stories

“Hallo Deutschland” – Getting Started with Bluemix in Germany

"Hallo Deutschland" or "Hello Germany" is an app I recently deployed. I didn't want to push a simple "hello world". The reason is that Bluemix Public is now available in Frankfurt, Germany!

Continue reading

Learn to use Entity Analytics on Bluemix DataConnect

Entity Analytics includes highly accurate and easy-to-use matching functionality that will link duplicate records to form a single entity, as well as discovering relationships and linkages between entities on various attributes.

Continue reading

Streaming Analytics + Message Hub: You’ll need more data

It’s never been easier to use IBM Streams and Kafka Messaging! Kafka-based Message Hub service is now available and fully integrates with the IBM Streams-based Streaming Analytics service. If your data can be streamed to Kafka, it is now trivial to process that data in real time with Streaming Analytics. Both IBM Streams and Apache Kafka are built for speed. Can your data keep up? More...

Continue reading