How-tos

Streaming Analytics metrics using the Streams REST API

Share this post:

Streaming Analytics Do you want to gain a deeper understanding of your Streams apps running either on-premises or in the Streaming Analytics Bluemix service? The Streams web console enables interactive Streams monitoring. However, the Streams REST API provides powerful programmatic access to detailed metrics for Streams instances, jobs, PEs, and more. The REST API is available wherever you can access the Streams web console.

Note that the Streams REST API deals with the underlying Streams infrastructure and is different than the Streaming Analytics REST API. To learn about Streaming Analytics REST API instead, see Streaming Analytics service instance REST API reference.

This is an introduction to the Streams REST API for both Streams on-premise and Streaming Analytics in the cloud.

Prerequisites

You need either a Streams installation or Streaming Analytics Bluemix service to follow along:

Getting the Streams instances REST URI

We start with the instances REST call and dig into other REST calls from there. We derive the URI from the Streams web console URI.

Open the Streams console:

  • With on-premise Streams, click the Streams Console icon usually found on the Desktop of an application host.
  • With Streams in the cloud, navigate to your Streaming Analytics service in the Bluemix dashboard and click Launch.

The browser window navigates to a URI similar to https://[host]/streams/domain/console[#anchor]. The [host] part might include a port number, and the [#anchor] part might or might not be present. Regardless, change this URI to https://[host]/streams/rest/instances.

Open the base URI in a browser and authenticate:

  • For on-premise Streams, enter your Streams username and password. The default for Quick Start Edition is `streamsadmin` / `passw0rd`.
  • For Streaming Analytics, get the username and password from the service credentials tab of the Streaming Analytics dashboard.

Exploring the Streams REST API

From this JavaScript Object Notation (JSON), we get some general instance information, and we can dig further into the REST API by following the URIs within.

Tip: To format the JSON nicely, find and install a JSON formatter browser extension.

API call to /streams/rest/instances

<code>{
  total: 1,
  instances: [
      {
         creationUser: "streamsdomainowner",
         creationTime: 1462734010505,
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef024d-4843-8741-cd1627a71cec/operators",
         startTime: 1464717735906,
         activeViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeviews",
         activeVersion: {
            buildVersion: "20151029154444",
            productVersion: "4.1.0.0",
            minimumOSVersion: "Red Hat Enterprise Linux Server release 6.1 (Santiago)",
            editionName: "IBM InfoSphere Streams",
            minimumOSPatchVersion: "1",
            fullProductVersion: "4.1.0.0",
            minimumOSBaseVersion: "6",
            productName: "IBM InfoSphere Streams",
            architecture: "x86_64"
         },
         id: "955d9eef-024d-4843-8741-cd1627a71cec",
         views: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/views",
         pes: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes",
         resourceAllocations: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations",
         owner: "streamsdomainowner",
         restid: "955d9eef-024d-4843-8741-cd1627a71cec",
         exportedStreams: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/exportedstreams",
         startedBy: "streamsdomainowner",
         hosts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts",
         jobs: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs",
         importedStreams: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/importedstreams",
         health: "healthy",
         activeServices: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeservices",
         configuredViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/configuredviews",
         domain: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/domains/standard1",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         peConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections",
         operatorConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operatorconnections",
         resourceType: "instance",
         status: "running"
      }
   ],
   resourceType: "instanceList"
}</code>

Jobs information using the Streams REST API

To get information about jobs in an instance, simply navigate to the URI given under the jobs key in the instances JSON. In the above JSON, that is https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs. Note that if no jobs are currently running, the API call result will have total: 0 and no job details will be shown. In this case, you can submit a job from the StreamsSPLSamples repository.

API call to /streams/rest/instances/<instanceid>/jobs

<code>{
   total: 1,
   jobs: [
      {
         instance: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         submitParameters: [],
         dataPath: "/home/streamsadmin/workspace/Counter/data",
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operators?job.restid=0",
         outputPath: "Distributed",
         activeViews: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/activeviews?job.restid=0",
         id: "0",
         applicationName: "sample::Counter",
         views: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/views?job.restid=0",
         pes: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes?job.restid=0",
         resourceAllocations: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations?job.restid=0",
         applicationVersion: "1.0.0",
         restid: "0",
         startedBy: "f452348f-c47f-48d7-b734-b41f2307c680",
         hosts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts?job.restid=0",
         health: "healthy",
         jobGroup: "default",
         adlFile: "Distributed/sample.Counter.adl",
         submitTime: 1464725273000,
         applicationPath: "toolkits/Counter",
         checkpointPath: "/home/streamsadmin/workspace/Counter/data/ckpt",
         domain: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/domains/standard1",
         name: "sample::Counter_0",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs/0",
         peConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections?job.restid=0",
         operatorConnections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operatorconnections?job.restid=0",
         resourceType: "job",
         status: "running",
         applicationScope: "Default"
      }
   ],
   resourceType: "jobList"
}</code>

This JSON provides details such as the job name, status, health, and more. It also gives URIs we can use to dig in even further. We’ll follow one of these URIs to see the job’s PE information.

PE information using the Streams REST API

Pick one of the jobs from the jobs JSON array and navigate to its pes URI.

API call to /streams/rest/instances/<instanceid>/pes?job.restid=<jobid>

<code>{
   total: 1,
   pes: [
      {
         instance: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec",
         restartable: true,
         relocatable: true,
         resourceTags: [],
         resourceAllocation: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/resourceallocations/10.121.213.173",
         indexWithinJob: 0,
         statusReason: "none",
         processId: "20317",
         operators: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/operators?pe.restid=0",
         requiredConnections: "connected",
         outputPorts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/outputports",
         host: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/hosts/10.121.213.173",
         id: "0",
         osCapabilities: [],
         connections: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/peconnections?pe.restid=0",
         launchCount: 1,
         restid: "0",
         currentWorkingPath: "/home/streamsuser/.streams/var/Streams.sab_DLZ-standard1-955d9eef-024d-4843-8741-cd1627a71cec/b44fdc99-7229-4544-9c1f-cd5d2c5f9397/currentWorkingDir/0",
         health: "healthy",
         tracingLevel: "error",
         optionalConnections: "connected",
         pendingTracingLevel: null,
         domain: "https://streams-console-c6d1.ng.bl you want to gain a deeper understanding of your Streams apps running either on-premises or in the Streaming Analytics Bluemix service? Whereas the Streams wuemix.net/streams/rest/domains/standard1",
         self: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0",
         inputPorts: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/inputports",
         metrics: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/pes/0/metrics",
         job: "https://streams-console-c6d1.ng.bluemix.net/streams/rest/instances/955d9eef-024d-4843-8741-cd1627a71cec/jobs/0",
         resourceType: "pe",
         status: "running"
      }
   ],
   resourceType: "peList"
}</code>

This gives details about the PEs in a particular job.

Conclusion

We have just scratched the surface of the Streams REST API. Dig into the other URIs inside the JSON to get even more details. It is also easy to access REST APIs programmatically, even from Streams itself. With programmatic access, you can proactively monitor your Streams instance, build your own custom dashboard, or store the instance state for later analysis.

  • Programmatically monitoring the REST API allows you to quickly diagnose and resolve Streams state anomalies. It also provides reassurance when everything is working correctly.
  • To easily access metrics not provided by the Streams web console, you can make a custom dashboard view from the Streams REST API metrics.
  • Persisting snapshots of the Streams REST API over time provides historical insight that you can use to pinpoint unexpected Streams state changes.
  • Using the REST API, you can now monitor your Streams instances both on-premise and in the cloud using the same REST interface.

Add Comment
No Comments

Leave a Reply

Your email address will not be published.Required fields are marked *

More How-tos Stories

Home automation powered by Cloud Functions, Raspberry Pi, Twilio and Watson

Over the past few years, we’ve seen a significant rise in popularity for intelligent personal assistants, such as Apple’s Siri, Amazon Alexa, and Google Assistant. Though they initially appeared to be little more than a novelty, they’ve evolved to become rather useful as a convenient interface to interact with service APIs and IoT connected devices.

Continue reading

Interpreting Spring Social Twitter Data with Watson Tone Analyzer

In this post, I'll show you how to build a basic Spring app with Twitter login using Spring Social. Then we'll use Watson Tone Analyzer to determine the dominant emotion from each of the tweets on the time of the logged-in user. The project we will create will be similar to the Accessing Twitter Data Spring guide, but with a few modifications.

Continue reading

Arria brings Natural Language Generation to IBM Cloud

The Arria Natural Language Generation APIs service is an addition to the Finance category on the IBM Cloud platform. This blog post shows you how to get started with Arria’s Natural Language Generation APIs service on the IBM Cloud platform.

Continue reading