October 26, 2017 By Oriana Zambrano 2 min read

Streaming Analytics Updates: IBM Streams Runner for Apache Beam

The IBM Streaming Analytics service is a cloud-based service for IBM Streams. Streams is an analytics platform that allows you to create applications that analyze data from a variety of sources in real time. Streaming Analytics continues to add enhancements to make it easy for you to create streaming applications however you choose. Previously, we announced integration with DSX to allow creating Streams applications in Python. Now, you can run a Beam application/pipeline in Streaming Analytics.

Imagine you are given the task to write an application for a website. The application needs to look at online users and their activity to identify popular content. You’ll need to look at logs, user clickstreams, and existing user data stored in a database. Which platform are you going to use to write this application: Apache Spark, Apache Flink, IBM Streams? Why not write the app with a single interface and choose where you run it later?

This is the goal of Apache Beam, a unified programming model for data processing—batch or streaming. Similar to Streams, Beam allows users to develop data processing applications using a set of functions to manipulate your data. Beam, however, simply provides a programming model, and leaves it up to you to select a runtime platform via a runner when you launch your application.

We’ve added the IBM Streams Runner for Apache Beam to the Streaming Analytics service so that you can run your Beam application on the Streams platform.

Beam on the Industry-leading IBM Streams Platform

IBM Streams offers a continuous, complete, and connected solution. If you use IBM Streams as your Beam runner, you’ll get a fast, stable, industry-leading platform. In addition, since the Streams runner can run in the cloud, you can develop Beam applications locally using the direct runner and then later deploy the applications to the Bluemix cloud.

No Streams Installation Required — The Streams runner allows you to directly send your applications to the Streaming Analytics service to be compiled and executed. This means there’s no need to install Streams on your system.

Interact with Beam pipelines with the newly updated Streams Console — Beam applications appear just like they are laid out in your source code. Additionally, you can view all custom metrics, console logs, data stream flow rates, and even congested streams.

Download today — The Streams Runner is now available to download through your existing Streaming Analytics service. Don’t have an existing service? Create one here.

IBM Streams Runner for Apache Beam Features

  • Support for Beam 2.0 Java SDK

  • Support primitive and custom composite Beam transforms

  • Support for custom Beam metrics

    • Counter, Distribution, and Gauge types

    • Watermark metrics are automatically created for you

  • Support for processing-time and event-time timers and window triggers

  • Support for stateful processing

  • Support for custom parameters specified at application runtime

  • Integration into the Streams Platform

    • Submit Beam applications to a Streaming Analytics service with no local Streams installation required

    • Specify local data files to be available for your application in the Streaming Analytics service

    • Support to cancel Streams job from the Beam application

    • View Beam Pipeline layouts in the Streams Graph

  • Specialized Beam SDK for Streams

    • Publish data streams for other Streams applications to utilize or subscribe to data streams for your application to consume

    • Read/write files to an IBM Object Storage OpenStack Swift for Bluemix service

Learn More

More from Announcements

IBM Hybrid Cloud Mesh and Red Hat Service Interconnect: A new era of app-centric connectivity 

2 min read - To meet customer demands, applications are expected to be performing at their best at all times. Simultaneously, applications need to be flexible and cost effective, and therefore supported by an underlying infrastructure that is equally reliant, performant and secure as the applications themselves.   Easier said than done. According to EMA's 2024 Network Management Megatrends report only 42% of responding IT professionals would rate their network operations as successful.   In this era of hyper-distributed infrastructure where our users, apps, and data…

IBM named a Leader in Gartner Magic Quadrant for SIEM, for the 14th consecutive time

3 min read - Security operations is getting more complex and inefficient with too many tools, too much data and simply too much to do. According to a study done by IBM, SOC team members are only able to handle half of the alerts that they should be reviewing in a typical workday. This potentially leads to missing the important alerts that are critical to an organization's security. Thus, choosing the right SIEM solution can be transformative for security teams, helping them manage alerts…

IBM and MuleSoft expand global relationship to accelerate modernization on IBM Power 

2 min read - As companies undergo digital transformation, they rely on APIs as the backbone for providing new services and customer experiences. While APIs can simplify application development and deliver integrated solutions, IT shops must have a robust solution to effectively manage and govern them to ensure that response times and costs are kept low for all applications. Many customers use Salesforce’s MuleSoft, named a leader by Gartner® in full lifecycle API management for seven consecutive times, to manage and secure APIs across…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters