Keep your streams flowing

5 min read

By: Preetam Kumar

Keep your streams flowing

Sign up for the new beta of IBM® Streaming Analytics to increase availability, boost performance and simplify scalability for high-speed data-flows

  • A new container-based architecture provides improved availability, even during patching

  • Dynamic resource allocation allows the environment to scale automatically to meet demand

  • New checkpointing features enable “at least once” and “exactly once” stream processing

Dealing with high-velocity data streams is one of the key problems that organizations need to solve as they start adopting real-time analytics.

Whatever your use case—whether you want to track footfall through a department store, deliver real-time offers to customers on your website, track a fleet of connected vehicles, or monitor a network of IoT devices—you need a stream processing engine that can capture, analyze and process thousands, or even millions, of messages per second.

Simpler stream processing

As the demand for such solutions has increased, so has the supply: there are more than a dozen open source projects that offer some kind of stream processing service, as well as numerous commercial solutions. But in most cases, setting these solutions up and running them on-premise is a complex task—and experts with the skills to maintain them are in short supply.

IBM Streaming Analytics aims to solve these challenges by providing an easy-to-use cloud-based stream processing service. The solution can simply be connected between the streaming data source and your target systems, and configured to perform whatever analytics or data processing operations you need while the data is in flight.

Get started with Streaming Analytics beta

A more flexible architecture

The latest version of Streaming Analytics—now available in open beta—has been rebuilt to run in Docker containers, instead of bare virtual machines, and uses Kubernetes for container orchestration.

The result is greater availability, because Kubernetes is able to adjust the environment in real time to maintain the desired service levels—for example, spinning up new containers to keep your streams online even when your current instance of the Streaming Analytics service needs to be taken offline for patching or upgrades.

The new architecture also enables a more dynamic approach to resource allocation—you can simply specify the maximum number of nodes that you want your environment to use, and the service will automatically scale up and down within that threshold. This helps to ensure that you only use—and pay for—the resources you actually need at any point in time, while still helping to maintain performance during sudden periods of peak load.

Self-healing streams

In a stream processing application, as in any complex distributed system, occasional failures are inevitable. Errors must be handled gracefully, and each component of the stream (known as an “operator”) must be able to recover if it gets into an inconsistent state.

Checkpointing is another feature in the new beta of Streaming Analytics, that helps to improve fault tolerance when such errors occur. It works by periodically saving the state of each operator as a checkpoint. Each checkpoint contains a delta of all the state changes that have occurred since the previous checkpoint. If an error occurs, the operator can quickly be restored to the most recent checkpoint, reducing the amount of data it needs to reprocess when it restarts.

The checkpointing feature has been tried and tested in several on-premise versions of IBM Streams, and is now ready for deployment in the cloud—bringing robust “at least once” and “exactly once” capabilities to Streaming Analytics.

Smoother integration

Finally, the beta now includes a new REST API that follows the same standards as other components of the IBM Watson Data Platform—making it even easier to compose different services and create sophisticated real-time analytics applications.

If you are already a Watson Data Platform user, the new Streaming Analytics beta could be the perfect time to take your first steps with stream processing. You can easily build a new data flow in Streams Designer, and instantly deploy it on the Streaming Analytics platform.

Alternatively, if you are currently using another open source or proprietary stream processing engine, this beta is a perfect opportunity to find out what you’ve been missing. Registering for the beta takes just a few clicks, and you can try out a basic configuration for free.

Get started with Streaming Analytics beta

How do I get started?

To learn more about how IBM Streaming Analytics can help you embed deep learning at the heart of your data science strategy, visit our website, or register for the beta here.

Be the first to hear about news, product updates, and innovation from IBM Cloud