What's new in Version 4.3.0 and Version 4.3.1

IBM Streams Version 4.3.1

Support for SUSE Linux Enterprise Server (SLES) 12 (SP4)

SLES 12 (SP4) is supported only on x86 systems.

Secure communication to and from TCPSink and TCPSource operators by using cryptographic protocols

TCPSink and TCPSource operators now support connections that use Secure Sockets Layer (SSL) and Transport Layer Security (TLS) cryptographic protocols. You can specify the sslProtocol and sslConfigName parameters to enable SSL or TLS for these two operators at runtime. For more information, see TCPSource and TCPSink.

New toolkits

The following new toolkits are included in IBM Streams Version 4.3.1. These toolkits are also compatible with IBM Streams Version 4.3.0.

Avro

The Avro Toolkit provides functions for serialization and deserialization of messages in an Apache Avro format.

Elasticsearch

The Elasticsearch Toolkit provides functions to store tuple data as JSON documents in Elasticsearch indices.

Eventstore

The Eventstore Toolkit provides a function to insert IBM Streams tuple data into a table in IBM Db2® Event Store.

JMS

The JMS toolkit provides operators that interact with Java Message Service (JMS) systems such as IBM MQ and Apache ActiveMQ.

Mail

The Mail toolkit provides an operator to send emails from an IBM Streams application by using the SMTP protocol. It also provides an operator to generate Streams input from an email account that uses the IMAP protocol.

PMML

The PMML Toolkit provides functions to score input records by using Predictive Model Markup Language (PMML) models and to interact with the Watson Machine Learning repository.

IBM Streams Version 4.3.0

Support for using Cloud Object Storage as a checkpoint data store

IBM Cloud Object Storage is a highly scalable cloud storage service, designed for high durability, resiliency, and security.

Now, you can use Cloud Object Storage as a checkpoint data store for IBM Streams. To do so, create a Cloud Object Storage service in IBM Cloud, and provide the service credential information to IBM Streams. The service credentials information includes endpoint, bucket name, and configuration entry name that contains service keys.

Learn more...

Dynamic resource allocation

You can dynamically allocate resources to a job to ensure that your environment supports the application workloads and that resources are used more efficiently. Application resources are allocated as jobs are submitted, and freed as jobs are canceled. You can increase the number of resources that are allocated to a running job without stopping the job. You can use tags to restrict allocation of dynamic resources by using the new streamtool restrictinstance and streamtool unrestrictinstance commands.

• For information about dynamic resource allocation, see Resource allocation for application services.
• For information about configuring an instance to use dynamic resource allocation, see Configuring an instance to use dynamic resource allocation.
• For information about the options for sharing job resources when you submit a job, see the --jobResourceSharing option in the streamtool submitjob command.
• For information about increasing application resources at run time, see Adjusting application resources at run time.

Change the width of a parallel region while the job is running

In previous versions of IBM Streams, you could only set the width (number of channels) of a parallel region when you submitted the job. Now you can adjust the job deployment to the workload by changing the parallel region width while the job is running. You can also add resources if needed. Learn more...

More ways to set the width of a parallel region at submission time

In previous releases of IBM Streams, you could set the width of a parallel region in the Streams Console or Streams Studio user interfaces or by applying the @parallel annotation to an invocation of a primitive or composite operator. Now, you can also set the parallel region width with parameters in the streamtool submitjob command or in the job configuration overlay file.

Enhanced application PE high availability

You can set high availability for application repositories to ensure that the processing elements (PEs) of these applications start on new hosts after a crash. Learn more...

Multi-dimensional elasticity at runtime

In addition to elastically adjusting the number of threads, the SPL runtime can also dynamically adjust the threading models for operators during execution. Learn more...

Nested expressions and arithmetic operators for the Import operator

You can use nested expressions and arithmetic operators for the Import operator. For details, see the section about import and export filter expressions. Learn more...

Customizable properties prevent overloading the system and improve the performance of a job after it is submitted and PE restarts occur

In previous versions of IBM Streams, after you submitted a job or restarted a PE, Streams considered only the CPU usage on each resource to determine where the job or PE should be deployed. With IBM Streams Version 4.3.0, you can now do the following:

• Configure IBM Streams to consider memory usage and network bandwidth
• Use new metrics for CPU utilization, memory utilization, and network bandwidth utilization to monitor a job's performance
• Set corresponding properties for upper and lower thresholds to optimize performance and prevent your environment from being overloaded

Learn more...

Moving services

You can move an instance-related or domain-related service from one host to another compatible host by using the streamtool mvservice or streamtool mvdomainservice command. You might want to move a service prior to a host being down for maintenance or to rebalance services after maintenance is complete.

New system metrics

New system metrics have been added. Learn more...

New operator metrics for inputPort

New operator metrics have been added for inputPort. Learn more...

New streamtool commands for metrics

The following streamtool commands have been added for accessing and managing domain and instance metrics: getdomainmetrics resetdomainmetrics resetinstancemetrics.

New default Transport Layer Security (TLS) cryptographic protocols - TLSv1.2

Transport Layer Security (TLS) version 1.2 or later is recommended for IBM Streams. This version is more secure than earlier TLS versions and SSL. TLSv1.2 is the new default value for new domains. The previous default value used for existing domains is not changed automatically. Learn more...

Stream processing applications with event time

You can now develop stream processing applications with event time. Event time is a simple model that supports streams processing where time is not derived from the system time of the machine IBM Streams is running on, but from a time value associated with each tuple. The new SPL annotation @eventTime and the timeInterval window are provided for grouping tuples within specified event-time intervals and running aggregate calculations on the group. Learn more...

Redis password to ensure a secure configuration

You can now configure IBM Streams to use Redis servers that have authentication enabled as a checkpoint data store. You can specify a Redis password to ensure the secure configuration of a checkpoint data store for IBM Streams domains and instances. Learn more...

Configuring restricted or sensitive information

You can use the streamtool setrestrictedconfig command to configure restricted or sensitive information for a domain or instance, such as connection information for a password-protected database. This information can be accessed by an application, but only a domain administrator (for domain information) or instance administrator (for instance information) can access or modify these settings by using the streamtool command.

Support for rolling upgrade option with jobs running

IBM Streams Version 4.3 provides rolling upgrade support for instances with jobs running. Rolling upgrade support enables you to upgrade a domain or instance while it is running without stopping and restarting all jobs and PEs.

After you use the rolling upgrade option on your instances, all running jobs and PEs continue to run in a healthy state. After the upgrade, any job you submit to the upgraded instance will run in the new upgraded version. You can use the streamtool lsjobs --long command option to track the release version used by each running job and PE. Learn more...

Using optional types in SPL

In previous versions of IBM Streams, you could not declare that a variable or attribute in SPL might have no data value associated with it. Now you can declare that certain variables or attributes can take a null value if they have no associated data value; these types of variables and attributes are referred to as having an optional type. You can also determine if a variable or attribute has an optional type or not. Learn more...

Encrypted data transport for applications with consistent regions

In previous versions of IBM Streams, you could only use the Transmission Control Protocol (TCP) as data transport protocol for applications with consistent regions. Now you can use the Transport Layer Security (TLS) or Secure Sockets Layer (SSL) cryptographic protocols to configure data transport for applications with consistent regions. This means that data transport can now be encrypted for applications with consistent regions.

Using the FileSource operator block format

When you specify the block format for the FileSource operator, you no longer have to specify the blocksize parameter. However, note that when you do not specify a block size, the entire file is read into a single tuple and you cannot specify the hotFile parameter. In this case, the tuple size might become very large.