Analyze enterprise data in real-time with Streaming Analytics + Secure Gateway

Share this post:

One of the basic challenges of bringing Streaming Analytics to the Cloud is how to efficiently make enterprise data accessible to the Streams applications running in the Bluemix network. The IBM Streams’ adapters support a wide variety of input and output protocols. However, enterprise data sources and repositories are often secured behind on premises firewalls, while Bluemix applications and services are secured behind Bluemix firewalls. Creating a Secure Gateway service instance to establish a Secure Gateway network tunnel through the respective firewalls, and adding enterprise specific destinations, enables Streaming Analytics sources and sinks to securely access on premises data in real-time while maintaining enterprise security and data management practices.

General Streaming Analytics Use Case:

The Reaching enterprise backend with Bluemix Secure Gateway tutorial provides the building blocks on creating and configuring a Secure Gateway tunnel to enterprise data at-rest. This post extends the use case to connect IBM Streams sources and sinks to enterprise data in-motion.

Streaming Analytics through Secure Gateway

The following IBM Streams protocol adapters are compatible candidates to leverage with the Bluemix Streaming Analytics and Secure Gateway services:

  • Kafka
  • FTP
  • inet
  • JDBC
  • MQTT
  • TCP/IP

Quick Guide Overview:

Here are the high level steps required to establish connections between IBM Streams applications running in Bluemix Streaming Analtyics services to real-time enterprise data sources and sinks tunneled through Bluemix Secure Gateway services. A TCP/IP Socket Sample with detailed instructions and screen shots follows below.

  • On enterprise systems, initialize/locate on premises data sources and repositories and record corresponding host-name/ip-address:port combinations
  • From the Bluemix dashboard, create a Secure Gateway service instance in a Bluemix organization
  • From the Bluemix Secure Gateway service instance dashboard, add a gateway
  • From the Bluemix Secure Gateway service instance dashboard, add destinations to the gateway that map Cloud host-name/ip-address:port combinations to on premises host-name/ip-address:port combinations
  • From Bluemix Secure Gateway service instance dashboard, generate gateway security token
  • On enterprise systems (with access to on premises data sources and repositories), run a Secure Gateway Client to connect to the corresponding gateway in Bluemix organization
  • On enterprise systems in Secure Gateway Client shell, add destination ACL’s to host-name/ip-address:port of on premises data sources and repositories
  • From the Bluemix dashboard, create a Streaming Analytics Service instance
    in the same Bluemix organization as the Secure Gateway service instance
  • From the Bluemix Streaming Analytics service instance dashboard, launch the Streams Console
  • From the Streams Console, submit Streams application with source and sink parameters of gateway destinations

Attention: To ensure that you comply with your company or corporate security policies, consult your Chief Information Security Officer before you create or install a Secure Gateway with the intention of making internal applications or data available to Bluemix through the Secure Gateway service.

TCP/IP Socket Sample following Quick Guide:

This section walks through the details of establishing Secure Gateway connections to run an end-to-end Streaming Analytics TCP/IP Socket sample following the Quick Guide above. The TCP/IP Socket sample consists of a Java application and a Streams SPL application that communicate through TCP/IP Sockets. In the context of Bluemix Secure Gateway and Streaming Analytics services, the Java application is intended to run on an on premises system and the Streams SPL application is intended to be submitted to a Streaming Analytics service instance.

The TCP/IP Socket sample mimics a simple dice game where the Java application writes dice roll tuples to the Streams application and the Streams application processes the tuples and writes roll result tuples back to the Java application.

The TCP Socket Java application contains a source Server Socket and a sink Server Socket. When the Java application is started, the source Server Socket and the sink Server Socket listen for TCP/IP socket connections on two specified ports. Once the Java application Server Sockets are listening, the Streams SPL application can be submitted. The Streams SPL application TCPSource and TCPSink operators will connect to the corresponding Java application source and sink sockets. Once the source Server Socket receives a socket connection, the Java application writes a dice roll tuple to the source socket every second. The Streams SPL application will process each dice roll tuple off the source socket and potentially write a roll result tuple to the sink socket. Once the sink Server Socket receives a socket connection, the Java application reads roll result tuples as they become available on the sink socket.

Note: The TCP/IP Socket sample does not leverage all of the security options available on the Secure Gateway service. Follow tutorial Securing Destinations with TLS in Bluemix Secure Gateway to secure public on premises destinations with TLS mutual authentication.

On Premises Setup

First we need to setup an on premises system with the necessary software and the TCP/IP Socket Sample applications.

  • Install Secure Gateway Client
    The Getting started with the Secure Gateway provides a number of options, however the Docker option is used in the screen shots and details below.
  • Install and verify Java version 1.7 or later
    Java can be downloaded directly from: IBM Developworks or Oracle.
  • Download the Streams Integration Samples Project
    Download the project and extract the TcpServerSockets and TcpClientSockets Eclipse project directories. (direct download) For review and customization, the TcpServerSockets directory can be imported into Eclipse Java IDE and the TcpClientSockets directory can be imported into the Eclipse based IBM Streams Studio 4.1.
  • Locate Streams SPL application
    The sample.TcpClientSockets.sab in the /StreamsIntegrationSamples/TcpClientSockets directory
  • Locate Java application
    Navigate to the bin directory in the /StreamsIntegrationSamples/TcpServerSockets directory
  • Identify the on premises host-name/ip-address and two open ports to be used later
    The remainder of article will use the following on premises host-name/ip-address:port combinations:

    • TCPSource – SourceHost:8080
    • TCPSink – SinkHost:8082
  • Test Java installation and server source and sink ports
    Note: The Java application is written so it be run stand alone to test the availability of the specified ports without connections from the Streams SPL application.
  • Run java samples/TcpServerSockets test 8080 8082 10 from the /StreamsIntegrationSamples/TcpServerSockets/bin directory. Where (arg[0] == test) will cause the application to self test the source port = arg[1] and sink port = arg[2] for the specified number of rolls = arg[3].
  • Start the Java application to listen for socket connections
    Run java samples/TcpServerSockets 8080 8082 100 from the /StreamsIntegrationSamples/TcpServerSockets/bin directory. The application will listen for socket connections from a submitted sample.TcpClientSockets Streams SPL application on TCPSource port = arg[0] and TCPSink port = arg[1]. Once connected the application will write dice roll tuples and read roll result tuples until the specified number of rolls = arg[2] is reached.
    Start TcpServerSockets

Create a Secure Gateway Service Instance

Now we need to go to Bluemix and create and configure your Secure Gateway service instance. If you are familiar with Bluemix, you can complete this step using the Bluemix cf commands.

  • From the Bluemix DASHBOARD, login and navigate to desired organization and space
  • Select the Secure Gateway service tile in the Bluemix Catalog
  • Verify desired space and service name
  • Select the “Standard” plan and click “Create”
    Create Secure Gateway Service

Add a Gateway

Next we add a gateway to the service instance with which we will tunnel the application socket connections.

  • Locate Services in the desired Bluemix organization and space
  • Click on the Secure Gateway service tile to launch the Manage tab of the Secure Gateway service dashboard
  • Click on “ADD GATEWAY”
  • Enter gateway name “TCPSockets” and click on “I’M DONE”
    Add Gateway

Add On Premises Destinations

Repeat the following steps for the on premises TCPSource and TCPSink host-name/ip-address:port combinations. Each destination constructs a mapping from a gateway Cloud Host:Port to an on premises host-name/ip-address:por

  • From the Manage tab of the Secure Gateway dashboard, click on the Gateway tile to view Gateway Details page
  • Click on “ADD DESTINATION”
  • Enter on premises destination name, host-name/ip-address, port, TCP and click on “OK”
    Add TcpSource Destination
  • Click on “i” in the lower right corner of the Destination tile to view Destination Details dialog
  • Identify the gateway destination Cloud Host:Port to be used later
    TcpSource Destination

Generate Gateway Security Token

To establish a secure tunnel from the Secure Gateway Client on the on premises system to the gateway, we will need to create a security token.

    • From the Gateway Details page, click on “CONNECT CLIENT” to launch the Connect dialog
    • Select “Docker” option
    • Click on “COPY” to copy Docker connect command for connecting the on premises Secure Gateway Client

Connect Client Command

Connect Secure Gateway Client

Returning to the on premises system , we will use the security token to authenticate the client on the on premises system to the gateway in Bluemix.

  • On the on premises system, launch a Docker Terminal session
  • Enter/paste the Docker connect command
  • From the Docker shell, validate message Secure Gateway tunnel is connected
  • From the Gateway Details page in the Secure Gateway dashboard, validate client is connected
    Client Connect in Docker

Add On Premises ACL’s

By default, there are no Access Control List entries on the tunnel, so we will need to add entries for the on premises system. Repeat the following step for the on premises TCPSource and TCPSink host-name/ip-address:port combinations.

  • From the Docker shell, call the acl allow cli to allow incoming communication to on premises host-name/ip-address:port combination
    • TCPSource – act allow SourceHost:8080
    • TCPSink – act allow SinkHost:8082

    Docker ACLs

Create a Streaming Analytics Service Instance

Returning the Bluemix again, we need to create your Streaming Analytics service instance to which we will submit the Streams application. Again, if you are familiar with Bluemix, you can complete this step using the Bluemix cf commands.

  • From the Bluemix DASHBOARD, login and navigate to desired organization and space
  • Select the Streaming Analytics Service tile in the Bluemix Catalog
  • Verify desired space and service name
  • Select the “Standard” plan and click “Create”
    Create Streaming Service

Launch Streams Console

The easiest way to submit a Streams job to the Streaming Analytics service is from the Streams Console available for launch from the service instance. Alternatively, the service instance VCAP references a set of REST API’s that also can be using to submit a Streams job.

  • Locate Services in the desired Bluemix organization and space
  • Click on the Streaming Analytics service tile to launch the Manage tab of the Streaming Analytics service dashboard
  • Click on “LAUNCH” to launch the Streams Console
    Streams Dashboard

Submit Streams SPL application

Finally, we are ready to connect the Streams SPL application running in Bluemix to our Java application running on your on premises system.
Prior to submitting the Streams SPL application, validate or start the Java application is running on your on premises system as described above.

    • From the Streams Console, navigate to Application Dashboard to view Streams instance details
    • Click on “Submit Job” to launch Submit Job dialog
    • Select “Upload an application bundle file from the local file system” and click “Browse”
    • Navigate to local file directory /StreamsIntegrationSamples/TcpClientSocket/bin that was previously extracted from the downloaded project and select sample.TcpClientSockets.sab file
      Select Streams Application
    • Click on “Submit” to launch Submission-time Values dialog
    • Enter TCPSource gateway destination Cloud Host:Port identified above in the sourceHost and sourcePort fields
    • Enter TCPSink gateway destination Cloud Host:Port identified above in the sinkHost and sinkPort fields
      Submission Time Parameters
    • Click on “OK” to submit sample.TcpClientSockets application job to the Streams instance
      Running TcpSockets

Upon submission, connection activity will be visible in the Docker shell, dice roll and result activity will be visible in the console where the Java application was started. Additionally, the Streams Console provides several Streams instance and job views. Once the Java application TcpServerSockets reaches the specified number of rolls, the Source and Sink Sockets will be closed and the application ends. The samples.TcpClientSockets Streams job will need to be canceled from the Streams Console.
Cancel Streams Job

Additional Information

More How-tos stories
May 6, 2019

Use IBM Cloud Certificate Manager to Obtain Let’s Encrypt TLS Certificates for Your Public Domains

IBM Cloud Certificate Manager now lets you obtain TLS certificates signed by Let’s Encrypt. Let’s Encrypt is an automated, ACME-protocol-based CA that issues free certificates valid for 90 days.

Continue reading

April 30, 2019

Introducing IBM Analytics Engine v1.2 and Announcing the Deprecation of IBM Analytics Engine v1.0

We are excited to inform you about the new version of IBM Analytics Engine v1.2 that will be available starting May 15, 2019. Along with this release, Analytics Engine v1.0 will be retired.

Continue reading

April 16, 2019

Announcing the Deprecation of the Decision Optimization Beta Service

The End of Beta date for the Decision Optimization service is May 17, 2019. The End of Beta Support date is June 20, 2019.

Continue reading