How-tos

Integrate your on-premises data with the cloud by leveraging Bluemix Hybrid Integration

Share this post:

The demands placed upon businesses in the twenty-first century are growing at an unprecedented rate. An increasing level of competition drives customers’ expectations ever higher in terms of reliability, performance, ease-of-use, and even a product’s time to market. In building cloud applications for their customers, many companies struggle to leverage highly important data that is often locked away in an on-premises datacenter.

IBM is committed to its clients’ success. We’re rapidly building enterprise-class Hybrid Integration services on top of our next generation Bluemix platform, thus enabling businesses to quickly build cloud applications at scale while leveraging their existing data in a completely secure manner. It’s not even necessary to move data from an on-premises system to the cloud! Hybrid Integration is all about letting you decide where the data should live, and providing all of the required tooling for accessing, discovering, shaping, and managing that information.

Understanding Hybrid Integration

To demonstrate the power of Hybrid Integration, let’s walk through a scenario which highlights the value of these integration services.

Boarding a plane and flying to destinations around the world is not nearly as simple as it used to be. Other than cramped flights and bad food, one of the primary complaints travelers have about flying is the inordinate amount of time they spend waiting in the security checkpoint before reaching the heart of the terminal. These checkpoints can quickly become bottlenecked, especially during typically busy travel days.

Governments and airports could globally improve customer experiences at these checkpoints by simply allowing each customer to check the current wait time at a given airport before leaving home!

Today, we’re going to build an app that will allow travelers to view and contribute security queue wait times from airports around the world.

In doing so, we’ll leverage a couple of different technologies that will allow us to:

  • Create a secure connection to an on-premises REST API using the Bluemix Secure Gateway.
  • Manage the on-premises API using Bluemix API Management.

When complete, the application will look like this:

In this tutorial we’ll be using the following Bluemix services which provide several features:

Secure Gateway

  • Create a secure tunnel from your internal network to the Bluemix cloud.
  • Expose only the systems or ports that contain data you want to access in the cloud.
  • Monitor how much data is transferred between the on-premises system and the cloud.

API Management

  • Provide a controlled facade or gateway in front of your real APIs.
  • Limit access to your APIs using authentication and authorization schemes.
  • Set rate limits on an API to prevent abuse or excessive load to the backend.

Let’s get started!

Before we begin, there are a few things you’ll need to download and install on your local machine or a VM. We recommend using Ubuntu Linux 14.04 or higher, but any platform should work. You’ll also need a Bluemix account.

Now that you’ve got the tools for the job, let’s get moving!

We’ll start by setting up the backend framework. After we’ve got a solid foundation, we’ll put the user interface in place to tie everything together.

Resources

Throughout the tutorial, we’ll refer back to the following resources:

Section 1 – Set up the datasource

Any good application needs data. Instead of needing to bring your own data endpoint, we’re providing a docker image to get you connected quickly.

The image contains a SQLite database file that has some sample data in it. To expose this database as an API, the image also runs a Loopback application. The unbundled Loopback application can be found in the source code repository that is provided at the end of this article, so feel free to check it out for yourself.

The easiest way to get the image is to pull it down from the docker registry.

Note: If you’re using boot2docker on a Mac or Windows machine, some steps will be slightly different. Watch for notes in the steps below with variances specific to boot2docker.

  1. After you’ve got Docker set up, run these commands:
    $ docker pull ibmcom/bluemix-integration-airport-sample
    $ docker run --name="hybrid_integration_data" -p 3000 ibmcom/bluemix-integration-airport-sample

    After a few seconds, you should see something like this in your console:
    INFO spawned: 'sshd' with pid 8
    INFO spawned: 'nodejs' with pid 9
    INFO success: nodejs entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

    Detached Mode:
    You can start the docker container in “detached mode” by using the -d flag before the image name. This allows you to start the container without having STDOUT/STDIN attached to the container. This allows you to reuse the terminal for later steps without needing to keep it open while the container is running. If you do this, you will need to check the logs to see if the the container started successfully by using docker logs

    $ docker run --name="hybrid_integration_data" -p 3000 <b>-d</b> ibmcom/bluemix-integration-airport-sample
    hybrid_integration_data
    $ docker logs hybrid_integration_data
  2. So far so good. The database connector is running and we should be able to browse the API created from it. To verify that everything is working, we first need to identify the container’s IP address and the local port it attached to.boot2docker note: Since Docker runs in a VM on Mac / Windows machines, you’ll need to substitute that VM’s IP address in place of the container’s IP address whenever you see <ip_address> below. To do this, simply run `boot2docker ip` after you finish initial setup and startup of boot2docker. Use that value instead of the value from `docker inspect`.Run: docker inspect hybrid_integration_data | grep IPAddress to inspect the container.
    The result will be a JSON-formatted list of variables, one of which will be labeled “IPAddress”. Copy this value.
    $ docker inspect hybrid_integration_data | grep "IPAddress"
    "IPAddress": "172.17.0.13",
  3. If you will be setting up Secure Gateway on the same machine, that’s all you need! You should now be able to access the running container by navigating your web browser to port 3000 on the IP address you just found. http://172.17.0.13:3000/explorer
  4. If you’re using another machine to host your Secure Gateway container, or you are unable to access the API using the IP address you found above, you’ll need to access it using the forwarded port on the machine hosting docker. Let’s figure out which port Docker mapped our application to.Run: `docker ps` to list the running containers. You should see something like this:
    48b9b323c705 ibmcom/bluemix-integration-airport-sample:1.0 "/usr/bin/supervisor 27 seconds ago Up 24 seconds 22/tcp, 0.0.0.0:<strong>49153</strong>-&gt;3000/tcp hybrid_integration_data

    The number above in bold represents the port that Docker mapped to our running application. If your machine is addressable as 192.168.1.13, you should be able to use the forwarded port found in this step to access the API via that IP address. With the examples provided, you would be able to access the API by navigating to http://192.168.1.13/explorer
  5. Regardless of which way you choose, you should take note of that IP/Port for later use. When navigating to the API explorer in your browser, a page similar to the following should be displayed:Capture_01Unfortunately, that data is not accessible outside of the network of the machine hosting the Docker container. We’ll fix that in the next step with our Secure Gateway.

Section 2 – Establish a Secure Gateway

To make our data accessible from anywhere, we’re going to using the Secure Gateway Bluemix service. With Secure Gateway, we get a secure tunnel to on-premises data, while still being in complete control of what’s shared.

This is where your Bluemix account comes in.

  1. If you haven’t already provisioned a new Secure Gateway instance, log in to Bluemix and navigate to the Catalog.
    Selection 173 Getting started with the Bluemix API Management Service
  2. Using the Category filter from the left, select the Integration option; then, from the filtered options, click on Secure Gateway.
    Selection_224
  3. Accept the default options, then click Create.
  4. In a moment, the Secure Gateway dashboard will appear, letting you know that you haven’t created any gateways yet. Click Add Gateway.Capture_02
  5. The “Add Gateway” page appears. Enter “Security Wait Times” into the name field, then select Connect it.Capture_10172015_0902
    We’re again going to make use of Docker by running the Secure Gateway client in a container alongside the one previously configured.
  6. Copy the Docker run command provided by the Secure Gateway, then open a new terminal on the same machine running our other container. Paste the copied command there, and press <enter> to run it.Capture_04Output similar to the following should be displayed:
    $ docker run -it bluemix/secure-gateway-client DYnGOlLXJ4u_prod_ng

    IBM Bluemix Secure Gateway Client version 1.0.2

    [2015-05-07 12:05:24.159] [INFO] secure tunnel connected

    In a few seconds, the Connected indicator should appear in place of the Disconnected message.
    Capture_10172015_0907

Finally, let’s add a destination to our gateway. Destinations point to specific endpoints that the Secure Gateway client can reach on the network where it is installed. In this case, we want to expose only the app that’s running on our data container.

  1. In the Create Destinations section, enter a descriptive name, the IP address and port we discovered back in Section #1. We also have some options to enable security on our destination to encrypt the data transmitted through the gateway, but for now we’ll leave this off by selecting No TLS.
  2. Click the + icon to add the destination.
    Capture_10172015_0917Using the Docker IP you gathered in the last step, enter the IP and the port the app is running on (3000)After you click the plus sign, you should see it enabled, as well as a cloud endpoint address that you can access the app from.
    Capture_10172015_0911Capture_10172015_0914
    Take note of the Cloud Host : Port, as that’s how you’ll be accessing the data from now on.
  3. Click I’m done.
  4. As a quick test, let’s make sure that we can access the backend data API we configured in Section #1 through the Secure Gateway we just configured.
    In a web browser, enter the following URL. It will look something like this: http://cap-sg-prd-1.integration.ibmcloud.com:15154/explorer
    http://&lt;CLOUD_HOST&gt;:&lt;CLOUD_PORT&gt;/explorer
  5. At this point, the same API Explorer we saw in Section #1 should appear, but this time it is being accessed through the Secure Gateway. To prove this is the case, try accessing this URL from another machine, or even another network. As long as it has Internet connectivity, the connection should work. Notice that attempting to use the 172.* or 192.* IP address from Section #1 does not work from another computer or network.

ACL Note: If you have issues while testing the API using the cloud endpoint, check the Secure Gateway client console. Some users may see a message noting that the connection was refused due to Access Control List. If this is the case, press the enter key to access the Secure Gateway client CLI. Then type the following (replacing with your IP/Port) to make an exception in the access control list.

cli&gt; acl allow 172.17.0.149:3000

Your test should now succeed.

Section 3 – Manage the API

So far, we’ve set up a datasource, created a Secure Gateway, and added a Destination pointing to our on-premises data. Our ultimate goal is to connect a Bluemix application to our backend API; however, we want to ensure that only those people with the right credentials can hit our data. Additionally, we may want to share this API with others down the road so that they can build their own applications using our data. In any scenario, we need to ensure that the data is not exposed to the wrong people, and that we can limit how much traffic hits the APIs themselves. Our backend can only handle so much load at one time!

  1. Open the Bluemix console and log in.
  2. Navigate to the catalog.
    Selection 173 Getting started with the Bluemix API Management Service
  3. Select the Integration filter from the left-hand menu to reduce the number of displayed options, then locate and click the API Management tile.
    Selection 174 Getting started with the Bluemix API Management Service
    The service information page will appear.
  4. API Management presently offers a single “Standard” plan.Click the Create button to provision and launch the API Management service. The Getting started page will appear.
    Selection 114 Getting started with the Bluemix API Management Service
  5. Click Import APIs, or compose a new one. The API Manager console will open in a new tab.
    Selection 115 1024x553 Getting started with the Bluemix API Management Service
    Once the API Manager console opens, it’s time to create our first API. Since we already have an API backing our underlying data, we can easily import this directly into API Management. We’ll do this using a Swagger document, which defines what our API looks like. Loopback has automatically generated this for us.
  6. Before continuing, we need to make one change to the Swagger JSON document that will be uploaded to API Management.
    Open wait-times-swagger.json from the git repository listed at the top of this article. Locate the following section (around line 796):
    Selection_209
    Change the value of host to reflect the host : port combination obtained from the Secure Gateway, and then save the file. Don’t include a protocol, like “http://”. The result should look similar to this:
    Selection_211
  7. Back in the API Manager, click the blue +API button, then select Import Swagger.
  8. Next, we can upload a Swagger document or reference one via a URL. In our case, the Swagger JSON document is provided as part of the code repository mentioned at the top of this article. Click Select a file, then locate and select the “wait-times-swagger.json” file on your disk.
    After a few moments, the APIs specified by the document are displayed. Click Add.
    Selection_207
    The newly created API appears in the list of available APIS. We’ll need to tweak a couple of things in the implentation of the API, so click the new API’s name to view the API details.Selection_208
  9. Once in the API details, you’ll need to edit the implementation details for each of the six operations.
    Capture_10172015_1711
    After clicking the edit icon, select the implementation tab and ensure proxy is selected. For the proxy URL, you will want to enter your Secure Gateway cloud endpoint followed by the applicable API call. Examples are shown below:
    Cloud Endpoint: cap-sg-prd-2.integration.ibmcloud.com:15158

    <b>POST /airports:</b> http://cap-sg-prd-2.integration.ibmcloud.com:15158/api/airports
    <b>GET /airports?filter:</b> http://cap-sg-prd-2.integration.ibmcloud.com:15158/api/airports
    <b>GET /airports/{id}?filter:</b> http://cap-sg-prd-2.integration.ibmcloud.com:15158/api/airports/{id}
    <b>POST /wait_times:</b> http://cap-sg-prd-2.integration.ibmcloud.com:15158/api/wait_times
    <b>GET /wait_times?filter:</b> http://cap-sg-prd-2.integration.ibmcloud.com:15158/api/wait_times
    <b>GET /wait_times/{id}?filter:</b> http://cap-sg-prd-2.integration.ibmcloud.com:15158/api/wait_times/{id}


    Afterwards, click the save button to solidify these changes.
    Capture_10172015_1717
  10. In order to manage our API, we need to create a plan. Begin by clicking the “Plans” glyph Selection_213 from the left navigation.
  11. Click the + Plan button.
  12. Enter Airport Wait Times in the Plan Name field, then click Add.
    Selection_214
  13. The new plan will appear in the list. Click the plan’s title to open the plan editor.
    Selection_215
  14. First, we need to add the resources from our API to the plan. Click the + Resource button.
  15. The list of APIs will appear on the left and the API’s resources on the right. If not already selected, pick the Airport Wait Times API, then select all of the resources that are part of the API. Then, click Add.
    Selection_217The selected resources will appear in the lower portion of the plan editor.
  16. Next, set a rate limit for the plan. Rate limits may be set either individually for each resource -or- for the entire plan. If a rate limit is applied at the plan-level, it is applied to all of the resources in that plan.
    Set the rate limit for the plan by clicking the edit icon under the Rate limit section. Set the rate limit to: 10,000 requests per 1 Days. Then, click Apply.
    Selection 132 Getting started with the Bluemix API Management Service
  17. Click Save in the upper-right corner of the page to persist the changes to the plan.
  18. Finally, deploy the plan.
    Click the Deploy icon to display the deployment menu. Wait until the Sandbox option appears, then select it.
    Selection 135 Getting started with the Bluemix API Management Service
    After a few moments, a success notification should appear.
    Selection_218
    At this point, the plan has been deployed and is ready to be published to Bluemix.
  19. To publish the plan to Bluemix, first open the Management page by clicking the management icon in API Manager.Selection 1411 Getting started with the Bluemix API Management Service
    The list of managed plans will be displayed, each indicating whether it’s staged, published, and so on.
    Selection_219
    Our “Airport Wait Times” plan is listed as “staged.” Let’s publish it.
  20. Expand the plan, then click the “Actions” gear and select Publish.
    Selection_220
    The publish dialog will appear. Select Custom from the “Visible to” dropdown, and then type Bluemix into each of the fields labeled “Type to add.” Then, press Publish.
    Selection_221
    After a moment, the plan status indicator should change to indicate that it is now published.
    Selection_222

At this point, the API should be available in your Bluemix organization’s private API catalog, which is simply noted by a new Custom APIs section in the Bluemix Catalog. An option with this category will also appear in the Category filter on the left-hand side of the catalog. You can browse to it in the catalog to see the resources attached to the API, provision it to your space, and so on. We’ll take care of these things in the next section as we deploy our front-end UI to Bluemix.
Selection_223

Section 4 – Deploy a front-end UI app to Bluemix

Next we’re going to deploy the app that consumes our Airport Wait Time API. If you want to dig around and see how the code works, you can obtain the source for this app by cloning the git repository mentioned earlier.

Before we actually push the button to deploy our app, we need to provision an instance of our API in our Bluemix space so that the app can bind to it.

  1. With the Bluemix Catalog still open and filtered to the Custom APIs category, click on the Airport Wait Times tile to open the API details page.
    Selection_228
  2. On the right-hand side of the page, select Leave unbound from the “App” dropdown. Then enter sample-airport-wait-times into the “Service name” field. When the app deploys, it will look specifically for an API that matches this name, so be sure you copy and paste it verbatim. Then, click Create.
    Selection_229
  3. After a few moments, the API should successfully finish provisioning and a page will be displayed showing some details about the API. From this point, you can simply return to the Bluemix Dashboard.

Now we’re ready to deploy our app. This is the final step in the process! We’ve provided a handy “Deploy to Bluemix” button right here to leapfrog some tedious steps of configuring and deploying an app manually. If you want to learn more about the details of that process, check out the Bluemix documentation.

The “Deploy to Bluemix” button will do several things for us:

  1. Create a new Node.js application in Bluemix.
  2. Bind the API we just provisioned to the Bluemix app.
  3. Push our client UI code to the newly created app.
  4. Start the app and let us know when everything is ready.

Push the button below, then click Deploy on the following page.

Deploy to Bluemix


Once the deployment process completes, that’s it! You now have an app deployed that accesses our backend dataset by making requests against the API managed through API Management, which in turn makes requests through the Secure Gateway back to the Docker container. To view the front-end UI, simply click the button labeled View your app at the bottom of the “Deploy to Bluemix” page.

At this point, you are starting to understand the power that Bluemix Hybrid Integration brings to your organization. You can expose only the specific data and systems that you want, and then manage and expose APIs those systems might provide. In the coming months, we’ll be introducing some data discovery and API generation capabilities that will make Hybrid Integration even more powerful. Stay tuned!

What solutions can you envision using Hybrid Integration? Let us know your thoughts, ideas, or even questions in the comments below. Also, feel free to ask questions over on Stack Overflow. Be sure to use the tags “bluemix” and “hybrid integration”.

Authored by:
– Cody Walker (@lostinthestory) – Software Engineer, Bluemix Hybrid Integration
– Matt Hamann (@mhamann) – Software Engineer, Bluemix Hybrid Integration

More stories
May 1, 2019

Two Tutorials: Plan, Create, and Update Deployment Environments with Terraform

Multiple environments are pretty common in a project when building a solution. They support the different phases of the development cycle and the slight differences between the environments, like capacity, networking, credentials, and log verbosity. These two tutorials will show you how to manage the environments with Terraform.

Continue reading

April 29, 2019

Transforming Customer Experiences with AI Services (Part 1)

This is an experience from a recent customer engagement on transcribing customer conversations using IBM Watson AI services.

Continue reading

April 26, 2019

Analyze Logs and Monitor the Health of a Kubernetes Application with LogDNA and Sysdig

This post is an excerpt from a tutorial that shows how the IBM Log Analysis with LogDNA service can be used to configure and access logs of a Kubernetes application that is deployed on IBM Cloud.

Continue reading