February 16, 2016 | Written by: Aaron Baughman
Categorized: Community | Hybrid Deployments
Share this post:
An emerging cognitive hybrid cloud can see and understand images and read multilingual news articles about the top 6 socially ranked players at the Australian Open 2016: Roger Federer, Milos Raonic, Novak Djokovic, Angelique Kerber, Serena Williams and Johanna Konta. Through nimble and lightweight containers, text and image comprehension can reveal trending topics such as the weather at the Open. The additional insight will whet the appetites for deep stories and content for the Australian Open 2017.
Future work such as understanding thousands of news sources, interpreting images, discovering trending topics and multilingual processing will bring fans closer to the action. Hybrid cloud improvements to support additional fan engagement encompass the integration of Docker with Software Defined Environments (SDE), Bluemix Virtual Private Network (VPN) service, local Docker registries and the reuse of Docker containers across clouds.
News sources provide trending topics and media sentiment
During the Australian Open 2016, hundreds of news reporters and media specialists gather within the media center located next to the Rod Laver Arena. Communication channels of all types are available including cable news, radio, social media and any other imaginable digital medium. Additionally, the Australian Open (AO) Radio broadcasts live interviews, stories and play-by-play coverage from the premises.
Globally, tennis fans have an increased appetite for tennis news coverage during the Australian Open. Media syndicates publish stories about projected popular topics at the Australian Open. The textual information is generally available within electronic form. On Bluemix, a microservice by AlchemyAPI, an IBM company, has a News API. Over 250,000 news articles from 75,000 unique sources are available through a Natural Language (NL) interface. A Concept Tagging microservice accepts the news articles and extract concepts. The retrieved concepts are aggregated together into trending topics.
In addition, sentiment analysis is performed on each trending topic. A Sentiment Analysis microservice in Bluemix performs positive and negative sentiment analysis based on documents, emoticons and entities. Each trending topic is associated with a sentiment connotation and displayed on a trending topic leaderboard. The trending topic leaderboard is correlated with the trending player leaderboard to depict popular topics about the most prevalent players.
Tweets are in multiple languages
The grand slam events occur in Australia, France, England and the United States. The tennis players that participate in the tournaments are from around the world. A single look at the ATP World Tour tennis player ratings shows the diverse home country origination of the top 100 players in both the male and female category. With the fan base of the tennis tournaments spanning the globe, the international reach of social media is only limited by access to the Internet.
As a result, about half of all tweets about any topic are not English. A Bluemix language translation microservice can translate tweets that are in Brazilian Portuguese, French, Modern Standard Arabic or Spanish to English. After the text of tweets are translated into English, our standard tennis dictionaries and lexicons are applied for a Multilingual Social Leadership Board.
Multimedia Images contain valuable information
Over 6,000 tweets are posted to Twitter every second. Of those, about 50% contain images or video. On Facebook, 350 million images are uploaded per day while 100 hours of video is uploaded to YouTube per hour. According to IBM’s Global Technology Outlook, multimedia makes up 60% of Internet traffic, 70% of mobile phone traffic and 70% of unstructured data. The prevalence of multimedia data and in particular images, can leverage computer vision techniques to parse and understand images.
Two microservices on Bluemix support image processing. The first computer vision service is the Visual Recognition microservice that applies semantic classifiers to images or video frames to recognize visual entities. The applied models are either pre-trained to a general domain or labeled data of positive and negative cases are uploaded to train custom classifiers. The second microservice is the AlchemyAPI Image Tagging service. An image is uploaded to the Image Tagging service to retrieve a list of semantic tags.
As a result, the Hybrid Cloud can see images!
Docker is integrated with Software Defined Environments
Docker images provide file system level encapsulation that includes runtime and environment libraries. The definition of layers encodes changes to an underlying operating system. The instantiation of a container is lightweight and does not include virtualization of hardware like virtual machines. Containers sit directly on top of a Linux operating system, which could increase the number of applications that run on a machine by 2x to 6x. Each deployed application within a container can then be stored within a Docker image and placed into a Docker registry for later usage.
Software Defined Environment may use configuration management tools such as Chef to define infrastructure as code through the use of the Ruby programming language. Cookbooks contain recipes, templates, attributes and resources to configure and install applications or libraries. A runlist defines the set of cookbooks that are to be run on top of an operating system. Each execution of the runlist is called converging a node, which should be idempotent such that each consecutive run produces the same results. A Docker image is created through a Cloud Foundry command such as:
cf ic build –t registry-name/namespace/image-name:tag dockerfile_location
The technique of image layering creates a Docker image that includes all the changes to a base image caused by a converged node. The Docker registry points to the IBM Private Cloud’s local Docker registry. A base image such as RedHat is pulled down and instantiated into a container. All of the required Chef utilities are installed on the node and registered with the Chef server. A second Docker image is created called RedHat-Chef and uploaded to the IBM Private Cloud’s local Docker registry. Next, a Dockerfile adds an image layer on the RedHat-Chef image that converges the node and install the appropriate cookbooks. In this case, we install Graphite, StatsD and the IBM HTTP Server. A third Docker image called RedHat-Chef-Graphite is created to encapsulate the application within the container. The RedHat-Chef-Graphite container is instantiated into a container on any Linux machine. The RedHat-Chef-Graphite image are uploaded to both the IBM Private Cloud and Bluemix Docker registries.
Bluemix offers a secure VPN service
The Social Sentiment Application currently sends UDP datagrams through the Internet to reach the IBM Private Cloud with a few percentage points of packet loss. To lesson the probability of packet loss and network hops, Bluemix’s Virtual Private Network provides a secure network between Bluemix and the IBM Private Cloud.
The public IP address of the Netty Docker container or the Graphite Docker container is exposed through the private IP address with a /16 subnet. A Vyatta IP address is attached to the IBM Bluemix Gateway and acts as the Bluemix endpoint. The management protocol of IPSEC, IKE, and IPSEC attributes must match the IBM Private Cloud endpoint. The agreed upon private key is shared between the endpoints for secure data transport over the network.
The service that is accessed within the private cloud has an internal IP address that is attached to a Network Address Translation (NAT) appliance. The appliance exposes an IP address that is accessed through the VPN. Any Bluemix service or container can access the on-premise resources through the secure IPsec Tunnel.
The video How to Configure IBM VPN Service provides more details and a Bluemix blog provides additional examples.
IBM’s Private Cloud uses a local Docker registry
The on-premise cloud, IBM Private Cloud, has several repositories for binaries (artifactory), source code (git), images (IBM Smart Cloud Orchestrator) and infrastructure as code (gitlab). However, as Docker image layers become more prevalent, the IBM Private Cloud needs a local image registry for the instantiation of containers.
A Docker local registry enables the additional function to store image layers that are created by converging nodes through Chef. Each base image that supports several Linux kernels is layered with Software Defined Environment user space environments. Each of the images are shared and published on the local registry, Bluemix registry and/or Docker hub for reuse.
As a result, the IBM Private Cloud team can quickly converge nodes by SDE and then to save image layers as Docker images into the Docker local registry.
Docker images can run anywhere
A Docker image that is built on a specific Linux distribution such as RedHat is deployed on a different Linux distribution such as Ubuntu. All of the user dependencies are included within the Docker image and each Linux distribution maintains backwards compatibility on system calls. As a result, the userspace is applied to any Linux kernel.
The Software Defined Environment layered image that contains RedHat as the base image with a layer that contains Chef utilities are deployed in Bluemix or the IBM Private Cloud. As a result, the Netty based Docker container is removed in Bluemix and replaced with the RedHat-Chef-Graphite container. The RedHat-Chef-Graphite image is pulled from the local Bluemix registry and instantiated. The IBM Streaming Analytic microservice sends UDP packets directly to a local container in Bluemix. The only network connection between the IBM Private Cloud and Bluemix is used to transfer Docker images that can run anywhere.
Any converged node by Chef within the IBM Private Cloud is layered into a base image and pushed to the Bluemix Docker registry. In essence, the IBM Private Cloud becomes an SDE factory for the creation of Docker images and in this case, Graphite layered images!
The next generation of our hybrid cloud becomes amorphous in shape and size as containers built in one environment can move to different environments. The movement of images and data is facilitated by a secure and efficient Virtual Network Connections microservice in Bluemix. Now, diverse data and containers can go to where they are needed at any point in time without concern for interoperability. As a result, we can produce understanding of multimedia data and multilingual language from the previously unused data, which accounts for as much of 70% of our content.
Contributor: Michelle Welcks
Editors: John Kent, Nik McCrory, Brian O’Connell, Herbie Pearthree and Dan Kehn