How-tos

Scaling and Auto-Recovery with IBM Containers

Share this post:

As of May 23rd IBM Bluemix Container Service now provides a native Kubernetes operations experience while removing the burden of maintaining master nodes. Kubernetes itself is based on the Docker engine for managing software images and instantiating containers. Get the details.

IBM Containers, based on Docker, is a relatively new addition to Bluemix. However, some of the IBM Container features like scaling and auto-recovery has been borrowed from Cloud Foundry because of their ease-of-use. In this tutorial, we will look at scaling and auto-recovery of IBM Containers using the ic plugin for the cf CLI as outlined in the article on Setting up the IBM Containers CLI.

We will go through the following steps:

  1. Pull the Spring Boot Docker image locally and run it
  2. Push the Docker image to Bluemix
  3. Create a Container group
  4. Understanding Scaling
  5. Understanding Auto-recovery

We will mainly use the CLI to accomplish the tasks using the GUI only as needed. For more details on the IBM Containers for Bluemix, see the official documentation.

Step 1: Pull the Docker image

Verify that Docker is running.


docker ps -a

Which should yield an output like below.


CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Use the following command to pull the Spring Boot image from the public Docker hub.

docker pull ragsns/spring-boot

The output from the command looks something like below.


Digest: sha256:c069009657927d95dec371b4e59945cacebea73d5918161097237e28048f18e8
Status: Downloaded newer image for ragsns/spring-boot:latest

Verify that you pulled it with the following command.


docker images

Which should yield an output that looks something like below.


REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ragsns/spring-boot latest 825fdce3f6d1 9 hours ago 655.3 MB
java latest 36621d4ab8e3 8 days ago 641.9 MB

There are a variety of ways of running the image locally. We will run it as a daemon as below.


export CONTAINER_ID=$(docker run -d -p 8080:8080 ragsns/spring-boot)

Ensure that the container started and is running.


echo $CONTAINER_ID

should yield the container ID of the container running locally. You can get the container ID by running the docker ps -a command as well


CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29d6c1c4cb8b ragsns/spring-boot "/bin/sh -c /run.sh" 19 seconds ago Up 20 seconds 0.0.0.0:8080->8080/tcp jolly_mcnulty

Run the following command to invoke the endpoint on the application substituting the name of the Docker VM (in my case it is default)


curl -L $(docker-machine ip default):8080

Which should yield the output


Hello World!

You can accomplish the same by visiting the URL http://<docker-vm-ip>:8080. In this example, it was http://192.168.199.100:8080.

Next, we will run it remotely on the cloud.

Step 2: Push the image to Bluemix

Verify that you are logged into Bluemix and IBM containers and that you have already set the namespace with the following command.


export CONTAINER_NAMESPACE=$(cf ic namespace get)

Tag the image.


docker tag ragsns/spring-boot registry.ng.bluemix.net/$CONTAINER_NAMESPACE/spring-boot

Push the image.


docker push registry.ng.bluemix.net/$CONTAINER_NAMESPACE/spring-boot

If necessary, configure the org’s vulnerability advisor settings from the UI manager to be able to run vulnerable images with caution instead of blocking it entirely as illustrated in the diagram below.

Container Vulnerability settings for orgs

Verify that the image is indeed pushed with the following command.


cf ic images

Which should yield an output that includes the image you just pushed.


registry.ng.bluemix.net/ragsns/spring-boot latest d8dc5ff8d27b 4 seconds ago 0 B
registry.ng.bluemix.net/ibm-mobilefirst-starter latest 5996bb6e51a1 4 weeks ago 770.4 MB
registry.ng.bluemix.net/ragsns/stackschool latest 4cd60603b82f 5 weeks ago 849 MB
registry.ng.bluemix.net/ragsns/etherpad_bluemix latest 49f7ae49f906 6 weeks ago 570 MB
registry.ng.bluemix.net/ibm-node-strong-pm latest ef21e9d1656c 6 weeks ago 528.7 MB
registry.ng.bluemix.net/ibmliberty latest 2209a9732f35 6 weeks ago 492.8 MB
registry.ng.bluemix.net/ibmnode latest 8f962f6afc9a 6 weeks ago 429 MB
registry.ng.bluemix.net/ragsns/hello-cloud latest da58df9d69c1 9 weeks ago 643.9 MB
registry.ng.bluemix.net/ragsns/etherpad_bluemix_watson latest 31bffdfe5ae4 9 weeks ago 589.3 MB

Step 3: Create a Container Group

Create a Container group with the following command


cf ic group create -p 8080 --name spring-boot --hostname spring-boot-$CONTAINER_NAMESPACE --domain mybluemix.net --memory 512 --max 3 --desired 3 --auto registry.ng.bluemix.net/$CONTAINER_NAMESPACE/spring-boot

Which should yield an output something like below.


Create group in progress
Created group spring-boot (id: 123dbe80-8ae8-434c-ba79-33a64aa82636)
Minimum container instances: 0
Maximum container instances: 3
Desired container instances: 3

Running the following command will show the running containers.


cf ic ps -a

Which should yield at output like below.


CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

6be5a071-3df registry.ng.bluemix.net/ragsns/spring-boot:latest "" 12 seconds ago Running 6 seconds ago 8080/tcp sp-ds25-fxj3zsmtqacg-6bnvxlpg6oj4-server-zu4po4fkqhhl
82b74326-96c registry.ng.bluemix.net/ragsns/spring-boot:latest "" About a minute ago Running a minute ago 8080/tcp sp-ds25-e7o6rmn7uuyk-yvo42pl7y5qd-server-hxsx7kaax6va
2dd2a96f-8cb registry.ng.bluemix.net/ragsns/spring-boot:latest "" About a minute ago Running 55 seconds ago 8080/tcp

Once the status shows Running you should be able to access the app in the container. You can invoke the following command to make sure the create completed.

The following command will list the container groups running.


cf ic group list

Which should yield and output that looks something like below.


Group Id Name Status Created Updated Port
123dbe80-8ae8-434c-ba79-33a64aa82636 spring-boot CREATE_COMPLETE 2015-11-20T16:38:33Z 8080

You can also inspect the container group or the individual containers by issuing a command like below.


cf ic group inspect spring-boot

Which should yield an output like below.


{
<b>"Autorecovery": "true",</b>
"Cmd": [],
"Creation_time": "2015-11-20T16:38:33Z",
"Env": [
"sgroup_name=spring-boot",
"metrics_target=logmet.opvis.bluemix.net:9095",
"logging_password=",
"tagseparator=_",
"tenant_id=49911e9c-cf3e-4913-9e20-ff52ed6d34e3",
"space_id=49911e9c-cf3e-4913-9e20-ff52ed6d34e3",
"logstash_target=logmet.opvis.bluemix.net:9091",
"sgroup_id=123dbe80-8ae8-434c-ba79-33a64aa82636",
"tagformat=tenant_id group_id uuid",
"group_id=123dbe80-8ae8-434c-ba79-33a64aa82636"
],
"Id": "123dbe80-8ae8-434c-ba79-33a64aa82636",
"Image": "f3d173bc-c440-4283-b18c-ba78a01e49e8",
"Memory": 512,
"Name": "spring-boot",
"NumberInstances": {
"CurrentSize": 3,
"Desired": 3,
"Max": 3,
"Min": 0
},
"Port": 8080,
"Route_Status": {
"in_progress": false,
"message": "registered route successfully",
"successful": true
},
"Routes": [
"spring-boot-ragsns.mybluemix.net"
],
"Status": "CREATE_COMPLETE",
"Updated_time": null
}

Note the route from the command above and also that the number of current instances is three with Autorecovery:true.

Now we’re ready to invoke the web service endpoint of the app running remotely on a container, similar to what we did earlier with Docker by invoking the following command.


curl -L spring-boot-$CONTAINER_NAMESPACE.mybluemix.net

Which should yield the same Hello World output as before. You can do the same by vising the URL spring-boot-$CONTAINER_NAMESPACE.mybluemix.net from a browser. Now, we’re ready to dive into understanding scaling and auto-recovery.

Step 4: Understanding Scaling

There is another web service endpoint /env that we will invoke now as below.


curl -L spring-boot-$CONTAINER_NAMESPACE.mybluemix.net/env

This lists all the environment variables but we’re particularly interested in the HOSTNAME it’s running on.

If you run the following command a few times you should be able to see that the HOSTNAME cycles through the number of current intances.


curl -L spring-boot-$CONTAINER_NAMESPACE.mybluemix.net/env | grep HOSTNAME

For illustration there were three different instances as below.


HOSTNAME = instance-0007d1bb
HOSTNAME = instance-0007d1c1
HOSTNAME = instance-0007d1b8

We are not doing anything particularly useful with the multiple instances but it’s possible to take some action by spreading out the load to multiple instances.

Actually the containers are automatically front ended by a High-Availability (HA) Proxy Load Balancer to distribute the load across multiple instances. Just including the containers in a group made this possible without any need to explicitly request the HA Proxy Load Balancer.

Finally, we will try and understand container recovery.

Step 5: Understanding Container Recovery

Again, no explicit programming is required to monitor the health of a container. If a container goes down for whatever reason, it’s automatically restarted. Bluemix is responsible for monitoring the state of the container and taking corrective action whenever the actual state does not match the desired state.

Issuing the command like below.


cf ic group instances spring-boot

Will yield the containers running in the container group as below.


Container Id Name Group Image Created Updated State Private IP Port
6be5a071-3df8-4daa-a740-8516e5450d46 sp-ds25-fxj3zsmtqacg-6bnvxlpg6oj4-server-zu4po4fkqhhl spring-boot registry.ng.bluemix.net/ragsns/spring-boot:latest 2015-11-20 10:40:26 -0600 CST Running 172.30.0.119 8080
82b74326-96c7-42ae-aec2-66048df377e7 sp-ds25-e7o6rmn7uuyk-yvo42pl7y5qd-server-hxsx7kaax6va spring-boot registry.ng.bluemix.net/ragsns/spring-boot:latest 2015-11-20 10:38:51 -0600 CST Running 172.30.0.118 8080
2dd2a96f-8cb3-4d8c-8c6c-e6dd99c6ffaf sp-ds25-chggwgjztarx-pwxe6g4pfnqv-server-xnrrhfdqdujd spring-boot registry.ng.bluemix.net/ragsns/spring-boot:latest 2015-11-20 10:38:50 -0600 CST Running 172.30.0.117 8080

We will stop one of the containers either using the container Id or by issuing a command as below which will extract the container ID of the last container.


cf ic rm --force $(cf ic group instances spring-boot | tail -1 | awk '{print $1}')

This should yield an output which is the container ID of the container we just stopped as below.


2dd2a96f-8cb3-4d8c-8c6c-e6dd99c6ffaf

When you re-run the following command


cf ic group instances spring-boot

You should see that it’s minus one instance from the earlier output since we stopped the container.

If you run the command to get the HOSTNAME of the container, it only cycles between two instance IDs as


HOSTNAME = instance-0007d1bb
HOSTNAME = instance-0007d1c1

If you issue the following command to list all the instances when the container has restarted as below.


cf ic ps -a

2af6f0f5-704 registry.ng.bluemix.net/ragsns/spring-boot:latest "" 54 seconds ago Building 53 seconds ago 8080/tcp sp-ds25-chggwgjztarx-pwxe6g4pfnqv-server-dxahovejiqyq
6be5a071-3df registry.ng.bluemix.net/ragsns/spring-boot:latest "" 8 hours ago Running 8 hours ago 8080/tcp sp-ds25-fxj3zsmtqacg-6bnvxlpg6oj4-server-zu4po4fkqhhl
82b74326-96c registry.ng.bluemix.net/ragsns/spring-boot:latest "" 8 hours ago Running 8 hours ago 8080/tcp sp-ds25-e7o6rmn7uuyk-yvo42pl7y5qd-server-hxsx7kaax6va

You will notice that a container was just restarted (as in 53 seconds ago) compared with the other containers (that were started 8 hours ago).

If you re-run the command to get the HOSTNAME of the container, it now cycles between three container IDs but with a new instance ID as below.


HOSTNAME = instance-0007d1bb
HOSTNAME = instance-0007d1c1
HOSTNAME = instance-0007de7e

Just like that, without any explicit programming or any user intervention, the container was restarted.

More stories
May 1, 2019

Two Tutorials: Plan, Create, and Update Deployment Environments with Terraform

Multiple environments are pretty common in a project when building a solution. They support the different phases of the development cycle and the slight differences between the environments, like capacity, networking, credentials, and log verbosity. These two tutorials will show you how to manage the environments with Terraform.

Continue reading

April 29, 2019

Transforming Customer Experiences with AI Services (Part 1)

This is an experience from a recent customer engagement on transcribing customer conversations using IBM Watson AI services.

Continue reading

April 26, 2019

Analyze Logs and Monitor the Health of a Kubernetes Application with LogDNA and Sysdig

This post is an excerpt from a tutorial that shows how the IBM Log Analysis with LogDNA service can be used to configure and access logs of a Kubernetes application that is deployed on IBM Cloud.

Continue reading