Best Practices: Enabling service instances in TCP/TLS Docker environments

Body

IBM Spectrum Symphony enables service instances to run in Docker containers. With 7.3.0, you can run these service instances in TLS and TCP Docker environments. 
 

Enabling service instances in TLS-enabled Docker environments

The Docker daemon (dockerd), by default, runs through a non-networked UNIX socket. It can also optionally communicate through an HTTP socket. If you need Docker to be safely accessible through the network, enable TLS by specifying the tlsverify flag and pointing Docker’s tlscacert flag to a trusted CA certificate. In the daemon mode, it only allows connections from clients authenticated by a certificate signed by that CA. In the client mode, it only connects to servers with a certificate signed by that CA.

In IBM Spectrum Symphony 7.3.0, we introduced three environment variables in the application profile - DOCKER_TLS_VERIFY, DOCKER_HOST_URL, and DOCKER_CERT_PATH - for SIM/SI communication with the dockerd via TLS.

Here's how you can configure service instances to run with TLS-enabled Docker:
 

IBM Spectrum Symphony must be installed inside a Docker container

1. Install Docker in Linux.
2. Start dockerd with TLS enabled:
   1. Create a CA, server and client keys with OpenSSL. See https://docs.docker.com/engine/security/https/.
   2. Edit /usr/lib/systemd/system/docker.service to start dockerd via TLS:

  [Service]  Type=notify  # the default is not to use systemd for cgroups because the delegate issues still  # exists and systemd currently does not support the cgroup feature set required  # for containers run by docker  ExecStart=/usr/bin/dockerd --tlsverify --tlscacert=/root/.docker/ca.pem --tlscert=/root/.docker/server-cert.pem --tlskey=/root/.docker/server-key.pem -H tcp://db05b06.example.com:2376 -D -H unix:///var/run/docker.sock  ExecReload=/bin/kill -s HUP $MAINPID  TimeoutSec=0  RestartSec=2  Restart=always

3. Stop Docker:

  systemctl stop docker

4. Reload Docker:

  systemctl daemon-reload

5. Start Docker:

  systemctl daemon-reload systemctl start docker

6. Check Docker status and make sure that dockerd is started via TLS:

  ps -ef|grep dockerd  root     24215     1  0 02:20 ?        00:00:20 /usr/bin/dockerd --tlsverify --tlscacert=/root/.docker/ca.pem --tlscert=/root/.docker/server-cert.pem --tlskey=/root/.docker/server-key.pem -H tcp://db05b06.example.com:2376 -D -H unix:///var/run/docker.sock  

3. Prepare two Docker images with the IBM Spectrum Symphony master and compute package installed. Here, the sym1 image has IBM Spectrum Symphony master installed, the sym2 image has IBM Spectrum Symphony compute installed.

  docker images  REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE  sym2                v1                  48db2a896ec0        2 weeks ago         4.92GB  sym1                v1                  571e034787fd        2 weeks ago         6.13GB  ubuntu              14.04               390582d83ead        5 weeks ago         188MB  ubuntu              16.04               b0ef3016420a        3 months ago        117MB

4. Start a Docker container with the ‘docker run’ command, use the ‘-e DOCKER_HOST_URL’ option to pass the DOCKER_HOST_URL environment variable to the container:

  docker run -it -d --privileged -v /usr/bin/docker:/usr/bin/docker -v /var/run/docker.sock:/var/run/docker.sock  -e "DOCKER_HOST_URL=tcp://db05b06.example.com:2376"  --name=master sym1:v1

5. Access this container using ‘docker exec’:

  docker ps    CONTAINER ID          IMAGE          COMMAND        CREATED    STATUS               PORTS     NAMES    0516cdba35c5          sym1:v1      "/bin/bash"    7 seconds ago Up 6 seconds          1527/tcp, 2222/tcp, 7870/tcp, 7874/tcp, 8050/tcp, 8443/tcp, 8543/tcp, 21000-22000/tcp   master    docker exec -it 0516cdba35c5 /bin/bash  root@0516cdba35c5:/#  

6. Copy the CA files that you generated in step 2.a to /root/.docker in the container. Make sure the Docker container contains the following three files (you can use the ‘docker cp’ command):

  -r--r--r-- 1 root root 2061 Jan 23 21:46 ca.pem  -r--r--r-- 1 root root 1842 Jan 17 04:30 cert.pem  -r-------- 1 root root 3243 Jan 17 04:30 key.pem  

7. In the Docker container, enable the Docker TLS feature in the application profile, for example, for the symping7.3 application:

1. Add enableDockerForServiceInstance="true" in the Consumer section.
2. Add a Docker section. The Docker image here is sym2, which has the IBM Spectrum Symphony compute package installed. If you don’t want a large image, install only dependent library/service/binary into this image.

   <Docker>      <ContainerDefinition  containerUser="root" imageName="sym2:v1" name="test">        <mount containerDir="/opt/data" hostDir="/opt/data" permission="rw"/>      </ContainerDefinition>    </Docker>  

3. Add Docker-related environment variables in the Service section:

  <Service name="sympingservice" dockerContainerDefinitionName="test" description="The Symping Service">     <osTypes>          <osType name="all" startCmd="${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/bin/sympingservice"              fileNamePattern="%sessionId%.%taskId%."              logDirectory="${SOAM_HOME}/work"              subDirectoryPattern="symping7.3/%sessionId%"              workDir="${SOAM_HOME}/work">              <env name="DOCKER_TLS_VERIFY">1</env>              <env name="DOCKER_HOST_URL">tcp://db05b06.example.com:2376</env>              <env name="DOCKER_CERT_PATH">/root/.docker</env>          </osType>      </osTypes>   ...  </Service>  

8. Register the application:

  soamreg symdocker.xml -f

9. Before submitting workload, ensure that the following preconditions are met:

1. In the container, make sure DOCKER_HOST_URL is exported to the container:

  root@7f60b78366d6:/# env|grep DOCKER  DOCKER_HOST_URL=tcp://db05b06.example.com:2376

2. In the container, make sure Docker CLIs are working properly (use ‘docker version’ to check):

  root@7f60b78366d6:/# docker version  Client:   Version:           18.09.2   API version:       1.26 (downgraded from 1.39)   Go version:        go1.10.4   Git commit:        6247962   Built:             Tue Feb 26 23:56:24 2019   OS/Arch:           linux/amd64   Experimental:      false    Server:   Engine:    Version:          1.13.1    API version:      1.26 (minimum version 1.12)    Go version:       go1.9.4    Git commit:       07f3374/1.13.1    Built:            Fri Dec  7 16:13:51 2018    OS/Arch:          linux/amd64    Experimental:     false

3. In IBM Spectrum Symphony, make sure elim.docker is working properly:

  root@7f60b78366d6:/# egosh resource list -o docker_active  NAME      docker_active  7f60b78*         18.09.2

10. Submit workload and enter the ‘docker ps’ command to check whether it is running in Docker.
11. Check the ssm/sim/dockercontroller logs for errors.

Enabling service instances in TCP-enabled Docker environments

When you use Docker to run IBM Spectrum Symphony service instances, UDS is disabled and TCP is used for SIM-SI communication. To then enable the Docker Controller to connect to the Docker daemon over TCP, you must configure the DOCKER_HOST environment variable. If the Docker Controller finds this variable defined, it uses the specified TCP value. If the variable is not defined, it uses the default UDS value (unix:///var/run/docker.sock).

This feature supports two scenarios: one with IBM Spectrum Symphony installed on the Docker host outside the Docker container, the other with IBM Spectrum Symphony installed inside the Docker container.

Scenario 1: IBM Spectrum Symphony installed on Docker host outside the Docker container

1. Install Docker in Linux.
2. Start dockerd with TCP enabled:
   1. Edit /usr/lib/systemd/system/docker.service to start the dockerd via TCP:

  [Service]  Type=notify  # the default is not to use systemd for cgroups because the delegate issues still  # exists and systemd currently does not support the cgroup feature set required  # for containers run by docker  ExecStart=/bin/dockerd -H tcp://9.21.53.197:2375  ExecReload=/bin/kill -s HUP $MAINPID  TimeoutSec=0  RestartSec=2  Restart=always

1. 2. Stop Docker:

  systemctl stop docker

1. 3. Reload Docker:

  systemctl daemon-reload

4. Start Docker:

  systemctl daemon-reload systemctl start docker

5. Check Docker status and make sure dockerd is started via TCP:

  ps -ef|grep dockerd  root      508664       1  0 Apr16 ?        00:07:02 /usr/bin/dockerd-current -H tcp://9.21.53.197:2375

3. Export the DOCKER_HOST environment variable on the Docker host:

  export DOCKER_HOST=tcp://9.21.53.197:2375

4. Enable the Docker TCP feature in the application profile, for example, for the symping7.3 application:

1. Add enableDockerForServiceInstance="true" in the Consumer section.
2. Add a Docker section. The Docker image here is the base Ubuntu image, which comes from Dockerhub. If you want to customize your image, prepare an image with IBM Spectrum Symphony installed, or only include dependent library/service/binary for SIs.

  <Docker>      <ContainerDefinition  containerUser="root" imageName="docker.io/ubuntu:16.04" name="test">        <mount containerDir="/opt/data" hostDir="/opt/data" permission="rw"/>      </ContainerDefinition>   </Docker>

3. Add the dockerContainerDefinitionName in the Service section:

  <Service name="sympingservice" dockerContainerDefinitionName="test" description="The Symping Service">      <osTypes>         <osType name="all" startCmd="${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/bin/sympingservice"             fileNamePattern="%sessionId%.%taskId%."             logDirectory="${SOAM_HOME}/work"             subDirectoryPattern="symping7.3/%sessionId%"             workDir="${SOAM_HOME}/work">          </osType>      </osTypes>   ...  </Service>

5. Register the application:

  soamreg symdocker.xml -f

6. Before submitting workload, ensure that the following preconditions are met:

1. On the Docker host, make sure DOCKER_HOST is exported to the container:

  env|grep DOCKER  DOCKER_HOST=tcp://9.21.53.197:2375

2. On the Docker host, make sure that Docker CLIs are working properly (use ‘docker version’ to check):

  docker version  Client:   Version:         1.13.1   API version:     1.26   Package version: docker-1.13.1-88.git07f3374.el7.centos.x86_64   Go version:      go1.9.4   Git commit:      07f3374/1.13.1   Built:           Fri Dec  7 16:13:51 2018   OS/Arch:         linux/amd64    Server:   Version:         1.13.1   API version:     1.26 (minimum version 1.12)   Package version: docker-1.13.1-88.git07f3374.el7.centos.x86_64   Go version:      go1.9.4   Git commit:      07f3374/1.13.1   Built:           Fri Dec  7 16:13:51 2018   OS/Arch:         linux/amd64   Experimental:    false

3. In IBM Spectrum Symphony, make sure elim.docker is working properly:

  egosh resource list -o docker_active  NAME      docker_active  7f60b78*         18.09.2

7. Submit workload and enter the ‘docker ps’ command to check whether it is running in Docker.
8. Check the ssm/sim/dockercontroller logs for errors.

Scenario 2: IBM Spectrum Symphony installed inside Docker container

1. Install Docker in Linux.
2. Start dockerd with TCP enabled:
   1. Edit /usr/lib/systemd/system/docker.service to start the dockerd via TCP:

  [Service]  Type=notify  # the default is not to use systemd for cgroups because the delegate issues still  # exists and systemd currently does not support the cgroup feature set required  # for containers run by docker  ExecStart=/bin/dockerd -H tcp://9.21.53.197:2375  ExecReload=/bin/kill -s HUP $MAINPID  TimeoutSec=0  RestartSec=2  Restart=always

1. 2. Stop Docker:

  systemctl stop docker

1. 3. Reload Docker:

  systemctl daemon-reload

4. Start Docker:

  systemctl daemon-reload systemctl start docker

5. Check Docker status and make sure dockerd is started via TCP:

  ps -ef|grep dockerd  root      508664       1  0 Apr16 ?        00:07:02 /usr/bin/dockerd-current -H tcp://9.21.53.197:2375

3. Prepare two Docker images with the IBM Spectrum Symphony master and compute package installed. Here, the sym1 image has IBM Spectrum Symphony master installed, the sym2 image has IBM Spectrum Symphony compute installed.

  docker images  REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE  sym2                v1                  48db2a896ec0        2 weeks ago         4.92GB  sym1                v1                  571e034787fd        2 weeks ago         6.13GB  ubuntu              14.04               390582d83ead        5 weeks ago         188MB  ubuntu              16.04               b0ef3016420a        3 months ago        117MB

4. Start a Docker container via ‘docker run’ command, use ‘-e DOCKER_HOST’ to pass the DOCKER_HOST environment variable to the container:

  docker run -it -d --privileged -v /usr/bin/docker:/usr/bin/docker -e "DOCKER_HOST = tcp://9.21.53.197:2375 " --name=master sym1:v1

5. Access this container using ‘docker exec’:

  docker ps  CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                                                                   NAMES    7f60b78366d6        sym1:v1             "/bin/bash"         2 days ago          Up 2 days           1527/tcp, 2222/tcp, 7870/tcp, 7874/tcp, 8050/tcp, 8443/tcp, 8543/tcp, 21000-22000/tcp   master    docker exec -it 7f60b78366d6 /bin/bash  root@7f60b78366d6:/#

6. Enable the Docker TCP feature in the application profile, for example, for the symping7.3 application:

1. Add enableDockerForServiceInstance="true" in the Consumer section.
2. Add a Docker section. The Docker image here is sym2, which has the IBM Spectrum Symphony compute package installed. If you don’t want a large image, install only dependent library/service/binary into this image.

  <Docker>      <ContainerDefinition  containerUser="root" imageName="sym2:v1" name="test">        <mount containerDir="/opt/data" hostDir="/opt/data" permission="rw"/>      </ContainerDefinition>  </Docker>

3. Add the dockerContainerDefinitionName in the Service section:

  <Service name="sympingservice" dockerContainerDefinitionName="test" description="The Symping Service">      <osTypes>         <osType name="all"                startCmd="${SOAM_HOME}/${VERSION_NUM}/${EGO_MACHINE_TYPE}/bin/sympingservice"              fileNamePattern="%sessionId%.%taskId%."              logDirectory="${SOAM_HOME}/work"              subDirectoryPattern="symping7.3/%sessionId%"              workDir="${SOAM_HOME}/work">          </osType>      </osTypes>   ...  </Service>

7. Register the application:

  soamreg symdocker.xml -f

8. Before submitting workload, ensure that the following preconditions are met:

1. On the Docker host, make sure that DOCKER_HOST is exported to the container:

  env|grep DOCKER  DOCKER_HOST=tcp://9.21.53.197:2375

2. On the Docker host, make sure that Docker CLIs are working properly (use ‘docker version’ to check):

  docker version  Client:   Version:         1.13.1   API version:     1.26   Package version: docker-1.13.1-88.git07f3374.el7.centos.x86_64   Go version:      go1.9.4   Git commit:      07f3374/1.13.1   Built:           Fri Dec  7 16:13:51 2018   OS/Arch:         linux/amd64    Server:   Version:         1.13.1   API version:     1.26 (minimum version 1.12)   Package version: docker-1.13.1-88.git07f3374.el7.centos.x86_64   Go version:      go1.9.4   Git commit:      07f3374/1.13.1   Built:           Fri Dec  7 16:13:51 2018   OS/Arch:         linux/amd64   Experimental:    false

3. In IBM Spectrum Symphony, make sure elim.docker is working properly:

  egosh resource list -o docker_active  NAME      docker_active  7f60b78*         18.09.2

9. Submit workload and enter the ‘docker ps’ command to check whether it is running in Docker:

  root@7f60b78366d6:/# docker ps    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                   NAMES    61228f28f86d        sym2:v1             "/bin/sh -c /opt/ibm…"   2 seconds ago       Up 1 second                                                                                                 symdocker-sympingservice-1028470-2  4cc8586182b1        sym2:v1             "/bin/sh -c /opt/ibm…"   2 seconds ago       Up 1 second                                                                                                 symdocker-sympingservice-1028470-1

10. Check the ssm/sim/dockercontroller logs for any errors.

References

Enabling service instances in TLS-enabled Docker environments

Enabling service instances in TCP-enabled Docker environments

Got questions?

We’d love to hear from you. If you’ve got comments or questions, add a comment below. You can also start a topic in our forum.