How-tos

Docker tip to work around lack of network connectivity between client and container

Share this post:

As of May 23rd IBM Bluemix Container Service now provides a native Kubernetes operations experience while removing the burden of maintaining master nodes. Kubernetes itself is based on the Docker engine for managing software images and instantiating containers. Get the details.

If you can invoke docker exec on a given container from a given client, then that client can open a real, fully general, SSH connection to that container — even if that client does not have network connectivity to the container! For example, your containers in a container group in IBM Containers do not have public IPv4 addresses — but you can open SSH connections to them from anywhere on the Internet. I am writing here from a Linux/Unix perspective.

This is valuable because, as you may know, SSH provides more functionality than docker exec does directly: with SSH you can do file transfers (scp, sftp) and create a variety of network tunnels/proxies (TCP host&port forwarding and SOCKS proxies); SSH can also do tun/tap tunnels but containers do not usually have the kernel capabilities needed to configure tun/tap tunnels.

The key idea is to combine (a) your SSH client’s openness to alternate ways of opening the underlying TCP connection and (b) the ability of docker exec to open a TCP connection to a container through the management path (rather than the usual data path). Although your SSH client normally uses its usual TCP/IP stack to open the TCP connection on which the SSH protocol is based, your SSH client can be configured to open the TCP connection by issuing a shell command. See the ProxyCommand configuration option, if you do not already know about it.

With a little imagination you can find ways to use docker exec to produce such a TCP connection. Since the management path is usually engineered for different load than the data path, you should temper your expectations accordingly. This is probably not a good way to get a lot of data through, but should be acceptable for occasional light use.

For a simple example, suppose you have a container that is already running an SSH daemon and already has netcat installed. Your first step, of course, is to determine the correct host key fingerprint for that container. Issuing the following command on your client would be one way to find the ECDSA host key fingerprint for a container that is running Debian or Ubuntu and whose ID starts with 5b569ba0:

docker exec 5b569ba0 ssh-keygen -l -f /etc/ssh/ssh_host_ecdsa_key.pub

Now that you are prepared to answer the host key verification question, you can open an SSH connection to that container. Issuing the following command would be one way to do so:

ssh -o 'ProxyCommand docker exec -i %h nc localhost %p' -i id_ecdsa root@5b569ba0

Of course you can get much fancier. For example, you can edit your SSH client configuration so that simply asking for a connection to a host named ${container ID prefix}.containers will apply this technique. Following is what you could put in your SSH client configuration file to accomplish that:

Host *.containers
ProxyCommand docker exec -i "$(echo '%h' | sed s/.containers//)" nc localhost %p

With that client configuration in place, a command like the following command will open a connection to a container:

ssh -i id_ecdsa root@5b569ba0.containers

Principal RSM - Workload Centric Computing

More stories
May 1, 2019

Two Tutorials: Plan, Create, and Update Deployment Environments with Terraform

Multiple environments are pretty common in a project when building a solution. They support the different phases of the development cycle and the slight differences between the environments, like capacity, networking, credentials, and log verbosity. These two tutorials will show you how to manage the environments with Terraform.

Continue reading

April 29, 2019

Transforming Customer Experiences with AI Services (Part 1)

This is an experience from a recent customer engagement on transcribing customer conversations using IBM Watson AI services.

Continue reading

April 26, 2019

Analyze Logs and Monitor the Health of a Kubernetes Application with LogDNA and Sysdig

This post is an excerpt from a tutorial that shows how the IBM Log Analysis with LogDNA service can be used to configure and access logs of a Kubernetes application that is deployed on IBM Cloud.

Continue reading