New Methods for Securing Containers

Share this post:

containers in the cloud

Although container technology has been used in the hosting market for nearly two decades as a cheap, dense hypervisor replacement, now that it’s hitting mainstream use cases, parties with various vested interests have been trying to portray them as insecure (or at least less secure than hypervisors). However, based on the hosting market experience, where root within the container was sold to any comer for a few dollars a month, we can see based on the exploit history that such fears look to be not well founded.

Exploit history of compute systems in perspective

It is important to bear in mind that the exploit history also shows that the operating system itself (which is where both the containers and the hypervisor interfaces to the host reside) is rarely the target of hackers. Their main goal is to exploit weaknesses in the applications or middleware to give them ownership of the machine. A large amount of the time, such ownership does not even require root access; the ability to store data and send out of the network being sufficient to the purpose of selling the compromised system on as a controlled spam DDoS or exploit bot. In spite of this, much ink has been expended in the vulnerability of containers to a two level exploit where the hacker breaks into the system and then uses a second exploit within the OS kernel itself to break out of the container. The theory goes that this is much easier than for a hypervisor because the attack surface for a container is the exposed system call interface of the entire kernel whereas for a hypervisor its the much smaller hypercall interface. While apparently compelling, this simplistic analysis fails to make any allowance for the discovery and fix velocity of the exposed attack surfaces.

The discovery and fix velocity of the kernel syscall interface is huge, principally because it is also the mechanism by which non-containerized Linux systems are attacked and exploited, thus it is scrutinized by every organization that runs Linux and as soon as exploits are discovered, they are fixed within a matter of days. Contrast that to the hypervisor interface: it may be much smaller, but because it isn’t used by anything other than hypervisors, it gets correspondingly much less scrutiny and, as the venom vulnerability demonstrated, exploitable flaws can reside in the interface for years before they’re discovered and finally fixed.

Securing the cloud

All of this leads to the conclusion that the measure of actual vulnerability is related to the attack surface size divided by the fix velocity and leads to the conclusion that both hypervisors and containers are about equally vulnerable to two level exploits, as rare as they are. However, as discussed in the previous section, most of the exploits seen in the field are in the middleware and applications, so the remainder of this post will concentrate on the far more important task of exploring mechanisms for securing the cloud by preventing and detecting application and middleware vulnerabilities.

Image sharing and signing

One of the principle advantages of the new Docker model of immutable infrastructure container packaged applications is that not only can anyone build container packaged applications, but thanks to the precise dependency replication properties, they can be shared widely via repositories like the Docker Hub and CoreOS Quay. Recent scans of both show somewhere around 70-80% of the images available still contain unpatched SSL implementations with the Heartbleed vulnerability, thus demonstrating the need for extreme caution in establishing the precise contents of any image you pull out of the repositories. The essential Container Security Guide by Red Hat notes that to trust any image from the cloud, you need some sort of provenance, which can be signified by a signature over the image identifying its source.

Of course, provenance is important, but it’s not the end of the security story, or, indeed, much more than a rest stop on the way to true security. In fact, signed images from trusted sources can still contain undiscovered security vulnerabilities which may be later discovered and exploited (or much worse, exploited long before they’re discovered), so signing merely shows that the image transmitted to you was not tampered with in the transmission path.

True security involves continual checking and updating of images as vulnerabilities are discovered, even while they’re in use within your infrastructure.

Continuous image scanning

One great way of constantly ensuring that your images, even though signed by a trusted source, do not contain vulnerabilities that might be discovered after signature, is to scan them against the latest vulnerability reports each time a new one is released. Such services are provided by the CoreOS Clair project and the IBM Bluemix Vulnerability Advisor. These services provide a completely up to the minute report about the vulnerabilities in an image the moment you choose to deploy it, and may, indeed, be programmed to prevent deployment of unfixed images if the security compromise is great enough.

Even though this method provides far greater assurance, it really only goes so far as vulnerabilities that are discovered by the white hats, what about those which have only been discovered by the black hats seeking to compromise your cloud deployment?

Dynamic introspection

In the foregoing, we’ve primarily dealt with scanning images at rest. Such scanning technologies are equally applicable to both hypervisors and containers; however, in taking security to the next level, we will exploit some of the transparency properties of containers which are not available in hypervisors. One of the greatest assets of containers is that they are fully transparent to inspection by they host operating system. The host can look at the process tree, the properties of the individual processes and even, should it wish, attach traces to the individual processes within a container, all without the consent of the container or, indeed, requiring any modification of the container itself. This very transparency provides the starting point for a great new line of security technology: dynamic introspection.

What dynamic introspection means is that you watch both what goes in and comes out of the container (the network and storage traffic) as well as how the processes within the container are executing. Once a profile of expected execution has been built up, it becomes possible to detect anomalies in both the execution signature as well as inbound or outbound traffic in the container. It is impossible for even the cleverest attacker to compromise the container, connect it to a bot network and repurpose it for nefarious ends while retaining the same introspection profile. Thus, a sophisticated introspection system can actually spot an infection within a running container, even if the channel by which it happens has not been reported though any official vulnerability aggregation site. Using careful dynamic introspection, therefore, offers the possibility of detecting infections even if they occur via as yet undiscovered exploits and thus provides the ultimate in security guarantees.

One of the great things about this dynamic introspection is that by exploiting the transparency properties of the container, it can all be done from the host without any co-operation (or installed agents) required within the container itself, meaning that (unlike the hypervisor case, which mostly functions via installed agents), there’s nothing the attacker can do to prevent detection via the introspection system. In this sense, dynamic introspection truly elevates container technology to greater security heights than are attainable by hypervisors. Indeed, it becomes a fair statement that by using all the listed technologies, we can move the state of the art in container security to the point where a properly signed, deployed and introspected container is, in fact, more secure than its hypervisor counterpart … now who would have thought that?


Dynamic introspection offers the hope of greatly advancing the state of the art in containers, and indeed the cloud itself. However, seeing its potential to radically shift the security landscape, we have produced as open source, an introspection toolkit to allow any interested party to plug into and introspect containers on their own, or to contribute to the growing project and help us take container security to the next level for everyone.

More Community stories
February 21, 2019

The Hero’s Journey to Cloud: Why Star Wars, Prometheus, and Cloud Are All Interconnected

In the Cloud Garage, we see some patterns over and over again—a development organisation wants to achieve a significant improvement and realises moving to cloud could be the way to do that. This journey often mimics the hero's journey in the traditional monomyth.

Continue reading

February 21, 2019

IBM Cloud Kubernetes Service Available in Mexico City

IBM Cloud Kubernetes Service is a managed Kubernetes offering that delivers powerful management tools, an intuitive user experience, and built-in security and isolation. Today, we are excited to announce the availability of the IBM Cloud Kubernetes Service in Mexico City, Mexico.

Continue reading

February 18, 2019

Build a Container Image Inside a Kubernetes Cluster and Push it to IBM Cloud Container Registry

We're going to show you how to build a source into a container image from a Dockerfile inside a Kubernetes cluster and push the image to IBM Cloud Container Registry with Google's Kaniko tool.

Continue reading