Share this post:
Although container technology has been used in the hosting market for nearly two decades as a cheap, dense hypervisor replacement, now that it’s hitting mainstream use cases, parties with various vested interests have been trying to portray them as insecure (or at least less secure than hypervisors). However, based on the hosting market experience, where root within the container was sold to any comer for a few dollars a month, we can see based on the exploit history that such fears look to be not well founded.
Exploit history of compute systems in perspective
It is important to bear in mind that the exploit history also shows that the operating system itself (which is where both the containers and the hypervisor interfaces to the host reside) is rarely the target of hackers. Their main goal is to exploit weaknesses in the applications or middleware to give them ownership of the machine. A large amount of the time, such ownership does not even require root access; the ability to store data and send out of the network being sufficient to the purpose of selling the compromised system on as a controlled spam DDoS or exploit bot. In spite of this, much ink has been expended in the vulnerability of containers to a two level exploit where the hacker breaks into the system and then uses a second exploit within the OS kernel itself to break out of the container. The theory goes that this is much easier than for a hypervisor because the attack surface for a container is the exposed system call interface of the entire kernel whereas for a hypervisor its the much smaller hypercall interface. While apparently compelling, this simplistic analysis fails to make any allowance for the discovery and fix velocity of the exposed attack surfaces.
The discovery and fix velocity of the kernel syscall interface is huge, principally because it is also the mechanism by which non-containerized Linux systems are attacked and exploited, thus it is scrutinized by every organization that runs Linux and as soon as exploits are discovered, they are fixed within a matter of days. Contrast that to the hypervisor interface: it may be much smaller, but because it isn’t used by anything other than hypervisors, it gets correspondingly much less scrutiny and, as the venom vulnerability demonstrated, exploitable flaws can reside in the interface for years before they’re discovered and finally fixed.
Securing the cloud
All of this leads to the conclusion that the measure of actual vulnerability is related to the attack surface size divided by the fix velocity and leads to the conclusion that both hypervisors and containers are about equally vulnerable to two level exploits, as rare as they are. However, as discussed in the previous section, most of the exploits seen in the field are in the middleware and applications, so the remainder of this post will concentrate on the far more important task of exploring mechanisms for securing the cloud by preventing and detecting application and middleware vulnerabilities.
Image sharing and signing
One of the principle advantages of the new Docker model of immutable infrastructure container packaged applications is that not only can anyone build container packaged applications, but thanks to the precise dependency replication properties, they can be shared widely via repositories like the Docker Hub and CoreOS Quay. Recent scans of both show somewhere around 70-80% of the images available still contain unpatched SSL implementations with the Heartbleed vulnerability, thus demonstrating the need for extreme caution in establishing the precise contents of any image you pull out of the repositories. The essential Container Security Guide by Red Hat notes that to trust any image from the cloud, you need some sort of provenance, which can be signified by a signature over the image identifying its source.
Of course, provenance is important, but it’s not the end of the security story, or, indeed, much more than a rest stop on the way to true security. In fact, signed images from trusted sources can still contain undiscovered security vulnerabilities which may be later discovered and exploited (or much worse, exploited long before they’re discovered), so signing merely shows that the image transmitted to you was not tampered with in the transmission path.
True security involves continual checking and updating of images as vulnerabilities are discovered, even while they’re in use within your infrastructure.
Continuous image scanning
One great way of constantly ensuring that your images, even though signed by a trusted source, do not contain vulnerabilities that might be discovered after signature, is to scan them against the latest vulnerability reports each time a new one is released. Such services are provided by the CoreOS Clair project and the IBM Bluemix Vulnerability Advisor. These services provide a completely up to the minute report about the vulnerabilities in an image the moment you choose to deploy it, and may, indeed, be programmed to prevent deployment of unfixed images if the security compromise is great enough.
Even though this method provides far greater assurance, it really only goes so far as vulnerabilities that are discovered by the white hats, what about those which have only been discovered by the black hats seeking to compromise your cloud deployment?
In the foregoing, we’ve primarily dealt with scanning images at rest. Such scanning technologies are equally applicable to both hypervisors and containers; however, in taking security to the next level, we will exploit some of the transparency properties of containers which are not available in hypervisors. One of the greatest assets of containers is that they are fully transparent to inspection by they host operating system. The host can look at the process tree, the properties of the individual processes and even, should it wish, attach traces to the individual processes within a container, all without the consent of the container or, indeed, requiring any modification of the container itself. This very transparency provides the starting point for a great new line of security technology: dynamic introspection.
What dynamic introspection means is that you watch both what goes in and comes out of the container (the network and storage traffic) as well as how the processes within the container are executing. Once a profile of expected execution has been built up, it becomes possible to detect anomalies in both the execution signature as well as inbound or outbound traffic in the container. It is impossible for even the cleverest attacker to compromise the container, connect it to a bot network and repurpose it for nefarious ends while retaining the same introspection profile. Thus, a sophisticated introspection system can actually spot an infection within a running container, even if the channel by which it happens has not been reported though any official vulnerability aggregation site. Using careful dynamic introspection, therefore, offers the possibility of detecting infections even if they occur via as yet undiscovered exploits and thus provides the ultimate in security guarantees.
One of the great things about this dynamic introspection is that by exploiting the transparency properties of the container, it can all be done from the host without any co-operation (or installed agents) required within the container itself, meaning that (unlike the hypervisor case, which mostly functions via installed agents), there’s nothing the attacker can do to prevent detection via the introspection system. In this sense, dynamic introspection truly elevates container technology to greater security heights than are attainable by hypervisors. Indeed, it becomes a fair statement that by using all the listed technologies, we can move the state of the art in container security to the point where a properly signed, deployed and introspected container is, in fact, more secure than its hypervisor counterpart … now who would have thought that?
Dynamic introspection offers the hope of greatly advancing the state of the art in containers, and indeed the cloud itself. However, seeing its potential to radically shift the security landscape, we have produced as open source, an introspection toolkit to allow any interested party to plug into and introspect containers on their own, or to contribute to the growing project and help us take container security to the next level for everyone.