Four reasons to pay attention to Apache Mesos

Share this post:

What a difference a year makes! While MesosCon 2015 was still a relatively small conference, it tripled in size since 2014. The buzz around Mesos has steadily grown and MesosCon demonstrated that clearly.


One impressive sign of the maturing of Mesos was the collection of early adopters sharing their real world experiences using Mesos, including stories from the Apple Siri team, Twitter and Verizon. While still a relatively new technology, Mesos has some key attributes and use cases that are indeed buzzworthy.

#1: Containers

Containers are quickly becoming the de facto approach to application packaging and delivery. Their simplicity and portability are well known. One of the side effects of the rise of containers is that infrastructure needs to deal with a large number of densely packed containers at runtime, especially when compared to virtual or physical machines. In addition, the lifecycle of those containers is much more dynamic, coming and going constantly based on updates and demand. Dealing with this dense and dynamic environment requires a more capable and fine-grained resource manager, one that can understand and hand out the physical resources needed quickly and correctly. Apache Mesos was built for environments like this and environments like this are becoming the norm.

#2: Spark

Apache Spark is all about processing information, lots of information, in both batch and steaming modes. And similar to containers, Spark workloads are dense and dynamic. But Spark workloads can also be sensitive to the physical characteristics of the infrastructure, such as memory size of the node, access to fast solid state disk, or proximity to the data source. So to run Spark workloads well you need a resource manager that not only can handle the rapid swings in load inherent in analytics processing, but one that can do to smartly. Matching of the task to the RIGHT resources is crucial and awareness of the physical environment is a must. Mesos is designed to manage this problem on behalf of workloads like Spark.

#3: Diversity

Spark and Containers are critical modern workloads. But the reality is most people will run both, along with many other workloads, including such things as search and batch processing. So while having an intelligent resource manager that understands how to handle a single workload is valuable, it becomes powerful when you can start to share the physical infrastructure dynamically between a diverse set of workloads simultaneously. Many individual applications are built from a diverse workload mix in their own right, with demand shifting say from a container hosted API tier to a Spark analytics backend from moment to moment. Apache Mesos provides this mixed workload capability that will be commonly needed in the future.

#4: Community

When it comes to open technology, the community is just as important as the code. The Mesos community has been steadily growing, adding 78 new contributors in the past year, this begins to put Mesos in the same league as projects like Docker and Spark. While there is room for improvement on core committers, the project is clearly on a good path. Another healthy sign is the ecosystem around the core Mesos project, such as the ever growing list of frameworks leveraging the Mesos platform. There are frameworks for Elasticsearch, Riak, Kibana, MySQL, Kafka, Logstash, Crate, Apache Myriad (YARN on Mesos), Docker Swarm, and Kubernetes. This collection of frameworks demonstrates the diversity of workloads that can be run side-by-side on a shared resource pool.

Next big thing

Resource Managers used to be restricted to niche usage in HPC scenarios. But the rise of cloud, containers, spark, and other scale out workloads has brought this technology to the forefront. In this new world, Mesos is fast becoming the next big thing.

IBM Fellow, VP, and CTO, Cloud Platform

More stories
April 16, 2019

Webinar: 3 New Ways to Get to Containers with WebSphere ND

For many enterprises, their inventory of WebSphere apps represent a significant investment that they want to preserve and extend via containerization. This webinar introduces the benefits of container-based development and operations and explains how to reduce the uncertainty and risk involved in migrating WebSphere-based workloads to the cloud.

Continue reading

April 12, 2019

IBM Cloud to Showcase How SAP Partners Can Grow Revenue at SAP Global Partner Summit 2019

This year, IBM Cloud is a platinum sponsor of the SAP Global Partner Summit. Check out what we have in store.

Continue reading

April 10, 2019

Learn How to Accelerate Your SAP Journey to Hybrid Cloud at SAPPHIRE NOW

Learn how to accelerate your SAP journey to hybrid cloud with the IBM Cloud team at SAPPHIRE NOW—May 7–9, 2019, in Orlando, Florida.

Continue reading