September 9, 2015 | Written by: Jason McGee
Share this post:
What a difference a year makes! While MesosCon 2015 was still a relatively small conference, it tripled in size since 2014. The buzz around Mesos has steadily grown and MesosCon demonstrated that clearly.
One impressive sign of the maturing of Mesos was the collection of early adopters sharing their real world experiences using Mesos, including stories from the Apple Siri team, Twitter and Verizon. While still a relatively new technology, Mesos has some key attributes and use cases that are indeed buzzworthy.
Containers are quickly becoming the de facto approach to application packaging and delivery. Their simplicity and portability are well known. One of the side effects of the rise of containers is that infrastructure needs to deal with a large number of densely packed containers at runtime, especially when compared to virtual or physical machines. In addition, the lifecycle of those containers is much more dynamic, coming and going constantly based on updates and demand. Dealing with this dense and dynamic environment requires a more capable and fine-grained resource manager, one that can understand and hand out the physical resources needed quickly and correctly. Apache Mesos was built for environments like this and environments like this are becoming the norm.
Apache Spark is all about processing information, lots of information, in both batch and steaming modes. And similar to containers, Spark workloads are dense and dynamic. But Spark workloads can also be sensitive to the physical characteristics of the infrastructure, such as memory size of the node, access to fast solid state disk, or proximity to the data source. So to run Spark workloads well you need a resource manager that not only can handle the rapid swings in load inherent in analytics processing, but one that can do to smartly. Matching of the task to the RIGHT resources is crucial and awareness of the physical environment is a must. Mesos is designed to manage this problem on behalf of workloads like Spark.
Spark and Containers are critical modern workloads. But the reality is most people will run both, along with many other workloads, including such things as search and batch processing. So while having an intelligent resource manager that understands how to handle a single workload is valuable, it becomes powerful when you can start to share the physical infrastructure dynamically between a diverse set of workloads simultaneously. Many individual applications are built from a diverse workload mix in their own right, with demand shifting say from a container hosted API tier to a Spark analytics backend from moment to moment. Apache Mesos provides this mixed workload capability that will be commonly needed in the future.
When it comes to open technology, the community is just as important as the code. The Mesos community has been steadily growing, adding 78 new contributors in the past year, this begins to put Mesos in the same league as projects like Docker and Spark. While there is room for improvement on core committers, the project is clearly on a good path. Another healthy sign is the ecosystem around the core Mesos project, such as the ever growing list of frameworks leveraging the Mesos platform. There are frameworks for Elasticsearch, Riak, Kibana, MySQL, Kafka, Logstash, Crate, Apache Myriad (YARN on Mesos), Docker Swarm, and Kubernetes. This collection of frameworks demonstrates the diversity of workloads that can be run side-by-side on a shared resource pool.
Next big thing
Resource Managers used to be restricted to niche usage in HPC scenarios. But the rise of cloud, containers, spark, and other scale out workloads has brought this technology to the forefront. In this new world, Mesos is fast becoming the next big thing.