As the use of cloud and web servers expanded, traditional log management was not keeping up. LogDNA saw a clear need to address data sprawl in the modern, cloud-native development stack. Its innovative software-as-a-service (SaaS) platform built on Kubernetes caught the attention of the IBM Cloud® team, which wove it into its global framework. As both an IBM Business Partner and a client, LogDNA was able to grow and strengthen its DevOps capabilities.
As applications and infrastructures grow more complex and dynamic, how could DevOps teams automate and streamline critical log collection, monitoring and analysis in cloud, on-premises and hybrid environments?
Synergy helped weave the LogDNA logging platform into the IBM global ecosystem, binding the two companies together as both partners and consults-clients to deliver innovative, managed Kubernetes services.
In 2013, while working on their third startup company, Christopher Ngyuen and Lee Liu had an epiphany: no log management tool was fully capable of handling the intense needs of a modern, cloud-native development stack. Traditional log analysis methodology simply wasn’t responsive or scalable enough to manage the level of data sprawl that happens in dynamic cloud environments.
If it frustrated them, they knew other DevOps teams were chasing data activity, too. So, they pivoted to start crafting a logging platform they could use, building on top of the popular Elasticsearch tool.
Logging has always provided critical information for developers on the lookout for code errors. When it involved a single server, logging was relatively straightforward, if often slow. But when virtualization moved into the data center, the process got significantly messier.
Centralized logging evolved to corral log files, but it was still up to developers, IT or infrastructure teams to organize the logging for themselves — a time-consuming, expensive use of resources. Worse, the growing clamor for zero downtime increased the speed at which developers needed real-time visibility into issues and changes. Today, containers that make up an application are proliferating as a new foundation for modernization.
“Anytime your application grows, you have more logs to deal with. Anytime you have new applications that come into your environment, or you’re launching new products, you have more logs,” says Norman Hsieh, VP of Business Development at LogDNA. “And effectively, somebody needs to spend more time managing and scaling that.”
After putting their integrated logging version to the test on Hacker News, Hsieh says, “We immediately saw a genuine demand and need for new perspective on logging.”
He continues: “At some point, you literally don’t have to touch any boxes, but you could have thousands of machines out there running your applications. Now multiply that by 10 or 100 in terms of the number of containers you’re running. You’re looking at a huge sprawl of data in these architectural frameworks, and it’s all automated but you don’t really know what’s going on. The only way to get that information is to save a log. And what you need now is a tool modernized enough to provide DevOps intelligence, know the frameworks and organize all that data automatically for you.”
And then the co-founders saw the writing on the wall: Kubernetes. “We thought it was an awesome technology to back our own architecture,” Hsieh adds. A groundswell of developers was fast adopting the lightweight, open-source platform for managing containerized workloads.
The LogDNA team seized on the advanced orchestration capabilities of Kubernetes in cloud environments, and looked for a leader in this space.
The work LogDNA was doing didn’t go unnoticed by IBM Cloud and Cognitive Software Chief Technology Officer (CTO) Jason McGee and his team, which itself was shifting focus to cloud, Kubernetes services and DevOps. Both companies were also working in the open source community, applying the momentum in the developer space.
“I give a lot of credit to IBM giving us a shot and saying, ‘These guys are doing something interesting with the technology they have today,’” says Hsieh. “We saw the same synergy between what Jason’s team was doing and where we were headed. When they talked about leading with Kubernetes on the IBM Cloud side, we knew this is exactly the space we wanted to push in.”
Like DNA strands winding together, LogDNA and IBM formed a tight relationship that transformed into both an integrated partnership and, happily, a client opportunity for LogDNA.
“We have had the chance to work extremely closely with IBM and not just from the business development side. We’re also working hand-in-hand with the internal teams that have built services within IBM,” says Hsieh. “We’re often asked by people in the Bay Area community how we get to do that. I think a lot of it is our willingness to say, ‘Hey, we are going to grow and expand with IBM.’”
One of the opportunities that added value to LogDNA and demonstrates the partnership was a new bare metal offering from IBM.
“You know our business is logging, but when you dig deep, we are also a storage and a big data company,” Hsieh explains. “So, when we initially looked at the IBM Cloud Kubernetes Service, we worried that current offerings were not going to meet our demands. We met with some of IBM’s distinguished engineers and they said, ‘Why don’t you take a look at the bare metal offering and see if there is something more flexible to get you to the requirements you need.’ We didn’t even know that existed.”
IBM had fast-tracked an offering to launch immediately, but this surprise was a game changer for LogDNA.
“We had taken our cloud-native architecture and provided a multichannel SaaS product using Kubernetes,” says Hsieh. “It enabled us to offer a product that can be deployed in all different types of managed Kubernetes environments but the bare metal option allowed us to get the IOPS [input/output operations per second] we needed to read and write quickly out of storage, and at a less-expensive price than network-based storage.”
With recommendations for some specification changes, LogDNA pressed IBM to get it done faster. It wasn’t the only time IBM leaned in.
“They have been extremely helpful. Early on, we were both learning how fast we were growing and so we weren’t even sure how many servers we needed. But the process of getting new servers and getting the new IBM Cloud Kubernetes Service up has just been getting better and better.”
Imagine you’re a disruptive ride-sharing application company, with millions of customers using your app around the clock—and driving a prodigious amount of log files in a massive infrastructure. Every car request, every ride taken, and the level of customer engagement leapfrogs the logging demands of yesterday.
“We think there is a higher market for that level of scalability versus what the incumbents are offering today. Thanks to our partnership with IBM, we can address that now,” says Hsieh.
“What is beautiful about LogDNA is that clients can start when their journey starts, and then stay with it in a consistent form regardless of where apps are running, or whether they are in early stages, modernization, or native application area,” says Sue Hahn, IBM Cloud Western Region Business Partner and Channel Sales and IBM Global Markets – Cloud Sales. “It can log and provide visibility across that whole spectrum—public, on-premises, hybrid, any cloud, seamlessly across various environments.”
The goal is to provide a log management tool that optimizes a developer’s data.
Hsieh says: “That means we focus heavily on things like automatic parsing. Any data that comes into LogDNA, we automatically take care of it for you, since we can recognize exactly what kind of logs they are. We bundle our services for simplicity and ease of use, so you don’t have to worry about logs.”
LogDNA attributes the ability to operate consistently around the world, no matter into which data center they push, to the global footprint that IBM offers. As IBM’s preferred logging service, the company’s product will be available in the IBM Cloud Service Catalog in all IBM service regions.
“We’ve expanded into Dallas, Frankfurt, with EU-Managed operations, Tokyo and London,” Hsieh says. “We’ll have Sydney soon. None of this would be possible without the consistency IBM brings—the exact, same deployment.”
Hahn adds: “That’s huge, to be able to support customers in a very short period of time because of the consistency of the offering. Wherever LogDNA deploys, it’s the same environment so we don’t have to change any operational processes. Customers can select and order our services, identify they want logging, and that pulls LogDNA directly into the order.”
In addition, IBM is using LogDNA as a log analysis tool for its internal systems.
The bulk of the partnership centers around LogDNA’s use of the IBM Cloud Kubernetes Service, but the company also engages with IBM Cloud Object Storage for testing, because the product offers integration directly into Cloud Object Storage. In the future, Hsieh says, the expectation is to apply other IBM services, such as IBM Watson® technology, to augment its machine learning capabilities, or a multicloud manager.
“Right now, our product is heavily focused on the DevOps space. Our strong suit is in providing better insights, better observability into development stacks. We’re building tools that enable developers to, essentially, not think about logs at all. We’re providing the convenience of a very robust log management tool without the inconvenience of having to manage or configure anything,” Hsieh says. “There are hidden costs to building out logging solutions, in resources and time. At the end of the day, we are letting our customers focus on the things they need to focus on.”
© Copyright IBM Corporation 2019. IBM Corporation, IBM Cloud, New Orchard Road, Armonk, NY 10504
Produced in the United States of America, September 2019.
IBM, the IBM logo, ibm.com, IBM Cloud, and IBM Watson are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright and trademark information” at ibm.com/legal/copyright-trademark.
This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates.
The performance data and client examples cited are presented for illustrative purposes only. Actual performance results may vary depending on specific configurations and operating conditions. THE INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING WITHOUT ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OR CONDITION OF NON-INFRINGEMENT. IBM products are warranted according to the terms and conditions of the agreements under which they are provided.