About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Transforming data into insight
How the IBM Chief Data Office (CDO) taps into the power of IBM Databand
Data can be a source of strategic advantage for any enterprise, but only if you can unlock its full potential. The paradox is a simple one: more data complexity leads to less visibility, making it challenging to deliver value from data at scale.
For the IBM® Chief Data Office (CDO), getting value from data is essential. Data is the fuel of IBM’s work to become a more productive company, and core to this strategy is the integration of data from various sources to deliver insights through data pipelines. A data pipeline is a sequence of operations that extracts, transforms, and loads data from diverse sources into a data warehouse or data lake for analysis—providing data consistency and efficiency.
For the CDO team, responsible for managing and maintaining the company's nearly 4,000 data pipelines, the traditional approach to managing pipeline health was cumbersome and time intensive. This involved manual monitoring and troubleshooting of issues which consumed a substantial amount of time and was prone to human error. “Managing pipeline health across our large volume of pipelines was a significant challenge, especially when teams are using different products,” explains Ashley Delport, senior engineering manager for CDO. “We needed a solution that could continuously monitor data health, proactively identify and resolve issues such as missing data and notify our teams if service was going to be impacted."
The team required something more efficient and effective as they continued to scale data pipelines for business insights and AI use cases. They wanted an innovative solution that provided real-time visibility into pipeline performance, automated monitoring and enhanced collaboration. Most importantly, the team didn’t want to lose the trust that they had earned from tens of thousands of users who relied on them for accurate, timely, high-quality data to run the business.
The CDO team started with a pilot focused on data from the company’s merger and acquisition motions and gained real-time visibility into pipeline performance by leveraging the Databand software. According to Sheeba Prakash, senior technical staff member from the team, “the real-time monitoring and alerting capabilities of IBM® Databand® saves us time and effort. Specifically, we've seen an estimated 85% reduction in manual monitoring and troubleshooting in a pilot!”
The solution’s intuitive interface enabled the team to easily track pipeline health, identify bottlenecks and resolve issues quickly. Automated workflows and customizable dashboards measurably streamlined pipeline management, freeing up valuable engineering time to work on accelerating acquisition integration.
The successful pilot led the CDO team to expand its use of Databand software as a single source of truth for insights into data pipeline performance. “IBM Databand has elevated our operational excellence. It makes us more proactive instead of reactive to data pipeline issues,” adds Delport.
The IBM Databand software helps the CDO team easily identify and resolve issues before they become major problems, create custom alerts and notifications for potential issues, quickly check the status of data pipeline runs, view historical run trends and pinpoint root causes within their pipeline directed acyclic graphs (DAGs).
The team has realized significant savings in engineering capacity, freeing up more time to integrate new data for strategic advantage rather than manually monitoring data pipelines, which is time consuming and doesn’t add value to the business.Examples include an estimated:
- 90% less time spent working on daily health reports
- 93% less time spent creating daily health reports for all data objects
- 80% less time spent viewing failure trends
- 85% less time spent manually monitoring and troubleshooting
Reflecting on the success and significant impact of implementing and expanding IBM Databand, Delport summarizes: “IBM Databand elevates our operational excellence. It makes more proactive instead of reactive to data pipeline issues, and delivers a better experience for our users and development teams.”
The IBM CDO is responsible for managing and governing IBM's overall data assets, including strategy, architecture, governance and management. They work to ensure that data is used effectively, efficiently and securely across the organization, and that it meets the needs of various business units and stakeholders.
Legal
© Copyright IBM Corporation 2024. IBM, the IBM logo, and Databand are trademarks or registered trademarks of IBM Corp., in the U.S. and/or other countries. This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates.
Client examples are presented as illustrations of how those clients have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.