Steve Gotlieb is Senior Manager for Data Engineering and Visualization at Autodesk, a multinational software corporation that provides software products across multiple industries. He manages the global data engineering and platform team across North America and Singapore. His team focuses on building reusable components that contribute to a robust and reliable data platform supporting data-driven solutions.
Under his leadership, Steve started championing data quality as a core platform component to support data mesh concepts that promote a bring-your-own-data approach and enable smooth data integration and utilization across the organization.
One significant challenge that repeatedly faced Steve’s team and other data engineering teams at Autodesk: they were often the last to know when data issues occurred. Steve’s team was forced to take a reactive approach to resolving issues, whether they were missing data, had late or stale data, or incorrect data with null values.
By the time the team was aware of a data issue, it might have existed for a month or more, costing the company valuable time and resources.
"We previously had a custom-built data quality management (DQM) system, but it was a passive and inextensible solution," says Steve. "The DQM system relied on running queries to monitor run counts, but it didn't proactively detect data quality issues. Notifications about data problems were inconsistent and delayed, often arriving via emails or Slack messages without clear ownership."
Steve and his team began evaluating data observability solutions, recognizing the need for a more proactive approach. They explored various options, including Monte Carlo Data and Datafold, but IBM® Databand® observability software stood out. Autodesk’s culture of innovation led it to arrange an innovation sprint, bringing together cross-functional teams to explore and showcase potential solutions. Preeti Taneja, Principal Data Engineer at Autodesk, played a pivotal role in this evaluation. Her team had just one week to demonstrate how Databand could transform its data quality processes.
They evaluated whether Databand could detect changes in source systems and provide real-time alerts in the event of workflow failures. The outcome was impressive. The seamless integration of Databand with Autodesk’s modern data stack, for example, Apache Airflow, dbt, Spark and Snowflake, and capability to deliver instant alerts left a strong impression.
“Databand’s ease of integration with our modern data stack allowed us to see value immediately,” says Preeti. “When we started getting instantaneous alerts, it was a true wow moment of Databand’s proactive data quality capabilities.”
Following an internal assessment, Databand emerged as the clear winner, leading the team to move forward with its implementation.
Steve’s team uses Databand daily to monitor data incidents across various use cases, including:
- Batch processing monitoring: Databand is used extensively in monitoring production batch processing. Over 1,000 DAGs are actively monitored by Databand.
- Inline testing: The team uses the inline testing capabilities of Databand to detect data quality issues in real time, which is crucial for maintaining data integrity.
- Data products support: Databand supports pipelines that deliver insights and in-product messaging for Autodesk’s customers.
- Machine learning (ML) and AI pipeline monitoring: Databand also monitors pipelines supporting ML and AI teams, helping to ensure that data quality is maintained across all stages of data processing.
The implementation of Databand brought immediate and significant improvements to Autodesk’s data quality management:
- Reduction in detection time: Databand reduced the time to detect data quality issues from days to minutes. This immediate detection allowed the team to address problems before they could cause major disruptions.
- Reduction in mean time to resolution (MTTR): With Databand, the mean time to resolve data issues dropped from weeks to days. Detecting incidents, such as late-arriving data, schema changes and pipeline failures, helps maintain trust and efficiency within the organization
- Root cause analysis: Databand provided advanced root cause analysis, enabling the team to quickly identify and fix issues at their source
- Seamless integration: The solution integrated smoothly with Autodesk’s existing platforms without needing to rewrite Spark, Airflow, and dbt core pipelines. This integration included monitoring batch processing, internal pipelines and data at rest in Snowflake environments
- Cost savings: Autodesk saw a decrease in cloud consumption costs by detecting issues early and avoiding reruns.
Autodesk has seen tangible results in improving data quality and operational efficiency. The transparent tracking of feature requests has further solidified the partnership, enabling continuous improvements and innovations.
The world’s designers, engineers, builders and creators trust Autodesk (link resides outside of ibm.com) to help them design and make anything—from the buildings we live and work in, to the cars we drive and the bridges we drive over. Even the products we use and rely on everyday and the movies and games that inspire us exist thanks to Autodesk. Autodesk’s Design and Make Platform unlocks the power of data to accelerate insights and automate processes, empowering our clients with the technology to create the world around us and deliver better outcomes for their business and the planet. For more information, visit autodesk.com (link resides outside of ibm.com).
Legal
© Copyright IBM Corporation 2024. IBM, the IBM logo, and Databand are trademarks or registered trademarks of IBM Corp., in the U.S. and/or other countries.
This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates.
Client examples are presented as illustrations of how those clients have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.