If bad data is in your warehouse, it’s already too late (plus the leading indicators of error)

This is precisely why we built IBM® Databand®. It’s a data pipeline observability tool created for this express purpose—to help engineers identify data pipeline issues early, and also track them back to their source to understand the root causes. With Databand, teams can identify which upstream data issues caused the downstream error they or someone else has noticed. What’s even better, they can set automatic alerts.

For example, in Databand, you can set alerts for leading indicators such as missing data, aberrant data, or suspicious values. As Databand’s founders put it, “To a downstream user, every problem will appear as a data quality problem. Our job is to find what’s really causing it, and ideally catch it before anyone realizes something’s amiss.”

Next, we explore two common data pipeline architecture scenarios, what can go wrong, and specifically, how an observability tool like Databand can help.

If you don’t catch it: 

Let’s say your pipeline takes a day to process. You schedule that process in the evening and thus it takes you one day to notice a column type change error. Now, not only do you have to rerun it, you have to also catch up on two additional days of data—the prior day’s data, and now the data from today. Now, your pipeline will take 20 hours to run instead of 10.

You could revert, but now the catchup multiplier clock is ticking. The more individuals throughout the business rely on that data, the more people are now using corrupted data, and worse: You don’t have a good way to communicate to them what’s happening or that they should pause and wait for new data. So it’s infiltrating all your other systems.

This happens quite often. In the case of one company processing applications, this scenario caused a month’s delay. They were expecting a certain subset of data in the monthly load, but it never showed. The engineers weren’t alerted to the issue until 72 hours later. It took another 72 hours to resolve, and due to scheduling issues for the meetings that followed, the fix was not implemented for 20 working days.

If you do catch it: 

The issue is fixed on the same day and you rerun the pipeline before it gets backed up. With a tool like Databand, you were able to create checkpoints with critical metric logging. When data didn’t come back as expected, you got alerts. You can contact the vendor, they reset things, and no harm is done.

You can run these ongoing checks for the dimensions of data quality health, at any point in your pipeline:

• Fitness: Is this data fit for its intended use?
• Lineage: Where did this data come from? When? Where did it change? Is it where it needs to be?
• Governance: Can you control the data?
• Stability: Is the data complete and available in the right frequency?

You may also hear the above referred to as the following:

• Freshness: Did the data arrive on time? (SLAs, durations, data landing times)
• Completeness: Did the full data arrive? (Nulls, counts, schemas)
• Accuracy: Did the true and correct data arrive? (Distributions, skew, domain-specific metrics)

If you don’t catch it: 

Lots of systems these days are connected through APIs and HTTP, and we all know you can receive a 200 (success) but no actual data. The “success” refers to connection itself, not the accuracy of data transfer. Most systems reliant on APIs simply weren’t built to check for data health and this means there’s really no check against the most common sources of errors, namely, users. It’s all too easy for an administrator to mistype, which throws off 100 entries that then propagate throughout the system, and then the catchup multiplier effect takes hold.

Depending on how your other connected systems are built, you may catch this error eventually. But not before some systems have accepted and processed all the wrong values. If you don’t catch those values when they are first deposited, you’re left looking for downstream effects—like a dashboard showing the wrong values (and causing difficult conversations). Or, as in one actual scenario, a payroll system stops sending paychecks to everyone including the data engineers.

If you do catch it: 

With an observability tool, you got an alert before the data was transferred, early in the data pipeline architecture, and you simply hit pause. You checked the error, traced it back to its source, corrected the misset values, and ran it again.

Furthermore, you could use these alerts to always validate the existence of data at every stage. If the check ever finds that the data doesn’t exist but should, it’ll rerun that stage. If the data exists but contains an error, you can set conditional actions, such as paging you—whether it’s for the wrong schema, skewed metrics, or any other issue.

As we covered in the scenarios above, you can catch more of the leading indicators of data pipeline architecture issues in four ways:

1. Introduce checkpoints
2. Establish trend baselines
3. Track data lineage and establish the root causes of issues
4. Measure metrics for pipeline executions and data input

When you build your data pipeline architecture with proper tracking, it becomes transparent and decodable. It ceases to be a black box. You can catch issues when they first occur, anywhere in your data pipeline, and act before those issues spread and the catchup multiplier effect causes the problem to creep into other departments.

As they say, the best data engineers are invisible. If you’re doing your job, everyone else is doing theirs. The more alerts you have, and the better leading indicators you can detect, the sooner you’re able to catch issues. And if you do that, everyone can trust the data—and you.

Learn how Databand uses data anomaly detection to automatically detect deviating behavior in your data pipelines and datasets. If you’re ready to take a deeper look, book a demo today.