If bad data is in your warehouse, it’s already too late (plus the leading indicators of error)

By the time you realize you have a problem, it’s typically too late. That is, a small problem has now grown into a big problem—one so large that you and others actually notice. And whatever you’ve noticed is, unfortunately, only a downstream effect, and probably one of many. To catch that root issue and keep it from recurring, you have to trace the error all the way back through your data pipeline architecture and that’s where things become tricky.

Without a firm sense of the leading indicators of data pipeline error, you’ll catch errors late. The longer errors exist in your pipeline, the more problems they cause. The more problems they cause, the busier you are addressing all the downstream issues, which keeps you from addressing the root causes. You will forever be fighting fires.

Take this simple example. If it takes an hour to process one day’s worth of data, and you notice an error three days late, you’re now backed up not just one hour, but three hours. If you’re lucky, the issue is reversible, and only internal employees are complaining. If you’re not so lucky, as is the case if your pipeline feeds an e-commerce system that’s now generated a stream of irreversible transactions, you also have customers to deal with. Problems beget more problems.

As we will argue, the cost of catching errors late is substantial. But the biggest casualty by far will be everyone’s trust in the data itself. And by extension, you.

Errors in your data pipeline architecture can have real costs. If your “consumer” is an internal sales team whose dashboards are all down, that’s one kind of cost. But if your data customer is external, the cost is greater and can cut into your brand reputation. If you’re the upstream data provider and customers now can’t run their machine learning models, you’ve now exported your data error to other companies like a bug, and they’re also now facing that same compounding cost of error.

We might call this the catchup multiplier effect: The longer it takes to fix a data pipeline architecture issue, the more the cleanup cost compounds.

IBM discovered this problem in the early 2000s. In a study published in the National Institute of Standards and Technology (NIST), engineers demonstrated that an issue that would take one hour to solve if caught immediately, would take 15 hours if caught at the design phase, and 100 hours if not caught until after launch.

Herein lies the problem in how most data pipeline architectures are built—they’re linear. They’re also often entirely without safeguards, checks, or any of the DevOps principles used to assure quality in software. Your job as the engineer is to implement these checks, alerts and controls. The better you can become at identifying leading indicators of data pipeline issues early, the faster you catch them and the smaller the price you pay.

Let’s say your pipeline takes a day to process. You schedule that process in the evening and thus it takes you one day to notice a column type change error. Now, not only do you have to rerun it, you have to also catch up on two additional days of data—the prior day’s data, and now the data from today. Now, your pipeline will take 20 hours to run instead of 10.

You could revert, but now the catchup multiplier clock is ticking. The more individuals throughout the business rely on that data, the more people are now using corrupted data, and worse: You don’t have a good way to communicate to them what’s happening or that they should pause and wait for new data. So it’s infiltrating all your other systems.

This happens quite often. In the case of one company processing applications, this scenario caused a month’s delay. They were expecting a certain subset of data in the monthly load, but it never showed. The engineers weren’t alerted to the issue until 72 hours later. It took another 72 hours to resolve, and due to scheduling issues for the meetings that followed, the fix was not implemented for 20 working days.

The issue is fixed on the same day and you rerun the pipeline before it gets backed up. With a tool like Databand, you were able to create checkpoints with critical metric logging. When data didn’t come back as expected, you got alerts. You can contact the vendor, they reset things, and no harm is done.

You can run these ongoing checks for the dimensions of data quality health, at any point in your pipeline:

• Fitness: Is this data fit for its intended use?
• Lineage: Where did this data come from? When? Where did it change? Is it where it needs to be?
• Governance: Can you control the data?
• Stability: Is the data complete and available in the right frequency?

You may also hear the above referred to as the following:

• Freshness: Did the data arrive on time? (SLAs, durations, data landing times)
• Completeness: Did the full data arrive? (Nulls, counts, schemas)
• Accuracy: Did the true and correct data arrive? (Distributions, skew, domain-specific metrics)

Lots of systems these days are connected through APIs and HTTP, and we all know you can receive a 200 (success) but no actual data. The “success” refers to connection itself, not the accuracy of data transfer. Most systems reliant on APIs simply weren’t built to check for data health and this means there’s really no check against the most common sources of errors, namely, users. It’s all too easy for an administrator to mistype, which throws off 100 entries that then propagate throughout the system, and then the catchup multiplier effect takes hold.

Depending on how your other connected systems are built, you may catch this error eventually. But not before some systems have accepted and processed all the wrong values. If you don’t catch those values when they are first deposited, you’re left looking for downstream effects—like a dashboard showing the wrong values (and causing difficult conversations). Or, as in one actual scenario, a payroll system stops sending paychecks to everyone including the data engineers.

With an observability tool, you got an alert before the data was transferred, early in the data pipeline architecture, and you simply hit pause. You checked the error, traced it back to its source, corrected the misset values, and ran it again.

Furthermore, you could use these alerts to always validate the existence of data at every stage. If the check ever finds that the data doesn’t exist but should, it’ll rerun that stage. If the data exists but contains an error, you can set conditional actions, such as paging you—whether it’s for the wrong schema, skewed metrics, or any other issue.

As we covered in the scenarios above, you can catch more of the leading indicators of data pipeline architecture issues in four ways:

1. Introduce checkpoints
2. Establish trend baselines
3. Track data lineage and establish the root causes of issues
4. Measure metrics for pipeline executions and data input

When you build your data pipeline architecture with proper tracking, it becomes transparent and decodable. It ceases to be a black box. You can catch issues when they first occur, anywhere in your data pipeline, and act before those issues spread and the catchup multiplier effect causes the problem to creep into other departments.

As they say, the best data engineers are invisible. If you’re doing your job, everyone else is doing theirs. The more alerts you have, and the better leading indicators you can detect, the sooner you’re able to catch issues. And if you do that, everyone can trust the data—and you.

Learn how Databand uses data anomaly detection to automatically detect deviating behavior in your data pipelines and datasets. If you’re ready to take a deeper look, book a demo today.