The incident starts looking like an application problem. A service is returning wrong values, or a feature is behaving strangely for a subset of users, or a number on a dashboard is obviously incorrect. Engineers dig into recent deployments, read the code, and find nothing wrong — because nothing is wrong with the code.
A schema changed upstream three weeks ago. A field that was never null is null for records from one source. A batch job ran twice during a retry and double-counted. The application is behaving perfectly with respect to inputs that are wrong.
Once you’ve been through this a few times, the pattern becomes hard to unsee: a large share of production incidents that present as application failures are data failures wearing a costume.
Why data failures are more expensive than outages#
An outage is loud. Error rates spike, alerts fire, someone is paged, and the clock starts. Unpleasant, but the system is working as designed — the failure is visible and the response is immediate.
Data failures have none of those properties.
They fail silently. No exception, no error rate, no page. The pipeline reports success because it did what it was told with what it was given.
They propagate before detection. By the time someone notices, downstream systems have consumed the bad data. Reports are built on it, models trained on it, decisions made from it. The remediation surface grows with every hour.
Detection is accidental. Usually someone notices a number looks wrong. That means detection latency is measured in days or weeks, and it’s a function of who happened to be paying attention.
They damage trust disproportionately. After an outage, people are annoyed and move on. After discovering a report was wrong for a month, they stop believing your numbers — and that’s much harder to repair than uptime.
The asymmetry is worth stating directly: silently wrong data is an outage where nobody gets paged. Most organizations treat pipeline uptime as an SLO and correctness as a softer, separate conversation. Consumers don’t experience it that way.
The seam is where things break#
Applications produce data. Pipelines consume it. Ownership usually stops at that boundary, and the boundary is where failures concentrate.
An application team changes a field’s meaning — reasonable, well-tested, ships cleanly. Downstream, that field feeds a transformation with an assumption baked in from two years ago. Nothing failed. The application team had no reason to know the pipeline existed. The data team had no signal the change was coming.
This is not a technical failure. It’s a missing contract, and it’s why data reliability work turns out to be mostly organizational once you’re past the obvious engineering.
Four failures that account for most of it#
Schema drift. A field is added, removed, renamed, or retyped. Additions are
usually benign; removals and type changes rarely are. Without explicit schema
validation these arrive as nulls and coercion errors rather than as failures.
Semantic drift. Harder, because the schema doesn’t change — the meaning does. A status field gains a new value. A currency amount switches unit. A timestamp changes from local to UTC. Every structural check passes. Every number downstream is wrong.
Duplicate processing. A job retries after a partial failure and reprocesses records already handled. Without idempotent, keyed writes, aggregates double-count. This one is particularly nasty because it’s often correct on rerun — the bug only manifests when a specific failure interleaves with a retry.
Silent volume changes. A source that normally delivers a million records delivers eighty thousand. Nothing errors — there just isn’t much data. Every downstream aggregate is quietly wrong, and no error-based monitoring will ever fire.
That last one is the clearest illustration of why standard observability practice doesn’t transfer directly. There is no error to alert on. The system is healthy. The data is wrong.
What actually helps#
Preserve raw input immutably#
Store what you received, unmodified, before any transformation. Every downstream step becomes replayable, and — critically during an incident — “was this always wrong, or did we break it?” becomes an answerable question rather than an argument.
It’s a recurring storage cost with no visible consumer, which is why it gets cut. Every organization that skips it rebuilds it later, usually mid-incident, without the history that would have made the incident tractable.
Validate in tiers, not as a gate#
A single pass/fail check forces a bad choice: strict enough to be unoperable, or permissive enough to be pointless.
Two tiers work better. Structural failures — unparseable, schema-violating — are quarantined. Semantic failures — implausible values, referential gaps — pass through flagged. Consumers can then decide what to do with flagged records rather than having the decision made for them upstream.
Quarantine, never drop#
Rejected records go to durable storage with the rejection reason attached. Dropped records are unrecoverable and, worse, invisible — you can’t investigate what you didn’t keep, and you can’t tell the difference between “no bad records” and “no validation.”
Monitor the things that have no error#
The highest-value data monitoring produces no exceptions and requires no labels:
- Freshness per dataset — time since last successful update against expected cadence. This is what consumers actually care about and what most platforms don’t expose.
- Volume against recent history — catches the silent-shortfall case that error monitoring structurally cannot.
- Rejection rate, segmented by source and rule — the single best leading indicator available. It moves before dashboards look wrong and long before anyone files a ticket. Alert on rate of change, not absolute level.
- Completeness against expected partitions — so missing data is detected rather than inferred from a chart that looks low.
- Schema drift as an explicit alertable event, not a log line nobody reads.
Make reprocessing routine#
You will need to reprocess. That requires idempotent, keyed writes — natural keys and upserts rather than appends — as a platform-wide invariant rather than a per-pipeline choice.
Retrofitting this onto append-oriented pipelines approaches a rewrite. It has to be an early decision, made before you have the evidence justifying it, which is precisely why it’s usually skipped.
Know what’s downstream#
When a dataset is wrong, the affected consumers should be queryable, not institutional knowledge. Lineage turns “we think this feeds some reports” into a list. During an incident that distinction is the difference between a contained correction and a week of discovery.
For application engineers#
If you build services rather than pipelines, three things make you a disproportionately good upstream neighbor:
Treat your emitted data as an API. Someone depends on its shape and meaning. Changing a field’s semantics is a breaking change even when no code fails.
Emit events, not just state. State tells consumers what is true now; events tell them what happened. Events are replayable and auditable, and they make downstream idempotency achievable.
Know who consumes your data. Not exhaustively — but if you can’t name a single downstream consumer, you can’t assess the blast radius of a change, and you will eventually make one that looks entirely safe.
The reframe#
The useful shift is treating data correctness as an availability concern rather than a quality initiative. Not because the technical work is identical — it isn’t — but because it changes how the work gets prioritized and who considers themselves responsible.
Silently wrong data has all the impact of downtime, none of the alerting, and a longer tail. Once it’s framed as availability, the questions become familiar ones: what’s the detection latency, what’s the blast radius, who’s on call for it. Those are questions organizations already know how to answer.