Designing for operation: what "on-call for data" actually means

Data products fail quietly

When a software application goes down, someone notices within minutes: a page fails to load, an alert fires, a customer complains. When a data product breaks, the failure is usually silent. A source schema changes upstream, a join starts fanning out, a feature drifts, and the system keeps running, just wrong. The assistant still answers. The agent still acts. Both are now quietly incorrect, and the first sign is often a business decision made on bad numbers, weeks later.

This is why "ship it and walk away" fails for data products in exactly the way it fails for application services, and why treating deployment as the finish line is a mistake regardless of how good the build was.

What "on-call" means when the thing on call is an agent

Application on-call works because someone (or something) is watching for a defined signal, has enough context to diagnose quickly, and has a bounded set of safe actions to take. Data products need the same three things: monitoring that ships with the product from day one, enough lineage and manifest context to diagnose without archaeology, and a clear, bounded answer to "what is this system allowed to do about it."

Three levels of autonomy, and why bounded is the default

Full autonomy sounds appealing until you consider what "the system silently rewrote a production pipeline" looks like in a postmortem. The safer default is bounded:

• Diagnose only: the system explains the cause. A human approves the fix. This is where every new product starts.
• Prepared fix: the fix is built and validated, waiting one click from deploy. Useful once a pattern of incident has been seen enough times to trust the diagnosis.
• Bounded autofix: for a specific, known, low-risk class of incident (a stale partition, a retry-safe failure), the system fixes and notifies. Never "anything, automatically."

The key word is bounded. Autonomy is scoped to incident classes that have been seen before and validated, not granted as a blanket capability.

A worked incident, start to finish

A CRM source changes a field name upstream. A freshness check fails against the manifest's SLO. The system traces the failure causally: this pipeline reads that field, the field disappeared at this commit, the join downstream is what broke. A fix is prepared: the mapping is updated, and a new version of the data product is built, not a patch applied in place. That new version runs through the same validation the original build did, the same thresholds, the same checks. Only then does it deploy, and the incident closes with the cause, the fix, and the proof all attached to the record.