Most data teams measure pipeline health the way DevOps measured web servers in 2010: did the job run, and did it exit zero. That tells you the truck arrived. It doesn’t tell you whether the cargo was on time, complete, correctly labeled, or the same shape as yesterday. By the time a stakeholder pings about a broken dashboard, the bad data has been in production for hours and downstream decisions have already been made on it.
Data observability replaces job-status monitoring with five signals that map to how the business actually consumes data: freshness, completeness, accuracy, schema drift, and lineage. Each gets an SLO, and the SLO tightens or relaxes based on what the data is being used for.
The five signals
Freshness — how recent is the newest record relative to expectation. A finance close table that should land by 06:00 UTC and arrives at 09:00 has failed freshness even if the row count is perfect. Measure as now() - max(loaded_at) against a per-table SLA.
Completeness — did all the rows we expect actually arrive. Two failure modes: row-count drops (a partition came in 40% light) and null-rate spikes (a join key started arriving null). Both need historical baselines; absolute thresholds break the moment seasonality kicks in.
Accuracy — do the values themselves match reality. This is the hardest signal to automate because “reality” lives outside the warehouse. Practical proxies: reconciliation against a system of record (Stripe revenue vs. warehouse orders.amount), distribution checks (P50 order value should sit inside a historical band), and referential integrity (every order.customer_id exists in customers).
Schema drift — has the contract changed without a coordinated release. New columns, type changes (int → string), nullability flips, dropped columns. Schema drift is the #1 cause of silent dashboard breakage because a renamed column doesn’t error — it just produces nulls downstream until somebody notices.
Lineage — when something breaks, what’s downstream. Lineage isn’t a signal you alert on; it’s the map you need the moment any of the other four fire. Without lineage, an analyst spends an hour figuring out which 12 dashboards depend on the broken table. With lineage, the alert routes itself.
SLOs by business-criticality tier
Not every table deserves a 15-minute freshness SLO. Tier the warehouse first; budget the alerting after.
Tier 1 — revenue-critical / regulator-facing. Billing, revenue recognition, customer-facing usage meters, anything tied to a financial filing. Freshness SLO measured in minutes (15–60). Completeness SLO ≥ 99.9% on row counts and key non-null rates. Schema drift = page on any breaking change. Owner: a named on-call rotation. Time-to-detect target: under 5 minutes. Time-to-resolve target: under 1 hour.
Tier 2 — operational decisions. Sales pipeline, marketing attribution, ops dashboards leadership reads daily. Freshness SLO measured in hours (4–24, aligned to the decision cadence). Completeness SLO ≥ 99% with anomaly detection on distributions. Schema drift = ticket within the same business day. Owner: a team, not a person. Time-to-detect: under 30 minutes. Time-to-resolve: same business day.
Tier 3 — exploratory / analytical. Cohort tables, ML feature stores being prototyped, ad-hoc marts. Freshness SLO measured in days. Completeness checked weekly. Schema drift = batched into a weekly review. Owner: the requesting team. The point of Tier 3 is that it is allowed to be flaky — over-alerting here is what trains people to ignore alerts everywhere else.
The tiering exercise is the deliverable, not the alerts. Most warehouses have 5–10% of tables in Tier 1, 20–30% in Tier 2, and the rest in Tier 3. If your tiering shows 60% Tier 1, you haven’t tiered — you’ve labeled.
Tooling
There is no one tool that does all five signals well; pick a stack, not a vendor.
Monte Carlo — strongest on automated freshness, volume, and distribution monitoring with low configuration overhead. Pulls metadata from warehouse query logs, so coverage is broad by default. Best fit when the bottleneck is “we don’t know what to monitor” rather than “we know exactly what to assert.” Lineage is solid for SQL warehouses, weaker for transformation logic outside dbt.
Soda — code-first checks (SodaCL) checked into the same repo as the dbt models. Best fit for teams that want quality assertions reviewed in pull requests and tied to the deploy. Lighter on automated anomaly detection than Monte Carlo; expects you to write the checks.
Great Expectations — the most expressive expectation language and the one to reach for when accuracy SLOs need bespoke business logic (e.g., “claim adjudication amount must equal units * unit_price - adjustments within $0.01”). Heaviest setup cost. Strongest fit inside Python-native pipelines (Airflow, Prefect, Dagster).
Open-source equivalents. dbt tests plus elementary-data covers freshness, volume, and uniqueness for any dbt project at zero license cost. re_data adds anomaly detection. OpenLineage + Marquez gives you cross-tool lineage without buying it. The trade is engineering hours instead of license dollars.
The right starting point for most teams: dbt tests + Elementary for Tier 2 and Tier 3 coverage, plus Monte Carlo or Soda only on the Tier 1 surface where missed-detection cost dominates the license cost. Buy at the tier where the business pays for the miss; build at the tier where the business pays for the alert fatigue.