Most data teams react to incidents by importing whatever they read about Google’s SRE practices, holding a sober blameless ceremony, and producing a Notion doc nobody opens twice. Three months later the same pipeline breaks the same way. The problem isn’t the absence of process; it’s process that was designed for a different shape of failure.

Data incidents are not service outages. The page doesn’t go down. A revenue dashboard is quietly wrong for fourteen hours, a finance reconciliation fails in a way nobody notices until close, a schema change lands at 4pm on a Friday and the model retrains on Monday with degraded features. The blast radius is real, but it propagates through trust and through downstream decisions, not through HTTP 500s. A response model that treats every incident like a paging emergency burns the team out. A model that treats nothing as urgent erodes the business’s faith in the data. You need something in between, and you need to write it down before you need it.

Severity tiers that mean something

Three tiers. Not five. Five tiers always collapses to three in practice because no one can distinguish SEV-3 from SEV-4 at 11pm.

SEV-1 — Business-stopping. Revenue reporting is wrong, a regulated metric is being submitted incorrectly, a customer-facing data product is returning bad numbers, or executive decisions are actively being made on broken data. Page on-call. Acknowledge in 15 minutes. Status updates every 30 minutes until mitigation. The bar to declare SEV-1 is high but not heroic — if a finance leader is going to walk into a board meeting tomorrow and quote a wrong number, that’s SEV-1 today.

SEV-2 — Degraded. A pipeline is late, a data quality test is failing, a non-critical dashboard is stale, a model is retraining on suspect features. The work is real but doesn’t require waking anyone up. Acknowledge during business hours within two hours. Updates daily until mitigation. Most actual incidents live here. Resist the temptation to upgrade them for visibility.

SEV-3 — Noisy or cosmetic. A dbt test is flapping, a freshness alert fired during a known maintenance window, a column nullability change broke a non-blocking downstream. Triage during normal sprint planning. The point of having SEV-3 is to capture these so the patterns become visible — flap rates, recurring offenders, the dbt model that’s been emitting warnings for six months — not to swarm them.

What makes the tiers work is the explicit declaration. Whoever notices the issue declares the tier in the channel. The on-call lead has 15 minutes to escalate, downgrade, or accept. After that the tier is locked for the duration of the incident. No quiet drift from SEV-2 into “I guess we should have woken people up.”

Communication templates

Fill-in-the-blank, posted to a single dedicated channel, every time. The template removes the cognitive load of writing under stress and gives stakeholders a predictable shape to read.

Declaration:

[SEV-1] <one-line description>
Started: <UTC timestamp>
Detected: <how — alert name, user report, internal observation>
Impact: <which dashboards / pipelines / decisions are affected>
IC: @<person>
Comms: @<person>  (separate from IC for SEV-1)
Updates: every 30 min in this thread

Update:

[SEV-1 update — 14:30 UTC]
Status: investigating | identified | mitigating | monitoring | resolved
What we know: <new facts since last update>
Working theory: <best current hypothesis>
Next checkpoint: <UTC timestamp>

Resolution:

[SEV-1 resolved — 18:42 UTC]
Duration: 4h12m
Root cause (preliminary): <one sentence>
Mitigation: <what was done to stop the bleed>
Backfill required: yes/no, owner @<person>, ETA <date>
Postmortem: scheduled <date>, owner @<person>

The Comms role is non-negotiable on SEV-1. The IC is debugging; somebody else is talking to stakeholders. Conflating the two is how you get an IC who hasn’t typed anything in 90 minutes because they’re context-switching between SQL and Slack.

Postmortem rules that survive contact with reality

Postmortems decay for predictable reasons. The doc takes too long to write, the meeting is scheduled too far out, the action items have no owner, and nothing changes. Counter all four.

  • One page, written within five business days. If it doesn’t fit on a page it’s not a postmortem, it’s a research project. Sections: timeline, impact, root cause, what went well, what didn’t, action items. Nothing else.
  • Action items have a single named owner and a date. “The team will improve monitoring” is not an action item. “@alice adds a freshness SLO on fct_orders by 2026-05-02” is.
  • Cap action items at five per incident. Anything below the top five lives in the backlog. Five things you actually do beats fifteen things you intend to do.
  • Postmortems are mandatory for SEV-1, optional but encouraged for repeat SEV-2s. SEV-3s do not get postmortems, they get patterns reviewed quarterly.
  • Blameless means blameless about people, not about systems. “The deploy process let a breaking schema change reach production without a contract test” is the right level of specificity. “Mistakes were made” is not.
  • Read the previous one before writing the new one. If three of the last five postmortems blame the same upstream system or the same gap in test coverage, the postmortem itself is no longer the artifact that matters — the unaddressed pattern is.

The smallest viable system here is one channel, one severity tier definition, three templates, and a five-business-day postmortem clock. That’s the table stakes. Anything beyond that you should add only when you can point to the specific incident that would have gone better if you had it.