Runbooks and SLOs: Making Reliability an Operating System

Most data and AI teams do not fail because they picked the wrong lakehouse, the wrong modeling tool, or the wrong LLM provider. They fail because they ship something that works on Tuesday and mysteriously does not on Friday. Or it "works," but only as long as the same two people are awake, reachable, and willing to cancel their plans.

That is the gap Runbooks and SLOs close.

They turn reliability from a vibe into an operating system. They define what "good" looks like, how you measure it, what you do when it is not true, and who owns the fix. They also create a fair contract with the business: we will meet these targets most of the time, and when we do not, we will respond predictably, learn, and improve. That is how you earn trust and keep it.

If your foundations philosophy is outcomes-first and thin-slice driven, runbooks and SLOs are not "later." They are Day 2 thinking applied on Day 0. They keep you out of hero culture and platform purgatory by making reliability a shipped feature, not a tribal memory.

Goal

Define operational procedures and performance targets to ensure reliability and accountability in data and AI services.

This is broader than uptime. In data and AI, the user experience is often about:

Freshness: did the data land and become usable when the business needed it?
Correctness: did the pipeline produce the right results and not silently corrupt metrics?
Availability: can consumers query, serve features, or call an inference endpoint right now?
Latency: does the system respond fast enough for the workflow it supports?
Quality: did a model degrade, drift, or start making nonsense outputs without anyone noticing?

SLOs are how you quantify these. Google's SRE guidance defines an SLO as a target level for a service level that is measured by an SLI (service level indicator). In other words, you pick what matters, you measure it, and you set a target that reflects the experience you intend to provide.

Runbooks are how you operationalize it. They make the right response the default response, especially at 2:00 a.m.

The Thin Slice

Document key operational tasks and establish basic SLOs. Create incident playbooks with triage, ownership, and reviews.

If you do only three things this month, do these.

1) Pick one service boundary and name the customers

Start with a single "service" that is real enough to run. Not "the data platform." Pick something people actually depend on.

Examples:

"Daily revenue reporting dataset for Finance"
"Customer 360 table consumed by the call center dashboard"
"Feature pipeline for churn model training"
"LLM-powered support summarization endpoint"

Then write the simplest possible customer statement:

Who uses it?
What decision or workflow depends on it?
What happens when it is wrong or late?

This keeps your SLOs honest. Without a customer and a workflow, SLOs become vanity metrics.

2) Define 2 to 4 SLIs that match the user journey

A common trap is picking dozens of indicators because you can measure them. Pick a few that map to the experience.

For data pipelines, the SRE community consistently treats "freshness" and "correctness" as first-class concerns, not just "job success." Google's SRE Workbook calls out the real cost of delayed or incorrect data and frames pipeline reliability as a product requirement, not an afterthought.

Practical SLIs for data:

Freshness lag: time since the latest successful publish to the consumption layer
Completeness: percentage of expected records received or processed
Correctness/error rate: percentage of records failing validation, constraint checks, or reconciliation
Availability for consumption: success rate of queries or API calls against the published interface

Practical SLIs for ML/LLM services:

Inference availability: successful responses over total requests
Latency: p95 or p99 response time for the endpoint
Quality proxy: task-specific score (evaluation set pass rate, human rating, guardrail pass rate)
Model health/drift: distribution shifts or performance degradation alerts

The key is to avoid confusing "what is easy to monitor" with "what users feel."

3) Set initial SLOs that are achievable and decision-linked

An SLO is not a wish. It is a negotiated target that reflects the business need and the reality of your system today. Microsoft's reliability guidance emphasizes defining targets with stakeholders and refining them through monitoring and testing, rather than inventing numbers in a vacuum.

Start with targets you can hit most of the time, then tighten as you reduce toil.

Examples you can steal:

Data freshness SLO

SLO: "Dataset is updated within 30 minutes of source availability for 95% of 30-minute windows, measured weekly."
Why it works: it encodes both the lag target and the reliability target (95% of windows). This pattern is common in real data pipeline SLO practice.

Data correctness SLO

SLO: "At least 99.5% of records pass validation rules per run, measured daily."
Add a clause: "If below 99.5%, dataset is flagged as degraded and downstream dashboards show a warning."

Inference latency SLO

SLO: "p95 latency under 800 ms for 99% of 5-minute windows."
Reality check: LLM workloads often have strict latency SLOs and real system design tradeoffs to meet them.

LLM safety/guardrail SLO

SLO: "99.9% of responses pass policy and safety checks; any failure triggers incident review."
Note: this is not just reliability, it is product risk management.

4) Write a runbook that fits on one page

Your first runbook is not a novel. It is a high-signal checklist that reduces cognitive load during an incident.

A good one-page runbook includes:

Service name + owner: one accountable team, one on-call rotation
What "good" looks like: the SLOs and the dashboards that show them
How to detect an issue: alert names and where they trigger
Triage steps: 5–10 minutes of actions that narrow the problem
Known safe mitigations: restart job, roll back deployment, disable feature flag, reroute traffic, pause downstream publishes
Escalation path: who to page next, and when
Communication template: what to tell users, where, and how often

If you do incident management formally, borrow structure from proven practices. Google's incident management approach emphasizes clear roles and coordinated communication, drawing on Incident Command System concepts. The point is not bureaucracy. The point is that during stress, role clarity beats improvisation.

5) Define incident playbooks with triage, ownership, and reviews

Runbooks are often "how to fix." Playbooks are often "how to respond."

At minimum, define:

Severity levels: Sev1/2/3 with clear impact definitions
Ownership: who is the incident commander, who is the technical lead, who handles comms
Cadence: update intervals for stakeholders
Post-incident review: blameless, timeboxed, with action items

Then enforce the habit: every Sev1 and Sev2 gets a review. Every review produces one or two concrete improvements, ideally automation or guardrails.

Tools can help, but the operating model matters more. Many incident platforms focus on making incident declaration, coordination, and postmortems consistent and measurable.

Scale Path

Automate runbook execution and integrate SLOs into monitoring dashboards. Add release checklists, Known Issues logs, and change windows.

Once the thin slice works, scale is about compounding, not adding paperwork.

1) Integrate SLOs into the same dashboards people already use

Do not create an "SLO portal" nobody opens. Put SLO burn-down, error budget, and the key SLIs next to the service's operational metrics.

SLOs are designed to drive decisions. In SRE practice, they are central because they help teams make data-driven tradeoffs between reliability work and feature work.

A simple dashboard layout that works:

Top: SLO status (met/not met), error budget remaining, burn rate
Middle: SLIs (freshness lag, correctness, latency, availability)
Bottom: leading indicators (queue depth, job duration, retries, cost anomalies)

2) Use error budgets to stop reliability debates

Without error budgets, every conversation becomes a values argument:

"We need to ship features."
"We need to stabilize."
"We cannot slow down."
"We cannot break things."

Error budgets convert that into a rule:

If you are burning too much budget, you pause risky releases and invest in stability.
If you have budget, you can move faster.

Google's guidance ties error budgets directly to how you balance reliability and change, including how planned maintenance and downtime interact with reliability targets.

3) Automate the top sources of toil

Runbooks should evolve toward automation. The goal is not to replace engineers. It is to stop spending human attention on repeatable tasks.

What to automate first:

Re-run a failed job with idempotent writes
Backfill a partition safely
Roll back a model or prompt version
Disable a feature flag or route traffic away from a failing endpoint
Quarantine "suspect" data and prevent downstream publishes
Open an incident with the right context pre-filled

Automation is also an accountability multiplier. If the runbook can be executed consistently, you get consistent outcomes, fewer missed steps, and faster recovery.

4) Add release checklists that treat data and AI as production software

If your data and AI releases do not have a checklist, you are shipping risk.

A strong release checklist includes:

Contract and schema change review, including backward compatibility
Data quality rule updates and expected impact
Lineage and downstream impact review
SLO impact assessment, including error budget status
Rollback plan verified
Monitoring and alerts verified for the new behavior
Communication plan if a metric definition changes

This is where reliability meets governance in a useful way. It is not red tape. It is how you avoid breaking the business with a "minor" change.

5) Maintain a Known Issues log that is user-facing

A "Known Issues" log is your honesty engine.

It should answer:

What is currently degraded?
What is the workaround?
When will it be updated next?
Who owns it?

This reduces noise, prevents duplicate incidents, and builds credibility. Users do not demand perfection. They demand transparency and predictable response.

6) Establish change windows for high-risk changes

Not every environment needs strict change windows. But data and AI systems often have batch cycles, month-end close, or peak business hours where failure is costly.

Use change windows when:

The blast radius is large
Rollback is slow
The workflow is time-sensitive

Tie it back to error budgets and SLO attainment, not tradition.

Anti-Patterns

Undocumented procedures and unrealistic SLOs. Reliance on heroics and lack of accountability.

These are the traps that quietly kill adoption and morale.

Anti-pattern 1: "We have SLOs" (but no one can find them) If SLOs live in a doc nobody reads, they do not exist. Put them in dashboards. Put them in alert definitions. Put them in incident reviews.

Anti-pattern 2: Unrealistic SLOs that force lying If you set 99.99% targets for a pipeline held together with manual backfills, you will get one of two outcomes:

constant "exceptions" that normalize failure
quiet manipulation of measurements to look compliant

Start with a target you can meet, then earn the right to raise it.

Anti-pattern 3: SLIs that do not represent the user experience If you measure job success but ignore freshness, you will celebrate a "successful" pipeline that shipped yesterday's data. Data processing reliability is about outcomes, not just green checkmarks.

Anti-pattern 4: The hero model If incidents are solved by heroics, you are accumulating operational debt. The fix is structural:

clear ownership
clear runbooks
post-incident reviews that produce automation and guardrails
capacity reserved for reliability work

Anti-pattern 5: Postmortems with no follow-through A review without action items is theater. Keep action items small, assign owners, and track completion.

What "Good" Looks Like in Six Months

SLOs are visible in the same place teams monitor systems.
Incidents have consistent roles, consistent comms, and consistent reviews.
Data consumers can trust that "fresh" means fresh, and "certified" means validated.
Model and LLM services have clear latency and quality targets, plus rollback paths.
Reliability work is prioritized using error budgets, not opinion.
The team no longer relies on heroics because the system has a memory.

That is the punchline: runbooks and SLOs are organizational memory, encoded into the way you operate. They make reliability scalable.

If your foundations are supposed to accelerate the business, this is how you keep the accelerator from becoming a liability.