Incident runbook design

What the Prompt Is Testing

This prompt is often phrased as: “Tell me about a time you designed or improved an incident runbook.” It tests whether you can make operational knowledge usable under stress, balance mitigation speed with safe decision-making, and improve the system after incidents. The interviewer is looking for evidence that you can:

• identify the people, failure modes, and decisions a runbook must support;
• turn ambiguous incident response into clear checks, branches, and escalation points;
• design for the on-call engineer who may be unfamiliar with the service;
• communicate severity, ownership, and uncertainty; and
• measure whether the runbook actually reduced time to mitigation or operational toil.

If you have not owned a runbook, choose a real example involving documentation, an on-call process, a deployment checklist, or a reliability improvement. Describe what you actually changed; present a proposed runbook or outcome as “I would…” rather than as personal experience.

30-Second Opening

State the incident problem, your responsibility, the design choice, and the result. Replace every bracketed item with evidence from your own story:

“When [incident or alert] repeatedly caused [impact] for [scope], I owned [specific responsibility]. I designed or improved a runbook around [decision tree, signal, or escalation gap], and after [implementation or training] we measured [result or current validation].”

Lead with the operational gap and the outcome. Save severity tables, alert thresholds, and escalation mechanics for the Action section or follow-up questions.

2–5 Minute Answer Structure

Use Situation, Task, Action, Result (STAR), with the runbook design details concentrated in Action:

• Situation (20–30s): What incident, alert, or recurring failure exposed the gap? Who was on call, and what made the response difficult?
• Task (10–20s): What did you own—incident response, runbook design, review, rollout, training, or measurement?
• Action (1–3 min): Explain how you mapped the incident lifecycle, chose severity and escalation rules, wrote decision points, validated commands and links, incorporated on-call feedback, and kept mitigation safe.
• Result (20–40s): State what changed in time to acknowledge, time to mitigate, repeat incidents, confidence, or toil. If you did not measure an outcome, say what you would measure and why.

Keep the story about one operational gap. The lifecycle, severity, rotation, communication, and postmortem sections below are a menu of details—not a requirement to recite every runbook component.

Choosing and Adapting a Real Story

Choose a real story where the runbook had a clear user: an on-call engineer, incident commander, service owner, or support partner. The strongest examples usually include:

• a confusing or repeated failure mode that exposed a missing decision or escalation path;
• a concrete change to steps, signals, ownership, or communication;
• a review or rehearsal with the people who would use the runbook; and
• evidence of improved response, reduced toil, or a clearly defined next measurement.

If you only contributed one part of a team effort, make that boundary explicit. You can adapt a real documentation or reliability story to this prompt, but do not claim to have designed the entire incident program if you only edited a section or tested a command.

Why Runbooks Matter

An incident at 3 AM is the worst time to figure out how your system works. Runbooks encode the knowledge of your most experienced engineers into step-by-step instructions that anyone on rotation can follow under stress without heroics.

A good runbook is not a manual — it's a decision tree. "If you see X, try Y. If Y doesn't solve it in 10 minutes, escalate to Z."

Incident Lifecycle

Every incident follows the same pipeline:

Detection -> Triage -> Mitigation -> Resolution -> Postmortem

Detection: alert fires, user report, on-call notified
Triage: assess severity, communicate to stakeholders, start incident channel
Mitigation: stop the bleeding (rollback, kill switch, scale up, redirect traffic)
Resolution: fix root cause, validate fix under production load
Postmortem: blameless review, action items assigned with owners and due dates

Detection to mitigation is the critical window. Optimize for this. Root cause investigation can happen after service is restored.

Severity Levels

Clear severity levels prevent both over-escalation (waking up the CTO for a minor UI bug) and under-escalation (spending an hour debugging alone while customers can't check out):

Declare severity early and adjust down if assessment improves. It's always better to escalate and stand down than to under-escalate and miss impact.

On-Call Rotation Design

Primary/secondary model: Primary is alerted first and handles the incident. Secondary is backup if primary doesn't acknowledge within 5 minutes and handles escalation.

Rotation structure: One-week rotations are standard. Shorter rotations spread load but increase context-switching cost. Longer rotations cause burnout. Overlap handoffs by 30 minutes.

Fair load: Track alert volume per rotation. If one rotation week consistently receives more alerts, that's a reliability problem — alert fatigue leads to ignored alerts, which leads to actual outages.

On-call compensation: Engineers should be compensated for on-call hours, especially when incidents occur. Uncompensated on-call is unsustainable and drives engineers away from reliability work.

Runbook Structure

Each runbook covers one alert or failure mode:

# Alert: database_primary_high_latency

## What This Alert Means
Primary write latency > 500ms for 5 minutes. Normal is < 80ms.
Potential causes: long-running queries, lock contention, disk I/O saturation.

## Immediate Actions (first 5 minutes)
1. Check: is this a spike or sustained? -> Grafana dashboard link
2. Check: are connections maxed out? -> `SHOW PROCESSLIST` or datadog link
3. Check: what changed in last 30 minutes? -> PagerDuty change feed link

## Diagnosis Steps
If connections are maxed:
  -> Identify long-running queries: `SELECT * FROM pg_stat_activity WHERE duration > 30s`
  -> Kill blocking queries if safe (no payment processing)
  -> Scale connection pool: [link to runbook: scale db connection pool]

If disk I/O saturation:
  -> Check disk metrics: [link]
  -> If Amazon RDS: trigger IOPS burst if not already active
  -> Consider read replica promotion for reads: [link to runbook]

If no obvious cause:
  -> Escalate to database oncall: @db-team

## Mitigation Options
- Enable read-only mode for non-critical paths: [feature flag link]
- Rollback last deployment: [deployment rollback link]
- Promote read replica: [runbook link]

## Escalation
If not resolved in 20 minutes: escalate to SEV2, add @db-team and @platform-lead

Communication During Incidents

Dedicated incident Slack channel, created by incident commander on declaration. Every 15-30 minutes, post a status update even if nothing has changed:

[14:45 UTC] - Investigating: error rate at 12%, source unknown. Working to identify.
[15:00 UTC] - Root cause found: connection pool exhausted after deploy at 14:30.
              Rolling back deployment now. ETA resolution: 10 minutes.
[15:12 UTC] - Rollback complete. Error rate back to baseline. Monitoring.
[15:30 UTC] - Incident resolved. Setting up postmortem for tomorrow 10am.

Never go silent to stakeholders. "I don't know yet but I'm working on it" is a valid update.

Blameless Postmortem

A postmortem's goal is to prevent the incident from happening again — not to assign fault. If engineers fear blame, they hide information that would help the investigation, and you repeat the incident.

Postmortem structure:

1. Timeline: minute-by-minute what happened, detected, done
2. Root cause: what changed, what failed, why
3. Contributing factors: what made the system fragile (lack of alerts, no canary deployment, missing circuit breaker)
4. Impact: duration, affected users, revenue estimate
5. Action items: each with an owner and a due date
6. What went well: things that helped contain or detect the incident faster

Action items without owners and due dates are not action items. Track them in the same system as engineering tickets.

Follow-Up Questions and How to Answer

“What makes a runbook usable at 3 AM?” Point to a clear alert meaning, safe first checks, decision branches, expected signals, escalation timing, and links that a new on-call engineer can follow.

“How did you choose severity levels?” Explain the impact-based criteria, the response time, and the people who validated them. Treat percentages and thresholds as context-specific policies, not universal laws.

“How did you keep the runbook safe?” Describe command review, permissions, rollback or stop conditions, read-only checks first, and a rehearsal or incident review.

“How did you know it worked?” Name the metric or feedback loop: time to acknowledge, time to mitigate, repeat pages, escalation quality, or on-call toil. If it was not measured, say what you would baseline next.

“What if the runbook becomes stale?” Explain ownership, review triggers after incidents or system changes, link checks, and a date or cadence for review.

Common Mistakes

• Writing an encyclopedia instead of a decision tree with clear next actions and stop conditions.
• Listing commands, dashboards, or contacts without verifying permissions, links, expected outputs, or rollback safety.
• Treating severity percentages, response times, or rotation patterns as universal standards rather than policies tailored to the service.
• Overclaiming the outcome—for example, claiming a reduction in mitigation time or toil without a baseline or measurement.
• Presenting a hypothetical runbook, rehearsal, or incident result as work you personally completed. Use “I would…” for proposed design.
• Describing a runbook as finished without naming an owner, review trigger, or way to keep it current.
• Focusing only on technical diagnosis and omitting communication, escalation, and the needs of the person on call.

Test Your Understanding

1. What is the difference between a manual and a runbook?

A manual explains a system broadly; a runbook guides a stressed operator through a specific alert or failure mode with decisions and escalation paths.

2. What should the first five minutes optimize for?

Reliable detection, severity assessment, communication, and safe mitigation—not a complete root-cause explanation.

3. What makes a runbook action safe to include?

The action has a clear purpose, expected signal, permission boundary, stop or rollback condition, and review by the people who will use it.

4. How do you answer if you have not measured the runbook's impact?

Say that the result is unmeasured, name the baseline and metric you would add, and avoid claiming improvement.

Recap

1. Runbooks encode institutional knowledge into step-by-step decision trees. Engineers under stress at 3 AM execute better with clear steps than from memory. Every alert should have a runbook.
2. Severity levels prevent escalation mistakes. Declare high and adjust down rather than under-declare. Clear criteria (% of users affected, specific functionality broken) remove ambiguity under pressure.
3. One-week rotations with primary/secondary roles are standard. Track alert volume per rotation week — alert fatigue leading to ignored alerts is a reliability risk in itself.
4. During an incident, optimize for mitigation speed, not root cause investigation. Rollback first, understand why later. Status updates every 15-30 minutes to stakeholders even when there is nothing new to report.
5. Blameless postmortems improve system resilience over time. Blame causes hiding, hiding causes recurrence. Every postmortem action item needs an owner and a due date or it won't happen.

Related Skills

• Incident triage, severity assessment, and escalation
• Alert design, observability, and operational metrics
• On-call communication and incident command
• Blameless postmortems, ownership, and continuous improvement