How to Create an Incident Runbook in Confluence
When a production incident hits at 2 AM, the last thing your on-call engineer should be doing is figuring out what to do from scratch. Incident runbooks codify your team's collective knowledge about how to diagnose, mitigate, and resolve known failure modes so that response is fast and consistent regardless of who is on call.
Confluence is a natural home for runbooks because it supports rich formatting, embedded diagrams, and easy cross-linking between related pages. This guide shows you how to structure incident runbooks that are genuinely useful under pressure — not just documentation theater that nobody reads when it matters.
Step-by-step guide
Create a Runbook Space and Hierarchy
Set up a dedicated Confluence space called 'Incident Runbooks' with a clear page hierarchy organized by service or system. Top-level pages should map to your major services (e.g., API Gateway, Payment Service, Auth Service), with child pages for specific incident types under each. This structure lets on-call engineers navigate directly to the relevant runbook by service name without searching.
- Add a pinned 'Runbook Index' page at the space root with a table listing all runbooks, their severity, and last-updated date
- Use Confluence labels like 'sev1-runbook' and 'sev2-runbook' to enable filtered views by severity
Define a Standard Runbook Template
Create a Confluence page template that every runbook follows. Include these mandatory sections: Incident Summary (one-line description of what's broken), Severity and Impact (who is affected and how), Detection (how you know this is happening — alerts, dashboards, customer reports), Diagnosis Steps, Mitigation Steps, Resolution Steps, and Escalation Path. Consistency across runbooks means engineers never waste time figuring out where to look for information.
- Add a metadata panel at the top with Owner, Last Tested, and Related Services fields
- Include a 'Prerequisites' section listing required access, tools, and credentials needed to execute the runbook
Write Diagnosis Steps with Decision Trees
Structure diagnosis as a decision tree, not a linear list. Start with the most common cause and use conditional branches: 'If metric X is above threshold, go to Step 3a. If metric X is normal, go to Step 3b.' Include specific commands to run, exact dashboard URLs to check, and what normal vs. abnormal values look like. Vague instructions like 'check the logs' are useless at 2 AM — specify which log group, what query to run, and what pattern to look for.
- Embed screenshots of what dashboards look like during normal operation vs. during the incident
- Include copy-pasteable CLI commands and database queries with placeholders clearly marked
Document Mitigation and Resolution Separately
Separate mitigation (stop the bleeding) from resolution (fix the root cause). Mitigation steps should be fast actions that reduce customer impact — restarting a service, scaling up instances, enabling a feature flag, or failing over to a backup. Resolution steps address the underlying cause and may require more time and expertise. Making this distinction explicit helps on-call engineers prioritize correctly under pressure.
- Add estimated time-to-mitigate and time-to-resolve for each runbook based on past incidents
- Include rollback procedures for each mitigation step in case it makes things worse
Add Escalation Paths and Communication Templates
Document exactly when and how to escalate: which Slack channel to post in, who the service owner is, what the PagerDuty escalation policy is, and at what point to involve leadership. Include pre-written communication templates for stakeholder updates — a fill-in-the-blank status update for Slack and a customer-facing message if needed. Under stress, people struggle to compose clear communications, so having templates removes that burden.
- Link to the relevant PagerDuty service and escalation policy directly in the runbook
- Include a 'Who to call if the primary owner is unavailable' backup contact list
Cross-Link Related Runbooks and Resources
Incidents rarely happen in isolation — a database runbook may need to reference the API gateway runbook if the root cause is upstream. Use Confluence's page linking to connect related runbooks, and embed links to relevant architecture diagrams, monitoring dashboards, and service dependency maps. The goal is to minimize context-switching during an active incident by keeping everything reachable within one or two clicks.
- Create a 'Related Incidents' section at the bottom linking to past postmortem pages for this failure mode
- Add inline links to Datadog, Grafana, or CloudWatch dashboards specific to each diagnosis step
Establish a Review and Testing Cadence
Runbooks rot fast. Set a quarterly review cycle where each service owner re-reads their runbooks and verifies that commands, URLs, and escalation paths are still accurate. Better yet, run 'runbook drills' where an engineer follows the runbook step-by-step in a staging environment to surface gaps. Add a 'Last Tested' date to each runbook's metadata panel so staleness is visible at a glance.
- Create a Confluence automation that flags runbooks not updated in 90 days
- After every real incident, update the relevant runbook as part of the postmortem action items
Common mistakes
Writing Runbooks After the Fact and Never Testing Them
Teams often write runbooks during a postmortem but never validate them against a real scenario. A runbook that hasn't been tested is a runbook that will fail when you need it most. Schedule dry runs where an engineer who didn't write the runbook follows it step by step — their confusion reveals the gaps.
Being Too Vague in Diagnosis Steps
Instructions like 'check the database' or 'look at the logs' are useless under pressure. Every step should include the exact command, URL, or query to run, what the expected output looks like, and how to interpret the results. Write for the least experienced person who might be on call.
Forgetting to Update Runbooks When Systems Change
Infrastructure changes, service ownership rotates, and dashboards get reorganized — but runbooks often don't get updated to reflect these changes. Tie runbook reviews to your deployment and architecture change processes so updates happen proactively, not after a failed incident response.
Burying Runbooks in a Documentation Graveyard
If engineers can't find the runbook within 30 seconds of being paged, it might as well not exist. Pin runbook links in your incident Slack channel, add them to PagerDuty service descriptions, and include direct links in alert notifications. Discoverability is as important as content quality.
Tips
Include the runbook URL directly in your PagerDuty or OpsGenie alert descriptions so on-call engineers land on the right page immediately when paged.
Use Confluence's 'status' macro to mark runbook sections as Verified, Needs Update, or Deprecated so engineers know which parts to trust.
Write runbooks in second person imperative ('Run this command,' 'Check this dashboard') — it's faster to parse under stress than passive voice.
Keep a 'Runbook Gaps' page where anyone can log incidents that didn't have a runbook, so you know what to write next.
How Vantage helps
Vantage helps product teams define the requirements and acceptance criteria for incident management systems alongside their product work. When building internal tooling or reliability features, Vantage's context-aware PRD generation ensures that operational requirements like runbook integration and alerting workflows are captured from the start.