Production Incident Kit

Severity levels

A severity level is a shared shorthand for "how bad is this and how fast do we move". Agree the levels before you need them, so nobody argues about labels while customers are waiting.

The response targets below are examples to adapt. Set numbers your team can actually meet, at the hours you actually cover. A target nobody meets is worse than a modest one people trust.

The levels

| Level | Name | Criteria (any one is enough) | Examples |
|---|---|---|---|
| SEV1 | Critical | Core service down or unusable for most customers. Data loss or corruption in progress. Confirmed security breach or exposure of customer data. Payments or money movement failing broadly. Legal, safety or regulatory impact | Site returns errors for everyone. Database corruption. Leaked admin credentials being used. Checkout failing for all cards |
| SEV2 | Major | A key feature is down or badly degraded for many customers, or for an important segment. Core service is working but with serious slowness or high error rates. A SEV1 is likely if nothing is done | Sign-in failing for 20% of users. One region down. Background jobs stalled so emails and exports aren't sent. Error rate well above normal |
| SEV3 | Minor | A non-critical feature is broken, or a small number of customers are affected, and a workaround exists | Reports page slow. Image uploads failing for one file type. A single enterprise customer's integration broken with a manual workaround |
| SEV4 | Low | Little or no customer impact. Internal tools, cosmetic problems, or a risk that needs attention soon but not now | Admin dashboard chart broken. Certificate expires in 10 days. Disk at 75% and rising slowly |

When in doubt, pick the higher level. Downgrading later costs nothing. Starting too low means the wrong people aren't there when it gets worse.

Response targets (examples)

| Level | Acknowledge page | Incident lead in place | First public update | Update cadence | Who gets paged | Hours |
|---|---|---|---|---|---|---|
| SEV1 | 5 minutes | 15 minutes | 15 to 30 minutes | Every 30 minutes, or sooner on change | Primary on-call, secondary on-call, incident lead rota, engineering manager | 24/7 |
| SEV2 | 15 minutes | 30 minutes | 30 to 60 minutes | Every 60 minutes | Primary on-call, incident lead rota | 24/7 or extended hours |
| SEV3 | 1 working hour | Optional | If customers are asking | When status changes | Primary on-call (notification, not a page) | Working hours |
| SEV4 | Next working day | Not needed | Not needed | Not needed | Ticket in the owning team's queue | Working hours |

Notes:

• "Acknowledge" means a person has seen the page and is looking, not that it's fixed.
• Public update times start when the incident is declared, not when it started.
• A SEV1 or SEV2 that runs outside covered hours still gets the full response. If you don't have 24/7 cover, say so in your SLA and status page, and set an out-of-hours phone tree for SEV1.

Who decides

• Anyone can declare an incident and set an initial level. Nobody should need permission to raise the alarm.
• The incident lead confirms or changes the level once they're in place.
• Change the level whenever the facts change, and say so in the incident channel: "Raising to SEV1: data loss confirmed in orders table."

Escalation triggers

Raise the level, or bring in more people, when:

• The incident has run for 30 minutes (SEV1) or 60 minutes (SEV2) with no clear mitigation in sight.
• Impact is spreading: more customers, more regions, more features.
• The cause is outside the on-call team's knowledge or access.
• There is any sign of a security cause.
• A customer with contractual commitments is affected and asking.
• Responders are tired. Incidents running longer than about four hours need a planned handover.

Customer-facing commitments

If you have SLAs or published uptime commitments, map them here so responders know which incidents count:

| Commitment | Applies to | Counts as downtime | Level it implies |
|---|---|---|---|
| [99.9% monthly uptime] | [API and dashboard] | [Error rate above 5% for 5 minutes or more] | [SEV1 or SEV2] |
| | | | |

After the incident

| Level | Postmortem | Due | Review |
|---|---|---|---|
| SEV1 | Required | Draft within 3 working days | Team review meeting, shared with leadership |
| SEV2 | Required | Draft within 5 working days | Team review meeting |
| SEV3 | Short write-up if there's something to learn | Within 2 weeks | Optional |
| SEV4 | Not needed | | |

Team decisions

Fill these in and keep them up to date.

• Paging tool: [name]
• Incident channel naming: [for example #inc-2026-10-22-checkout]
• Status page: [URL]
• Incident lead rota: [link or names]
• Out-of-hours phone tree for SEV1: [link]
• Last reviewed: [date] by [name]

Incident runbook

This is the full process for a production incident, from the first alert to the postmortem. It's written for a team of three to thirty engineers. Smaller teams can combine roles; larger teams can add more.

An incident is any unplanned event that harms, or is about to harm, the service your customers rely on. If you're asking "is this an incident?", treat it as one. Standing one down costs a few minutes; missing one costs much more.

The principles

1. Mitigate first, root cause later. Your first job is to stop the harm. Rolling back a deploy you don't yet understand is a good outcome.
2. One person leads. The incident lead coordinates. They don't have to be the most senior or the person typing commands.
3. Write everything down, as it happens. A timeline written live is accurate; one reconstructed tomorrow is not.
4. Communicate early and on a schedule. Silence feels like neglect to customers and colleagues.
5. Blameless. People act on the information they had at the time. Ask how the system let it happen.

Phase 1: detect

Incidents start from an alert, a customer report, a colleague's message or a provider notice.

• The person who notices checks one independent signal to confirm it's real.
• If customer impact is possible, they declare (phase 2). They don't wait for certainty.
• If it's clearly not customer-affecting, they open a ticket at SEV4 instead.

Good alerts page on symptoms customers feel (error rate, latency, failed checkouts), not on every cause (CPU at 80%). If your team gets paged for things that don't need action, fix the alert after the incident. See on-call-checklist.md.

Phase 2: declare

1. Create a dedicated incident channel, named #inc-[YYYY-MM-DD]-[short-name]. Keep all incident discussion there.
2. Post the declaration: what's broken, since when, initial severity, who is acting lead, link to the main dashboard.
3. Set the severity with severity-levels.md.
4. Page the right people for that severity.
5. Start the incident log (incident-log-template.md) or use the channel as the log, with times on every entry.
6. Open a video or voice bridge for SEV1 and SEV2. Link it in the channel. Decisions made on the call get written in the channel.

Phase 3: assign roles

For SEV1 and SEV2, fill these three roles as soon as possible. For SEV3, one person can hold all three.

Incident lead

Owns the incident until it's closed or handed over.

• Keeps the big picture: what's the impact, what are we trying, what's next.
• Assigns work to named people with a time to report back: "[Name], check the queue depth and report back in 10 minutes."
• Makes decisions when the group is stuck. A reasonable decision now beats a perfect one in an hour.
• Sets and changes the severity.
• Decides when to escalate, bring in other teams, or call vendors.
• Keeps the team from going down rabbit holes: "Is this helping us mitigate?"
• Doesn't do hands-on debugging on a SEV1 if anyone else can. Leading is a full-time job.

Communications

Owns every message that leaves the incident channel.

• Posts status page updates on schedule (communications/status-page-templates.md).
• Sends internal updates to leadership, support and sales (communications/internal-update-template.md).
• Gives support a holding reply for customers.
• Drafts the customer email after resolution if needed (communications/customer-email-template.md).
• Gets technical facts from the lead, not from interrupting the people debugging.
• Reminds the lead when an update is due.

Operations

The people investigating and making changes. There can be several.

• Announce every change in the channel before making it: what, where and how to undo it.
• Make one change at a time, then watch the effect.
• Report findings as they happen, including dead ends ("Database CPU is normal, not the cause").
• Ask the lead before trying anything risky or irreversible.

Optional roles for large incidents

| Role | When to add it | What they do |
|---|---|---|
| Scribe | SEV1 lasting more than 30 minutes | Keeps the timeline so the lead can think |
| Subject expert | The cause is in a system the on-call team doesn't know | Joins to advise, takes direction from the lead |
| Liaison | A partner, major customer or vendor is involved | Single point of contact with them |
| Security lead | Any sign of an attack or data exposure | Owns evidence preservation and legal notification decisions |

Phase 4: investigate and mitigate

Work in loops of about 10 to 15 minutes: observe, form a theory, try the safest action, check the result, report.

Questions that find most causes:

1. What changed just before it started? Deploys, config, flags, infrastructure, data, traffic, dependencies, certificates, DNS, scheduled jobs.
2. What's the scope? All users, a region, a customer, a feature, a platform?
3. What do the four golden signals say? Latency, traffic, errors, saturation.
4. Is a dependency unhealthy? Check provider status pages and your own dependency dashboards.

Mitigations, safest first:

| Mitigation | Use when | Watch out for |
|---|---|---|
| Roll back the last deploy | Timing lines up with a deploy | Database migrations that can't be reversed |
| Revert a config or flag change | Timing lines up with a change | Caches holding the old value |
| Drain a bad node, pod or region | Errors are concentrated | Remaining capacity must handle the load |
| Scale up or out | Saturation is the problem | Costs, and a slow dependency can't be fixed by more callers |
| Shed load or rate limit | Overload from traffic or a single client | Tell customers if their requests are being rejected |
| Disable a feature | One feature is causing wider harm | Turn it back on only after a fix |
| Fail over | Primary system or region is down | Data replication lag |
| Restore from backup | Data is corrupted | Last resort. Know your restore time and data loss window first |

Once impact stops, say so: "Mitigated at 14:42 UTC. Error rate back to baseline. Monitoring." Move the status page to "Monitoring". The root cause can now be investigated calmly, often the next working day.

Phase 5: keep the timeline

Record, with UTC times:

• When the problem started (from graphs, not when it was noticed)
• When it was detected, and how
• When it was declared, and at what severity
• Each significant finding, theory, decision and action, with who did it
• Severity changes
• Each external update sent
• When impact was mitigated, and when it was fully resolved

Short lines are fine: "14:31 [Name]: rolled back api to v2.41.3." The postmortem will be built from this.

Phase 6: escalate

Escalate when:

• There's no clear mitigation after 30 minutes (SEV1) or 60 minutes (SEV2).
• The incident needs access or knowledge the current responders don't have.
• Impact is growing.
• There is any sign of a security cause. Bring in your security lead and switch to evidence-preserving actions.
• A vendor is the cause. Open a support case at the highest priority your contract allows, and post the case number in the channel.
• People are tired. Plan handovers every few hours on long incidents.

How to escalate: page the next person on the escalation list, tell them the severity, the impact and what you need from them in one message.

Phase 7: hand over

On long incidents, or when the lead changes:

Handing over incident lead to [name] at [time]. Impact: [summary]. Status: [mitigated / not mitigated]. Current work: [who is doing what]. Next status update due: [time]. Open questions: [list].

The new lead confirms in the channel: "I have the lead."

Phase 8: resolve and close

Resolved means customers are no longer affected and the system is stable. It doesn't mean the cause is fixed for good.

Before closing:

• Metrics have been at normal levels for an agreed period (for example 30 minutes for SEV1).
• Temporary mitigations are listed, with owners to make them permanent or remove them.
• The status page shows "Resolved" with a short summary.
• Internal stakeholders have a final update.
• Support knows what to tell customers who ask.
• A postmortem owner and due date are set (SEV1 and SEV2).
• The incident log is saved where the postmortem author can find it.
• Thank the people who helped. Note who was woken up, and make sure they can take time back.

Post the closing message in the channel:

Resolved at [time UTC]. Duration: [x]. Impact: [summary]. Postmortem owner: [name], draft due [date]. Thanks all.

Phase 9: learn

Write the postmortem with postmortem-template.md, review it as a team and track the action items to completion. An incident without follow-up is likely to happen again.

Quick reference

| Step | Who | Done when |
|---|---|---|
| Confirm | First responder | One independent signal checked |
| Declare | First responder | Channel created, first message posted |
| Set severity | First responder, then lead | Level posted in channel |
| Assign roles | Incident lead | Lead, comms and ops named |
| Mitigate | Operations | Customer impact stopped |
| Communicate | Communications | Updates on schedule |
| Resolve | Incident lead | Stable for the agreed period, status page resolved |
| Learn | Postmortem owner | Postmortem reviewed, actions tracked |

Status page templates

Short, honest and on time beats detailed and late. Customers want to know three things: do you know, what's affected, and when will they hear more.

Writing rules

• Say what customers experience, not what broke inside: "Some payments are failing" rather than "Elevated 5xx on payments-api pods".
• Be specific about scope: which feature, which region, which customers, if you know.
• Give a next update time and keep it, even if the update is "no change, still working on it".
• Don't guess at causes or fix times in public. "We expect to have this fixed within the hour" becomes a broken promise if you're wrong.
• Don't blame a vendor, a customer or a person. "A third-party provider" is fine when it helps customers understand.
• Don't minimise. Avoid "minor", "brief" or "a small number" unless you're sure. Customers who are fully blocked read that as dismissive.
• Use plain words. No internal names, ticket numbers or jargon.
• Use UTC, or UTC plus your main customers' time zone, and always say which.
• Keep the same incident open from start to finish, adding updates, so the history reads as one story.

Fill-in fields are in [square brackets]. Delete any sentence you can't fill truthfully.

---

Investigating

Use as soon as the incident is declared and customers may be affected.

Title: [Feature or service] [errors / slowness / unavailable]

We're investigating reports of [errors / slow responses / failures] when [customers do what, for example "signing in" or "loading dashboards"]. [Customers in [region] / Some customers / All customers] may be affected. We'll post an update by [HH:MM UTC].

Shorter version for the first few minutes:

We're investigating [errors when signing in]. Next update by [HH:MM UTC].

Identified

Use when you know the cause or have a clear fix underway.

We've identified the cause of [the errors when signing in] and are [rolling back a recent change / applying a fix / moving traffic to healthy servers]. [Customers may still see errors until this is complete.] Next update by [HH:MM UTC].

If you have a workaround:

While we work on a fix, you can [use the mobile app / retry the request / use the export from Settings instead].

Monitoring

Use when the fix is in place and you're watching to confirm it holds.

A fix has been applied and [sign-in / payments / dashboards] [are working normally again / are recovering]. We're monitoring to make sure the issue is fully resolved. [If you still see problems, please [retry / refresh / contact support at [link]].] Next update by [HH:MM UTC].

Resolved

Use when impact has ended and the system is stable.

This issue is resolved. Between [HH:MM] and [HH:MM UTC], [some / all] customers [could not sign in / saw errors when paying / saw delayed data]. [Data was not lost / Delayed [emails / jobs] have now been processed / Affected [payments] can be retried.] We're reviewing what happened and will take steps to prevent it happening again. We're sorry for the disruption.

If you'll publish a summary:

We'll publish a summary of what happened and what we're changing by [date].

If customers need to act:

If you [made a payment / submitted a form] between [HH:MM] and [HH:MM UTC], please [check / retry / contact support at [link]].

Scheduled maintenance

Post at least [3 to 7] days ahead for disruptive work, and remind customers [24 hours] before.

Title: Scheduled maintenance: [system]

On [day, date] from [HH:MM] to [HH:MM UTC], we'll be carrying out maintenance on [system]. During this time, [describe the impact: "the dashboard will be read-only" / "the API may return errors for up to [5] minutes" / "no impact is expected"]. [No action is needed from you. / Please [action] before [time].] We'll update this page when the work starts and when it's complete.

When it starts:

Maintenance is now in progress. [Impact reminder.] We'll update this page when it's complete.

When it's complete:

Maintenance is complete and [system] is working normally. Thank you for your patience.

If it overruns:

Maintenance is taking longer than planned. [Impact reminder.] We now expect to finish by [HH:MM UTC] and will update this page by then.

Third-party outage

Use when a provider you depend on has a problem that affects your customers.

Title: [Feature] affected by a third-party outage

[Feature, for example "card payments" or "email notifications"] [is unavailable / is delayed / may fail] because of an outage at one of our service providers. We're monitoring their progress [and have [switched to a backup provider / paused [jobs] so nothing is lost]]. [Other features are working normally.] Next update by [HH:MM UTC].

When the provider recovers:

Our provider has resolved their issue and [feature] is working normally again. [Delayed [emails / jobs] are being processed now and should complete by [HH:MM UTC].]

Naming the provider is a judgement call. It helps customers if they use the same provider and can see its status page; it can read as passing blame. Either is fine; be consistent.

Partial or regional impact

[Feature] is [unavailable / slow] for customers in [region / on [plan] / using [integration]]. Other customers aren't affected. Next update by [HH:MM UTC].

Security-related incidents

Don't post detail about an active security incident on a public status page until your security lead and legal contact agree the wording. A neutral holding message is fine:

We're investigating an issue affecting [feature]. [Feature] [is temporarily unavailable] while we do this. Next update by [HH:MM UTC].

Update schedule by severity

| Severity | First update | Then |
|---|---|---|
| SEV1 | Within 15 to 30 minutes of declaring | At least every 30 minutes |
| SEV2 | Within 30 to 60 minutes | At least every 60 minutes |
| SEV3 | If customers are noticing or asking | When status changes |

Adjust to match severity-levels.md.