Note: Company names, engineers, incidents, numbers, and scaling scenarios in this article are hypothetical — even when they resemble real ones. See the full disclaimer.

Incident response tooling

It is 02:47 on a Saturday at PaisaBridge and Aditi's phone is buzzing with a PagerDuty page — CheckoutAPIErrorRateBurnRate14_4_1h. She is the Incident Commander on rotation this week, which means her first job is not to debug; it is to run the room. In the next ninety seconds she has to declare a severity, open a war-room channel, page the checkout-team's secondary, ping customer-support so they can pre-stage a holding line, start a timeline that will become the post-mortem record, and decide whether to wake up the VP of engineering. If she does this from muscle memory across six tools she will get one of them wrong and lose four minutes she does not have. The team that runs incidents well has built a single command that does all of it — /incident declare sev2 checkout in Slack — and the IC's job becomes thinking, not clicking. Incident response tooling is the discipline of making the operationally correct path also the path of least resistance at 03:00. Most orgs leave it to the tribal knowledge of whoever ran the last big outage; the orgs that ship reliably do not.

The first ten minutes — what the tooling has to do

The first ten minutes of an incident are mostly procedure, not debugging. The on-call has to acknowledge the page, decide a severity, open a channel, page partners, start a timeline, notify customer-facing teams, and only then start looking at logs. Every minute spent on procedure is a minute customers see the outage. A team without tooling spends 6–9 of those 10 minutes on procedure; a team with mature tooling spends under 90 seconds. The difference compounds — a SEV-2 with a 10-minute mitigation window has 8.5 minutes of actual debugging time in the first case and 9.5 in the second, an 11% buy-back. Across a year of incidents, that buy-back is easily 30 saved customer-impact-minutes.

Why the per-minute buy-back compounds: incidents are not independent samples. A team that loses 7 minutes per incident on procedure also tends to skip the post-mortem ritual ("we were too tired"), which means the next incident has the same procedural overhead. Tooling breaks the loop — the procedure happens regardless of how exhausted the IC is, the timeline is captured even if no human typed it, and the post-mortem template starts pre-populated. The buy-back is not just minutes per incident; it is the team's ability to learn from incidents accumulating over months.

The seven things the tooling has to do in those 90 seconds are a fixed list — the team's job is not to invent the list each time but to encode it once and re-run it forever:

1. Acknowledge the page — bidirectional sync with PagerDuty so the alert stops escalating.
2. Declare a severity — picks the playbook the rest of the bot will execute.
3. Open a war-room channel — #inc-YYYY-MM-DD-<service> with a templated topic, pinned dashboards, pinned runbook.
4. Open a voice bridge — Zoom/Meet, link posted in channel.
5. Page partners — secondary on-call, customer-support lead, sometimes a database SME, by service.
6. Start the timeline — every chat message, every command run, every dashboard click captured automatically.
7. Pre-stage comms — a draft customer-facing message in #status-pages-draft for the Comms Lead.

A team that has built this — /incident declare sev2 <service> in Slack — does all seven in parallel before the IC has finished typing the next message.

The /incident bot — the load-bearing artefact

The /incident slash command is the centrepiece. Most teams write theirs in Python because every API in the chain (PagerDuty, Slack, Zoom, Statuspage, Jira) has a clean Python SDK, and because the bot is the kind of code on-calls debug at 03:00 — Python is what the on-call already reads. The minimal version is under 200 lines. Here is the load-bearing core, the part that runs when the IC types /incident declare sev2 checkout:

# incident_bot.py — Slack /incident command handler (FastAPI + slack-bolt + Python SDKs)
# pip install slack-bolt slack-sdk pdpyras pyjwt requests pydantic fastapi uvicorn
import os, time, datetime as dt, requests
from slack_bolt import App
from slack_bolt.adapter.fastapi import SlackRequestHandler
from pdpyras import APISession  # PagerDuty REST client
from fastapi import FastAPI, Request

slack = App(token=os.environ["SLACK_BOT_TOKEN"], signing_secret=os.environ["SLACK_SIGNING_SECRET"])
pd_session = APISession(os.environ["PAGERDUTY_API_KEY"])

# Severity → policy: which on-calls page, what comms template, retention class
SEVERITY_POLICY = {
    "sev1": {"page": ["primary", "secondary", "manager", "vp_eng"], "comms": "public_status_page",
             "post_mortem": "required_within_3d", "war_room_required": True},
    "sev2": {"page": ["primary", "secondary", "cs_lead"], "comms": "in_app_banner_only",
             "post_mortem": "required_within_5d", "war_room_required": True},
    "sev3": {"page": ["primary"], "comms": "internal_only",
             "post_mortem": "optional", "war_room_required": False},
}

# Service → escalation policy IDs in PagerDuty (data-driven, lives in a YAML)
SERVICE_ROUTING = {
    "checkout":    {"primary": "P3CHKO1", "secondary": "P4CHKO2", "cs_lead": "PCS1234", "owner_team": "checkout-platform"},
    "rider-pos":   {"primary": "P9RPOS1", "secondary": "P0RPOS2", "cs_lead": "PCS1234", "owner_team": "rider-platform"},
    "auth":        {"primary": "P5AUTH1", "secondary": "P6AUTH2", "cs_lead": "PCS1234", "owner_team": "identity"},
}

@slack.command("/incident")
def declare_incident(ack, body, client):
    ack()  # Slack expects an ack within 3 seconds; everything below runs async
    # Parse: "/incident declare sev2 checkout" → action=declare, sev=sev2, service=checkout
    parts = body["text"].split()
    if len(parts) < 3 or parts[0] != "declare":
        client.chat_postEphemeral(channel=body["channel_id"], user=body["user_id"],
                                   text="Usage: /incident declare <sev1|sev2|sev3> <service>")
        return
    _, sev, service = parts[0], parts[1], parts[2]
    if sev not in SEVERITY_POLICY or service not in SERVICE_ROUTING:
        client.chat_postEphemeral(channel=body["channel_id"], user=body["user_id"],
                                   text=f"unknown sev '{sev}' or service '{service}'")
        return

    pol = SEVERITY_POLICY[sev]
    routing = SERVICE_ROUTING[service]
    today = dt.datetime.utcnow().strftime("%Y-%m-%d-%H%M")
    channel_name = f"inc-{today}-{service}"

    # Step 1: create the war-room channel with a templated topic
    chan = client.conversations_create(name=channel_name, is_private=False)
    channel_id = chan["channel"]["id"]
    client.conversations_setTopic(channel=channel_id,
        topic=f"{sev.upper()} :: {service} :: IC: <@{body['user_id']}> :: started {today} UTC")
    client.conversations_setPurpose(channel=channel_id,
        purpose=f"Incident {sev.upper()} on {service}. Runbook: https://playbooks/{service}")

    # Step 2: page the right people via PagerDuty
    paged = []
    for role in pol["page"]:
        if role in routing:
            incident = pd_session.rpost("incidents", json={
                "incident": {
                    "type": "incident",
                    "title": f"{sev.upper()} {service} — IC {body['user_name']}",
                    "service": {"id": routing[role], "type": "service_reference"},
                    "urgency": "high" if sev in ("sev1", "sev2") else "low",
                    "body": {"type": "incident_body", "details": f"Slack channel: #{channel_name}"},
                }
            })
            paged.append(role)

    # Step 3: pin the runbook, dashboards, and timeline-bot kickoff
    client.chat_postMessage(channel=channel_id, text=(
        f":rotating_light: *{sev.upper()}* on `{service}` — IC <@{body['user_id']}>\n"
        f"- Runbook: https://playbooks.example.com/{service}\n"
        f"- Dashboard: https://grafana.example.com/d/{service}-overview\n"
        f"- Tempo: https://tempo.example.com/search?service.name={service}\n"
        f"- Paged: {', '.join(paged)}\n"
        f"- Comms policy: *{pol['comms']}*\n"
        f"- Post-mortem: *{pol['post_mortem']}*"
    ))

    # Step 4: write the incident metadata to the timeline store; bot will tail this channel
    requests.post(os.environ["TIMELINE_BOT_URL"] + "/start", json={
        "incident_id": channel_name, "severity": sev, "service": service,
        "ic_user_id": body["user_id"], "started_at": time.time(),
    }, timeout=2)

    # Step 5: pre-stage the comms draft if customer-facing
    if pol["comms"] != "internal_only":
        client.chat_postMessage(channel="C_STATUS_PAGES_DRAFT", text=(
            f"Draft for {channel_name}: 'We are currently investigating an issue affecting "
            f"{service}. We will share an update within 30 minutes.' — Comms Lead, please refine."
        ))

api = FastAPI()
handler = SlackRequestHandler(slack)
@api.post("/slack/events")
async def endpoint(req: Request): return await handler.handle(req)

Sample run when Aditi types the command at 02:47:38 UTC:

[02:47:38] /incident declare sev2 checkout  (from @aditi.r)
[02:47:38] ack returned to Slack (within 220ms)
[02:47:39] channel #inc-2026-04-22-0247-checkout created (id C09XYZ001)
[02:47:39] topic set: "SEV2 :: checkout :: IC: @aditi.r :: started 2026-04-22-0247 UTC"
[02:47:40] PagerDuty incident created PD-INC-871234 -> primary on-call
[02:47:40] PagerDuty incident created PD-INC-871235 -> secondary on-call
[02:47:41] PagerDuty incident created PD-INC-871236 -> cs_lead
[02:47:41] runbook + dashboard + tempo links posted in channel
[02:47:42] timeline-bot /start: incident_id=inc-2026-04-22-0247-checkout
[02:47:42] comms draft posted to #status-pages-draft

elapsed: 4.2 seconds

Walking the load-bearing lines:

• SEVERITY_POLICY and SERVICE_ROUTING — the policy is data, not code. New severities, new services, new escalation chains land as YAML PRs reviewed by the platform team. The IC at 03:00 never edits the policy; they execute it. Why policy-as-data beats policy-in-code: a code change to add a new service requires a deploy, a code review, and someone awake to merge it. A YAML change in the policy file can be made by any platform engineer at any time and reloaded by the bot without a deploy. More importantly, the YAML is readable by non-engineers — the head of customer-support can confirm the cs_lead routing is correct without reading Python. The bot's logic stays small; the policy grows freely.
• pd_session.rpost("incidents", ...) — paging via the PagerDuty REST API rather than asking the IC to log into the dashboard and click. The IC never loads pagerduty.com during an incident; the bot does it for them. This is the single biggest time-saver, because the PagerDuty UI is a four-click path even for an experienced user.
• conversations_setTopic(...) — the channel topic is the IC's at-a-glance status. Anyone joining the channel sees severity, service, IC, and start time without reading a single message. War-rooms with no topic discipline turn into chat threads where new joiners ask "wait what's going on" and burn five more minutes.
• requests.post(... + "/start", ...) — the timeline bot is a separate service. It tails the channel via the Slack RTM API and writes every message to a structured timestream (Postgres + S3). The IC does not maintain the timeline; the bot does. This decouples timeline-keeping from the IC's attention budget, which during a SEV-1 is approximately zero.
• The comms draft to #status-pages-draft — the Comms Lead does not have to remember to start writing. The bot's draft is a starting point; the lead refines it and pushes it to the actual status page. Comms is the second-most-skipped step (after timeline) when there is no automation; pre-staging the draft is what makes it consistent.

The bot is opinionated by design. There is no --severity custom flag, no --don't-page escape hatch, no way to skip the comms draft. The opinionatedness is the feature — the IC at 03:00 should not be making bespoke choices; they should be running the team's pre-decided procedure. Bespoke choices belong in the post-mortem ritual, not the first ten minutes.

The timeline bot — the silent witness

The timeline is the most-skipped artefact in incident response. Every team intends to keep one; almost no team actually does. The default failure mode: the IC says "someone scribe please" in the war-room channel, two engineers half-heartedly copy-paste a few messages into a Google Doc, then both get pulled into actual debugging and the timeline ends at minute four of a forty-minute incident. The post-mortem author the next morning has to reconstruct events from chat history, kubectl history, alert history, and memory — and gets it wrong.

The corrective is a bot that listens silently and records everything with timestamps:

Why a typed timeline beats a chat-log dump: a chat log is text. A typed timeline distinguishes between human chat ("I think it's the consumer rebalance"), tool output ("kubectl rollout status returned 0 ready"), and system events ("alert resolved"). The post-mortem author can filter by type — show me only the system events between 02:47 and 03:11 — and reconstruct the causal sequence rather than the narrative sequence. The narrative sequence is what the engineers remember; the causal sequence is what actually happened. The two are different roughly 40% of the time on incidents longer than 30 minutes, and the difference is where the post-mortem's real lessons live.

The kubectl audit hook is the part most teams skip and later regret. Plumbing the cluster's audit log to the timeline bot adds a row every time anyone runs a kubectl command during the incident — the user, the verb, the resource, the timestamp. When the post-mortem author reconstructs "what mitigation actually worked", they have a precise log of every command tried, in order, with attributions. Without it, the team is reduced to "I think Karan tried restarting the deployment around 03:02 but I'm not sure who did the actual rollout".

Severity definitions — the policy that drives everything

Every other tool in this article is parameterised by severity. The /incident bot's escalation chain depends on it. The comms-template generator depends on it. The post-mortem requirement depends on it. Severity definitions are therefore the most load-bearing piece of policy you write — and the one teams write most carelessly. The default failure mode is a one-liner per severity ("SEV-1 is bad", "SEV-2 is also bad") that gives the IC at 03:00 no actual decision rule. The IC then guesses, often wrong, often under-classifying because the social cost of declaring SEV-1 feels high.

The corrective is a written rubric with mechanical triggers. PaisaBridge's:

SEV-1. Customer-visible outage of a P0 surface (UPI payment, login, checkout) lasting >5 minutes, OR data loss of any volume, OR a security incident with confirmed exfiltration. Mechanical triggers: error-rate burn-rate >14.4 over 1h on a P0 SLO; OR data-loss-detected alert; OR security team's cse-confirmed tag. Auto-pages: primary, secondary, owner-team manager, VP-eng, on-call CISO. Comms: public statuspage update within 15 minutes, every 30 minutes thereafter. Post-mortem: required within 3 business days, executive review.

SEV-2. Customer-visible degradation of a P0 surface (latency >2× baseline, or partial-region outage) OR full outage of a P1 surface (dashboard, settings, non-payment flows). Mechanical triggers: burn-rate >6 over 1h on a P0 SLO; OR full unavailability of a P1 service; OR >5% of customer support tickets in the last 30 minutes name the service. Auto-pages: primary, secondary, customer-support lead. Comms: in-app banner, no public statuspage. Post-mortem: required within 5 business days.

SEV-3. Internal-only impact, OR customer-visible issue affecting <1% of users, OR a near-miss caught before customer impact. Mechanical triggers: any alert that the IC believes warrants the war-room ritual but does not meet SEV-2 thresholds. Auto-pages: primary only. Comms: internal Slack only. Post-mortem: optional, encouraged for novel near-misses.

The mechanical triggers are the load-bearing detail. "Burn-rate >14.4 over 1h on a P0 SLO" is a number the IC can check on a dashboard in five seconds. "Customer-visible outage" is a judgement call that costs minutes of debate. A severity rubric that does not reduce to a checkable number for at least one trigger per severity will be argued about during every incident. The argument adds time, the time costs customers, and the IC under-declares. Numbers eliminate the argument.

The comms policy varies by severity for a reason. SEV-1 customer-impact is large enough that customers will already be tweeting; the public statuspage update is informing a population that already knows. SEV-2 is small enough that posting publicly creates more anxiety than it resolves; an in-app banner reaches the affected users without alarming the unaffected ones. SEV-3 is internal-only because there are no customers to inform. Cargo-culting a "always update the public status page" policy from a different team's playbook is one of the most common mistakes — it disguises the fact that the policy varies by severity for sound reasons.

Common confusions

• "PagerDuty is the incident-response tool." PagerDuty is the paging tool — it routes alerts to humans and tracks acknowledgement. It is one of seven things the response tooling does. Teams that conflate the two end up with great paging and chaotic everything-else. The /incident bot, the timeline bot, the comms-draft generator, the war-room channel template, and the severity rubric all live above PagerDuty. PagerDuty is a load-bearing dependency, not the system.
• "The /incident bot needs to support every edge case." The bot needs to support the 90% case in 90 seconds; the 10% edge cases are handled by the IC adding a --escalate vp_eng after the fact, in chat. A bot that supports every edge case has a CLI surface so large that the IC has to read documentation at 03:00 — which they will not. Opinionatedness is the feature. The boring path must be the fast path; the unusual path can be slow because it is unusual.
• "Severity is a judgement call by the IC." Severity should be 90% mechanical (the burn-rate is over 14.4, the data-loss alert fired, etc.) and 10% IC judgement (a novel failure mode that doesn't quite fit). Teams that frame severity as primarily a judgement call always under-classify, because under-classifying feels socially safer than over-classifying. The mechanical triggers force the IC to declare SEV-1 when the numbers demand SEV-1, which is the entire point.
• "Timeline-keeping is the scribe's job." Timeline-keeping is the bot's job. Asking a human scribe to maintain a timeline during a SEV-1 is asking them to do the least cognitively important job during the most cognitively expensive moment. The scribe role survives in some teams as an "executive summarizer" — someone who summarises every 10 minutes for the VP joining late — but the raw timeline must come from the bot.
• "War-room channels should be reused." War-room channels should be one-per-incident, archived after the post-mortem ships. Reusing a channel ("#incidents-checkout") confuses the timeline bot, mixes severities, and makes searching for "the SEV-2 from last Tuesday" a context-switching nightmare. The marginal cost of creating a channel is approximately zero; the marginal cost of confusing two incidents in the same channel during a post-mortem is hours.
• "Customer-support notification can wait." Customer-support is the team that hears about the incident before your alerts in roughly 25% of customer-impacting incidents. Notifying CS as part of the auto-page chain (not as an afterthought 20 minutes in) cuts your time-to-detect on the next incident, because CS now has a pre-staged "we're aware, investigating" macro instead of escalating to engineering through a slow channel.

Going deeper

Chat-ops and the runbook-as-command pattern

Mature incident-response tooling extends /incident into a richer chat-ops surface where every common operation is a command in the war-room channel: /dashboard checkout posts the current state of the checkout dashboard as an inline image; /promql rate(http_requests_total[5m]) runs the query and posts the result; /runbook restart-consumer shows the runbook page inline; /escalate vp_eng adds a paged human; /incident resolve triggers the post-mortem creation flow. The pattern is runbook-as-command — every step the runbook documents is also runnable as a slash command, so the IC's eyes never leave the channel. PaisaBridge-shape platforms with mature chat-ops report the IC's mean number of context switches (Slack ↔ Grafana ↔ Tempo ↔ kubectl) drops from ~14 in the first 10 minutes to ~3, which directly reduces cognitive load and time-to-mitigate.

Statuspage automation and the public-comms ladder

Public statuspages (Atlassian Statuspage, Instatus, custom) are the externally-visible side of incident-response tooling. The default failure mode: the Comms Lead writes a status update by hand at minute 12 of the incident, posts it, and forgets to post the next one until customer-support escalates at minute 45. The corrective is a tiered comms ladder — the bot drafts every update on a schedule (initial within 15 min, then every 30 min, then every 60 min once mitigated), the Comms Lead approves with a single emoji reaction, the bot posts. Skipping the schedule requires an explicit /comms pause with a reason logged to the timeline. The discipline is not "remember to update"; the discipline is "explain why you are pausing the auto-update". Razorpay-shape teams running this pattern report customer complaints about "we didn't know what was going on" drop by ~70% after introducing scheduled comms.

Tooling for blameless post-mortem follow-through

The /incident bot's job extends past resolution: when the IC types /incident resolve, the bot creates the post-mortem document pre-populated with the timeline, the severity, the IC, the paged team, the duration, and a list of the action-item placeholders. The Action Item tracker (a separate Jira project tagged incident-followup) gets a weekly automated sweep that surfaces overdue AIs in the platform-team's Slack. The tooling for follow-through is what separates teams that learn from incidents from teams that hold post-mortems and forget the AIs by the next sprint. See /wiki/playbooks-post-mortems-and-blameless-culture for the cultural rules; the tooling here is the enforcement layer for those rules.

Reproduce this on your laptop

python3 -m venv .venv && source .venv/bin/activate
pip install slack-bolt slack-sdk pdpyras fastapi uvicorn requests pyjwt pydantic
# Configure Slack app at api.slack.com (slash command -> /incident, scopes: channels:manage,
#   chat:write, conversations.connect:manage), then export SLACK_BOT_TOKEN and SLACK_SIGNING_SECRET.
# Get a PagerDuty REST API key with incidents.write scope; export PAGERDUTY_API_KEY.
# Run the bot:
uvicorn incident_bot:api --host 0.0.0.0 --port 8000
# Expose with ngrok and point Slack's slash-command URL to https://<ngrok>/slack/events
# Test in a Slack workspace: /incident declare sev3 checkout

A working /incident declare end-to-end (Slack → Python bot → PagerDuty test incident → war-room channel created) takes roughly an afternoon to build for a single team. The full version with timeline bot + chat-ops + statuspage automation is a 1–2 week project for a platform engineer, and pays back its cost in the first SEV-2 of the next quarter.

Where this leads next

/wiki/playbooks-post-mortems-and-blameless-culture covers the cultural and ritual layer that the tooling in this article enables — the post-mortem template, the blameless lint, the action-item discipline. The tooling automates the procedural work of running the room; the cultural rules govern what happens after.

/wiki/reducing-on-call-pain covers the pre-incident half of the same problem: alert design, on-call rotation health, paging hygiene. Incident-response tooling shaves minutes off mitigation; on-call pain reduction shaves incidents off the calendar entirely. Both are required; neither is sufficient.

/wiki/the-observability-maturity-model places incident-response tooling on the maturity scale — "team has a /incident bot", "team has automated timeline capture", "team has statuspage scheduled comms" are concrete checkpoints. Most platform teams reach the first two within their second year and the third within their third.

References

• PagerDuty Incident Response documentation — the canonical public reference for IC role responsibilities, severity definitions, war-room patterns. The severity rubric in this article is structurally similar.
• Atlassian Incident Management Handbook — pairs well as a counterweight; the comms-ladder pattern is articulated more crisply here.
• Google SRE Workbook, "Incident Response" (Chapter 9) — the IC role, the role separation between IC, Operations Lead, and Comms Lead, and the war-room rituals all draw from this chapter.
• Slack API — Slash Commands and slack-bolt Python SDK — the load-bearing primitives for the /incident bot in this article.
• PagerDuty REST API — Incidents — for the programmatic paging layer; pdpyras is the Python wrapper.
• Charity Majors, "WTF is SRE? — On-Call" and the broader Honeycomb on-call writeups — sharp practitioner take on what tooling actually changes versus what it merely papers over.
• Niall Murphy et al., Site Reliability Engineering (Google / O'Reilly, 2016), Chapter 14 "Managing Incidents" — the original published statement of the IC / OL / Comms role separation that frames this article.
• /wiki/playbooks-post-mortems-and-blameless-culture — internal: the cultural rules that the tooling here enforces; without those rules, the tooling becomes elaborate scaffolding around chaos.