---
id: hatch3r-incident-response
description: Handle production incidents with structured triage, mitigation, and post-mortem. Use when responding to production issues, outages, or security incidents.
tags: [devops]
quality_charter: agents/shared/quality-charter.md
efficiency_patterns: agents/shared/efficiency-patterns.md
cache_friendly: true
---
# Incident Response Workflow

## Quick Start

```
Task Progress:
- [ ] Step 0: Detect ambiguity (P8 B1)
- [ ] Step 1: Classify severity (P0-P3) based on impact
- [ ] Step 2: Triage — identify affected systems, user impact, blast radius
- [ ] Step 3: Mitigate — apply hotfix or rollback, verify mitigation works
- [ ] Step 4: Root cause analysis — trace the failure chain
- [ ] Step 5: Write post-mortem with timeline, root cause, action items
- [ ] Step 6: Create follow-up issues for permanent fixes and preventive measures
```

## Step 0 — Detect Ambiguity (P8 B1)

Before any work, scan the invocation for unresolved questions in scope, intent, acceptance criteria, target environment, or irreversibility. If any are found, ask the user via the platform-native question tool per `agents/shared/user-question-protocol.md`. Do not proceed under silent assumption. Default path, not an exception. Triggers for THIS skill: user-facing impact vs internal-only, blast radius known (single tenant vs all users), rollback safety verified, stakeholder notification scope (engineering vs exec vs public), and whether mitigation requires data write (irreversible) vs config flip (reversible).

## Fan-out Discipline (P8 B2)

This skill delegates per task size:
- Tier 1 (trivial single-file): inline execution acceptable.
- Tier 2 (multi-file or multi-concern): spawn parallel sub-agents per concern via the Task tool.
- Tier 3 (multi-module / high-risk): one fresh sub-agent per independent module or gate; orchestrator integrates only.

Never under-fan-out to save tokens. Token cost is dominated by quality and completeness gains. Emit `sub_agents_spawned: { count, rationale }` in your output.

## Step 1: Classify Severity

| Severity | Definition                                  | Examples                                     |
| -------- | ------------------------------------------- | -------------------------------------------- |
| P0       | Complete outage, data loss, security breach | App unusable, auth down, data exposed        |
| P1       | Major degradation, significant user impact  | Sync failing, billing broken, >1% error rate |
| P2       | Partial degradation, limited impact         | Single flow broken, slow performance       |
| P3       | Minor issue, workaround available            | Cosmetic bug, edge case                     |

- Check for related issues or prior incidents using the platform tools (check `platform` in `.hatch3r/hatch.json`):
  - **GitHub:** Use **GitHub MCP** (`issue_read`, `search_issues`) or `gh issue list --search "..."`
  - **Azure DevOps:** `az boards query --wiql "SELECT [System.Id] FROM WorkItems WHERE [System.Title] CONTAINS '...'"` or `az boards work-item show --id N`
  - **GitLab:** `glab issue list --search "..."` or `glab issue view N`
- For external library docs and current best practices, follow the project's tooling hierarchy.

## Step 2: Triage

- **Affected systems:** Frontend, backend, database, auth, payment, third-party services?
- **User impact:** How many users? Which flows? Which plans (free/paid)?
- **Blast radius:** Is the issue contained or spreading?
- **Data:** Any data corruption, loss, or exposure? Check project privacy/security specs for implications.
- **Timeline:** When did it start? Any recent deploys, config changes, or dependency updates?

## Step 3: Mitigate

- **Immediate actions:** Rollback last deploy, disable feature flag, revert config, scale up, or apply hotfix.
- **Verification:** Confirm mitigation works — error rate drops, affected flow recovers.
- **Communication:** Notify stakeholders if P0/P1. Document status in incident channel or issue.
- Do not spend time on perfect fixes during active incident — stabilize first.

## Step 4: Root Cause Analysis

- Trace the failure chain: what changed, what failed, why.
- Review logs (correlationId, userId), metrics, deploy history.
- Check ADRs in project docs for architectural context.
- For external library docs and current best practices, follow the project's tooling hierarchy.

## Step 5: Post-Mortem

Write a structured post-mortem document:

- **Summary:** One-paragraph description of the incident.
- **Timeline:** Key events (detection, mitigation, resolution) with timestamps.
- **Root cause:** What went wrong and why.
- **Impact:** Users affected, duration, business impact.
- **Action items:** Permanent fixes, preventive measures, process improvements.
- **Lessons learned:** What we'll do differently.

Store in project incident docs or as an issue/wiki page on the platform. Follow project conventions.

## Step 6: Follow-Up Issues

- Create follow-up issues/work items for each action item from the post-mortem (check `platform` in `.hatch3r/hatch.json`):
  - **GitHub:** `gh issue create --title "..." --body "..." --label "incident-follow-up"` (or use **GitHub MCP** `issue_create`)
  - **Azure DevOps:** `az boards work-item create --type "Bug" --title "..." --description "..." --fields "System.Tags=incident-follow-up"`
  - **GitLab:** `glab issue create --title "..." --description "..." --label "incident-follow-up"`
- Label appropriately (e.g., `incident-follow-up`, `P0`, `P1`).
- Link issues/work items to the post-mortem and to each other.
- Assign owners and due dates for critical fixes.

## Error Handling

- **Cannot reproduce the incident locally**: Use production logs and traces to build the timeline. If local reproduction is blocked by environment differences, document the gap and recommend a staging environment test.
- **Mitigation introduces new issues**: Roll back the mitigation immediately, reassess the approach, and apply a more targeted fix. Document both the original incident and the mitigation regression in the post-mortem.
- **Root cause spans multiple services or teams**: Document the cross-service dependency chain, assign follow-up items to the responsible teams, and coordinate a joint post-mortem.

## Definition of Done

- [ ] Incident mitigated and verified
- [ ] Post-mortem written with timeline, root cause, and action items
- [ ] Follow-up issues created for permanent fixes and preventive measures
- [ ] Stakeholders notified (if P0/P1)
- [ ] No sensitive data (secrets, PII, code content) in post-mortem or logs

## Additional Resources

- Privacy/security specs: project documentation
- Observability: project logging and correlation conventions
- Error handling: project error handling patterns
