---
description: Structured post-mortem template for incidents, outages, or failed investigations. Guides teams through root cause analysis, timeline reconstruction, and actionable follow-up items.
use_case: Use after production incidents, failed deployments, or any investigation that warrants a formal retrospective with timeline and action items.
patterns_used:
  - section: "Classification"
    pattern: "Pattern 1: Section Header (Overview & Context)"
  - section: "Timeline"
    pattern: "Pattern 4: Process Workflow"
  - section: "Action Items"
    pattern: "Pattern 5: Closeout & Follow-up"
---

# Post-Mortem: [Title of Incident or Investigation]

## 1. Classification

| Field | Value |
| :--- | :--- |
| **Post-Mortem Type** | [e.g., Production Incident, Process Inefficiency, Wasted Effort, Configuration Issue] |
| **Severity** | [P0, P1, P2, P3, N/A] |
| **Time Investment** | [e.g., 4 days, 2 weeks, 3 hours] |
| **Impact** | [e.g., User-Facing, Development Blocked, Team Morale, Resource Waste] |
| **Issue Status** | [Resolved, Abandoned, Mitigated] |
| **Date Resolved** | YYYY-MM-DD |

---

## 2. Executive Summary

[2-3 sentence summary of what happened, what the impact was, and how it was resolved.]

### Key Finding

[A plain-English description of the single most important discovery or lesson. (e.g., "The root cause was underestimating authentication complexity; the token's 15-minute expiry was the real blocker.") ]

---

## 3. Timeline of Events

[A chronological narrative of the investigation, focused on the "false summits." This tells the *story* of the problem.]

### [Month Day]: Initial Planning (False Summit #1)
* **What We Thought:** ["We'll just install the Terraform provider and import existing resources. Easy."]
* **Reality Check:** [Discovered the OAuth credentials generated by the UI lacked resource management permissions.]
* **Time Lost:** [e.g., 4 hours debugging 403 errors.]

---

### [Month Day]: [Activity] (False Summit #2)
* **What We Thought:** ["Okay, the Bearer Token works! `curl` succeeds. We are unblocked."]
* **Reality Check:** [The Bearer Token expired every 15 minutes. The process failed again the next morning.]
* **Time Lost:** [e.g., 2 hours re-debugging a "401 Unauthorized" error.]

---

### [Month Day]: [Activity] (False Summit #3)
* **What We Thought:** [e.g., "The SSH tunnel is up and stable."]
* **Reality Check:** [e.g., "The IAP-based tunnel dies after 10 minutes of inactivity, breaking the connection."]
* **Time Lost:** [e.g., "3 hours debugging 'Connection refused' errors."]

---

### [Month Day]: Final Resolution
* **What We Did:**
    1.  [Resolution step 1, e.g., "Scripted the bearer token refresh process."]
    2.  [Resolution step 2, e.g., "Added a health check to restart the SSH tunnel."]
    3.  [Resolution step 3, e.g., "Mapped the Google ADC credential path for WSL."]
* **Success:** [Description of the final, working state.]

---

## 4. Quantified Impact

[Use this to make the pain of the "false summits" clear and justify the prevention work.]

| False Summit | Estimated Time | Actual Time | Ratio |
| :--- | :--- | :--- | :--- |
| #1: [Title] | [hours/min/sec] | [hours/min/sec] | [N]x |
| #2: [Title] | [hours/min/sec] | [hours/min/sec] | [N]x |
| #3: [Title] | [hours/min/sec] | [hours/min/sec] | [N]x |
| **TOTAL** | **[N hours/min/sec]** | **[N hours/min/sec]** | **[N]x** |

---

## 5. Root Cause Analysis

### Primary Root Cause: [Summary of the core issue]

[Explain the fundamental issue. (e.g., "The fundamental issue was **underestimating the authentication architecture's complexity**. We were debugging 5 independent auth layers as if they were one problem.") ]

* **[Component/Layer 1]:** [e.g., "Airbyte Authentication (OAuth vs. Bearer)"]
* **[Component/Layer 2]:** [e.g., "SSH Tunnel (IAP-based, dies on inactivity)"]
* **[Component/Layer 3]:** [e.g., "Google ADC (for Secret Manager)"]

### Contributing Factors

[List other factors that made the problem worse or harder to solve.]

1.  **[Factor 1]:** [e.g., "No integration tests for the end-to-end workflow."]
2.  **[Factor 2]:** [e.g., "Documentation was written before the workflow was validated."]
3.  **[Factor 3]:** [e.g., "Provider immaturity (v0.6.5) led to poor error messages."]

### Analysis of "False Summits"

**Pattern:** [Describe the *pattern* of failure. (e.g., "Each solution appeared complete in isolation but failed when integrated with the next component. Each component worked individually, but integration failures emerged progressively.") ]

* (e.g. Bearer token **authentication worked** (curl succeeded)...)
* (e.g. But bearer token **expiry wasn't tested** (15 min TTL).)
* (e.g. SSH tunnel **connectivity worked** (at setup time)...)
* (e.g. But tunnel **stability wasn't monitored** (dies on inactivity).)

---

## 6. Mistakes Catalog & Analysis

[A "blameless" catalog of mistaken assumptions. This is often the most valuable section for learning.]

### Mistake #1: [Brief summary of the mistaken assumption]
* **What we believed:** [e.g., "OAuth credentials from the UI would have full permissions."]
* **What was actually true:** [e.g., "They lack resource management scopes."]
* **Why the confusion:** [e.g., "The documentation doesn't explain permission scopes."]
* **How discovered:** [e.g., "HTTP 403 Forbidden errors on all API operations."]
* **Prevention:** [e.g., "Test actual resource operations immediately after auth setup, not just connectivity."]

### Mistake #2: [Brief summary of the mistaken assumption]
* **What we believed:** [e.g., "Bearer tokens are like API keys (long-lived)."]
* **What was actually true:** [e.g., "These tokens expire in 15 minutes."]
* **Why the confusion:** [e.g., "The UI/docs don't show an expiry time."]
* **How discovered:** [e.g., "401 Unauthorized" on day 2 of work.]
* **Prevention:** [e.g., "Always test token/session expiry or build refresh automation immediately."]

### Mistake #3: [Brief summary of the mistaken assumption]
* **What we believed:** [e.g., "Cron expressions are universal (5 parts)."]
* **What was actually true:** [e.g., "This tool uses Quartz Scheduler (6-7 parts)."]
* **Why the confusion:** [e.g., "Most systems use Unix cron."]
* **How discovered:** [e.g., "'Cron expression contains 5 parts but we expect...' error."]
* **Prevention:** [e.g., "Check cron format requirements in the tool's specific docs first."]

---

## 7. Lessons Learned

### What Made This Hard?

[List the specific challenges or "gotchas" encountered.]

1.  **[Challenge 1]:** [e.g., "Multiple Interdependent Authentication Layers"]
2.  **[Challenge 2]:** [e.g., "Misleading Error Messages"]
3.  **[Challenge 3]:** [e.g., "Implicit 15-minute Token Expiry"]

### What Worked Well?

[What tools, processes, or people were crucial to solving this?]

1.  **[Success 1]:** [e.g., "Investment in generic, reusable scripts (`refresh-token.sh`)"]
2.  **[Success 2]:** [e.g., "Systematic debugging and note-taking in the incident channel"]
3.  **[Success 3]:** [e.g., "Version-controlling the connector YAML manifests"]

### What Made This Take So Long?

[Be brutally honest about process failures.]

* **Did we follow a systematic process?** [e.g., "No, we jumped between 3 different auth layers without isolating variables."]
* **Were we going in circles?** [e.g., "Yes, we re-diagnosed the token expiry issue twice."]
* **What assumptions blocked us?** [e.g., "We assumed the SSH tunnel was stable, which cost us 3 hours."]
* **What should have been obvious?** [e.g., "The 401 error on day 2 should have immediately pointed to token expiry, not a different auth problem."]

---

## 8. Documentation & Prevention

[All the permanent and canonical updates to documentation to ensure this never happens again ]

### Canonical Documentation Changes

| Document | Type | Location | Status | Permanent Fix |
|----------|------|----------|--------|---------------|
| [title] | [runbook/faq/script/ADR/doc] | [path] | [new/updated/removed] | [detailed description of what was changed to ensure this never happens again] |
| [title] | [runbook/faq/script/ADR/doc] | [path] | [new/updated/removed] | [detailed description of what was changed to ensure this never happens again] |
| [title] | [runbook/faq/script/ADR/doc] | [path] | [new/updated/removed] | [detailed description of what was changed to ensure this never happens again] |

### Testing Changes
| Test | Type | Location | Status | Permanent Fix | Date Passed |
|------|------|----------|--------|---------------|-------------|
| [title] | [unit/integration/e2e/other] | [new/updated/removed] | [detailed description of what was changed to ensure this never happens again] | [datetime] |
| [title] | [unit/integration/e2e/other] | [new/updated/removed] | [detailed description of what was changed to ensure this never happens again] | [datetime] |
| [title] | [unit/integration/e2e/other] | [new/updated/removed] | [detailed description of what was changed to ensure this never happens again] | [datetime] |

### IaC Changes
| Infrastructure | Document | Location | Status | Permanent Fix | Passing Tests |
|----------------|----------|----------|--------|---------------|---------------|
| [provisioning/configuration/templating] | [title]  | [path] | [new/updated/removed] |  [detailed description of what was changed to ensure this never happens again] | [Tests that prove the IaC is fixed] |
| [provisioning/configuration/templating] | [title]  | [path] | [new/updated/removed] |  [detailed description of what was changed to ensure this never happens again] | [Tests that prove the IaC is fixed] |
| [provisioning/configuration/templating] | [title]  | [path] | [new/updated/removed] |  [detailed description of what was changed to ensure this never happens again] | [Tests that prove the IaC is fixed] |


#### Runbook update checklist (optional)
* [ ] **Root cause documented with link** (optional)
* [ ] **Troubleshooting steps added** (optional)
* [ ] **Quick diagnosis checklist added**  (optional)

#### Configuration Documentation (optional)
* [ ] **Ambiguous options clarified** (optional)
* [ ] **Example values provided:** (optional)

#### Architecture/Design Docs (optional)
* [ ] **Failure modes added** (optional)

---

## 9. Related Work & Artifacts

### Related Cards

| Card ID | Card Name | Card Type | Brief Description |
|---------|-----------|-----------|-------------------|
| [ID] | [Name] | chore/bug/feature/docs | [What this card was about] |
| [ID] | [Name] | chore/bug/feature/docs | [What this card was about] |

### Artifacts Created
* [Link to script]: (eg. `scripts/refresh-airbyte-token.sh`)
* [Link to script]: (eg. `scripts/publish-connector-via-api.sh`)
* [Link to doc]: (eg. `docs/airbyte-iac-workflow.md`)

### Code Changes
* [Link to PR/Commit]: (eg. `ad5eb7d` - Deploy ShowMojo connector (SUCCESS))
* [Link to PR/Commit]: (eg. `bbe825f` - Fix API authentication (bearer token discovery))

---

## 10. Meta: Template Usage & Process Connection

### When to Use This Template

Use this template AFTER resolving an issue to:
- Reflect on investigations that took way longer than they should have
- Document root cause analysis for production incidents
- Share knowledge about non-obvious solutions
- Identify process failures that allowed inefficiency
- Create searchable reference for similar future problems
- **Learn whether following `chore-troubleshooting.md` would have prevented wasted time**

Do NOT use this template during active investigation - use `chore-troubleshooting.md` instead.

### The Connection Between Templates

**During investigation:** Use `chore-troubleshooting.md` to systematically track attempts, hypotheses, and outcomes in real-time. This prevents going in circles and helps identify when to pivot.

**After resolution:** Use this template to reflect on whether you FOLLOWED that systematic process, and what the cost of NOT following it was. The goal is to learn from inefficiency and drive adoption of systematic troubleshooting for future work.

**Key question:** If you had documented your "false summits" systematically in real-time (using the troubleshooting template), would you have caught the pattern earlier and saved days of work?
