# DD-008: DLQ Drain During Graceful Shutdown

**Status**: ✅ Implemented (V1.0)
**Date**: 2025-12-21
**Deciders**: DataStorage Team
**Supersedes**: N/A
**Related**: DD-007 (Kubernetes-Aware Graceful Shutdown)

---

## 📋 Context and Problem Statement

DataStorage service uses a Dead Letter Queue (DLQ) to store audit messages that failed to write to PostgreSQL. During graceful shutdown (DD-007), these DLQ messages could be lost if the service terminates without processing them.

**Key Requirements**:
- BR-AUDIT-001: Complete audit trail with no data loss
- DD-007: Kubernetes-aware graceful shutdown
- Audit messages in DLQ must be persisted before shutdown

### Problem Scenarios

1. **Kubernetes Pod Termination**: Service receives SIGTERM signal
2. **DLQ Contains Messages**: Audit events that failed primary database write
3. **Risk**: Without DLQ drain, messages are lost when service terminates

---

## 🔍 Alternatives Considered

### Alternative 1: No DLQ Drain (Original DD-007)
**Approach**: Continue with DD-007's 4-step graceful shutdown without DLQ drain

**Pros**:
- ✅ Simpler implementation
- ✅ Faster shutdown (no DLQ processing time)
- ✅ Existing DD-007 already production-ready

**Cons**:
- ❌ **Data Loss**: DLQ messages lost during shutdown
- ❌ Violates BR-AUDIT-001 (complete audit trail)
- ❌ Defeats purpose of DLQ (error recovery)

**Confidence**: 40% (rejected - violates audit completeness)

---

### Alternative 2: Infinite DLQ Drain (Wait Until Empty)
**Approach**: Process all DLQ messages regardless of how long it takes

**Pros**:
- ✅ Guaranteed no DLQ message loss
- ✅ Complete audit trail preservation

**Cons**:
- ❌ **Kubernetes Forced Termination**: K8s will kill pod after `terminationGracePeriodSeconds`
- ❌ Unpredictable shutdown time
- ❌ May block pod termination indefinitely if DLQ continuously fills

**Confidence**: 30% (rejected - not practical for Kubernetes)

---

### Alternative 3: DLQ Drain with Timeout ✅ APPROVED
**Approach**: Process DLQ messages with a maximum time budget (10 seconds)

**Sequence**:
1. Complete in-flight HTTP connections (DD-007 Step 3)
2. **Drain DLQ with timeout** (DD-008 - NEW)
3. Close resources (DD-007 Step 4, now Step 5)

**Pros**:
- ✅ **Best Effort Data Preservation**: Most DLQ messages persisted
- ✅ **Predictable Shutdown**: Maximum 10s delay
- ✅ **Kubernetes Compliant**: Fits within `terminationGracePeriodSeconds`
- ✅ **Graceful Degradation**: If timeout, at least some messages saved

**Cons**:
- ⚠️ **Partial Drain Possible**: If DLQ drain times out, remaining messages stay in Redis
  - **Mitigation**: 10 seconds is sufficient for typical DLQ depth
  - **Note**: Messages are NOT lost — they remain in Redis and will be retried on next startup

**Confidence**: 95% (approved - pragmatic balance)

---

## 🎯 Decision

**APPROVED: Alternative 3** - DLQ Drain with Timeout

**Rationale**:
1. **Kubernetes Compliance**: 10s timeout fits within `terminationGracePeriodSeconds` (90s)
2. **Best Effort Preservation**: Significantly reduces data loss compared to no drain
3. **Operational Reality**: DLQ depth is typically low (monitored via metrics)
4. **Graceful Degradation**: Partial drain is better than no drain

**Key Insight**: Perfect data preservation (Alternative 2) is impossible in Kubernetes without risking forced termination. A time-bounded best-effort approach (Alternative 3) provides practical audit completeness.

---

## 💻 Implementation

### Graceful Shutdown Sequence (DD-007 + DD-008)

```
STEP 1: Set shutdown flag              (DD-007)
  ↓ 5s delay for K8s endpoint propagation
STEP 2: Wait for endpoint removal       (DD-007)
  ↓
STEP 3: Drain HTTP connections (30s max) (DD-007)
  ↓
STEP 4: Drain DLQ (10s max)              (DD-008) ← NEW
  ├─ Process notification DLQ
  ├─ Process events DLQ
  └─ Write messages to database
  ↓
STEP 5: Close resources                  (DD-007)
  ├─ Flush audit store
  └─ Close PostgreSQL
  ↓
✅ SHUTDOWN COMPLETE
```

### Primary Implementation Files

**DLQ Client Enhancement**:
- `pkg/datastorage/dlq/client.go`:
  - `DrainWithTimeout(ctx, notificationRepo, eventsRepo)` - Main drain method
  - `drainStream(ctx, auditType, repo)` - Process single stream
  - `DrainStats` - Statistics tracking

**Server Graceful Shutdown Enhancement**:
- `pkg/datastorage/server/server.go`:
  - `Shutdown(ctx)` - Updated to call new step 4
  - `shutdownStep4DrainDLQ(ctx)` - NEW step for DLQ drain
  - `shutdownStep5CloseResources()` - Renamed from step 4

**Testing**:
- `test/unit/datastorage/dlq/drain_test.go` - 5 unit tests (100% passing)

### Data Flow

```go
// 1. Server initiates shutdown with timeout
ctx, cancel := context.WithTimeout(context.Background(), dlqDrainTimeout) // 10s
defer cancel()

// 2. DLQ client drains both streams
stats, err := s.dlqClient.DrainWithTimeout(ctx, s.repository, s.auditEventsRepo)

// 3. For each stream (notifications, events):
//    Two-pass cursor-based iteration (ARCH-F1 fix):
//    Pass 1: forward sweep, XDel only after successful write
//    Pass 2 (if any writes failed): retry from "-" for remaining messages
for each message in stream {
    // Check timeout before processing
    if ctx.Done() { return processed, nil }

    // Parse message
    dlqMsg := parseStreamMessage(msg)

    // Write to database
    if err := writeMessageToDB(ctx, auditType, dlqMsg, repo); err != nil {
        // Message stays in Redis — NOT deleted (DF-3 fix)
        continue
    }

    // Remove from DLQ only on success
    XDel(ctx, streamKey, msg.ID)

    processed++
}

// 4. Return stats
return DrainStats{
    NotificationsProcessed: X,
    EventsProcessed: Y,
    TotalProcessed: X+Y,
    TimedOut: bool,
    Duration: time.Duration,
    Errors: []error,
}
```

### Graceful Degradation

**If DLQ drain times out**:
- ✅ Already-processed messages are persisted
- ✅ Shutdown continues (timeout is non-fatal)
- ✅ Remaining DLQ messages stay in Redis for the next startup's retry worker (#1048 DF-3)
- 📊 Drain statistics logged for monitoring

**Metrics** (via Prometheus, see `pkg/datastorage/metrics/metrics.go`):
- `datastorage_dlq_drain_batch_total` - Total DLQ drain batch operations during shutdown
- `datastorage_shutdown_dlq_drain_errors_total` - Total DLQ drain errors during shutdown

---

## ✅ Consequences

### Positive Consequences

1. ✅ **Reduced Data Loss**: DLQ messages persisted before shutdown
2. ✅ **Audit Completeness**: BR-AUDIT-001 compliance improved
3. ✅ **Kubernetes Compatible**: Predictable shutdown within grace period
4. ✅ **Monitoring**: Drain statistics provide operational visibility
5. ✅ **Graceful Degradation**: Partial drain better than none

### Negative Consequences

1. ⚠️ **Shutdown Delay**: Additional 10s maximum
   - **Mitigation**: Acceptable for audit completeness, configurable timeout
2. ⚠️ **Partial Drain Possible**: If timeout, remaining messages stay in Redis
   - **Mitigation**: Typical DLQ depth low (monitored), 10s sufficient for most cases
   - **Note**: Messages are NOT lost — they remain in Redis for the next startup's retry worker (#1048 DF-3)
3. ⚠️ **Complexity**: Additional shutdown step
   - **Mitigation**: Well-tested (5/5 tests passing), clear logging

### Neutral

- 🔄 **DLQ Depth Matters**: Drain effectiveness depends on DLQ message count at shutdown time
- 🔄 **Database Performance**: Drain speed depends on PostgreSQL write performance

---

## 🧪 Validation Results

### Unit Tests (100% Passing)

**File**: `test/unit/datastorage/dlq/drain_test.go`

| Test | Purpose | Status |
|------|---------|--------|
| `should drain notification DLQ messages successfully` | Basic notification drain | ✅ PASS |
| `should drain event DLQ messages successfully` | Basic event drain | ✅ PASS |
| `should drain both notification and event DLQ messages` | Mixed stream drain | ✅ PASS |
| `should handle timeout during drain gracefully` | Timeout handling | ✅ PASS |
| `should handle empty DLQ gracefully` | Empty DLQ edge case | ✅ PASS |

**Coverage**: 100% of `DrainWithTimeout` code paths tested

### Confidence Assessment Progression

- Initial assessment: 85% confidence (concept validated)
- After implementation: 92% confidence (tests passing)
- After production testing: 95% confidence (timeout tuning validated)

### Key Validation Points

- ✅ DLQ messages successfully written to database
- ✅ Timeout handled gracefully (no errors)
- ✅ Empty DLQ handled correctly
- ✅ Both notification and event streams processed
- ✅ Drain statistics accurately tracked

---

## 📊 Configuration

### Timeout Settings

```go
// pkg/datastorage/server/server.go
const (
    endpointRemovalPropagationDelay = 5 * time.Second  // DD-007
    drainTimeout = 30 * time.Second                    // DD-007
    dlqDrainTimeout = 10 * time.Second                 // DD-008 ← NEW
)
```

**Total Shutdown Time Budget**:
- Endpoint propagation: 5s (DD-007 Step 2)
- HTTP drain: up to 30s (DD-007 Step 3)
- **DLQ drain: up to 10s** (DD-008 Step 4) ← NEW
- Resource close: ~1s (DD-007 Step 5)
- **Total Max**: ~46s (within `terminationGracePeriodSeconds: 90s` per deployment.yaml)

---

## 📚 Related Decisions

**Builds On**:
- **DD-007**: Kubernetes-Aware Graceful Shutdown (4-step pattern)
- **DD-009**: Dead Letter Queue Pattern

**Supports**:
- **BR-AUDIT-001**: Complete Audit Trail with no data loss
- **BR-STORAGE-017**: DLQ Fallback on Database Unavailability

---

## 🔄 Review & Evolution

### When to Revisit

- If DLQ capacity increases significantly (>1000 messages typical)
- If shutdown timeouts become frequent (monitored via Prometheus)
- If Kubernetes `terminationGracePeriodSeconds` changes
- If database write performance degrades

### Success Metrics

- **DLQ Drain Completion Rate**: Target >95% complete within timeout
- **Data Loss Rate**: Target <1% of DLQ messages lost during shutdown
- **Shutdown Duration**: Target <45s average (including DLQ drain)
- **Timeout Frequency**: Target <5% of shutdowns experience DLQ drain timeout

---

## 🎯 Implementation Status

**Version**: V1.1
**Status**: ✅ Implemented and Tested
**Date**: 2025-12-21
**Last Reviewed**: 2026-05-14

**Changes Made**:
1. ✅ Added `DrainWithTimeout` method to DLQ client
2. ✅ Enhanced graceful shutdown with Step 4 (DLQ drain)
3. ✅ Created 5 unit tests (100% passing)
4. ✅ Updated server shutdown constants
5. ✅ Added logging and statistics tracking

**Deployment**: Ready for V1.0 release

### Implementation Status (v1.5)

V1.5 wiring surfaces **DLQ drain failures** upward through `Server.Shutdown` (aggregated alongside HTTP drain/`sql.DB` close errors via `errors.Join`) while retaining **non-blocking cleanup** semantics so Postgres still closes after a failed bounded drain (#1048). **Capacity monitoring / near-cap gauges** (`dlq/monitoring.go`, `dlq/metrics.go`, `metrics.DLQ*`) expose depth pressure so draining alone is no longer the only DLQ lifecycle signal Helm/Prometheus observes.

---

**Document Status**: ✅ Authoritative
**Approved By**: DataStorage Team
**Implementation Required**: ✅ Complete
**Next Review**: 2026-03-21 (or upon significant DLQ depth changes)

