# DD-007: Kubernetes-Aware Graceful Shutdown Pattern

## Status
**✅ Approved** (2025-10-31)  
**Last Reviewed**: 2025-10-31  
**Confidence**: 95%

---

## Context & Problem

**Problem**: Standard Go `http.Server.Shutdown()` and Kubernetes controller-runtime shutdown don't coordinate with Kubernetes pod lifecycle, leading to:
- **Request failures during rolling updates** (5-10% error rate)
- **Aborted reconciliations** in CRD controllers
- **Resource leaks** (unclosed database/cache connections)
- **Data corruption** from interrupted writes

**Why This Matters**:
- Kubernetes takes 1-5 seconds to propagate endpoint removals
- During this window, new requests/reconciliations start on terminating pods
- Standard shutdown immediately stops accepting connections, causing connection refused errors
- Controllers may abort critical reconciliations mid-operation

**Key Requirements**:
1. **Zero request failures** during rolling updates and pod terminations
2. **Complete ongoing work** before shutdown (HTTP requests, reconciliations, database transactions)
3. **Clean resource cleanup** (connections, file handles, goroutines)
4. **Kubernetes-native behavior** (coordinated with readiness/liveness probes)
5. **Timeout protection** (prevent infinite hangs during shutdown)

**Scope**: Applies to:
- HTTP/REST API services (Gateway, Context API, HolmesGPT API, Data Storage, Notification)
- CRD Controllers (RemediationProcessing, AIAnalysis, RemediationOrchestrator, ActionExecution, WorkflowExecutor)
- Any service that handles stateful operations in Kubernetes

---

## Alternatives Considered

### Alternative 1: Standard Go Shutdown (Default Approach)

**Approach**: Use `http.Server.Shutdown(ctx)` directly without Kubernetes coordination

```go
func (s *Server) Shutdown(ctx context.Context) error {
    return s.httpServer.Shutdown(ctx)
}
```

**Pros**:
- ✅ Simple implementation (1 line of code)
- ✅ Standard Go pattern
- ✅ Completes in-flight requests

**Cons**:
- ❌ No Kubernetes endpoint coordination (5-10% request failures)
- ❌ New connections arrive during shutdown window
- ❌ No resource cleanup
- ❌ Not production-ready for zero-downtime deployments

**Confidence**: 40% (rejected - unsuitable for production)

---

### Alternative 2: Sleep-Based Delay (Naive Approach)

**Approach**: Add fixed 5-second sleep before shutdown

```go
func (s *Server) Shutdown(ctx context.Context) error {
    time.Sleep(5 * time.Second)  // Wait for endpoint removal
    return s.httpServer.Shutdown(ctx)
}
```

**Pros**:
- ✅ Reduces request failures (partial fix)
- ✅ Simple to implement

**Cons**:
- ❌ No readiness probe coordination (Kubernetes still routes traffic)
- ❌ Fixed delay wastes time (not adaptive)
- ❌ No resource cleanup
- ❌ Race condition (new requests still arrive if endpoints not updated)

**Confidence**: 50% (rejected - incomplete solution)

---

### Alternative 3: Kubernetes-Aware 4-Step Shutdown (Production Pattern)

**Approach**: Coordinate shutdown with Kubernetes pod lifecycle using readiness probe

```go
func (s *Server) Shutdown(ctx context.Context) error {
    // STEP 1: Set shutdown flag (readiness probe returns 503)
    s.isShuttingDown.Store(true)
    s.logger.Info("Shutdown flag set, readiness probe returning 503")
    
    // STEP 2: Wait for Kubernetes endpoint removal propagation
    // Kubernetes typically takes 1-3 seconds to update endpoints
    s.logger.Info("Waiting 5 seconds for endpoint removal propagation")
    time.Sleep(5 * time.Second)
    
    // STEP 3: Drain in-flight connections
    if err := s.httpServer.Shutdown(ctx); err != nil {
        return fmt.Errorf("HTTP shutdown failed: %w", err)
    }
    
    // STEP 4: Close resources
    if err := s.dbClient.Close(); err != nil {
        s.logger.Error("Failed to close database", zap.Error(err))
    }
    if err := s.cacheManager.Close(); err != nil {
        s.logger.Error("Failed to close cache", zap.Error(err))
    }
    
    return nil
}

func (s *Server) handleReadiness(w http.ResponseWriter, r *http.Request) {
    // Check shutdown flag FIRST
    if s.isShuttingDown.Load() {
        w.WriteHeader(503)
        return
    }
    // ... normal health checks ...
}
```

**Pros**:
- ✅ **Zero request failures** during rolling updates
- ✅ **Kubernetes-native coordination** via readiness probe
- ✅ **Complete in-flight work** within timeout
- ✅ **Clean resource cleanup**
- ✅ **Production-proven** (industry best practice)
- ✅ **Timeout protection** (30-second context deadline)

**Cons**:
- ⚠️ More complex implementation (50 lines vs 1 line)
- ⚠️ Requires readiness probe configuration
- ⚠️ Fixed 5-second delay (industry standard, not configurable)

**Confidence**: 95% (approved)

---

## Decision

**APPROVED: Alternative 3** - Kubernetes-Aware 4-Step Shutdown Pattern

**Rationale**:

1. **Zero-Downtime Requirement**: Production services must have 0% request failure rate during deployments
   - Alternative 1: 5-10% failure rate ❌
   - Alternative 2: 2-5% failure rate ⚠️
   - Alternative 3: 0% failure rate ✅

2. **Kubernetes-Native Behavior**: Must coordinate with Kubernetes pod lifecycle
   - Alternative 1: No coordination ❌
   - Alternative 2: Partial coordination ⚠️
   - Alternative 3: Full coordination ✅

3. **Resource Cleanup**: Must prevent connection leaks and resource exhaustion
   - Alternative 1: No cleanup ❌
   - Alternative 2: No cleanup ❌
   - Alternative 3: Complete cleanup ✅

4. **Industry Best Practice**: Used by major projects (Istio, Linkerd, NGINX Ingress)
   - 5-second wait is industry standard for Kubernetes endpoint propagation
   - Readiness probe coordination is documented in Kubernetes best practices
   - Pattern proven in production at scale

**Key Insight**: The readiness probe is the critical missing piece - without it, Kubernetes continues routing traffic during shutdown, causing inevitable request failures. The 5-second wait alone is insufficient; coordinated signaling via readiness probe is essential.

---

## Implementation

### HTTP Services Pattern

**Primary Implementation Files**:
- `pkg/{service}/server/server.go` - Server struct with shutdown logic
- `cmd/{service}/main.go` - Signal handling and shutdown orchestration
- Deployment YAML - Readiness probe configuration

**Required Changes**:

#### 1. Add Shutdown Flag to Server Struct

```go
import "sync/atomic"

type Server struct {
    httpServer     *http.Server
    logger         *zap.Logger
    dbClient       DatabaseClient
    cacheManager   CacheManager
    
    // REQUIRED: Shutdown coordination flag
    isShuttingDown atomic.Bool  // Thread-safe flag for readiness probe
}
```

#### 2. Update Readiness Probe Handler

```go
func (s *Server) handleReadiness(w http.ResponseWriter, r *http.Request) {
    // STEP 0: Check shutdown flag FIRST (before any other checks)
    if s.isShuttingDown.Load() {
        s.logger.Debug("Readiness check during shutdown - returning 503")
        s.respondJSON(w, http.StatusServiceUnavailable, map[string]string{
            "status": "shutting_down",
            "reason": "graceful_shutdown_in_progress",
        })
        return
    }
    
    // Normal health checks
    dbHealthy := s.dbClient.Ping(r.Context()) == nil
    cacheHealthy := s.cacheManager.Ping(r.Context()) == nil
    
    if !dbHealthy || !cacheHealthy {
        w.WriteHeader(503)
        return
    }
    
    w.WriteHeader(200)
}
```

**CRITICAL**: Do NOT check shutdown flag in liveness probe - it should always return 200 during shutdown

#### 3. Implement 4-Step Shutdown Method

```go
func (s *Server) Shutdown(ctx context.Context) error {
    s.logger.Info("Initiating Kubernetes-aware graceful shutdown")
    
    // STEP 1: Set shutdown flag (readiness probe → 503)
    // This signals Kubernetes to remove pod from Service endpoints
    s.isShuttingDown.Store(true)
    s.logger.Info("Shutdown flag set - readiness probe now returns 503",
        zap.String("effect", "kubernetes_will_remove_from_endpoints"))
    
    // STEP 2: Wait for Kubernetes endpoint removal propagation
    // Kubernetes typically takes 1-3 seconds to update endpoints across all nodes
    // We wait 5 seconds to be safe (industry best practice)
    const endpointPropagationDelay = 5 * time.Second
    s.logger.Info("Waiting for Kubernetes endpoint removal propagation",
        zap.Duration("delay", endpointPropagationDelay),
        zap.String("reason", "ensure_no_new_traffic"))
    time.Sleep(endpointPropagationDelay)
    s.logger.Info("Endpoint removal propagation complete - no new traffic expected")
    
    // STEP 3: Drain in-flight HTTP connections
    // This completes any requests that arrived BEFORE endpoint removal
    // Uses context timeout from caller (typically 30 seconds)
    s.logger.Info("Draining in-flight HTTP connections",
        zap.String("method", "http.Server.Shutdown"),
        zap.Duration("max_wait", 30*time.Second))
    
    if err := s.httpServer.Shutdown(ctx); err != nil {
        s.logger.Error("HTTP server shutdown failed", zap.Error(err))
        return fmt.Errorf("HTTP shutdown failed: %w", err)
    }
    s.logger.Info("HTTP connections drained successfully")
    
    // STEP 4: Close external resources
    // Continue cleanup even if one step fails (don't return early)
    var shutdownErrors []error
    
    // Close database connections
    s.logger.Info("Closing database connections")
    if err := s.dbClient.Close(); err != nil {
        s.logger.Error("Failed to close database", zap.Error(err))
        shutdownErrors = append(shutdownErrors, fmt.Errorf("database close: %w", err))
    } else {
        s.logger.Info("Database connections closed successfully")
    }
    
    // Close cache connections
    s.logger.Info("Closing cache connections")
    if err := s.cacheManager.Close(); err != nil {
        s.logger.Error("Failed to close cache", zap.Error(err))
        shutdownErrors = append(shutdownErrors, fmt.Errorf("cache close: %w", err))
    } else {
        s.logger.Info("Cache connections closed successfully")
    }
    
    if len(shutdownErrors) > 0 {
        s.logger.Error("Shutdown completed with errors",
            zap.Int("error_count", len(shutdownErrors)))
        return fmt.Errorf("shutdown errors: %v", shutdownErrors)
    }
    
    s.logger.Info("Graceful shutdown complete - all resources closed")
    return nil
}
```

#### 4. Signal Handling in main.go

```go
func main() {
    // ... server creation ...
    
    // Start server in background
    errChan := make(chan error, 1)
    go func() {
        if err := srv.Start(); err != nil {
            errChan <- err
        }
    }()
    
    // Setup signal handling for SIGTERM and SIGINT
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
    
    // Wait for shutdown signal or server error
    select {
    case err := <-errChan:
        logger.Fatal("Server failed", zap.Error(err))
    case sig := <-sigChan:
        logger.Info("Shutdown signal received", zap.String("signal", sig.String()))
    }
    
    // Graceful shutdown with 30-second timeout
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    
    logger.Info("Initiating graceful shutdown...")
    if err := srv.Shutdown(shutdownCtx); err != nil {
        logger.Error("Graceful shutdown failed", zap.Error(err))
        os.Exit(1)
    }
    
    logger.Info("Server shutdown complete")
}
```

#### 5. Kubernetes Deployment Configuration

```yaml
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: service
        # CRITICAL: Readiness probe is REQUIRED for graceful shutdown
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 1  # Fast endpoint removal on shutdown
        
        # Liveness probe should NOT check shutdown flag
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        
        # Termination grace period must exceed shutdown timeout
        # 30s shutdown + 5s propagation + 5s buffer = 40s minimum
        terminationGracePeriodSeconds: 40
```

---

### CRD Controller Pattern

**Primary Implementation Files**:
- `internal/controller/{controller}/controller.go` - Reconcile shutdown logic
- `cmd/{controller}/main.go` - Controller manager shutdown

**Graceful Degradation Flow**:

1. **STEP 1**: Kubernetes sends SIGTERM to controller manager
2. **STEP 2**: Manager stops accepting new reconcile requests (LeaderElection releases leadership)
3. **STEP 3**: Manager waits for in-flight reconciliations to complete (up to timeout)
4. **STEP 4**: Manager closes Kubernetes client connections

**Implementation** (using controller-runtime):

```go
// In cmd/{controller}/main.go
func main() {
    // ... manager setup ...
    
    // Configure manager with graceful shutdown
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        // REQUIRED: Graceful shutdown timeout
        GracefulShutdownTimeout: pointer.Duration(30 * time.Second),
        
        // Health probes for Kubernetes coordination
        HealthProbeBindAddress: ":8081",
        
        // Leader election (prevents split-brain during shutdown)
        LeaderElection:   true,
        LeaderElectionID: "controller-leader-election",
    })
    
    // Add readiness check that fails during shutdown
    if err := mgr.AddReadyzCheck("ready", func(req *http.Request) error {
        // Manager automatically fails this during shutdown
        return mgr.GetCache().WaitForCacheSync(req.Context())
    }); err != nil {
        logger.Error(err, "Failed to add readiness check")
        os.Exit(1)
    }
    
    // Start manager (blocks until shutdown)
    logger.Info("Starting controller manager")
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        logger.Error(err, "Controller manager failed")
        os.Exit(1)
    }
    
    logger.Info("Controller manager shutdown complete")
}
```

**Reconcile Loop Shutdown**:

```go
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := r.Log.WithValues("resource", req.NamespacedName)
    
    // Check context cancellation (manager shutdown)
    select {
    case <-ctx.Done():
        log.Info("Reconciliation cancelled due to shutdown")
        return ctrl.Result{}, ctx.Err()
    default:
        // Continue normal reconciliation
    }
    
    // Perform reconciliation work
    // ... business logic ...
    
    // Check context again before long operations
    if ctx.Err() != nil {
        log.Info("Shutdown detected mid-reconciliation - aborting safely")
        return ctrl.Result{Requeue: true}, nil  // Requeue for next pod
    }
    
    return ctrl.Result{}, nil
}
```

**Key Differences from HTTP Services**:
- **No manual endpoint coordination** (controller-runtime handles leader election)
- **Work unit = reconciliation** (vs HTTP request)
- **Requeue on abort** (vs complete HTTP request)
- **30-second hard timeout** (prevents infinite reconciliation loops)

---

## Graceful Degradation

**HTTP Services**:
1. Readiness probe returns 503 → Kubernetes removes from endpoints
2. In-flight requests complete within 30-second timeout
3. Requests exceeding timeout: HTTP connection forcibly closed (client gets broken pipe)
4. Resource cleanup continues even if HTTP shutdown fails

**CRD Controllers**:
1. Leader election released → Other pods take over reconciliations
2. In-flight reconciliations complete within 30-second timeout
3. Reconciliations exceeding timeout: Context cancelled, work requeued
4. Kubernetes client cache closed after timeout

**Resource Cleanup Priority**:
1. **Critical**: Database connections (prevent connection pool exhaustion)
2. **High**: Cache connections (prevent Redis connection leaks)
3. **Medium**: File handles (prevent file descriptor exhaustion)
4. **Low**: Goroutines (garbage collected automatically)

---

## Consequences

### Positive

- ✅ **Zero request failures** during rolling updates (0% vs 5-10% baseline)
- ✅ **No data corruption** from aborted reconciliations
- ✅ **Clean resource cleanup** (no connection leaks)
- ✅ **Kubernetes-native behavior** (coordinated with pod lifecycle)
- ✅ **Timeout protection** (prevents infinite hangs)
- ✅ **Production-proven pattern** (used by Istio, Linkerd, NGINX Ingress)

### Negative

- ⚠️ **5-second deployment delay** per pod (Kubernetes endpoint propagation wait)
  - **Mitigation**: Industry standard, necessary for zero-downtime
  - **Impact**: 10-pod deployment takes 50 extra seconds (acceptable for production stability)

- ⚠️ **Increased code complexity** (50 lines vs 1 line)
  - **Mitigation**: Template provided in DD-007, copy-paste pattern
  - **Impact**: One-time implementation cost, long-term benefit

- ⚠️ **Requires readiness probe** in Kubernetes deployment
  - **Mitigation**: Already standard practice for production services
  - **Impact**: None (already required for health checks)

### Neutral

- 🔄 **Fixed 5-second wait** (not configurable per-service)
  - Kubernetes endpoint propagation time is variable (1-5 seconds across cluster)
  - 5 seconds is conservative industry standard
  - Making it configurable adds complexity without significant benefit

---

## Validation Results

**Confidence Assessment Progression**:
- **Initial assessment**: 80% confidence (theory-based, Gateway implementation exists)
- **After Gateway validation**: 95% confidence (production-proven pattern)
- **After triage analysis**: 95% confidence (confirmed gaps in Context API)

**Key Validation Points**:
- ✅ Gateway service uses this pattern successfully (zero request failures observed)
- ✅ Context API triage confirmed 5-10% failure rate without pattern
- ✅ Industry best practice documented by Kubernetes documentation
- ✅ Pattern used by major projects (Istio, Linkerd, NGINX Ingress)

**Test Results** (from Gateway implementation):
- ✅ Readiness probe returns 503 immediately on SIGTERM
- ✅ Zero request failures during rolling updates (measured)
- ✅ In-flight requests complete successfully within timeout
- ✅ Resources cleaned up without leaks

---

## Related Decisions

- **Builds On**: [DD-005 Observability Standards](DD-005-OBSERVABILITY-STANDARDS.md) - Health check endpoints
- **Supports**: 
  - BR-GATEWAY-007: Zero-downtime deployments
  - BR-CONTEXT-007: Production readiness
  - BR-DATA-STORAGE-007: High availability
- **Referenced By**:
  - Gateway Service: Fully implemented (reference implementation)
  - Context API: Implementation pending (DD-007)
  - All future HTTP services: Must follow DD-007

---

## Review & Evolution

**When to Revisit**:
- If Kubernetes endpoint propagation time changes significantly (currently 1-3 seconds)
- If new Kubernetes lifecycle hooks become available (e.g., `PreStop` improvements)
- If in-flight request timeout (30s) proves insufficient for specific use cases
- If controller-runtime graceful shutdown behavior changes

**Success Metrics**:
- **Request Failure Rate**: 0% during rolling updates (baseline: 5-10%)
- **Deployment Success Rate**: 100% (no failed deployments due to timeout)
- **Resource Leak Rate**: 0% (no leaked connections after shutdown)
- **Average Shutdown Time**: < 10 seconds (5s propagation + 2s drain + 1s cleanup)

---

## Implementation Status (v1.5)

As implemented in **`pkg/datastorage/server`** (`Server.Shutdown`), graceful shutdown aggregates non-fatal step failures via **`errors.Join`** so operators observe every partial failure instead of masking earlier errors. Sequence additions include **`DLQRetryWorker.Stop()`** before **`DrainWithTimeout`** (DD-009), **`RetentionWorker.Stop()`** ahead of **`sql.DB.Close()`** (FedRAMP **AU-11**, #1048), and **`config.Server.shutdownTimeout`** (YAML `shutdownTimeout`) budgeting **`http.Server.Shutdown`** alongside the propagation/poll timers.

---

## References

- **Kubernetes Pod Termination Lifecycle**: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
- **Go http.Server.Shutdown()**: https://pkg.go.dev/net/http#Server.Shutdown
- **Controller-Runtime Graceful Shutdown**: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/manager
- **Industry Best Practices**:
  - Istio: https://istio.io/latest/docs/ops/deployment/performance-and-scalability/
  - Linkerd: https://linkerd.io/2/tasks/graceful-shutdown/
  - NGINX Ingress: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/

---

**Last Updated**: October 31, 2025  
**Next Review**: April 30, 2026 (6 months)

