# System Resilience: Recovery & Stability

> **Navigation**: [← Index Hub](../../INDEX.md) | [Audit Framework](../governance/AUDIT.md#3-the-shield-security--baseline)

This document describes the high-availability, self-healing, and security mechanisms that ensure Serverless Claw remains operational and safe.

## 🛡️ Security & Baseline Control

The Shield acts as the authoritative gate for all tool executions, enforcing least-privilege resource access and blast-radius limits.

### Layer 1: Unified Security Gateway

The `ToolExecutor` delegates all security decisions to the `SafetyEngine`. No tool can execute without an explicit policy allowance.

### Layer 2: Blast Radius Enforcement

The system enforces hard limits on high-impact actions to prevent runaway costs or destructive propagation:

- **Class C Limit**: Hard cap of 5 Class C actions per hour per agent (Principle 10).
- **Promotion**: Class C actions require human approval unless an agent is in `EVOLUTION_MODE="AUTO"` and has a `TrustScore >= 95`.
- **Persistence**: The `BlastRadiusStore` uses a **Two-Phase Atomic Update** pattern to ensure integrity during window transitions:
  - **Phase 1 (Active Window)**: Conditional `UpdateCommand` using `ADD` to increment count within current window.
  - **Phase 2 (Expired Window)**: Conditional `UpdateCommand` using `SET` to atomically reset the window if it has expired.
- **Retry Safety**: Maximum 3 retries to prevent infinite recursion during persistent race conditions. After max retries, the operation is allowed to proceed to prevent system blockage.

```text
   [ Class C Action ]
          |
          v
   [ BlastRadiusStore.incrementBlastRadius() ]
          |
          +---- [ Phase 1: Window Active? ] ----+
          |                                     |
    [ Yes: Conditional ADD ]             [ No: Conditional SET ]
          |                                     |
     <SUCCESS>                            <SUCCESS>
          |                                     |
    [ Update Cache ]                     [ Reset Cache ]
          |                                     |
          +------------------+------------------+
                             |
                      [ Return Result ]
```

**Key Features**:

- DynamoDB-backed storage with key pattern `WS#{workspaceId}#safety:blast_radius:{agentId}:{action}`
- **Atomic Integrity**: Handles window resets atomically during the increment phase, eliminating "get-then-delete" race conditions.
- Local cache for performance within same Lambda instance.
- 1-hour TTL (WINDOW_MS) with automatic cleanup via `expiresAt` GSI.

### Layer 3: Loop Interdiction

Reasoning loops and repetitive "semantic grinding" are caught by the **SemanticLoopDetector**.

- **Penalty**: Detected loops result in automatic trust penalties recorded via `SafetyBase.recordFailure`.
- **Multi-Tenant Isolation**: Output history is cached with `workspaceId` scoping to prevent reasoning loop poisoning across different tenants (Anti-Pattern 19).

---

## 🌊 Distributed Resilience Primitives

To ensure consistency across concurrent Lambda executions, the system utilizes a generic **DistributedState** utility grounded in the `MemoryTable`.

### 1. Token Bucket Rate Limiter

The `consumeToken` primitive implements a distributed token bucket using atomic conditional updates to prevent race conditions.

```text
     [ Request ]
          |
          v
     [ DistributedState.consumeToken ]
          |
          v
     [ Atomic Conditional Update ]
     (if_not_exists(#c, :cap) - :one)
     Condition: #c > :zero
          |
    +-----+-----+
    |           |
 [ SUCCESS ]  [ FAIL ]
    |           |
 (Tokens > 0) (Tokens = 0)
    |           |
 Return TRUE  Return FALSE
 [ ALLOW ]    [ REJECT ]
```

**Key Features**:

- Uses window-based keys (`safety:token_bucket:{key}:{windowId}`) for automatic window reset
- Atomic conditional update in single DynamoDB operation - no read-then-write race
- Fail-closed: if DDB fails, request is rejected to prevent rate limit bypass
- 1-hour TTL via DynamoDB expiry

### 2. Global Circuit Breaker

The `isCircuitOpen` primitive provides a shared failure state across the entire agent swarm. When a specific threshold of failures is reached within a sliding timeout window, the circuit trips globally for all active instances.

Serverless Claw employs a persistent circuit breaker to prevent runaway deployments and cost spikes.

### Layer 1: Daily Deployment Limit

- **Mechanism**: Atomic counter in DynamoDB.
- **Default**: 5 deployments per UTC day.
- **Reset**: Automatically resets at UTC midnight.

### Layer 2: Sliding Window

The system tracks `deploy` and `health` failures in a 1-hour sliding window.

- **Closed**: System operating normally.
- **Open**: Failures > 5. Autonomous deployments are blocked for 10 minutes.
- **Half-Open**: One probe deployment allowed after cooldown.

---

## 🆘 Dead Man's Switch (Emergency Recovery)

The **Dead Man's Switch** is an external heartbeat monitor that triggers an automated rollback if the system becomes "braindead" (unresponsive).

1. **Schedule**: Triggered every 2 hours by EventBridge (Production only).
2. **Probes**: Combined HTTP (`GET /health`) and Cognitive Health check.
3. **Emergency Action**: If probes fail, CodeBuild triggers an emergency rollback.
4. **Git Revert**: The system performs a `git revert HEAD` (or to a Last Known Good hash) and redeploys the stack.

---

## 🩹 Self-Healing Loop

When a deployment fails, the system automatically attempts a repair before escalating to human intervention.

```text
    +-----------+           +-----------+
    |  Monitor  | --------> | SuperClaw |
    | (Enrich)  |           | (Diagnose)|
    +-----------+           +-----------+
          ^                       |
          |                       v
    +-----------+           +-----------+
    | Deployer  | <-------- | Coder     |
    | (Fail)    |           | (Repair)  |
    +-----------+           +-----------+
```

## 🛡️ Security Interaction Flow

The `SafetyEngine` decomposes every evaluation into four specific validation stages to ensure deterministic results and maintainable logic.

```text
  [ Agent Output ] -> [ SemanticLoopDetector ] -- (Loop Found) --> [ SafetyBase.recordFailure ]
          |                                                             (Trust Penalty)
          v
  [ Tool Call ] -> [ Shield (SafetyEngine) ]
          |                  |
          |          (1) Static Check (Class D Block?)
          |                  |
          |          (2) Access Control (Blocked Patterns? System Protected?)
          |                  |
          |          (3) Rate Limits (Concurrent Throttling)
          |                  |
          |          (4) Dynamic Check (Time Windows? Principle 9 Promotion? Blast Radius?)
          |                  |
          +------------------+-------------------> [ Result (Allow/Block/Approval) ]
```

### Self-Healing Flow:

1. **Diagnosis**: `BuildMonitor` extracts the last 3000 chars of build logs.
2. **Enrichment**: The `traceId` and `gapId` of the failing task are bundled.
3. **Repair**: The **Coder Agent** receives the error logs and reasoning context to propose a fix.
4. **Guard**: This loop is subject to the Circuit Breaker to prevent "Infinite Repair Harm."

---

## 🌊 Concurrency & Lock Management

To prevent race conditions between multiple agents modifying the same code or database records, the system uses a distributed **Lock Manager**.

- **Implementation**: DynamoDB-backed atomic locks with TTL.
- **Default TTL**: 5 minutes for general tasks; 30 minutes for Strategic Gaps.
- **Conflict Handling**: If a lock is busy, the agent emits a `TASK_PAUSED` signal and retries after a jittered backoff.
