# Error Recovery Pattern

## Overview

The **Error Recovery Pattern** enables agentic systems to gracefully handle failures, diagnose issues, implement corrective actions, and learn from errors to build resilient and reliable AI applications. By combining exception handling, self-correction mechanisms, and adaptive strategies, this pattern ensures systems can recover from both expected and unexpected failures.

Unlike traditional error handling that simply catches and logs exceptions, the Error Recovery pattern actively attempts to diagnose root causes, apply appropriate recovery strategies, verify successful recovery, and maintain historical context for continuous improvement.

## Why Use This Pattern?

Real-world agentic systems face numerous sources of failure:
- **Unreliable external services**: API rate limits, timeouts, temporary outages
- **Invalid or unexpected inputs**: Malformed data, edge cases, adversarial inputs
- **Resource constraints**: Memory limits, quota exhaustion, network issues
- **LLM-specific challenges**: Hallucinations, output format errors, token limits
- **Environmental variability**: Transient network conditions, service degradation

Traditional approaches have limitations:
- **Simple try-catch**: Fails to distinguish transient from permanent errors
- **Immediate failure**: No attempt at recovery or alternative approaches
- **Manual intervention**: Requires human operators to diagnose and fix
- **No learning**: Repeats the same mistakes without adaptation

Error Recovery solves these by:
- **Intelligent classification**: Distinguishes transient, retryable, and permanent errors
- **Multi-strategy recovery**: Retries, fallbacks, circuit breakers, alternative approaches
- **Self-correction loops**: Validates outputs and fixes issues automatically
- **Pattern learning**: Tracks common errors and successful recovery strategies
- **Graceful degradation**: Provides partial results when full recovery isn't possible

### Example: Without vs. With Error Recovery

```
Without Error Recovery (Brittle):
User: "Summarize this document"
System: [API timeout]
Error: Request failed after 30s
→ Complete failure, no result

With Error Recovery (Resilient):
User: "Summarize this document"
System: [API timeout detected]
→ Attempt 1: Retry with exponential backoff (2s delay)
→ Attempt 2: [Still timeout] Try with reduced token limit
→ Attempt 3: [Success] Return summary with degraded quality notice
Result: "Here's a summary (note: using shorter analysis due to service constraints)..."
→ Successful recovery, partial result delivered
```

## How It Works

The Error Recovery pattern operates through a five-stage cycle:

1. **Detect**: Monitor operations and catch failures early
2. **Diagnose**: Classify error type and determine root cause
3. **Recover**: Apply appropriate recovery strategy based on diagnosis
4. **Verify**: Validate that recovery was successful
5. **Learn**: Store error patterns and successful recovery paths

```
┌─────────────────────────────────────────────────────────┐
│                    Execute Operation                     │
└───────────────────┬─────────────────────────────────────┘
                    ↓
            ┌───────────────┐
            │    Detect     │ Monitor for errors/failures
            └───────┬───────┘
                    ↓
         ┌──────────────────┐
         │ Error Occurred?  │
         └────┬─────────┬───┘
              No        Yes
              ↓         ↓
         [Success]  ┌───────────────┐
                    │   Diagnose    │ Classify error type
                    └───────┬───────┘
                            ↓
                    ┌───────────────┐
                    │  Select       │ Choose recovery strategy
                    │  Strategy     │ (retry, fallback, etc.)
                    └───────┬───────┘
                            ↓
                    ┌───────────────┐
                    │   Recover     │ Execute recovery action
                    └───────┬───────┘
                            ↓
                    ┌───────────────┐
                    │    Verify     │ Validate recovery
                    └───────┬───────┘
                            ↓
                    ┌───────────────┐
                    │     Learn     │ Store patterns
                    └───────┬───────┘
                            ↓
                    [Return to Execute or Fail]
```

## When to Use This Pattern

### ✅ Ideal Use Cases

- **Production systems**: Critical applications requiring high availability
- **External API integrations**: Services with rate limits, timeouts, or variable reliability
- **User-facing applications**: Where failures impact customer experience
- **Long-running workflows**: Complex multi-step processes prone to intermittent failures
- **Resource-constrained environments**: Systems with memory, quota, or cost limits
- **LLM-based generation**: Handling format errors, hallucinations, invalid outputs
- **Autonomous agents**: Self-healing systems that minimize human intervention
- **Batch processing**: Large-scale operations where partial failures are acceptable

### ❌ When NOT to Use

- **Non-critical prototypes**: Early development where failure is acceptable
- **Deterministic operations**: Pure computation with no external dependencies
- **Real-time constraints**: When retry latency is unacceptable
- **Security-critical paths**: Where recovery attempts might mask attacks
- **Simple scripts**: One-off tasks with no reliability requirements

## Rule of Thumb

**Use Error Recovery when:**
1. System interacts with **unreliable external services**
2. Failures are **often transient** and retryable
3. **Partial success** is better than complete failure
4. **Downtime costs** exceed recovery implementation costs
5. System should be **self-healing** with minimal human intervention

**Don't use Error Recovery when:**
1. Errors indicate fundamental logic bugs (fix the code instead)
2. Failures must immediately alert operators (don't mask critical issues)
3. Recovery attempts would compound the problem (e.g., cascading failures)
4. Latency is more critical than reliability

## Core Components

### 1. Error Detection & Monitoring

Early detection prevents cascading failures:
- **Exception catching**: Try-catch blocks around failure-prone operations
- **Health checks**: Proactive monitoring of service availability
- **Timeout management**: Prevent indefinite hangs
- **Output validation**: Catch malformed results before they propagate

### 2. Error Classification

Different errors require different recovery strategies:

**Transient Errors** (retry):
- Network timeouts
- Rate limit errors (429)
- Temporary service unavailability (503)
- Connection resets

**Retryable Errors** (retry with modification):
- Token limit exceeded (reduce input size)
- Invalid format (adjust prompt)
- Authentication expired (refresh token)

**Permanent Errors** (fallback or fail):
- Invalid API credentials (401)
- Resource not found (404)
- Malformed request (400)
- Quota exhausted (no recovery)

### 3. Recovery Strategies

Multiple strategies for different failure modes:

**Retry with Exponential Backoff**:
- Immediate retry for transient issues
- Increasing delays prevent overwhelming services
- Jitter prevents thundering herd

**Fallback Strategies**:
- Alternative APIs or services
- Cached or stale data
- Simplified or degraded functionality
- Default safe values

**Self-Correction Loops**:
- Parse errors → Fix format → Retry
- Validation failures → Adjust parameters → Retry
- Output errors → Regenerate with constraints

**Circuit Breaker**:
- Stop retrying after repeated failures
- Fail fast during outages
- Periodically test for recovery

### 4. Verification

Ensure recovery was successful:
- **Output validation**: Check format, completeness, correctness
- **Health verification**: Confirm service is responsive
- **Quality checks**: Ensure degraded results meet minimum standards
- **Side effect validation**: Verify state consistency

### 5. Error History & Learning

Build institutional knowledge:
- **Pattern tracking**: Common error types and frequencies
- **Success rate metrics**: Which strategies work best
- **Failure correlation**: Related errors that occur together
- **Adaptive thresholds**: Adjust retry limits based on history

## Implementation Approaches

### Approach 1: Basic Retry with Exponential Backoff

Using the `tenacity` library for robust retry logic:

```python
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type
)

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    retry=retry_if_exception_type(TransientError)
)
def call_api(endpoint: str) -> dict:
    """Call API with automatic retry on transient failures"""
    response = requests.get(endpoint)
    response.raise_for_status()
    return response.json()
```

### Approach 2: Multi-Strategy Recovery

Implement multiple recovery paths:

```python
def execute_with_recovery(operation, *args, **kwargs):
    """Execute operation with fallback strategies"""

    # Strategy 1: Try primary approach with retry
    try:
        return retry_with_backoff(operation, *args, **kwargs)
    except RetryableError as e:
        log_error("Primary strategy failed", e)

    # Strategy 2: Try with reduced parameters
    try:
        return operation(*args, **{**kwargs, 'max_tokens': 500})
    except Exception as e:
        log_error("Reduced parameters failed", e)

    # Strategy 3: Use fallback service
    try:
        return fallback_operation(*args, **kwargs)
    except Exception as e:
        log_error("Fallback failed", e)

    # Strategy 4: Return degraded result
    return get_cached_or_default()
```

### Approach 3: Self-Correction Loop

For LLM outputs with validation:

```python
def generate_with_correction(prompt: str, validator: Callable) -> str:
    """Generate output with automatic correction"""

    max_attempts = 3
    errors = []

    for attempt in range(max_attempts):
        # Generate output
        output = llm.invoke(prompt)

        # Validate
        is_valid, error_msg = validator(output)

        if is_valid:
            return output

        # Diagnose and correct
        errors.append(error_msg)
        prompt = f"{prompt}\n\nPrevious attempt failed: {error_msg}\nPlease correct this issue."

    raise ValidationError(f"Failed after {max_attempts} attempts: {errors}")
```

### Approach 4: Circuit Breaker Pattern

Prevent cascading failures:

```python
class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, timeout: int = 60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.state = "closed"  # closed, open, half-open
        self.last_failure_time = None

    def call(self, func, *args, **kwargs):
        if self.state == "open":
            if time.time() - self.last_failure_time > self.timeout:
                self.state = "half-open"
            else:
                raise CircuitBreakerOpen("Service unavailable")

        try:
            result = func(*args, **kwargs)
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise

    def on_success(self):
        self.failure_count = 0
        self.state = "closed"

    def on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = "open"
```

## Key Benefits

### 🛡️ Resilience

- **Fault tolerance**: System continues operating despite failures
- **Self-healing**: Automatic recovery without manual intervention
- **Graceful degradation**: Provides partial functionality when full recovery isn't possible
- **Reduced downtime**: Quick recovery from transient issues

### 📈 Improved User Experience

- **Higher availability**: Users experience fewer failures
- **Transparent recovery**: Issues resolved before users notice
- **Informative feedback**: When recovery fails, users get actionable information
- **Consistency**: Predictable behavior even during service disruptions

### 💰 Cost Efficiency

- **Reduced manual intervention**: Fewer on-call alerts and manual fixes
- **Optimized retries**: Smart backoff prevents wasted API calls
- **Resource management**: Circuit breakers prevent resource exhaustion
- **Learning from patterns**: Continuous improvement reduces future failures

### 🔍 Observability

- **Error tracking**: Comprehensive logs of failure modes
- **Recovery metrics**: Success rates for different strategies
- **Pattern detection**: Identify systemic issues early
- **Diagnostic data**: Rich context for debugging

## Trade-offs

### ⚠️ Increased Complexity

**Issue**: Error recovery adds significant code complexity

**Impact**: More code to write, test, and maintain

**Mitigation**:
- Use well-tested libraries (tenacity, circuit breaker packages)
- Start with simple retry logic, add sophistication as needed
- Document recovery strategies clearly
- Test error paths thoroughly

### ⏱️ Increased Latency

**Issue**: Retries and fallbacks add delay to operations

**Impact**: Slower response times, especially during failures

**Mitigation**:
- Set aggressive timeouts for fast failure
- Use exponential backoff with reasonable max delays
- Implement parallel fallback strategies where possible
- Fail fast for permanent errors

### 💸 Higher Costs

**Issue**: Retries consume additional API calls and resources

**Impact**: Increased operational costs, potential quota exhaustion

**Mitigation**:
- Set reasonable retry limits (typically 3-5 attempts)
- Use circuit breakers to prevent runaway retries
- Implement smart backoff to reduce API call frequency
- Monitor and alert on excessive retry rates

### 🎭 Masking Real Issues

**Issue**: Successful recovery might hide systemic problems

**Impact**: Underlying issues go undetected and unfixed

**Mitigation**:
- Always log all errors, even when recovered
- Track recovery rates and alert on patterns
- Distinguish between normal transient errors and systemic issues
- Implement monitoring and alerting for recovery metrics

## Best Practices

### 1. Error Classification

```python
class ErrorClassifier:
    """Classify errors to determine recovery strategy"""

    TRANSIENT_ERRORS = [
        "timeout",
        "connection reset",
        "503",
        "429"
    ]

    RETRYABLE_ERRORS = [
        "token limit",
        "invalid format",
        "rate limit"
    ]

    PERMANENT_ERRORS = [
        "401",
        "404",
        "invalid credentials"
    ]

    @classmethod
    def classify(cls, error: Exception) -> str:
        error_str = str(error).lower()

        if any(e in error_str for e in cls.TRANSIENT_ERRORS):
            return "transient"
        elif any(e in error_str for e in cls.RETRYABLE_ERRORS):
            return "retryable"
        elif any(e in error_str for e in cls.PERMANENT_ERRORS):
            return "permanent"
        else:
            return "unknown"
```

### 2. Exponential Backoff with Jitter

```python
import random
import time

def retry_with_backoff(func, max_attempts=5, base_delay=1, max_delay=32):
    """Retry with exponential backoff and jitter"""

    for attempt in range(max_attempts):
        try:
            return func()
        except RetryableError as e:
            if attempt == max_attempts - 1:
                raise

            # Calculate delay: min(base * 2^attempt, max_delay)
            delay = min(base_delay * (2 ** attempt), max_delay)

            # Add jitter: random value between 0 and delay
            jittered_delay = delay * random.random()

            print(f"Attempt {attempt + 1} failed, retrying in {jittered_delay:.2f}s")
            time.sleep(jittered_delay)
```

### 3. Comprehensive Error Logging

```python
import logging
from datetime import datetime

def log_error_with_context(error: Exception, context: dict):
    """Log errors with rich diagnostic context"""

    error_record = {
        "timestamp": datetime.now().isoformat(),
        "error_type": type(error).__name__,
        "error_message": str(error),
        "classification": ErrorClassifier.classify(error),
        "operation": context.get("operation"),
        "attempt": context.get("attempt"),
        "recovery_strategy": context.get("strategy"),
        "stack_trace": traceback.format_exc()
    }

    logging.error(f"Error occurred: {json.dumps(error_record, indent=2)}")

    # Store for pattern analysis
    error_history.append(error_record)
```

### 4. Output Validation

```python
from typing import Tuple

def validate_output(output: str, expected_format: str) -> Tuple[bool, str]:
    """Validate LLM output meets expected format"""

    validations = {
        "json": lambda x: validate_json(x),
        "code": lambda x: validate_code_syntax(x),
        "structured": lambda x: validate_structure(x)
    }

    validator = validations.get(expected_format)
    if not validator:
        return True, ""

    try:
        validator(output)
        return True, ""
    except ValidationError as e:
        return False, str(e)

def validate_json(output: str) -> None:
    """Validate JSON format"""
    try:
        json.loads(output)
    except json.JSONDecodeError as e:
        raise ValidationError(f"Invalid JSON: {e}")

def validate_code_syntax(output: str) -> None:
    """Validate Python code syntax"""
    try:
        ast.parse(output)
    except SyntaxError as e:
        raise ValidationError(f"Invalid Python syntax: {e}")
```

### 5. Circuit Breaker Pattern

```python
# Use existing circuit breaker library
from pybreaker import CircuitBreaker

# Configure circuit breaker
breaker = CircuitBreaker(
    fail_max=5,          # Open circuit after 5 failures
    timeout_duration=60,  # Keep circuit open for 60 seconds
    exclude=[PermanentError]  # Don't count permanent errors
)

@breaker
def call_external_service(endpoint: str):
    """Call external service with circuit breaker protection"""
    return requests.get(endpoint)
```

## Performance Metrics

Track these metrics to evaluate error recovery effectiveness:

### Recovery Metrics
- **Recovery success rate**: % of errors successfully recovered
- **Mean Time To Recovery (MTTR)**: Average time from error to successful recovery
- **Recovery attempts per error**: How many strategies tried before success
- **Strategy effectiveness**: Success rate by recovery strategy type

### Error Metrics
- **Error frequency**: Errors per hour/day/operation
- **Error classification distribution**: Transient vs. retryable vs. permanent
- **Top error patterns**: Most common error types
- **Error correlation**: Related errors occurring together

### Performance Impact
- **Latency overhead**: Added delay from recovery attempts
- **Cost overhead**: Additional API calls from retries
- **Circuit breaker state**: Time spent in open/closed/half-open states
- **Degraded operation rate**: % of requests served with degraded quality

### System Health
- **Overall availability**: % of successful operations (including recoveries)
- **Partial success rate**: % of operations with degraded results
- **Complete failure rate**: % of operations that couldn't be recovered
- **Recovery trend**: Improving or degrading over time

## Example Scenarios

### Scenario 1: API Timeout with Retry

```
Operation: Call weather API
Error: Connection timeout after 30 seconds

Recovery Attempt 1:
→ Classify: Transient error (network timeout)
→ Strategy: Retry with exponential backoff
→ Wait: 2 seconds
→ Retry: [Success]
→ Result: Weather data retrieved

Total Time: 32 seconds (30s timeout + 2s backoff)
Recovery: Successful (1 retry)
```

### Scenario 2: Token Limit with Self-Correction

```
Operation: Generate code documentation
Error: Maximum token limit exceeded (4096 tokens)

Recovery Attempt 1:
→ Classify: Retryable error (resource limit)
→ Strategy: Reduce input size by 50%
→ Retry: [Still exceeds limit]

Recovery Attempt 2:
→ Classify: Still retryable
→ Strategy: Generate in chunks
→ Retry: [Success with 3 chunks]
→ Result: Documentation generated in parts

Total Time: 45 seconds
Recovery: Successful (2 attempts, strategy adaptation)
```

### Scenario 3: Cascading Failure with Circuit Breaker

```
Operation: Call payment processing API
Error: 503 Service Unavailable

Recovery Attempt 1-5:
→ Classify: Transient error
→ Strategy: Retry with backoff
→ All retries: [Failed]
→ Circuit breaker: OPEN (failure threshold reached)

Subsequent Operations:
→ Circuit state: OPEN
→ Strategy: Fail fast, use fallback
→ Result: Queue payments for later processing

After 60 seconds:
→ Circuit state: HALF-OPEN
→ Test request: [Success]
→ Circuit state: CLOSED
→ Normal operation resumed

Recovery: Prevented cascading failures, graceful degradation
```

### Scenario 4: Invalid Output Format

```
Operation: Extract structured data from text
Error: LLM output is not valid JSON

Recovery Attempt 1:
→ Classify: Retryable (format error)
→ Diagnose: Missing closing brace
→ Strategy: Self-correction prompt
→ Prompt: "Previous output was invalid JSON (missing }). Please provide valid JSON."
→ Retry: [Success]
→ Validation: JSON parses correctly
→ Result: Structured data extracted

Total Time: 12 seconds
Recovery: Successful (self-correction)
```

## Advanced Patterns

### 1. Adaptive Retry Limits

Learn optimal retry counts based on historical success:

```python
class AdaptiveRetry:
    def __init__(self):
        self.success_history = defaultdict(list)

    def get_retry_limit(self, error_type: str) -> int:
        """Determine optimal retry limit based on history"""
        history = self.success_history[error_type]

        if not history:
            return 3  # Default

        # Calculate average attempts to success
        avg_attempts = sum(history) / len(history)

        # Add buffer
        return int(avg_attempts * 1.5)

    def record_success(self, error_type: str, attempts: int):
        """Record successful recovery for learning"""
        self.success_history[error_type].append(attempts)
```

### 2. Bulkhead Pattern

Isolate failures to prevent resource exhaustion:

```python
from concurrent.futures import ThreadPoolExecutor

class Bulkhead:
    """Isolate resources to prevent cascading failures"""

    def __init__(self, max_workers: int = 5):
        self.executor = ThreadPoolExecutor(max_workers=max_workers)

    def execute(self, func, *args, timeout: int = 30):
        """Execute with resource isolation"""
        future = self.executor.submit(func, *args)
        try:
            return future.result(timeout=timeout)
        except TimeoutError:
            future.cancel()
            raise OperationTimeout(f"Operation exceeded {timeout}s")
```

### 3. Saga Pattern for Distributed Transactions

Coordinate recovery across multiple services:

```python
class Saga:
    """Coordinate multi-step operations with compensation"""

    def __init__(self):
        self.steps = []
        self.compensations = []

    def add_step(self, action, compensation):
        """Add step with compensation logic"""
        self.steps.append(action)
        self.compensations.append(compensation)

    def execute(self):
        """Execute saga with automatic rollback on failure"""
        completed = []

        try:
            for step in self.steps:
                result = step()
                completed.append(result)
            return completed
        except Exception as e:
            # Rollback completed steps
            for i in reversed(range(len(completed))):
                try:
                    self.compensations[i](completed[i])
                except Exception as comp_error:
                    logging.error(f"Compensation failed: {comp_error}")
            raise
```

### 4. Health Check with Auto-Recovery

Proactive monitoring and recovery:

```python
import asyncio

class HealthMonitor:
    """Monitor service health and trigger recovery"""

    def __init__(self, check_interval: int = 60):
        self.check_interval = check_interval
        self.services = {}

    def register_service(self, name: str, health_check: Callable):
        """Register service for monitoring"""
        self.services[name] = {
            "check": health_check,
            "status": "unknown",
            "consecutive_failures": 0
        }

    async def monitor(self):
        """Continuously monitor and recover services"""
        while True:
            for name, service in self.services.items():
                try:
                    is_healthy = await service["check"]()
                    if is_healthy:
                        service["status"] = "healthy"
                        service["consecutive_failures"] = 0
                    else:
                        await self.handle_unhealthy(name, service)
                except Exception as e:
                    await self.handle_failure(name, service, e)

            await asyncio.sleep(self.check_interval)

    async def handle_unhealthy(self, name: str, service: dict):
        """Handle unhealthy service"""
        service["consecutive_failures"] += 1

        if service["consecutive_failures"] >= 3:
            logging.warning(f"Service {name} unhealthy, attempting recovery")
            await self.attempt_recovery(name)
```

## Comparison with Related Patterns

| Pattern | Focus | Recovery Approach | When to Use |
|---------|-------|-------------------|-------------|
| **Error Recovery** | Fault tolerance | Multi-strategy recovery | Production systems with external dependencies |
| **HITL** | Human oversight | Human intervention | High-stakes decisions requiring judgment |
| **Guardrails** | Prevention | Block invalid operations | Ensure safety and compliance |
| **Monitoring** | Observability | Alert and diagnose | Track system health and performance |
| **Reflection** | Quality | Iterative refinement | Improve output quality through critique |
| **Circuit Breaker** | Failure isolation | Stop cascading failures | Protect against service outages |

**Combining patterns:**
- Error Recovery + HITL: Auto-recover simple errors, escalate complex ones
- Error Recovery + Guardrails: Prevent errors upfront, recover when they occur
- Error Recovery + Monitoring: Track recovery success, identify patterns
- Error Recovery + Reflection: Recover from errors, then improve quality

## Common Pitfalls

### 1. Infinite Retry Loops

**Problem**: Retrying permanent errors indefinitely

**Solution**:
- Classify errors properly (permanent vs. transient)
- Set hard limits on retry attempts
- Implement circuit breakers

### 2. Insufficient Backoff

**Problem**: Overwhelming services with rapid retries

**Solution**:
- Use exponential backoff with jitter
- Respect rate limit headers (Retry-After)
- Implement circuit breakers for repeated failures

### 3. Ignoring Error Context

**Problem**: Generic recovery doesn't address root cause

**Solution**:
- Classify errors based on type and context
- Use different strategies for different error types
- Log detailed diagnostic information

### 4. Hiding Critical Issues

**Problem**: Successful recovery masks systemic problems

**Solution**:
- Always log all errors, even when recovered
- Monitor recovery rates and patterns
- Alert on high recovery rates or unusual patterns
- Distinguish normal transient errors from systemic issues

### 5. Over-Engineering

**Problem**: Complex recovery logic for simple use cases

**Solution**:
- Start with simple retry logic
- Add sophistication only when needed
- Use proven libraries instead of custom implementations
- Measure the value of each recovery strategy

### 6. Lack of Verification

**Problem**: Assuming recovery succeeded without validation

**Solution**:
- Always validate outputs after recovery
- Check side effects and state consistency
- Implement health checks
- Use assertions and contracts

## Conclusion

The Error Recovery pattern is essential for building resilient, production-ready agentic systems. By combining intelligent error classification, multi-strategy recovery approaches, self-correction loops, and continuous learning, it enables systems to handle failures gracefully and maintain high availability.

**Use Error Recovery when:**
- Building production systems with reliability requirements
- Integrating with unreliable external services
- User experience depends on system availability
- Failures are often transient and recoverable
- System should self-heal with minimal human intervention

**Implementation checklist:**
- ✅ Classify errors into transient, retryable, and permanent
- ✅ Implement exponential backoff with jitter for retries
- ✅ Use circuit breakers to prevent cascading failures
- ✅ Validate outputs after recovery attempts
- ✅ Log all errors with rich diagnostic context
- ✅ Track recovery metrics and patterns
- ✅ Set reasonable retry limits and timeouts
- ✅ Implement graceful degradation for unrecoverable errors
- ✅ Test error paths thoroughly
- ✅ Monitor recovery success rates

**Key Takeaways:**
- 🛡️ Multi-strategy recovery provides resilience against diverse failures
- 🔄 Classify errors properly to apply appropriate recovery strategies
- ⚡ Exponential backoff prevents overwhelming services during recovery
- 🔍 Always log errors, even when successfully recovered
- 📊 Track recovery metrics to identify systemic issues
- 🎯 Balance recovery attempts with latency and cost constraints
- 🧠 Learn from error patterns to improve future recovery
- ✋ Use circuit breakers to prevent cascading failures

---

*Error Recovery transforms fragile systems into resilient ones—enabling autonomous operation, graceful degradation, and continuous improvement in the face of inevitable failures.*

## Corporate SSL proxy note

If you're behind a corporate SSL-inspecting proxy, run examples with:

```bash
AGENTIC_DISABLE_SSL=1 bash run.sh
```

