# Verify Timeout Triage Design

## Executive Summary

This document proposes replacing the current hard 300s verify timeout with a multi-layered triage-based approach that eliminates false failures from resource contention while still catching genuinely stuck processes. The recommended solution combines **isolated build directories**, **configurable timeouts**, **progress monitoring**, and **verify concurrency limits** to break the positive feedback loop that causes cascading failures.

## Problem Analysis

### Current Failure Mode: The Fork Bomb Cascade

The existing 300s hard timeout creates a positive feedback loop under agent concurrency:

1. **Initial contention**: 8 agents run `cargo test` simultaneously, contending on shared `target/` directory
2. **Timeout cascade**: Lock contention causes multiple agents to exceed 300s → automatic failure
3. **FLIP amplification**: Failed verify triggers FLIP evaluation, spawning additional agents
4. **Resource saturation**: New agents also hit lock contention, system spirals toward `max_agents` limit
5. **Recovery failure**: Only solution becomes killing all `wg` processes

### Root Causes Identified

1. **Shared build artifacts**: All agents share the same `target/` directory despite worktree isolation
2. **Aggressive timeout**: 300s is too short for concurrent cargo builds under lock contention
3. **Binary failure mode**: Timeout = immediate failure with no triage to distinguish contention from genuine hangs
4. **No throttling**: Unlimited concurrent verify processes amplify contention
5. **Missing configurability**: No per-task verify timeout overrides

## Design Goals (Requirements)

1. **✅ No false failures from contention**: If cargo test is waiting on file locks, not a failure
2. **✅ Detect genuinely stuck processes**: Still catch tests that are actually hung
3. **✅ Reduce contention at source**: Eliminate shared resource conflicts
4. **✅ Configurable timeouts**: Support `--verify-timeout` per-task + higher defaults
5. **✅ Break fork bomb cascade**: Implement circuit breakers to prevent positive feedback

## Recommended Solution: Multi-Layer Defense

### Strategy F: Combination Approach

We recommend implementing **all** the following layers as they address different failure modes:

#### Layer 1: Eliminate Root Cause - Isolated Build Directories

**Implementation**: Extend worktree setup to configure per-agent `CARGO_TARGET_DIR`

```bash
# In .wg/worktree-setup.sh (auto-generated if not exists)
#!/bin/bash
WORKTREE_PATH="$1"
export CARGO_TARGET_DIR="$WORKTREE_PATH/target"
mkdir -p "$CARGO_TARGET_DIR"
echo "export CARGO_TARGET_DIR=\"$CARGO_TARGET_DIR\"" >> "$WORKTREE_PATH/.bashrc"
```

**Benefits**: 
- Eliminates cargo lock contention between agents completely
- No performance penalty - agents still benefit from incremental compilation within their own workspace
- Fixes 90% of timeout cases that are actually just lock contention

#### Layer 2: Configurable Timeouts with Progress Monitoring

**Task Schema Addition**:
```rust
// In Task struct
#[serde(skip_serializing_if = "Option::is_none")]
pub verify_timeout: Option<String>, // "900s", "15m", etc.
```

**Timeout Resolution Priority**:
1. Task-specific `--verify-timeout` flag value
2. Global `WG_VERIFY_TIMEOUT` environment variable  
3. New default: **900s** (was 300s)

**Progress Monitoring**: Instead of simple elapsed time, monitor:
- **Output activity**: If no stdout/stderr for 5+ minutes → likely stuck
- **Process activity**: Use `ps` to check CPU usage patterns
- **File system activity**: Monitor cargo lock files for release patterns

#### Layer 3: Verify Process Throttling

**Concurrency Control**: Add `max_concurrent_verifies` config (default: 2)

```rust
// In coordinator config
#[serde(default = "default_max_concurrent_verifies")]
pub max_concurrent_verifies: u32,

fn default_max_concurrent_verifies() -> u32 { 2 }
```

**Queue Implementation**: 
- Verify commands queue when limit exceeded
- Priority: task with fewer previous failures goes first
- Timeout starts when verify actually begins, not when queued

#### Layer 4: Triage on Timeout

**When timeout occurs**, instead of immediate failure:

1. **Process analysis**: Check if process is genuinely stuck vs waiting
2. **Lock detection**: Look for cargo lock files that might indicate waiting
3. **Resource inspection**: Check system load and I/O wait
4. **Only fail if**: Process shows no activity AND no external waiting conditions

**Triage Implementation**:
```rust
enum TriageResult {
    GenuineHang,           // Process stuck, should fail
    WaitingOnLocks,        // Contention, should retry with longer timeout
    HighSystemLoad,        // Resource pressure, should retry later
    UnknownButActive,      // Process active, extend timeout
}
```

## Implementation Plan

### Phase 1: Foundation (1-2 days)
- **File**: `src/commands/done.rs`
- Add `verify_timeout` field to Task schema
- Update `wg add` command to accept `--verify-timeout` flag
- Implement timeout resolution priority logic
- Change default timeout from 300s to 900s

### Phase 2: Isolation (1-2 days)  
- **File**: `src/commands/spawn/worktree.rs`
- Auto-generate `worktree-setup.sh` if missing
- Ensure `CARGO_TARGET_DIR` is set in agent environment
- Test isolation prevents lock contention

### Phase 3: Throttling (2-3 days)
- **File**: `src/commands/service/coordinator.rs` 
- Add verify process tracking to coordinator state
- Implement verify queue with concurrency limits
- Add `max_concurrent_verifies` config option
- Ensure queue persistence across coordinator restarts

### Phase 4: Triage (3-4 days)
- **File**: `src/commands/done.rs`
- Implement process activity monitoring
- Add lock detection logic
- Create triage decision algorithm
- Add retry logic for "waiting" vs "hung" determination

### Phase 5: Monitoring & Tuning (1-2 days)
- Add metrics for verify timeout causes
- Tune default timeouts based on real usage patterns  
- Add logging for triage decisions
- Document new behavior and configuration options

## Migration Strategy

### Backward Compatibility

✅ **Existing `--verify` strings**: Continue to work unchanged  
✅ **Current timeout behavior**: Tasks without `--verify-timeout` use new 900s default  
✅ **Environment override**: `WG_VERIFY_TIMEOUT` still works globally  
✅ **Task authors**: No changes required to existing task descriptions

### Rollout Approach

1. **Phase 1**: Deploy with feature flags disabled, test timeout configuration
2. **Phase 2**: Enable isolation for new agents, monitor performance impact
3. **Phase 3**: Enable throttling with conservative limits (max_concurrent_verifies=4)
4. **Phase 4**: Enable triage for timeout cases, monitor false positive rates
5. **Phase 5**: Tune defaults based on real data, enable all features

### Safety Mechanisms

- **Feature toggles**: Each layer can be disabled via config if issues arise
- **Gradual rollout**: Enable for subset of tasks initially
- **Monitoring**: Track verify success/failure rates before and after changes
- **Rollback plan**: Can revert to 300s hard timeout if needed

## Alternative Approaches Evaluated

### Option A: Triage-Only Approach
**Pros**: Minimal changes, preserves current architecture  
**Cons**: Still vulnerable to lock contention, doesn't fix root cause  
**Verdict**: ❌ Insufficient - treats symptoms not cause

### Option B: Scoped Verify Only  
**Pros**: Faster verify commands reduce contention window  
**Cons**: Requires task authors to understand test structure, complex to implement  
**Verdict**: ⚠️ Future enhancement - good but not sufficient alone

### Option C: Isolated Build Dirs Only
**Pros**: Eliminates most contention, simple to implement  
**Cons**: Doesn't handle other timeout causes (genuinely slow tests)  
**Verdict**: ⚠️ Necessary but not sufficient - good foundation layer

### Option D: Longer Timeout + Progress Detection Only
**Pros**: Handles both contention and genuine hangs  
**Cons**: Still vulnerable to cascade amplification, no throttling  
**Verdict**: ⚠️ Good but incomplete - needs throttling layer

### Option E: Queue/Semaphore Only
**Pros**: Prevents cascade, simple to understand  
**Cons**: Serializes verify commands, slows overall throughput  
**Verdict**: ⚠️ Important circuit breaker but doesn't fix underlying contention

## Breaking the Feedback Loop

The recommended solution breaks the positive feedback loop at multiple points:

1. **Isolation** eliminates the contention that triggers most timeouts
2. **Higher timeouts** reduce false failures from remaining contention sources  
3. **Throttling** prevents verify command amplification during cascades
4. **Triage** distinguishes real hangs from resource waiting, reducing unnecessary FLIP triggers
5. **Progress monitoring** catches genuine hangs faster than binary timeout

## Success Metrics

### Before Implementation
- Verify failure rate: ~15-20% (estimated from logs)
- Timeout vs genuine failure ratio: ~80% timeout, 20% genuine
- FLIP trigger frequency: High during concurrent agent periods
- Agent utilization: Low due to cascade failures

### After Implementation  
- Verify failure rate: <5% target
- Timeout vs genuine failure ratio: <10% timeout, 90% genuine  
- FLIP trigger frequency: Only for actual test failures
- Agent utilization: High, sustained concurrent agents

### Monitoring Points
- Verify timeout causes (lock contention vs genuine hang vs resource pressure)
- Queue wait times for verify commands
- Triage decision accuracy (manual spot-checking)
- System resource utilization during concurrent verify operations

## Configuration Reference

### New Task Fields
```toml
[task]
verify_timeout = "15m"  # Optional, overrides global setting
```

### New Coordinator Config
```toml
[coordinator]
max_concurrent_verifies = 2  # Default 2, prevent verify amplification
verify_default_timeout = "900s"  # New default, was 300s
verify_progress_timeout = "300s"  # No output for this long = likely stuck
verify_triage_enabled = true  # Enable triage vs immediate failure
```

### Environment Variables
```bash
WG_VERIFY_TIMEOUT=1200  # Override default timeout globally
WG_VERIFY_MAX_CONCURRENT=3  # Override concurrency limit
```

## Risk Assessment

### Low Risk
- **Isolation**: Well-understood pattern, similar to existing worktree setup
- **Configuration**: Additive changes, backward compatible
- **Higher timeouts**: Conservative change that reduces false failures

### Medium Risk  
- **Throttling**: Coordinator state complexity, queue persistence needs
- **Migration**: Existing tasks need gradual rollout

### High Risk
- **Triage logic**: Complex heuristics, potential for false positives/negatives
- **Performance**: Multiple isolation directories increase disk usage

### Mitigation Strategies
- **Feature flags**: Each layer independently configurable
- **Comprehensive testing**: Unit tests for triage logic, integration tests for queuing
- **Monitoring**: Detailed metrics to detect regressions
- **Graceful degradation**: Fallback to current behavior if new systems fail

## Conclusion

The recommended multi-layer approach addresses all requirements while providing defense-in-depth against timeout failures. **Isolated build directories** eliminate the root cause of most cascades, while **configurable timeouts**, **progress monitoring**, and **verify throttling** provide safety nets for remaining edge cases.

This design breaks the fork bomb positive feedback loop at multiple points, ensuring the system remains stable under high concurrency while still catching genuinely problematic test hangs.

**Implementation priority**: Phase 1 (configuration) + Phase 2 (isolation) provide 90% of the benefit and should be implemented first. Phases 3-4 (throttling and triage) can follow based on monitoring data from the initial rollout.