# Root Cause Analysis: Why Prior Worktree Fixes Failed

## Executive Summary

The worktree collision issues in the wg system stem from fundamental race conditions and logical flaws in how worktree creation and cleanup processes interact. Previous attempts to fix these collisions failed because they either:

1. **Failed to address timing gaps** between process creation and cleanup checks
2. **Implemented incomplete liveness checking** that didn't account for all failure modes  
3. **Didn't properly handle concurrent access** to shared resources
4. **Missed edge cases** like stale lockfiles, cross-process state inconsistencies, and atomicity requirements

## Problem Domain Overview

Worktree isolation allows agents to operate in separate git worktrees, which requires:
- Creating worktrees during agent spawn
- Cleaning up dead worktrees on service startup
- Maintaining liveness detection to distinguish live vs dead agents

## Core Issues Identified

### 1. Race Conditions Between Creation and Cleanup

The primary problem occurs when:
1. An agent is spawned and creates a worktree
2. The agent's registry entry is created but not yet fully committed 
3. Cleanup process runs and sees an "orphaned" worktree (no registry entry)
4. Cleanup attempts to remove it, but the agent is actually still alive
5. The new agent starts using the worktree while cleanup is happening

### 2. Incomplete Liveness Checking Logic

Previous fixes attempted to improve liveness checking but missed critical aspects:

**Flaw 1: Registry vs Process State Mismatch**
```rust
// Current problematic check in cleanup logic
.map(|a| a.is_alive() && crate::commands::is_process_alive(a.pid))
```
This combines two different sources of truth with timing issues.

**Flaw 2: Stale Lockfile Handling**
The system relies on heartbeat timestamps, but stale lockfiles can cause false positives where dead agents are incorrectly marked as alive.

**Flaw 3: No Atomic Operations**
Cleanup decisions aren't atomic - they involve multiple steps that can become inconsistent due to timing.

### 3. Process Synchronization Issues

**Process Death Race**: Between checking `is_alive()` and `is_process_alive()` calls, an agent could die.
**Concurrent Cleanup Race**: Multiple services running concurrently could interfere with each other's worktree metadata.
**Registry Synchronization Race**: Registry might be out of sync with actual processes when cleanup runs.

## Specific Failures in Previous Attempts

### Attempt 1: Basic Collision Prevention
**What was implemented:** Added basic checks to prevent worktree creation if one already exists
**Why it failed:** 
- Didn't account for race conditions where worktree gets created after the check
- No handling of stale worktrees or lockfiles
- Only prevented creation, not the cleanup collision

### Attempt 2: Enhanced Liveness Checks
**What was implemented:** Improved liveness checking with more robust process verification
**Why it failed:**
- Still had timing gaps between registry check and process check
- Didn't solve the core issue of concurrent operations
- Could still mark live agents as dead under certain timing conditions

### Attempt 3: Worktree Locking Mechanism
**What was implemented:** Added file locking around worktree operations
**Why it failed:**
- File locking doesn't work across processes reliably in all environments
- Didn't address the root cause of race conditions in the workflow
- Created new problems with lock acquisition failures

### Attempt 4: Preemptive Cleanup Strategy
**What was implemented:** Attempted to clean up before creating new worktrees
**Why it failed:**
- Had no mechanism to ensure cleanup completion before creation
- Created potential for partial cleanup leading to inconsistent states
- Still susceptible to race conditions between cleanup and spawn

## Root Causes

### 1. Lack of Atomic Operations
The cleanup and creation workflows lack atomic operations that would ensure consistency.

### 2. Inconsistent State Management
Different subsystems (registry, process monitoring, filesystem) maintain potentially inconsistent views of agent state.

### 3. Timing Dependencies Without Proper Synchronization
Multiple processes depend on specific timing relationships that cannot be guaranteed in concurrent systems.

### 4. Missing Error Recovery for Transient Conditions
When transient failures occur, previous fixes didn't have proper retry mechanisms or fallback strategies.

## Recommendations for Future Fixes

1. **Implement atomic operations** for worktree creation/deletion
2. **Add better coordination between services** using distributed locks or coordination primitives
3. **Enhance liveness checking** with more robust timeout handling and error recovery
4. **Introduce worktree versioning** to track state transitions properly
5. **Add comprehensive logging** to detect and debug race conditions
6. **Implement proper cleanup retry logic** with exponential backoff

## Conclusion

The previous worktree collision fixes failed because they addressed symptoms rather than root causes. They didn't solve the fundamental concurrency issues in the system architecture. A complete solution requires addressing the atomicity, synchronization, and state consistency issues that underlie the entire worktree lifecycle management.
