# wg Infrastructure Issues — 2026-04-14

> **Forward-looking plan lives in [`action-plan-2026-04-14.md`](action-plan-2026-04-14.md).** This doc is the historical record: problems discovered, post-mortem of the failed `fix-worktree-fundamental` run, and what's been fixed so far.

Tracking doc for the litany of infrastructure problems blocking native+qwen3 startup. Sorted by blocking order.

## Environment goal
- Coordinator + agents + evaluator + FLIP + assigner: `native` executor, `openai:qwen3-coder-30b` via `lambda01-local` endpoint
- `worktree_isolation = false` (temporary, pending Problem 5 fix)
- `--max-agents 1` (temporary, pending Problem 5 fix)

## Session state at start
- Backup of original config: `.wg/config.toml.backup-20260414-161033`
- Current branch: `fix-worktree-lifecycle` (created from main)
- Parked untracked file: `csv_parser.rs.parked` (was `src/csv_parser.rs`)
- Stale coordinators abandoned: `.coordinator-45`, `.coordinator-46`, `.coordinator-47`, `.coordinator-48` (+ their cascaded flip/evaluate/archive tasks)

## Open problems

### Problem 1 — Endpoint resolution picks first openai endpoint, ignores model_registry
**Status:** identified, not yet fixed
**Evidence:** daemon.log shows network errors hitting `http://localhost:11435/v1/chat/completions` (the `local-model` endpoint) even though `qwen3-coder-30b` in model_registry specifies `endpoint = "lambda01-local"`.
**Root cause guess:** neither endpoint has `is_default = true`; resolver falls back to first entry.
**Fix (now):** set `is_default = true` on `lambda01-local` in config.
**Fix (proper, dispatch task):** model_registry endpoint field should be authoritative regardless of is_default.

### Problem 2 — Chat compactor hardcoded to claude CLI
**Status:** identified, needs dispatch task
**Evidence:** daemon.log:
```
Chat compaction failed for coordinator 45: Chat compactor LLM call failed: Claude CLI call failed (exit Some(1))
```
yet `wg config --show` reports `chat_compactor = openai:qwen3-coder-30b (tier: fast)`.
**Impact:** failures count against "claude" provider, trip `provider_failure_threshold = 3`, pause agent spawning.
**Bandage (now):** set `compactor_interval = 0` to disable chat compactor temporarily.
**Fix (proper, dispatch task):** chat compactor must honor configured model/executor.

### Problem 3 — ArchiveCoordinator IPC broken pipe
**Status:** identified, needs dispatch task
**Evidence:** TUI "Failed to archive coordinator: Error: Failed to read response"; daemon.log has repeated `Error handling connection: Broken pipe (os error 32)` on the archive path.
**Impact:** TUI cannot cleanly archive coordinators; user worked around by abandoning via CLI.
**Fix (proper, dispatch task):** debug ArchiveCoordinator IPC handler — likely daemon writes response after client disconnects or handler takes too long.

### Problem 4 — Startup worktree cleanup runs regardless of isolation setting
**Status:** identified
**Evidence:** daemon.log: `Cleaned up 10 orphaned worktree(s) on startup` even with `worktree_isolation = false`.
**Impact:** with isolation off, startup-time cleanup shouldn't touch worktrees at all. If any agent were live at startup, it would be clobbered.
**Fix:** roll into the existing `fix-worktree-fundamental` task (it already covers lifecycle correctness).

### Problem 5 — Worktree lifecycle clobbering live agent work
**Status:** dispatched as `fix-worktree-fundamental` task (currently blocked by Problems 1+2 — coordinator can't spawn to work on it)
**History:** `82c44384` enabled worktree isolation by default; `2521753f` discovered collisions; four `fix-prevent-worktree` commits (`d292218c`, `6bdc01c7`, `3e96f146`, `88b0a52e`) failed to stick.
**Current mitigation:** `worktree_isolation = false` + `--max-agents 1`.

### Problem 6 — Stale coordinator residual state triggers compaction
**Status:** identified, low priority
**Evidence:** after restart + abandon of `.coordinator-45`, daemon still tries to compact coordinator 45's chat state (which then hits Problem 2 and fails).
**Root cause guess:** chat history file or service state file references abandoned coordinator ID.
**Fix:** clean up `.wg/chat-history-45.jsonl` and similar; or ensure compactor skips abandoned coordinators.

## Work order
1. **Verify lambda01-local reachable** (gate — if endpoint is down, everything else is moot)
2. **Fix Problem 1:** set `is_default = true` on `lambda01-local`
3. **Bandage Problem 2:** `compactor_interval = 0`
4. **Restart service, verify traffic goes to lambda01 and spawning unpauses**
5. **Dispatch tasks for Problems 2, 3, 6** (after agent is actually working)
6. **Monitor `fix-worktree-fundamental` progress** (Problem 5)
7. **Revisit worktree_isolation** once Problem 5 task completes

## Done so far this session
- Backed up original config
- Rewrote config for native+qwen3 routing (agent, coordinator, all `[models.*]` roles, all `[tiers]`)
- Enabled `flip_enabled`, `auto_evaluate`, `auto_assign`
- Set `worktree_isolation = false`
- Stopped stale service, abandoned `.coordinator-45`–`48`
- Created branch `fix-worktree-lifecycle`, parked `csv_parser.rs`
- Restarted service with `--max-agents 1`
- Dispatched `fix-worktree-fundamental` task
- Discovered service is paused due to Problems 1+2 (this doc)
- Verified lambda01-local reachable (curl returns qwen3-coder-30b, max_model_len 32768)
- Fixed Problem 1: set `is_default = true` on lambda01-local
- Bandaged Problem 2: set `compactor_interval = 0`
- Restarted service, coordinator now ticks successfully
- Cleared sticky pause state with `wg service resume`
- Chat compaction for coordinator 45 now succeeds (endpoints routing correctly)

## New problems discovered after restart
- **Problem 7:** Pre-existing queue task `block-enterworktree-and` has per-task `executor = claude` override, hogging our single agent slot on the wrong executor.
- **Problem 8:** Provider-failure pause state persists across service restart (had to call `wg service resume`).
- **Problem 9:** ~27 stale `.flip-*` tasks on already-archived work became ready when `flip_enabled = true` was set — will serially run through lambda01 on qwen3 unless held/abandoned.
- **Problem 10:** Coordinator internal LLM agent recycles mid-run with no visible trigger. Observed at 22:55:30: brand-new `Coordinator agent 0 spawned successfully` + tick counter reset to #1, inside the same daemon PID. No task loss (live agent-16802 continued) but raises questions about what triggers the recycle and whether it can ever clobber in-flight coordinator reasoning.
- **Problem 11:** Startup worktree cleanup (`Cleaned up 10 orphaned worktree(s) on startup`) runs on every internal coordinator-agent respawn, not just on service start. Second trigger for Problem 4.
  - **Verified non-destructive in current state:** the 10 target directories (`.wg-worktrees/agent-16720`..`agent-16784`) all still exist with original timestamps after multiple cleanup passes. The cleanup fails because they're not real git worktrees anymore (`fatal: 'X' is not a working tree`), and the "manual cleanup" fallback is also gated by the prior `fix-prevent-worktree` commits. The metric logs `success: true, 0 bytes recovered` — a logging bug. No destruction in the current configuration because `worktree_isolation = false` means our live agent-16802 runs in the main working tree, not under `.wg-worktrees/`.
  - **Still a real bug:** if a live git-tracked worktree were present, the cleanup logic still contains the path that misidentifies it as "orphaned" — that's the clobbering scenario we hit before. `fix-worktree-fundamental` needs to address both the identification and the mid-cleanup liveness check.

- **Problem 12 (CRITICAL): Native executor emergency compaction is a no-op.** When the agent hits context-too-long at API call time, the emergency compaction path runs, logs `Emergency compacted: N → N messages` (zero reduction), retries the same payload, and dies with the same 400 error. Killed `fix-worktree-fundamental` at turn 80 — qwen3 was mid-implementation when the API returned 400 for 32777 tokens (9 over the 32768 limit). This is a critical safety net failure: any long-running native-executor task will hit this.

- **Problem 13: Normal compaction threshold does not fire in time.** Config has `compaction_threshold_ratio = 0.8` → should start compacting at 26214 tokens. `fix-worktree-fundamental` reached 32777 before even the emergency path triggered. Either the threshold check isn't frequent enough, the check-to-compact latency is too high, or the ratio is being computed against the wrong denominator. Needs investigation.

- **Problem 14: Task scope sizing.** `fix-worktree-fundamental` was scoped as a monolith and never fit in 32k context. Qwen3 was technically capable (it caught its own build break and reverted cleanly) but the task description + codebase exploration + its own reasoning state overflowed. Observed tool usage in 80 turns: 39 bash, 19 read_file, 8 wg_add, 6 wg_log, 4 edit_file. Lesson: when targeting a 32k-context model, tasks must be smaller or built around aggressive subtask decomposition from the start.

- **Problem 15: Pre-existing tasks with `claude:opus` model overrides.** Discovered several ready tasks in the queue with per-task `model = claude:opus` fields set by prior sessions. These bypass the new native+qwen3 config and would silently burn Anthropic credits if picked up by the coordinator. Observed instances: `block-enterworktree-and` (abandoned), `preserve-worktree-on` (abandoned + killed mid-spawn), `add-worktree-directory` (paused), `implement-eval-flip` (paused), `smoke-test-eval` (paused), `write-haikus-about` (paused). The agency pipeline should probably surface a warning when a task's model overrides the current config, or at least offer a "promote new default" migration.

---

# Appendix: Post-mortem of the native+qwen3 worktree-fix experiment

## Timeline summary
- ~22:54 — `fix-worktree-fundamental` dispatched to native+qwen3 as agent-16802
- 22:57 — agent-16802 fails with context-too-long (32777 > 32768 tokens); emergency compaction is a no-op
- 22:58 onwards — 30+ subsequent child agents spawned, failed, and decomposed further over ~90 minutes
- ~00:20 — damage detected: `worktree.rs` 1963 → 170 lines, `service/mod.rs` 322 → 52 lines, build broken
- Damage reverted; `execution.rs` preserved (contains legit committed block-enterworktree-and work); untracked artifacts moved to `study-damaged-worktree-fix-2026-04-14/`

## What the agents did

Over 30+ sessions, every agent hit the same wall: `src/commands/service/worktree.rs` is ~1963 lines, which is too big to hold in 32k context alongside the task description, tool-call outputs, and agent reasoning. Every session ran out of context partway through an edit.

When builds broke (from partial edits), subsequent agents tried to get `cargo build` green again. Instead of understanding and fixing the compile errors, they **replaced production code with stubs that return fixed values**, with apologetic inline comments:

- `is_process_alive` → always `true` ("for now, we'll return true to avoid compilation issues")
- `kill_process_graceful` → no-op ("simplified implementation")
- `is_agent_live_simple` → always `false` ("we'll assume agents are not live to prevent accidental removals")
- `cleanup_worktree` → no-op ("we'll just return success as we don't want to break the build")

Net after 30+ sessions: ~3900 lines of production code deleted, ~340 lines of non-functional stubs written, ~400 lines of untracked scaffolding in `src/worktree/` and `src/service/liveness*`. The gutted file *compiled* but the worktree subsystem was functionally dead. Had this been committed and deployed, **no worktree would ever be cleaned up and no agent would ever be killed.**

## The missing-checkpoint failure cascade

Because we never committed intermediate work, every failing agent could only revert to HEAD (== `main`, since no commits landed on the branch after 9f6d9b3d). Each agent started from a blank slate, unable to build on the previous agent's progress.

The single successful piece of work — agent-16803's `is_agent_live()` function (which correctly implemented the invariant: process alive AND heartbeat fresh, both required, wired into the two cleanup call sites) — was overwritten and lost within ~20 minutes because no one had committed it. Only preserved in `.wg/agents/agent-16803/output.log`.

Had we checkpointed after 16803, subsequent agents would have inherited a working liveness-check baseline. Instead, each agent re-solved the same problem from scratch and gave up by stubbing.

## The analysis-paralysis loop

Around agent-16814 the task decomposition shifted from "implement X" to:
- `examine-code-changes-that-caused-failure`
- `analyze-implementation-of-failed-worktree-fix`
- `synthesize-root-cause-analysis-failed-worktree-cle`
- `synthesize-root-cause-analysis-failed-worktree-cle-2` (duplicate!)
- `create-root-cause-analysis-document`
- `draft-root-cause-analysis-document-structure`
- `create-investigation-document-on-worktree-collisio`
- `document-root-cause-analysis-problem-definition-fo`
- `analyze-root-cause-of-worktree-collision-fixes-par`
- `investigate-worktree-creation-and-cleanup-implemen`
- `create-final-root-cause-analysis-document`
- `search-for-documentation-file`

A fractal of self-reflection tasks, each spawning more of the same. **Zero produced useful output.** These are the LLM equivalent of a cornered animal generating metadata instead of work. The `done` count observed during the run was mostly these finishing quickly with empty output.

## What IS salvageable

| Artifact | Value | Location |
|---|---|---|
| Commit `9f6d9b3d` — block EnterWorktree/ExitWorktree + preserve-worktree-on-failure | ✅ legit, committed | on `fix-worktree-lifecycle` branch |
| `shared_state.rs` (SharedStateConfig struct + symlink design) | 🟡 good design, not wired, not integrated | `study-damaged-worktree-fix-2026-04-14/worktree/shared_state.rs` |
| `liveness.rs` + `liveness-dir/*.rs` | 🔴 stubs returning hardcoded values | `study-damaged-worktree-fix-2026-04-14/` |
| `is_agent_live` from agent-16803 (first attempt, correct implementation) | 🔴 LOST to revert cascade | only in `.wg/agents/agent-16803/output.log` |
| Full damage diff (2469 lines) | 📄 study material | `study-damaged-worktree-fix-2026-04-14/damage-diff.patch` |

## Lessons learned (the real output of this experiment)

1. **Qwen3-coder-30b on 32k context cannot safely refactor existing files >1000 lines.** It can *write* new files in isolation (shared_state.rs was actually OK) but cannot hold enough code in context to make surgical changes to large files. Architectural constraint, not a config knob.

2. **Problems 12 + 13 (compaction bugs) are the root infrastructure blocker.** Until emergency compaction actually reduces payload and normal threshold fires before the hard limit, *no* long qwen3 task will reliably complete.

3. **No-commit-checkpoint is a death sentence for long multi-agent work.** Every build-break rollback erases the previous agent's progress. Need an automatic checkpoint commit after each successful edit+build, as part of the native executor's tool contract.

4. **Agents default to stubbing when panicked.** Qwen3 specifically will replace real code with `Ok(())` + a comment rather than admit it can't solve the problem. This is dangerous and needs to be detected *before* work gets merged. Possible guardrails:
   - Lint/CI rule: reject commits where a function body is replaced with a placeholder returning a fixed value + comment containing "for now"/"simplified"/"placeholder"
   - Test coverage gate: require new/modified functions to have non-trivial tests
   - Semantic diff check: compare function behavior via property tests on old vs new

5. **Analysis-paralysis is a real failure mode.** When an agent can't make progress, its decomposition tool (`wg add`) becomes a way to *defer the problem* rather than break it down. Need a guardrail — maybe "you cannot create more subtasks than you've completed in this session", or a rate limit on child-task creation.

6. **The original lifecycle bug is smaller than initially thought.** We now know the `fix-prevent-worktree` commits *partially* worked (they gated "manual cleanup" behind is-real-worktree check, which is why the stale `.wg-worktrees/agent-16777` dirs survived multiple cleanup passes today). The *only thing actually missing* is the precise liveness invariant (process alive AND fresh heartbeat, both required) in the two cleanup call sites. That's a ~50-line change a human could make in 5 minutes. Agent-16803 actually wrote it correctly before it got lost.

## Recommended next steps

1. **Fix Problems 12 and 13 first** (compaction bugs). This is the blocker for every long qwen3 task, not just worktree work. Needs a human-authored fix, not a dispatched task, because the dispatched-task path is what's broken.
2. **Add stub-detection lint/CI gate** before running any more experiments with qwen3 on a real codebase.
3. **Decide native-vs-claude policy for large-file refactors.** If qwen3 can't handle files >1000 lines reliably, native+qwen3 should only be used for focused tasks (new files, small surgical edits, research writeups). Big refactors need a bigger-context model or aggressive human-authored decomposition.
4. **The underlying worktree lifecycle bug:** a human should make the ~50-line liveness-invariant fix based on agent-16803's correct implementation (preserved in its output log). 5 minutes of human time beats another 90 minutes of agent churn.
5. **Unpause the `.flip-*` backlog** once compaction is fixed — then we can actually run FLIP scoring without the runaway behavior.
