# SPARK v3 Retrospective

**Date:** 2026-03-07
**Period:** 2026-02-27 to 2026-03-07 (9 days)
**Branch:** safety-mandatory-validation

---

## 1. What We Set Out to Do

SPARK v3 was the third major iteration of wg's self-organizing development cycle. The goal: make wg capable of managing its own development autonomously -- dispatching tasks, evaluating results, evolving its own prompts, and recovering from failures without human intervention.

### Planned Tracks

| Track | Scope |
|-------|-------|
| **Safety** | Fail/retry/abandon commands, service guard, zero-output detection, dependency enforcement, mandatory validation |
| **Infrastructure** | TOCTOU race fixes, liveness detection, compactor (context.md generation), atomic task mutations, per-role model routing |
| **Agency** | Evaluation pipeline, FLIP (Faithful Likelihood of Implementation Precision), evolver (prompt self-improvement), cost tracking, executor weight tiers |
| **Human** | Notification channels (Telegram, webhook), NotificationChannel trait, human-in-the-loop approval flows |
| **TUI** | Inspector views, firehose, service health badge, token display, markdown rendering, graph visualization, touch support, message input |
| **Lifecycle** | Task state machine, cascade abandon, retry+re-eval, supersession, zombie detection, convergence |
| **Observability** | Agent activity dashboard, stream events, lifecycle indicators, FLIP pipeline visibility |

### Original Scope

The task graph contained 335 total tasks (including system-generated .assign, .evaluate, and .verify-flip tasks). Of these, 108 were user-defined implementation tasks covering the tracks above, with 227 system tasks auto-generated by the coordinator for assignment, evaluation, and FLIP verification.

---

## 2. What We Actually Did

### Summary Numbers

| Metric | Value |
|--------|-------|
| Tasks completed | 323 of 335 (96.4%) |
| User tasks completed | 108 |
| System tasks completed | 215 |
| Tasks failed | 8 |
| Tasks blocked/pending | 4 |
| Commits | 446 (since project inception); ~290 in SPARK v3 period |
| Agents spawned | 703 |
| Evaluations recorded | 1,207 |
| Mean eval score | 0.803 |
| FLIP evaluations | 233 |
| Tests passing | 2,730 |
| Rust source lines | 152,202 |
| CLI commands | 89 |

### Track-by-Track Results

#### Safety (28 tasks completed)

**Shipped:**
- `wg fail`, `wg retry`, `wg abandon`, `wg pause`, `wg resume` -- full task lifecycle safety commands
- Service guard: prevents agents from stopping/pausing the coordinator
- Zero-output detection with 3-layer circuit breaker (agent-level, per-task, global API-down)
- Live dependency enforcement: recall agents with unmet deps
- `wg reset` command for clean-slate task reset
- Mandatory validation (`--verify` flag) with machine-checkable criteria
- Agent git hygiene guidelines

**Quality:** Safety subsystem scored 4/5 in autopoietic validation. All 19 unit tests and 12 CLI tests pass.

#### Infrastructure (35 tasks completed)

**Shipped:**
- TOCTOU race fix: flock-based atomic graph mutations (3-phase migration: core, command, service)
- Liveness detection: sleep-aware stuck agent handling
- Compactor MVP: 3-layer context artifact generation (code + 11 unit tests, wired into coordinator)
- Zero-output agent detection and circuit-breaking respawn
- Per-role provider routing (opus for code, haiku for eval, sonnet for FLIP inference)
- Validation prompt improvements
- Previous agent context injection on task retry
- Agent activity protocol with enriched stream events

**Quality:** Infrastructure scored 3/5. The compactor code is tested but has not yet produced runtime artifacts (no context.md generated). TOCTOU fix is solid.

#### Agency (224 tasks completed -- includes 91 .evaluate + 107 .assign + 30 .verify-flip system tasks)

**Shipped:**
- Evaluation pipeline: 1,207 evals with non-degenerate distribution (mean 0.806, 77 unique score values)
- FLIP pipeline: 233 FLIP evals, low-FLIP threshold triggers .verify-flip tasks (14 created, 12 completed)
- Evolver: consumed 831 evals, produced 12 data-driven amendments across 2 evolution runs
- Cost tracking data layer (cost_usd, token_usage on evaluations)
- Executor weight tiers: 4-tier ExecMode enum (Shell, Bare, Light, Full)
- Coordinator prompt evolution via `wg evolve`
- Self-dispatch loop with ~2s latency
- Auto-evaluate: every completed task gets evaluated without human intervention

**Quality:** Agency scored 4/5. The eval-to-evolver pipeline works but doesn't close the loop autonomously (evolver auto-trigger never fired, evolved-amendments.md is empty).

#### Human Integration (20 tasks completed)

**Shipped:**
- NotificationChannel trait and NotificationRouter
- Telegram notification channel (inline keyboards for approvals)
- Webhook notification channel (universal integration glue)
- `wg ask` command for human-in-the-loop queries

**Quality:** Scored 2/5 -- the weakest track. FLIP scored Telegram at 0.29 and webhook at 0.46. No evidence of notifications actually firing in production. The infrastructure exists but is unvalidated at runtime.

#### TUI (78 tasks completed)

**Shipped:**
- Inspector view cycling with slide animation (Alt+Left/Right)
- Firehose inspector view: combined live stream of all agent activity
- Service health badge: red/yellow/green in status bar with tap-to-inspect control panel
- Token display: novel vs cached input split (arrow/disc icons)
- Lifecycle indicators: Unicode symbols with pink coloring for agency phases
- Unified markdown rendering (pulldown-cmark + syntect syntax highlighting)
- Graph visualization: direction-aware arrowheads, dangling dep viz, outbound edge fix
- Touch drag-to-pan for graph viz (Termux/mobile support)
- Session and agent time counters
- Dynamic model registry lookups (replacing hardcoded model lists)
- Message input UX: click-to-type, draft persistence
- Remap panel cycling to Alt+arrows, i/I for viz pane resize
- Fix paste cursor position, fix scrollbar colors
- Temporal navigation

**Quality:** TUI scored 4/5. All features verified with tests. Health badge uses status bar rather than upper-right widget (aesthetic divergence from spec, functionally equivalent).

#### Lifecycle (30 tasks completed -- includes validation tasks)

**Shipped:**
- Cascade abandon with transitive propagation
- Retry + re-eval workflow
- Task supersession
- Zombie detection (11 integration tests)
- Done-blocks-on-incomplete-deps
- PendingValidation status
- Full lifecycle from creation through evaluation

**Quality:** Scored 4/5. 11 integration tests pass covering the complete lifecycle.

#### Observability (32 smoke tests + validation tasks)

**Shipped:**
- Agent activity dashboard view in TUI
- Enriched stream events with token tracking
- Lifecycle phase labels with distinct coloring
- `wg status` and `wg agents` for monitoring
- FLIP pipeline visibility

**Quality:** Operational. Smoke tests cover agency loops, notification routing, compactor, TUI graph rendering, and multi-provider routing.

### Notable Achievements

1. **Agent isolation design:** Research -> synthesis -> implementation pipeline completed. Git worktrees chosen over GitButler virtual branches. Worktree spawn, CWD isolation, and lifecycle cleanup all implemented.

2. **Zero-output detection:** 3-layer protection (agent, per-task circuit breaker, global API-down backoff) with 17 unit tests. Prevents infinite respawn loops.

3. **FLIP pipeline:** Independent second-opinion scoring that auto-generates verification tasks for low-quality work. 233 FLIP evals produced, 14 verification tasks created and dispatched.

4. **Self-dispatch loop:** Task creation to agent execution in ~2 seconds, fully automated. No human intervention needed for the common case.

5. **Evolver:** Data-driven prompt evolution consuming 831 evaluations. Produced role mutations (Programmer -> Programmer-SelfReviewing), motivation tuning (Fast -> Fast-Validated), and retirement of redundant variants.

6. **Service health badge as control panel:** Combined monitoring with actions (start/stop/pause/panic) in the TUI status bar.

7. **152K lines of Rust:** From zero to a full-featured task orchestration system with TUI, agency pipeline, evaluation, and service management.

---

## 3. What We Didn't Finish or Deferred

### Failed Tasks (8)

| Task | Reason |
|------|--------|
| .verify-flip-synthesis-tui-render | FLIP verification failed (score 0.65) |
| .evaluate-.verify-flip-implement-atomic-task | Eval of FLIP verify failed |
| .evaluate-.verify-flip-agency-executor-weight | Eval of FLIP verify failed (score 0.29) |
| .evaluate-.verify-flip-human-notificationchannel-trait-2 | Eval of FLIP verify failed (score 0.29) |
| .evaluate-.verify-flip-infra-per-role-2 | Eval of FLIP verify failed (score 0.52) |
| .evaluate-safety-self-healing | Safety self-healing eval failed |
| .evaluate-.verify-flip-toctou-phase3-service | Eval of FLIP verify failed (score 0.62) |
| .evaluate-agency-compactor-role | Compactor role eval failed |

Most failures are in the meta-evaluation layer (evaluating the evaluators). The primary tasks themselves completed successfully.

### Designed But Not Implemented

- **Graceful agent drain on service stop:** Service kills immediately rather than waiting for agents to finish current work
- **Compactor runtime artifacts:** Code and tests exist, recently wired into coordinator, but no context.md has been generated yet
- **Evolver auto-trigger:** Infrastructure exists but never fired autonomously (no evolver_state.json)
- **Notification delivery at runtime:** Channels built but never validated with actual Telegram/webhook delivery

### Known Gaps from Autopoietic Validation

1. **Learning loop doesn't close:** Evolver produces amendments, but they aren't consumed by agents (evolved-amendments.md is empty). System observes its performance but doesn't autonomously improve.
2. **38 git stashes accumulated:** Root-caused to shared working tree. Guidelines written. Worktree isolation designed and partially implemented.
3. **7 unmerged feature branches:** 139 commits across branches never merged to main. Code marked "done" based on feature branch commits.
4. **Human integration unvalidated:** Lowest-scoring track. Trait and channels exist but no runtime evidence of working notifications.

---

## 4. Process Learnings

### What Worked

**Fan-out/fan-in pattern:**
The most productive pattern was research -> design -> implementation with parallel fan-out. Example: agent isolation went through research (GitButler, git worktrees) -> synthesis (decision document) -> parallel implementation (spawn, CWD, lifecycle cleanup). Each phase produced artifacts that the next phase consumed.

**FLIP verification pipeline:**
Having an independent second opinion on task quality caught genuinely low-quality work. Tasks with FLIP < 0.70 triggered verification, and several were rejected and restarted. This created a quality feedback loop without human oversight.

**Research -> design -> implement pattern:**
42 research documents and 41 design documents were produced. Having agents research before implementing led to better decisions (e.g., choosing worktrees over GitButler, choosing flock over advisory locks).

**Evaluation-driven evolution:**
The evolver's data-driven approach -- citing specific score data to justify mutations -- produced meaningful improvements. Retiring redundant variants (Programmer-Usable, Documenter-StyleAware) kept the role space manageable.

**Conventional commit discipline:**
47.7% of commits include task ID tracing, enabling post-hoc analysis of which tasks produced which code. Strong commit message conventions (53.5% feat, 23.9% fix) make the git log readable.

### What Didn't Work

**Shared working tree conflicts:**
The #1 process failure. All agents operating in a single git worktree led to 38 stashes, cross-agent contamination, and lost work. Root cause: `spawn_agent_inner()` doesn't isolate agent file systems. Workaround: worktree isolation was designed and implemented mid-cycle, but the historical damage (stashes, unmerged branches) remains.

**Zombie agents and eval timeouts:**
Agents that produced zero output consumed spawn slots and required circuit-breaker intervention. The zero-output detector was built reactively to address this problem. Evaluation timeouts created noise in the eval pipeline.

**Unmerged branches:**
Only 2 of 9 feature branches were ever merged to main. Tasks are marked "done" when committed to a feature branch, but code doesn't reach main. This means the current state depends on which branch you're on.

**Human integration quality:**
The notification channel track produced the lowest FLIP scores (0.29-0.46). The code compiles and has structure, but there's no evidence it actually works end-to-end with real Telegram/webhook endpoints.

**Evolver auto-trigger gap:**
The evolver ran manually (via `wg evolve`) rather than auto-triggering from the coordinator. The auto-trigger infrastructure exists in code (thresholds, reactive triggers, budget caps) but `evolver_state.json` was never created. The loop from observation to self-improvement doesn't close automatically.

### How the System Self-Organized

The coordinator's 7-phase tick loop drove all automation:
1. Dead agent cleanup
2. Zero-output sweep
3. Auto-evaluate completed tasks
4. FLIP scoring
5. Spawn new agents on ready tasks
6. Compactor check
7. Evolution check (never triggered)

Phases 1-5 ran continuously. The system successfully self-organized task execution: creating .assign tasks, dispatching agents, evaluating results, and generating FLIP verification for low scores. This happened 323 times without human intervention.

Human intervention was needed for:
- Setting strategic direction (which tasks to create)
- Branch management (merging, resolving divergence)
- Running evolution manually (`wg evolve`)
- Monitoring for systemic issues (watching agent churn, checking stash accumulation)

### Cost Analysis

| Metric | Value |
|--------|-------|
| Total agents spawned | 703 |
| Total evaluations | 1,207 |
| Mean eval score | 0.803 |
| Eval cost (tracked) | Not recorded in eval files (cost_usd fields are 0) |
| Agent cost (tracked) | Not recorded in agent metadata |
| Development period | 9 days (Feb 27 - Mar 7, 2026) |

**Cost tracking gap:** While the `cost_usd` and `token_usage` fields exist on evaluation records, they are consistently zero -- the evaluation pipeline doesn't capture actual API costs. Agent metadata records executor and model but not token usage or cost. This means we cannot provide a precise cost-per-task or total-cost figure.

**Estimate based on agent count:** With 703 agents spawned (most using Claude Opus for execution, Haiku for evaluation, Sonnet for FLIP), and typical session lengths of 2-15 minutes, the total API cost is significant but untracked. The cost tracking data layer was implemented (`agency-cost-tracking-3`) but is not yet populated with real data.

---

## 5. Current State of wg

### Capabilities

wg is a functional self-organizing task orchestration system:

- **89 CLI commands** covering task management, query, analysis, agency, service, monitoring, and utilities
- **2,730 passing tests** across unit, integration, doc-test, and smoke test suites
- **152,202 lines of Rust** in the src/ directory
- **TUI dashboard** with graph visualization, inspector views, markdown rendering, and agent monitoring
- **Agency pipeline:** eval -> FLIP -> verify -> evolve (manual trigger)
- **Service management:** coordinator with automatic dispatch, dead agent cleanup, zero-output detection
- **Agent isolation:** git worktree-based isolation for concurrent agents

### Tech Debt

| Item | Severity | Notes |
|------|----------|-------|
| 38 git stashes | Medium | Root-caused, guidelines written, worktree isolation partially addresses |
| 7 unmerged branches (139 commits) | High | Code on feature branches not on main. Merging required. |
| Compactor no runtime output | Medium | Code exists, tested, wired in. Needs first run. |
| Evolver auto-trigger not active | Medium | Infrastructure exists. Needs evolver_state.json creation. |
| Empty evolved-amendments.md | Medium | Evolver produces amendments but they don't reach agent prompts. |
| Service guard tests need --test-threads=1 | Low | Unsafe env var manipulation; benign test isolation issue. |
| No graceful drain on service stop | Low | Kills immediately. Not yet a problem at current scale. |
| Cost tracking unpopulated | Medium | Fields exist but always zero. No real cost visibility. |
| Documentation staleness | High | Many docs from earlier phases, not updated for SPARK v3 changes. |

### User Experience

**For the human user:**
- `wg init` + `wg add` + `wg service start` gets a project running
- `wg status` provides a one-screen overview
- The TUI (`wg viz --tui`) shows everything: task graph, agent activity, lifecycle phases, token usage
- `wg msg` enables communication with running agents
- Setup wizard (`wg setup`) handles initial configuration

**For AI agents:**
- Rich task descriptions with validation criteria
- Context injection from dependency artifacts
- Message-based coordination between tasks
- Automatic evaluation and FLIP scoring of their output
- Worktree isolation for safe concurrent work

**Gaps:**
- No getting-started guide beyond README
- Manual pages exist but may be stale
- No website
- Agent guide exists but is 33KB -- might be overwhelming
- 89 commands with limited discoverability (grouped help helps, but steep learning curve)

---

## 6. Documentation Gap Analysis

### Inventory

| Document | Path | Size | Last Modified | Status | Priority |
|----------|------|------|---------------|--------|----------|
| **README.md** | README.md | ~3KB | Recent | Partially current. Install and basic usage correct. Missing: TUI, agency pipeline, evaluation, FLIP, service management. | Critical (user-facing) |
| **CLAUDE.md** | CLAUDE.md | 67 lines | Recent | Current. Concise project instructions for AI assistants. | Critical (agent-facing) |
| **CLI help** | `wg --help` | N/A | Current | Current. 89 commands with grouped display. Missing detailed docs for newer commands. | Critical (user-facing) |
| **COMMANDS.md** | docs/COMMANDS.md | Large | Older | Stale. Covers core commands but missing many SPARK v3 additions (evaluate, evolve, agent, firehose, etc.). | Critical (user-facing) |
| **AGENT-GUIDE.md** | docs/AGENT-GUIDE.md | 33KB | Mar 2 | Partially current. Comprehensive but may not reflect latest workflow changes. | Critical (agent-facing) |
| **AGENT-SERVICE.md** | docs/AGENT-SERVICE.md | - | Older | Likely stale. Service architecture may not reflect 7-phase coordinator tick. | Important (agent-facing) |
| **Manual** | docs/manual/*.md | 9 files | Mixed | Partially stale. Chapters 01-05 cover overview, task graph, agency, coordination, evolution. May not reflect SPARK v3 state machine changes. | Important (user-facing) |
| **AGENCY.md** | docs/AGENCY.md | - | Older | Likely stale. Pre-dates FLIP pipeline, SPARK v3 eval rubric, evolver runs. | Important (agent-facing) |
| **AGENCY_AUDIT.md** | docs/AGENCY_AUDIT.md | - | Older | Historical. May have inaccurate claims about current state. | Nice-to-have |
| **DEV.md** | docs/DEV.md | - | Older | Likely stale. Developer setup may not mention worktree isolation, agency setup. | Important (internal) |
| **WORKTREE-ISOLATION.md** | docs/WORKTREE-ISOLATION.md | 27KB | Feb 28 | Recent. Likely current for worktree isolation design. | Important (internal) |
| **LOGGING.md** | docs/LOGGING.md | - | Older | Unknown staleness. | Nice-to-have |
| **KEY_DOCS.md** | docs/KEY_DOCS.md | - | Older | Index file. May reference moved/deleted docs. | Nice-to-have |
| **COORDINATOR_ENTITY.md** | docs/COORDINATOR_ENTITY.md | - | Mar 5 | Recent. Likely current. | Important (internal) |
| **MODEL_REGISTRY.md** | docs/MODEL_REGISTRY.md | - | Mar 5 | Recent. Likely current for model routing. | Important (agent-facing) |

### Research Documents (40 files in docs/research/)

Mostly current -- produced during SPARK v3. Key documents:
- `flip-pipeline-ordering.md` (Mar 7) -- current
- `gitbutler-virtual-branches.md` (Mar 7) -- current
- `git-worktrees-agent-isolation.md` (Mar 7) -- current
- `organizational-patterns.md` -- current, referenced by agent prompts
- `human-in-the-loop-channels.md` -- current but implementation unvalidated

### Design Documents (41 files in docs/design/)

Mostly current -- produced during SPARK v3:
- `agent-isolation-implementation.md` (Mar 7) -- current
- `mandatory-validation.md` (Mar 4) -- current
- `unified-lifecycle-state-machine.md` (Mar 4) -- current
- `liveness-detection.md` (Mar 4) -- current

### Report Documents (8 files in docs/reports/ + 1 in docs/audit/)

All current -- produced in final SPARK v3 validation:
- `autopoietic-validation.md` -- capstone synthesis (current)
- `validate-core-dispatch.md` -- dispatch tests (current)
- `validate-safety-resilience.md` -- safety tests (current)
- `validate-agency-pipeline.md` -- eval/FLIP/evolver analysis (current)
- `validate-tui-observability.md` -- TUI feature verification (current)
- `agent-work-integrity.md` -- stash archaeology and root cause (current)

### What's Missing

| Gap | Priority | Notes |
|-----|----------|-------|
| Getting started tutorial | Critical | README covers install but not "build your first project" |
| TUI user guide | Critical | No documentation of TUI features, keybindings, views |
| Agency pipeline overview | Critical | No user-facing explanation of eval -> FLIP -> verify -> evolve |
| Updated COMMANDS.md | Critical | Missing ~30 commands added during SPARK v3 |
| Website | Important | No web presence (no `website/` directory) |
| Architecture overview | Important | No high-level system architecture document |
| Configuration reference | Important | 89 commands, many with config -- no centralized reference |
| Changelog / release notes | Important | No changelog tracking what changed between SPARK versions |
| Contributing guide | Nice-to-have | No CONTRIBUTING.md |
| API documentation | Nice-to-have | `cargo doc` works but no published docs |

### Priority Summary

**Critical (user-facing, must fix):**
1. README expansion (TUI, agency, service)
2. COMMANDS.md update (30+ missing commands)
3. Getting started tutorial
4. TUI user guide

**Important (agent-facing or internal):**
5. AGENT-SERVICE.md update for 7-phase coordinator
6. AGENCY.md update for FLIP, evolver, SPARK v3 rubric
7. Manual chapters review and update
8. Architecture overview document
9. Website

**Nice-to-have:**
10. Changelog
11. Contributing guide
12. Published API docs

---

## Summary

SPARK v3 accomplished its primary goal: wg can dispatch its own tasks, evaluate results, and recover from failures without human intervention. The system completed 323 of 335 tasks (96.4%) across 7 tracks with a mean evaluation score of 0.803 over 1,207 evaluations.

The key gap is the learning loop: the system observes its own performance (via evals and FLIP) but doesn't yet autonomously apply those observations to improve (evolver auto-trigger inactive, evolved amendments not consumed). Closing this loop would move wg from "self-sustaining for execution" to "genuinely self-improving."

The other major debt items are git hygiene (38 stashes, 7 unmerged branches) and documentation staleness. Both are tractable problems with clear solutions.

**Autopoietic score: 3.5/5** -- functional and largely self-sustaining, with specific gaps that prevent a higher rating.
