# Optimization log — running scores ledger

Chronological record of eval-driven prompt-optimization runs on flow-next skills/agents.
**Append a row when a mutation is kept OR deliberately discarded** — the discards are as
valuable as the wins (they stop the next agent re-running a dead end). Methodology:
[`optimizing-skills.md`](optimizing-skills.md). Harnesses live under `optimization/<target>/`.

Columns: **quality** = accuracy/coverage/detection metric (the eval that guards against
silent regression); **efficiency** = prompt or output tokens; **status** = kept / held
(no change, guarded a trim) / discarded / shipped.

| date | target | lever | quality | efficiency | status | notes |
|---|---|---|---|---|---|---|
| 2026-07-04 | `prospect` skill (fn-84.6, ~10.3k tok) | new suite: Phase-3 critique classification (7-slug taxonomy) + over-reject guard + taxonomy-precision lever; 2 batches, facts-only, DESIGN-REVIEWED pre-run | **EARNED 4/4**: E1 9/9 (slugs precise, strategy cites verbatim, dups anchored to fn-70), E2 over-reject guard CLEAN (5 good kept, 17%<40% floor, no padding), E4 precision 4/4 | no change (0 tok) | **held / skipped** | run-trick: blind Phase-3 EMISSION at sonnet. Pre-run fable design review caught 2 CRIT+3 MAJOR before any run (the fn-84.5 process working). TWO subtle key imprecisions caught + fixed transparently: C2-4 (intra-vs-cross-run grounding, caught by the RUN) + C1-3 (double-defect made E4 ill-posed, caught by the POST-run fable QA — MAJOR; replaced with a single-defect on-mission-XL candidate, re-ran -> too-large, E4 genuinely earned). Quality lever + trim honestly discarded. Both a pre-run design review AND a post-run QA review (fable) gated this suite. |
| 2026-07-04 | `make-pr` skill (fn-84.4b, review-ease) | CAPABILITY add (not optimization): 4 render-only sections from unused payload fields — Not-in-this-PR (boundaries), Verification (tests + honest test-gap), Review plan (attention-budget file buckets), R-ID inferred chip | **E1-E10 15/15** — E7-E10 new capabilities render faithfully (risky: careful surface ~707/940, honest flowctl.py test-gap, R15 inferred chip), E1-E6 NO regression | +1280 tok (capability, quality↑) | **kept** | Gordon steer 'go further — easier to review'; fable DESIGN review first (the new process). Ports the HTML lens §5.4 review-intent ordering to the always-on body. Deferred to fn-86: removed-export refs + evidence.files traceability (need flowctl). |
| 2026-07-04 | `audit` skill (fn-84.5, ~14.9k tok) | new suite: classification (Keep/Update/Consolidate/Replace/Delete) + over-flag guard + consolidate-vs-delete lever; 3 stores incl. a STRESS store | **EARNED 4/4 AFTER a fable-review-forced fixture rewrite** (the first baseline was spoon-fed — frozen contexts stated conclusions; rewritten to RAW FACTS). E1 6/6 (incl. M1-c answer-key CORRECTION Delete->mark-stale — audit was right, my key wrong), E2 over-flag 0 FP incl. hard M3-a, E4 consolidate both directions | no change (0 tok) | **held / considered-and-skipped** | run-trick: `mode:autofix` EMISSION at `sonnet` (per-entry verdicts vs a frozen answer key + frozen codebase context). Delete-vs-mark-stale lever DISCARD+REVERTED (chased my own mis-key, not a real gap); quality lever discarded; trim inspection-backed-skipped (fn-N=rule-carrying, calibration=proximity-load-bearing, fn-82 already −549 tok). Added an M3 stress store so 'ceiling' is earned not spoon-fed (fn-84.4 lesson). Worktree mechanic (Major-1) still deferred — audit is write-happy but classification is emission-scorable. |
| 2026-07-04 | `make-pr` skill (fn-84.4, heaviest prompt ~35k tok) | re-baseline on current main + add E6 Where-to-look quality eval (+ a risk-differentiated `payload-risky` fixture); test BOTH axes — quality lever + efficiency trim | **held on both**: E1-E5 behavioral 10/10 (rich+risky); E6 Where-to-look 1/1 (renders risk-led Security bullets when `security_sensitive_paths[]` set) | **efficiency: -189 tok kept, MEASURED** (rationale asides + a STRUCTURAL omission-clause dedup vs the §2.13 table; body held rich E1-E5 + risky E1-E6; boundary = per-section unique imperatives like §2.8's 'no fallback line') | **kept (trim) + held/discarded (quality)** | harness `optimization/make-pr/`. QUALITY: Where-to-look already risk-prioritized by design (5 field-triggered categories Arch->Security->...->Tests) -> lever discarded, no headroom; the hypothesized gap was a FIXTURE artifact (pre-fn-84 payloads had no risk-differentiated diff). EFFICIENCY (fable-review-corrected: measured, not asserted): -189 tok kept incl. a structural §2.13 dedup; fn-82's fold already took the -4.5k win; the measured boundary is the per-section unique imperatives + field-rules (accuracy-risky per-section backlog). Fixture made exporter-faithful. Run-trick: `--dry-run` render at `sonnet`. |
| 2026-07-04 | `interview` skill (fn-84.3) | quality lever: restraint/prune cue ('match count to completeness; prune [high]-confidence + pure-taste questions') at SKILL.md Question Order (LEAN +6 lines) | **fixed the target** — I4 E5 fail→PASS (over-asking 5 Qs → 3 sharp), tightened I3 (E4 noisy→solid N=2); BUT NOT zero-regression — on the thin fixture I1 the prune cue DROPPED the symlink/Windows NFR probes (bled from well-specified into under-specified specs); E1–E3 accuracy floor held 12/12 but a real qualitative coverage loss | +144 tok | **discarded (reverted)** | harness `optimization/interview/`. Run-trick: question EMISSION at `sonnet`, **judged by an independent `fable` subagent** (E4 NFR-coverage + E5 question-quality) per user steer. Directionally correct but the auditable aggregate quality stayed 7/8→7/8 — the E5 'is-this-padding' judgment is ~50%-noisy on hard fixtures (baseline I3 itself flipped 5/5→3/5 across N=2), so a keep would launder a flat aggregate (Major-3 violation). FOLLOW-UP: majority-vote E5 at N≥5 / sealed holdout to ratchet subjective question-quality changes. Diagnosed real blind spot: interview over-asks on well-specified specs (its 'expect 40+' bias). |
| 2026-07-04 | `capture` skill (fn-84.2) | re-baseline on current main + add E6 `[inferred]`-tagging-accuracy eval; attempt [inferred]-discipline quality lever + a non-routing trim | **held 18/18** (accuracy 15/15 + new E6 quality 3/3): C1 6/6 (N=2), C2 6/6 (cat-3 → outcome-AC + Motivation, SUBSTRUCTURED), C3 6/6 (override refused, exit 2, hand-edit preserved) | no change (0 tok) | **held / considered-and-skipped** | ceiling skill — quality lever discarded (E6 at ceiling, no headroom; conservative tagging is the safe direction); trim discarded (routing tables load-bearing per pre-fn-84 exp1; only non-routing candidate §5.0 heredoc is unexercised by the fixtures → weak-ratchet). Confirms fn-82 dedupe did not regress capture. |
| 2026-07-03 | `plan` skill (fn-84.1) | quality lever: "**finalization folds into one task**" + "7+ is a ceiling, not a floor" cue at `steps.md` Task Sizing (proximity, LEAN +6 lines) | **accuracy 15 → 16/16** — E3 (task sizing) fail→PASS on the P4 ordering/sizing-stress fixture (over-split docs+CI finalization → one task), confirmed N=2; E5 ordering held 4/4 | **+117 tok** | **kept** | harness `optimization/plan/`. **PIVOT:** the a-priori lever was dependency-ordering (E5), but the baseline showed ordering already correct on all 4 fixtures incl. the stress case — the DATA redirected the lever to the E3 sizing gap (both pre-authored evals → Major-B clean). Run-trick: frozen-research emission at `sonnet`; P2 = non-flow-next (DocIQ-Sphere) anti-overfit fixture; P3 = override-respect (hand-edited spec preserved). |
| 2026-07-03 | `plan` skill (fn-84.1) | prompt trim: compact `examples.md`'s verbose BAD full-code example blocks (~500 tok) | n/a — **unverifiable by this suite** | ~−500 tok potential | **considered-and-skipped** | the trim governs the "no implementation code in the emitted spec" behavior that NO eval here scores; and the emission run-trick only exercises the authoring-judgment surface (Steps 2–5), so any trim outside it "holds 16/16" **trivially** = weak-ratchet false hold. Keeping an unverifiable trim is the "ratchet worse than none" failure fn-84 forbids. **FOLLOW-UP:** add an E6 "no-code" eval + research/mermaid path coverage, re-baseline (Major-B), then re-attempt. |
| 2026-07-03 | worker Phase-1 re-anchor (`flowctl anchor` single-call bundle vs the ~8 discrete reads in worker.md) | round-trip reduction: bundle assembled from the verbatim captured stdout of the same production `cmd_*` functions; worker.md Phase 1 → ONE call | **PASS — zero loss, twice**: comprehension-equivalence eval **bundle 7/7 ≥ status-quo 7/7** on 3 frozen real tasks (incl. a non-satisfies-section question) + deterministic byte-for-byte superset test locked in CI (`test_anchor_bundle.py`) | ~8 CLI/git round-trips → **1** per worker start (content identical by construction — the win is round-trips/latency, not token count) | **shipped** (fn-83.3 command, fn-83.4 wiring) | harness: `optimization/worker-anchor/`. Bundle is a floor, not a ceiling: memory keyword-search + all further reads retained; fail-open per section. New proof shape for pure packaging mutations — see `optimizing-skills.md` "comprehension-equivalence". |
| 2026-07-03 | plan-sync gate (`flowctl plan-sync-probe` skip-gating of the post-task plan-sync spawn) | cross-repo ship gate (fn-83.6): 27 replayed real-history scenarios from DocIQ-Sphere/gno/transcribe at pinned SHAs, frozen real-agent keys (opus, N=3, wobble⇒drift) | **FAIL** — probe SAFE on this repo's corpus (0/14 false skips, fn-83.2) but cross-repo: **1 false skip** (transcribe fn-25.3 — production plan-sync provably updated the downstream body at that exact state) | true-negative skip-rate **1/15 = 6.7%** (need ≥50%) — the gate would pay ~nothing | **discarded / shelved** (gate NOT shipped; fn-83.4 keeps the unconditional spawn and REMOVED the probe/ledger/config from the shipped CLI; harnesses archived under `optimization/plan-sync-gate/`) | "monolith vs sliced" hypothesis dead: feature-sliced multi-stack DocIQ also 0% skips. Provable disjointness ~doesn't exist in real task graphs (downstream bodies genuinely name touched dirs/files/symbols); the one false skip landed in the stated deviation-only residual on the FIRST skip opportunity. Full verdict: `optimization/plan-sync-gate/cross-repo/README.md`. **Do-not-re-attempt decision record: `.flow/memory/knowledge/decisions/plan-sync-skip-gate-not-viable-2026-07-03.md`.** |
| 2026-07-02 | `work` + `pilot` skills — progressive-disclosure gating | move default-OFF machinery behind a forcing-sentinel gate into `references/*.md` (work tracker-touchpoints; pilot qa-stage freshness probe) + backlog-mode prose single-sourced to `backlog-mode.md` | default (bridge-inactive / qa-off) path behavior byte-unchanged; gate fails **open** on probe/parse error (Read still forced); Phase-5 sync-check + 4-state summary + qa Phase 5/6 routing kept inline (safety net) | **work −984 tok, pilot −2207 tok = ~3.2k tok/run** off the always-loaded set (never loaded on the default path) | **kept** (fn-82.1) | gate skeleton: `RAW="$(<probe> 2>/dev/null)" \|\| ACTIVE=1` on BOTH probe and parse (no unguarded `\| jq`), one-level link, default branch bare no-op. New `references/*.md` auto-mirror to Codex (sync-codex wholesale dir copy). Early-proof task — validated the pattern before the dedupe classes ran. |
| 2026-07-02 | `tracker-sync` + `qa` + `map`/`memory-migrate` skills — Class-1 archaeology | strip build-time `fn-N`/`fn-N.M` provenance + flowctl.py absolute line-refs from always-loaded prose (allowlist: keep `R\d+`, `S-[A-Z]` oracles, versions); relocate adapter round-trip spike harnesses to `agent_docs/tracker-sync-spikes.md` (dev archive, NOT shipped/mirrored) | no behavior — dead-text removal only; strip-allowlist protected rule-carrying tags + live S-A…S-G fixtures; qa's fn-53 ownership scaffolding + transient proof-receipt removed (§6.3 qa_verdict always runs) | **tracker-sync −182 tok, qa −720 tok** always-loaded (+ map/memory-migrate on-demand-ref dead-text cleanup) | **kept** (fn-82.2) | ~93 fn-NN citations swept from tracker-sync SKILL.md+steps.md; line-number refs → constant/function names (survives refactors). Spikes moved OUT of the plugin tree (jira had none). |
| 2026-07-02 | `impl-review`/`spec-completion-review`/`interview`/`audit`/`prospect` skills — Class-3 intra-skill dedupe | collapse duplicated blocks to one authoritative site, keep short imperative rules repeated at action sites (verbatim, never paraphrase) | review skills resolve the backend in exactly ONE place (workflow-common Phase 0 — kills the double `review-backend` round-trip); interview template-cascade walker survives byte-for-byte (case-insensitive-FS + 4-tier fallback) with both cross-refs repointed; audit Replace/supersede single-sourced to phases.md; prospect python-picker defined once | **impl-review −132, spec-compl −123, interview −481, audit −549, prospect −192 ≈ −1.5k tok** always-loaded | **kept** (fn-82.3) | dedupe rule: merge *explanatory/context* blocks; KEEP load-bearing imperative repetition. Proximity rule honored (no routing/taxonomy table relocated out of its phase). |
| 2026-07-02 | `make-pr` skill (~31k tok) | fold phases.md Done-when checklists inline into workflow.md; stub + un-force-load phases.md | body held **5/5** (behavioral + accuracy evals, payload-rich; fresh baseline row recorded first) | **−18.1KB (~4.5k tok)/run** off the always-loaded set | **kept** (fn-82.4) | fold-not-gate: checklists are consumed every run. workflow.md +6.4KB (phases.md detail folded verbatim, incl. new Phase-1 block + Phase-0/1.5 failure modes); phases.md 24.5KB → 1.4KB stub (stable link target — deletion forbidden); SKILL.md force-load + `$FLOWCTL` preamble swept. Stale phases.md-only "Body ≤8000 chars" invariant (contradicted §4.4's 65,000) died with the stub. |
| 2026-07-02 | `capture` skill | far-copy dedupe: biz-routing table single-sourced at the CONSUMER (workflow.md §2.6) | held **15/15** incl. C2 cat-3 → outcome-AC + `### Motivation` (SUBSTRUCTURED) and C3 override refusal (exit 2); fresh baseline row recorded first | −2.5KB (~630 tok)/run off the always-loaded pair | **kept** (fn-82.4) | inverse of the reverted ~2026-06 DRY trim: the survivor sits BESIDE the drafting step; phases.md's far copy → pointer. Trigger-phrasing column + the 2 phases-only rules folded into §2.6 verbatim. |
| 2026-07-02 | `capture` skill | source-tag taxonomy + forbidden-behaviors recap dedupe | — | — | **considered-and-skipped** (fn-82.4) | taxonomy copies are complementary, not duplicates (workflow §2.1 drafting-format w/ examples vs phases.md calibration w/ acceptance-tests + when-to-use-which) — merging = column restructure/paraphrase, out of scope; forbidden-behaviors recap = short imperative rules repeated at an action site, which the dedupe rule explicitly KEEPS (and the recap carries a readiness row absent from SKILL.md's list). When in doubt, don't dedupe. |
| 2026-07-02 | 12 hot-path skills — runtime plumbing (capture, interview, plan, plan-review, impl-review, spec-completion-review, export-context, deps, make-pr, tracker-sync, resolve-pr, work) | single-emission writes (Write render = read-back) + file composition for RP prompts + single-entry review responses + round-trip dedupe | read-back contract fully preserved (full draft in Write render; full-file Read per edit cycle); guards ADDED (fix-loop cap all 4 backends, snapshot-scoped staging, injection-free prompt assembly); smoke 138/138 + pytest 1393 green | **11 full-content re-emission sites + 13 redundant CLI round-trips removed** (computed from fn-81.1–.3 diffs: 2 capture heredoc re-authorings + autofix re-print + 7 `[PASTE]`/content-retype prompt sites + tracker-sync merged-body re-emit; 7 double config-gets → `LEAF`, plan post-write `show`+`cat` + dup `show`, interview dup fetch, deps' 2nd N-call per-spec loop, make-pr happy-path `gh pr view`) | **shipped** (fn-81) | survey-driven (all 28 skills), not eval-harness. land/drive/strategy/sync/ralph-init surveyed clean; deliberate re-probes kept (land fn-66 R3 merge-evidence probe, pilot pre/post-dispatch `gh pr list` pair). Path-persistence rule: vars die across tool calls — draft/prompt paths are literal agent-composed uniques. |
| 2026-07-01 | `impl-review` prompt (all backends) | code-smell baseline + rubric trim + output-format trim | detection **7 → 10/10** on ground-truth corpus (smells 2.5 → 5/5; correctness 5/5 held) | prompt **−27% (−950 tok)**; output −16% | **shipped** (fn-74, PR #184 `47068f9c`) | `optimization/review-prompt/`. Baseline reliably missed Feature Envy / Data Clumps / Primitive Obsession (0/4). No over-flag on clean code. Validated codex + RP (GPT-5.5-high). |
| 2026-07-01 | `impl-review` — full 14-smell list | broad smell list | same detection as 8-smell | +75 tok vs lean | **discarded** | the 6 rare smells (Shotgun Surgery, Message Chains, Middle Man, …) added tokens, no detection. Lean 8-smell won. |
| 2026-07-01 | `plan-review` prompt (all backends) | targeted 4-item spec-quality checklist | detection **8.0 → 9.3/10** (test strategy **0/3 → 3/3**, observability 1/3 → 3/3) | +74 tok (trim already applied) | **shipped** (fn-74, PR #184 `611a77b2`) | plan reviewer already strong; checklist targets its blind spots. P6 (subtle task-ordering) stays hard (1/3). No over-flag. |
| 2026-07-01 | `plan-review` — broad 11-item checklist | broad list | 9.0/10 (< lean's 9.7); **regressed** task-ordering 2→1 | +181 tok | **discarded** | broad list *diluted* focus — the lean, targeted 4-item version beat it on quality AND cost. Less-is-more (2nd instance). |
| ~2026-06 | `repo-scout` agent | output budget | eval set 83% → 100%, accuracy held | output **~40–50% smaller** | shipped | free-form scout prose → planner. The output-budget lever's home turf. |
| ~2026-06 | `context-scout` agent | output budget | accuracy held | output **60–70% leaner** | shipped | ditto. |
| ~2026-06 | `flow-gap-analyst` agent | output budget (per-item verbosity, not item count) | 26/27 gaps preserved | output **50–70% leaner** | shipped | proof the lever generalizes past scouts. Coverage answer-key = the no-feature-loss guard. |
| ~2026-06 | `capture` skill | DRY trim (relocate routing tables) | 15/15 → **14/15** (Decision Context flattened) | — | **discarded** (reverted) | proximity is load-bearing: a routing/taxonomy table beside the step that uses it is applied more reliably. Do NOT relocate. |
| ~2026-06 | `make-pr` skill (~31k tok) | prompt trim | body held 5/5 | **~170 tok** (stale fn-42 archaeology only) | kept (modest) | mostly load-bearing render prose; deeper trims are accuracy-risky per-section work. |

## Standing lessons (distilled from the rows)

- **Less-is-more, twice.** A lean/targeted list beat a broad one on both quality and cost
  (impl 8-vs-14 smells; plan 4-vs-11 checklist). Broad lists dilute the model's focus.
- **Over-flag guard is mandatory for "find X" prompts.** A quality lever that catches more
  on bad input must be checked on *clean* input — the fn-74 winners added valid depth, not
  noise (finding-rate ≈ baseline, `false-missing == 0`).
- **Validate cross-backend** for anything feeding `build_review_prompt` (codex/copilot/
  cursor) — and remember RP keeps a **parallel rubric copy** in the skill markdown
  (`workflow-rp.md` / plan-review `workflow.md`); a prompt change must land in both.
- **Proximity is load-bearing** (capture): don't relocate routing/taxonomy/guardrail
  tables out of the phase that consumes them, even to DRY.
- **Position within a prompt barely matters** (fn-74): a block validated at the top scored
  identically wired lower — the model reads the whole prompt. Wire at the clean code seam.
