# T-2215 — CLI Error-Pattern Survey (supplemental to T-2210 / IW-1)

**Parent inception:** T-2209 (Capability-overlay arc)
**Supplements:** T-2210 (IW-1 delivery-shape spike — recommends **B** over **C** at 1.30 vs 0.55 value-per-cost)
**Status:** Research memo for the operator. Producer ≠ judge — no Sovereign acts taken. Filed 2026-06-05.
**Question owner:** the operator wants *empirical* CLI-error evidence to confirm or pivot the a-priori BVP ranking of B vs C.

---

## Question

Does the empirical distribution of CLI errors / friction observed in real framework usage **support B's ranking** (CLI-overlay-only / `--json` on existing verbs), or does it **pivot to C** (both-as-siblings / `--json` underlay + thin MCP wrapper)?

Concretely: of the CLI pain we actually observe, which classes does B's `--json` already cover, and do C's extra MCP-leg fixes target classes that appear frequently enough to justify ~2.5× the cost (F8 5.40 vs 2.20)?

---

## Method

**Sources surveyed (time window ≈ last 60 days, 2026-04-05 → 2026-06-05):**

| Source | What it yields | Volume sampled |
|--------|----------------|----------------|
| `.context/working/.gate-bypass-log.yaml` | which flag/env-var agents reach for under friction | 2,857 lines; full env-var + flag frequency tally |
| `.context/working/watchtower.log` | stderr/stdout of web ops shelling out to `fw` | 3,242 lines; grepped for error/parse/exit |
| `.context/project/concerns.yaml` | registered structural gaps (G-NNN) | 70 G-entries; CLI-class filtered |
| `.context/project/learnings.yaml` | codified failure classes (L-NNN) | L-291/364/365/387/392-402/417/449 read in full; class-keyword tally |
| `.context/working/feedback-stream.yaml` | operator corrections + task references | 48,622 lines; CLI-class task_file refs extracted |
| `git log --oneline -300` | recent CLI-touching commits | last 300 (≈ 2 weeks dense) |
| T-2210 + parent T-2209 artifacts | B-vs-C framing, §5 auth, §7 shapes | full re-read |

**Sampling caveats (expanded in §Caveats):** the error corpus is generated by exactly **two consumer classes** — (1) shell agents running `## Verification` blocks and gate-bypass commands, and (2) the single Watchtower Python frontend shelling out to `bin/fw`. There are **zero MCP-client consumers** in the corpus, because no framework MCP server exists yet. This is the dominant self-selection effect and it cuts directly at the B-vs-C question — see §Caveats.

---

## CLI Error Classes Observed

### Class P — PARSE (text / ANSI / pipe fragility) — **DOMINANT, ~15 instances**

Agents and the web frontend parse the CLI's human-formatted (ANSI/prose) output through `grep`/`head`/`sed`/`awk`/regex, and the pipe or the regex breaks.

- **SIGPIPE-141 cluster — 4 codified learnings + 2 fix commits.** `cmd | grep -q PATTERN` in `## Verification` closes the pipe on first match → SIGPIPE 141 from upstream under `set -o pipefail` → the gate reports failure on a successful match. L-387 (`learnings.yaml:2973`, T-1862), L-393 (`:3033`, T-1848), L-394 (`:3040`, T-1856 — *recurred within 30 min of writing L-393*), L-402 (`:3096`, T-1900). Recent fix commits: `7f2362645` (T-2168, `/bvp` verification head-pipe → here-string), plus T-2090 follow-up.
- **ANSI / markup stripping.** T-1638 (`feedback-stream.yaml:3700` — "termlink extract stripAnsiCodes to shared"); T-1618 (`:3411` — strip HTML comments from Human-AC parser); T-2156 (`f067325f7` — HTML-comment strip in `_parse_agent_acs`).
- **Regex/sed parse hardening.** T-1748 (`:5502` — harden `parseVerdictEnvelope` regex); T-1967 (`:8035` — L-414 AC-parser sed range); T-1756 (`:5638` — re-parse historical parse-fail outcomes).
- **Web frontend scraping CLI prose (also Class O).** `watchtower.log`: 3× `inception decide T-XXXX failed: ... stdout='...ERROR: Cannot complete inception...' stderr=''` (T-2204, T-2201, T-2203). The CLI emitted its error as **prose on stdout**, and the Flask frontend had to string-match it to learn what failed.
- **Emission-side parse breakage.** L-392 (`:3026`, T-1873) — episodic generator emitted YAML with unescaped backticks → `yaml.ScannerError`; episodic silently unreadable to `fw recall`/`fw timeline`. 19 corpus victims.

### Class S — SCHEMA (output shape / type changes silently; downstream breaks with no signal) — **~5 instances, HIGH severity**

- **Type-coupling drift, web↔shell — 11 occurrences.** `watchtower.log`: `Failed to parse bypass log .../bypass-log.yaml: '<' not supported between instances of 'datetime.datetime' and 'str'` ×11. The shell writes a YAML log; one entry's timestamp deserialises as `datetime`, another as `str`; the Python reader crashes comparing them. No schema contract between writer and reader.
- **Silent-corpus on format migration.** L-397 (`learnings.yaml:3062`, T-1874–1879) — storage format migrated (`arc:slug` tags → `arc_id` frontmatter); consumers reading only the old shape became silent zero-emitters. 9 sites across 4 commits.
- **UI silently drops data on parse error.** G-entry "UI silently drops data on parse errors" (`concerns.yaml:486`, T-403) — malformed-but-parseable YAML still slips through; no schema validation.

### Class D — DISCOVERY (caller doesn't know which verbs/flags exist) — **~4 instances, mostly already mitigated**

- **META-GAP: doc/help/code drift.** G-035 (`concerns.yaml:1001`) — "every new required flag or subcommand creates a doc-drift bug … Agent CLAUDE.md gets read at session start AND saved into memory; stale forms cache indefinitely." Trigger: "multiple times per release cycle." **Resolved** via `fw doctor` Quick-Reference check (T-1147) + `help-router-parity.bats` (T-1185) + `config-registry-parity.bats` (T-1187).
- T-1962 (`feedback-stream.yaml:8460` — `fw arc review` slug verb didn't exist as expected); T-2124 (`:9750` — `hook-enable --script` flag); L-007 (`learnings.yaml:1690` — context-aware `bin/fw` vs `.agentic-framework/bin/fw` path confusion, T-1257).

### Class C — CONTRACT (producer/consumer parity gap) — **2 named, HIGH operational frequency**

- **L-399 / T-1890 — the origin.** (`learnings.yaml:3075`) A PreToolUse hook shipped a `--switch-focus` bypass contract on the hook side only; `update-task.sh` + `lib/{learning,pattern,decision}.sh` + `git commit` all rejected the flag with **"Unknown option"**, so agents worked around via direct-invoke `bash agents/...` — *a path the hook regex didn't match, silently circumventing the gate with no audit trail* for ~3 weeks.
- **The dual-mechanism contract, in live use:** gate-bypass-log shows `FW_SWITCH_FOCUS=1` env-var prefix **403×** vs `--switch-focus` flag **148×** — the two mechanisms L-399 mandates (env-var covers external tools like `git`; flag covers fw-internal parsers). This is the contract *working as designed* but it is a standing source of friction every session.

### Class V — VERB-SHAPE (flag names / response shape diverge across sibling verbs) — **~5 instances**

- **Enum-divergence cluster (L-006 enumeration class).** G-024/G-037/G-038/G-039 (`concerns.yaml:1090`, `:1148` and siblings) — enums hardcoded in `web/blueprints/tasks.py` diverge from `status-transitions.yaml`; canonical example: `fw task create --owner orchestrator` **loads in the CLI but renders broken in Watchtower and is unsettable via the dropdown**. Same input value, two surfaces, two answers.
- T-1937 (`feedback-stream.yaml:7865` — `fw bvp arcs` CLI parity); T-1568 (`:1913` — replace `--force` with narrow flags); T-1790/T-1784 (`:5927`/`:6777` — `fw orchestrator status` route/`--task-type` filter shape). CLAUDE.md itself notes `--task` vs `--task-id` divergence across verbs.

### Class A — AUTH / SOVEREIGNTY (direct-invoke bypasses hook gates) — **~3 instances, HIGH severity**

- **Direct-invoke bypass** is the *mechanism* of the L-399 failure above — agents escape the hook by calling the script directly.
- **CLAUDECODE gate.** `concerns.yaml:396` + T-1259/T-1260 — CLI-driven completion refused under `$CLAUDECODE=1`; the framework had to add a structural refusal because the CLI surface was agent-spoofable.
- **MCP carries no identity (forward hazard).** Parent T-2209 §5 / T-2210 strawman: stdio JSON-RPC carries no agent identity by default; env-inherit `$CLAUDECODE` auth is "trivially spoofable." (Relevant because C *adds* this surface.)

### Class O — OBSERVABILITY (can't tell which op ran / why it failed) — **~4 instances**

- **Error-to-stdout, empty stderr.** `watchtower.log` 3× `... failed: ... stdout='...ERROR...' stderr=''` — the failure signal went to stdout as prose; nothing structured to branch on.
- T-1475 (`feedback-stream.yaml:563` — "CTL-013 audit diagnostic: capture stderr"); T-1507 (`:1158` — inception-decide CLI/Watchtower output truncation); L-365 (`learnings.yaml` — cron `2>/dev/null` swallowed the only ModuleNotFoundError signal, 9 days blind).

---

## Fix-Coverage Matrix

Coverage of each observed class by candidate. **Key distinction:** the observed consumers are *shell agents* and *the Watchtower Python frontend* — both consume the CLI/JSON **underlay**, which is exactly **B**. C's MCP wrapper adds value only for **MCP clients**, of which the corpus contains zero.

| Class | Freq (instances) | B (`--json` overlay) | C (B underlay + MCP wrapper) | Neither / orthogonal |
|-------|:---:|---|---|---|
| **P — PARSE** | ~15 (dominant) | **YES** — structured fields replace ANSI/prose scraping; `jq -e` reads-all (no SIGPIPE) | **YES** (inherits B underlay; MCP leg adds nothing the observed consumers use) | — |
| **S — SCHEMA** | ~5 (×11 type-coupling) | **PARTIAL** — fixes *iff* underlay ships a `schema_version` field (parent §7 Q4); then web/shell parsers fail loud | **PARTIAL→BETTER** for MCP clients only (typed tool schema); observed web↔shell drift still rides the underlay = B's coverage | needs schema_version contract regardless of shape |
| **D — DISCOVERY** | ~4 (mostly mitigated) | **NO** | **YES** for MCP clients (`tools/list`) — but observed D instances are shell/doc-drift, already fixed by parity bats | doc/help/code parity tests (already shipped) |
| **C — CONTRACT** | 2 named, ~551 live bypass uses | **NO** | **NO** | **orthogonal** — lives in hook↔consumer space, independent of output format (L-399) |
| **V — VERB-SHAPE** | ~5 | **PARTIAL** — normalises *response* envelope across verbs; does not fix *input* enum/flag divergence | **PARTIAL→BETTER** for MCP clients (one typed schema/tool); observed consumers ride underlay | input-enum divergence needs single-source-of-truth (status-transitions.yaml pattern) |
| **A — AUTH** | ~3 (high severity) | **NEUTRAL** — adds no new surface (parent §5 "trivially safe") | **WORSE** — MCP leg adds the §B-005 spoofable auth surface; does *not* fix direct-invoke bypass | direct-invoke gate-coverage is a hook-matcher problem |
| **O — OBSERVABILITY** | ~4 | **YES** — `{ok, error_code, …}` lets web branch on a field; structured stderr | **YES** (inherits B) + typed MCP errors for MCP clients | — |

---

## Frequency-Weighted Score

Weighting by observed instance count (P≈15, S≈5+11-log, O≈4+3-log, C≈2-named/551-live, V≈5, D≈4, A≈3):

- **Mass fully covered by B alone:** Class P (~15) + Class O (~7) + the type-coupling half of Class S (×11 web↔shell) = **the clear majority of all observed friction**, and *100% of it consumes the CLI/JSON underlay*. None of it is an MCP-client error.
- **Where C beats B:** only Classes D, S-uplift, V-uplift — and **every one of those C-only fixes targets MCP clients**. The corpus contains **zero** MCP-client errors, and the observed D/V instances are shell/web problems already mitigated (parity bats) or needing single-source-of-truth (orthogonal to both candidates).
- **Where C is net-negative:** Class A — C's MCP leg *adds* the §B-005 spoofable auth surface against observed direct-invoke-bypass pain that it does not fix.
- **Orthogonal to both:** Class C (CONTRACT) — the single highest *operational*-frequency signal (551 live bypass invocations) is unaffected by delivery shape.

The frequency-weighted distribution is therefore **dominated by classes B fully covers, and the classes that would justify C are empirically empty** in the observed corpus.

---

## Verdict

**The empirical CLI-error evidence SUPPORTS B. It does not pivot to C.**

1. **The dominant observed pain is parse-fragility and web→CLI prose-scraping (Classes P + O + the type-coupling half of S) — ~25+ instances, all of it 100% inside B's `--json` coverage.** The 3 `inception decide … stdout='…ERROR…' stderr=''` scrapes and the 11 `datetime vs str` type-coupling crashes are the single most concrete pieces of evidence, and they are *exactly* what a structured `--json` response (B) eliminates: the Watchtower frontend would branch on `{"ok": false, "error_code": "open-questions-under-disposed"}` instead of string-matching prose.

2. **Every class where C beats B targets MCP clients — a consumer that generates zero errors in the corpus because it does not yet exist.** C's incremental fixes (D-discovery via `tools/list`, S/V schema-convergence via typed tools) are coverage for a hypothetical future consumer, not for observed pain. The deciding fact is *who consumes the output*: both observed consumer classes (shell agents, the Flask frontend) ride the CLI/JSON underlay = B; the MCP wrapper sits unused by everything in the evidence.

3. **On the one axis where C and B genuinely differ for an observed class (Class A / AUTH), C is net-negative** — it adds the §B-005 spoofable surface without fixing the observed direct-invoke bypass.

4. **The single highest operational-frequency signal (Class C / CONTRACT, 551 live bypass invocations + the L-399 "Unknown option" origin) is orthogonal to delivery shape** — neither candidate addresses it, so it cannot tip the choice toward C.

**This reinforces, rather than merely re-states, T-2210's a-priori ranking.** T-2210 scored C 0.11 BVP above B on the forward-looking F-ORCH driver (routable MCP surface) and still chose B on value-per-cost. The backward-looking error corpus independently confirms that the present-day pain is *entirely B-addressable*, and that C's extra cost buys coverage for a consumer class the evidence cannot yet see. **Ship B.** If/when MCP clients appear and generate their own error corpus, re-run this survey — the B underlay is the de-risking spine that makes the B→C evolution cheap at that point.

**One concrete build-slice requirement surfaced by the evidence:** B's Class-S coverage is *conditional* on the `--json` underlay shipping a `schema_version` field (parent §7 open-Q4). A naive `--json` that dumps current output shape would still let the ×11 type-coupling class break silently. **The build slice must include `schema_version` for B to close Class S** — flag this for IW-2/T-2211 and the build task.

---

## Caveats

1. **Self-selection is the headline caveat and it is structural, not incidental.** The corpus has zero MCP-client errors *because there are no MCP clients*. The absence of MCP-client pain is not evidence that MCP clients wouldn't generate pain — it is evidence that they don't exist yet. A backward-looking error survey *structurally cannot* see the forward value of C's MCP leg. That forward value is real and is precisely what BVP F-ORCH scoring (C=5 vs B=3) was designed to capture a-priori — the operator already weighed it at 0.11 BVP and judged it insufficient for 2.5× cost. **This survey does not move that needle; it confirms the present-day half of the ledger.** If the *purpose* of the arc is to bootstrap a future multi-agent/MCP ecosystem (a strategic F-ORCH bet), that is an operator call the error evidence cannot make.

2. **Log-volume thinness on the highest-severity classes.** Class S type-coupling is 11 occurrences but they are repeats of *one* root cause (`bypass-log.yaml` datetime/str). Class A and Class O are ~3-4 instances each. The counts are real but the *distinct-root* count is lower than the raw tally — I have shown the raw grep math so the operator can discount.

3. **Class C (CONTRACT) over-weights if counted by invocation.** 551 live `FW_SWITCH_FOCUS`/`--switch-focus` uses inflate its apparent frequency; these are the *designed* bypass working, not failures. Counted by *incidents* it is ~2 (L-399 origin + the dual-mechanism need). Either way it is orthogonal to B-vs-C.

4. **`git log` window is short.** Last 300 commits ≈ 2 dense weeks; the deeper history lives in the L-NNN / G-NNN corpus, which I weighted more heavly. ANSI=7 / parse-mentions in commit *titles* under-count the real Class-P volume (the cluster is in learnings, not commit subjects).

5. **Producer ≠ judge.** This memo is decision input for the operator, not a decision. It confirms B for the *observed* pain; the strategic forward-bet on C remains a Sovereign call.
