# ADR 0004: context-mode Tool-Output Sandbox for test-runner

> Status: ACCEPTED · session main-2026-05-19-deep-2 · issue #439
> Source research: docs/research/2026-05-19-context-mode-evaluation.md
> Project-instruction file resolution: `CLAUDE.md` and `AGENTS.md` (Codex CLI) are transparent aliases — see [skills/_shared/instruction-file-resolution.md](../../skills/_shared/instruction-file-resolution.md). Wherever this ADR says `CLAUDE.md`, the alias rule applies.

## Context

Issue #439 asks whether `mksglu/context-mode` — a 15.1k★ MCP server claiming ~98% context reduction by sandboxing tool output — should back the `/test` runner's `ux-evaluator` path to cap artifact token cost. The hook was that context-mode's single most-cited benchmark row is *a Playwright snapshot* (56.2 KB → 299 B, 99%), an apparent direct hit on our exact artifact class (research note § External Findings; [BENCHMARK.md](https://github.com/mksglu/context-mode/blob/main/BENCHMARK.md)).

**The token-cost concern.** The `ux-evaluator` agent (`agents/ux-evaluator.md`, opus, read-only) ingests run-dir artifacts and applies the 4-check UX rubric (`skills/test-runner/rubric-v1.md`). AX-tree YAML can theoretically balloon — `skills/playwright-driver/SKILL.md:108` warns *"50–200K tokens"* — and an opus agent holding full artifact content is the stated growth vector.

**What we already do — the `${RUN_DIR}` disk-handoff.** The coordinator-side problem context-mode sells is already solved differently in our architecture. Drivers write all artifacts to `.orchestrator/metrics/test-runs/<run-id>/` on disk; the test-runner orchestrator dispatches `ux-evaluator` via the Agent tool with only a `<scope: ${RUN_DIR}, rubric, output>` string — `skills/test-runner/SKILL.md:151-160`, explicitly *"the coordinator does NOT need to forward findings through prompt context."* The single dominant text artifact, `results.json` (≈21.5K tok in the largest real run), is parsed programmatically by the orchestrator and is **provably outside the ux-evaluator glob set** (`agents/ux-evaluator.md:39-42` globs only `ax-snapshots/axe-*.json`, `screenshots/*.png`, `console.log`). Binary `report/trace/**` (2.2 MB bulk) is path-referenced and never enters any context — a retention concern (`test-runner.retention-days`), not a token concern.

**The fidelity contract `issue-reconcile.mjs` depends on.** `agents/ux-evaluator.md:15` mandates byte-identical `findings.jsonl` across two invocations on identical run-dir contents — the invariant `scripts/lib/test-runner/issue-reconcile.mjs` cross-run dedup relies on. The research note's per-rubric-check table (§ Per-rubric-check raw-vs-summary tolerance) establishes that **2 of 4 checks hard-require raw bytes**: Check 2 (axe-violations) embeds the verbatim axe `Result` object and derives the fingerprint `locator` from `nodes[0].target`; Check 3 (console-errors) needs per-line `file:line` origins. context-mode's 99% figure is its **lossy `ctx_execute_file` class** (BENCHMARK.md: *"compresses data into 1-2 line summaries"*) — non-deterministic LLM summaries that would break the fingerprint contract. Its fidelity-preserving class (`ctx_index`+`ctx_search`) yields only 50–93% and still requires rewriting the read-only-from-disk agent to a `ctx_search`-driven retrieval model, plus an MCP wiring seam absent from our `.mcp.json`.

**Empirical reality (W3).** Across all three real on-disk runs, the heaviest evaluator-facing load was `console.log` at **≈4.3K tokens** (17,279 B / 4) — ~1/12 of the claimed 50K lower bound. AX-tree YAML and axe JSON were **0 bytes in every run** (no tested target has `@axe-core/playwright` or `page.accessibility.snapshot()` fixtures; soft-skip path `scripts/lib/playwright-driver/runner.mjs:184` fired each time). The 50–200K figure is the SKILL author's stated worst case, **never exercised locally**.

## Decision

**Decision: Skip (Stay).** Keep the on-disk `${RUN_DIR}` handoff; do not adopt context-mode and do not build a wrapper now. The disk-handoff already achieves the coordinator-side goal context-mode sells; the only residual gap — keeping heavy artifacts out of the *evaluator's* opus context — is narrow, fidelity-constrained (2/4 rubric checks forbid lossy summarization), and **empirically undemonstrated**: the heaviest measured evaluator-facing load across all three production runs is ≈4.3K tokens, far below any threshold that would justify the integration, supply-chain (ELv2, 1-day-old releases, a documented v1.0.137–139 silent-suppression regression — research note § External Findings), and fingerprint-correctness cost. Adopting context-mode's headline mode would *degrade* evaluation quality — the explicit #439 disqualifier — because the 99% number is its lossy 1–2-line-summary class, not a fidelity-preserving reduction. This is consistent with `docs/adr/0001-context-vs-orchestration.md`: an external context-reduction claim is an input signal, not a reflexive substrate swap, and here the re-derivation collapses 99% to "compresses a file (`results.json`) we already keep out of every agent's context by construction."

**Quantified revisit trigger.** Reopen this decision (toward build-own, see Consequences) **if and only if a single `/test` run's ux-evaluator agent input-token count exceeds 40,000 tokens**, measured from the agent dispatch record (`agents/ux-evaluator.md:145-161` machine-readable contract block) cross-referenced with the `.orchestrator/metrics/test-runs.jsonl` roll-up (`skills/test-runner/SKILL.md:300-317`), on a fixture-rich target that actually emits `ax-snapshots/*.yaml` + `axe-*.json`. Sub-threshold runs, or threshold breaches caused by artifacts outside the ux-evaluator glob set (e.g. `results.json`), do **not** trip the trigger — only evaluator-glob-matched artifacts count.

## Consequences

**What we keep.** The `${RUN_DIR}` disk-handoff (`skills/test-runner/SKILL.md:151-160,332`) and the read-only-from-disk `ux-evaluator` (`agents/ux-evaluator.md`) are unchanged. Zero new dependency, zero regression surface, full fingerprint fidelity for `issue-reconcile.mjs` cross-run dedup. No security re-review (the `sandbox-tier: read-only` contract at `agents/ux-evaluator.md:7` is preserved — `ctx_execute*` would have widened it to "execute a sandboxed runtime"). No `harness-audit` Category 8 perturbation (`scripts/lib/harness-audit/categories/category8.mjs` MCP-config surface is untouched).

**What we explicitly do NOT take on.** (1) An **ELv2 source-available, non-OSI dependency** — inconsistent with the repo's zero-dependency in-repo `.mjs` posture (`scripts/lib/test-runner/fingerprint.mjs` deliberately uses only node `crypto`). (2) An **11-tool MCP server** (`ctx_execute`, `ctx_index`, `ctx_search`, + 8 more) with weekly upstream churn (v1.0.140 was 1 day old at evaluation, with a documented `ctx_*`-silent-suppression regression in v1.0.137–139, repo issue #623). (3) A **`ux-evaluator` rewrite** from Glob+Read to `ctx_search`-driven retrieval — a non-trivial change to a fingerprint-critical agent whose determinism invariant (`agents/ux-evaluator.md:15`) would be jeopardized even by the lossless path's BM25/RRF tie-break ranking.

**What we'd build-own instead if the trigger fires** (NOT now): a ~30–60 LOC fidelity-preserving chunked reader in `scripts/lib/test-runner/` (e.g. `chunked-axe-reader.mjs` / `summarize-console-errors.mjs`) that ux-evaluator calls for the two raw-required checks only, preserving `(ruleId, nodes[0].target)` and per-error `file:line` `locator` verbatim. Zero new dependency, no license question, scoped exactly to our rubric — strictly preferable to importing an 11-tool MCP for one summarization need. Build-own is the chosen escalation path; adopt remains rejected at any threshold because the fidelity-safe context-mode mode caps at 93% while still requiring the agent rewrite.

**Cross-connection (#437/#438).** This decision shares the theme of #437 (Native Agent Teams) and #438 (Anthropic Routines): an external primitive's headline metric must be re-derived against *our* architecture before adoption. Here the re-derivation is the load-bearing finding — context-mode's 99% is real *for the lossy class on a file we already isolate*, and collapses to "no benefit at our scale, with a fidelity regression if forced." Any future reconsideration must repeat that re-derivation, not import the headline.

## Follow-ups

- **Issue (file): parametrize the revisit-trigger measurement into test-runner.** Add a post-dispatch step that records the ux-evaluator agent's input-token count to `.orchestrator/metrics/test-runs.jsonl` and emits a `revisit-trigger-breach` flag when evaluator-glob-matched artifacts exceed 40K tokens on a run. This converts the trigger from a manual audit into a mechanical signal so a real pain surfaces loudly (priority: low; depends on a fixture-rich target existing — see next item).
- **Issue (file, optional): fixture-rich `/test` target for trigger validation.** Stand up or register a target with `@axe-core/playwright` + a `page.accessibility.snapshot()` fixture so the 50–200K AX claim (`skills/playwright-driver/SKILL.md:108`) can be measured at all. Until this exists the trigger can never fire — the W3 honest gap (research note § Empirical Step 3) remains open by construction.
- **Conditional (NOT now): build-own spec.** Only if the trigger fires — author a thin-slice spec for the `scripts/lib/test-runner/` chunked reader (≤60 LOC, fingerprint-preserving, zero-dep). Do not pre-build; the appetite rule (`mvp-scope.md`) and ADR-0001 both forbid speculative adoption against an undemonstrated need.
