# Program Status

<!-- cspell:ignore OVERCLAIMED tiktoken overclaimed rpncalc realsoftware tinymd characterised -->

Living status doc for the prompt-language thesis. Single page an operator reads to know where the thesis stands and what is next. Facts only; anything not directly observable in the repo is flagged "not measured".

Last synced: 2026-05-12 against `main` after the 2026-05-11 Concord/VHO
experiments, operator signer provisioning, signed-bundle milestone, and
multi-agent scrutiny pass.

## 1. Thesis status

Thesis (short form): prompt-language becomes the higher-order engineering medium engineers edit first, with code as a downstream artifact the agents produce and maintain. Full statement: [`docs/strategy/thesis.md`](thesis.md).

Current verdict: **inconclusive — still no claim-eligible thesis run**. Design,
scaffolding, verifier/signer surface, bounded runtime-backed proof, and
local-model diagnostics all exist, but no run satisfies the full
claim-eligibility rule in §3a. The operator signer is now provisioned in
`docs/security/trusted-signers.json`, but the first signed bundle still fails
claim verification because of trace, witness, path, and real cross-family-review
blockers. L3's live-smoke falsifier still stands: **E6 = PARTIAL** (stuck at
node 7, cold-start `memory:`/`remember` race) and **E7 = FAIL** (82s silent exit,
no state persisted, `#` comments converted to prompt nodes). The shipped
end-to-end factory flows therefore do not yet execute cleanly against live
`claude -p`.

The strongest product evidence today is narrower: prompt-language is a
verification-first supervision runtime with real gates, state, review/retry loops,
headless runner paths, and CI-backed tests. The checked-in `factory-runtime-proof`
series gives one clean bounded Codex completion (`20260418-083500/codex-medium`)
and bounded Claude runner evidence that still falls short of a clean completion
claim (`20260418-074500/claude-medium`).

The strongest local-model evidence is the FSCRUD R30-R45 sequence. It does **not**
prove autonomous local full-stack implementation. It does support a bounded
architecture: local 30B Ollama can act as a selector/ranker/rationale/risk-response
module when PL owns deterministic validation, normalization, rendering, protected
artifacts, and verification. Local-model ladder reruns on 2026-04-28 remain useful
but non-claim-eligible: H15 shows PL can lift a local model to green (`10/10` vs
solo `6/10`), while H12 ties at higher cost and H14 shows over-staged PL losing to
solo. The thesis boundary is therefore: PL helps when the flow gives task-fit
control, verifier feedback, and repair structure; it is not automatically better
and it is not proven as a primary engineering medium.

## 2. Programs

| Program                            | Goal                                                                                   | Status                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Evidence                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Next actionable step                                                                                                                                                  |
| ---------------------------------- | -------------------------------------------------------------------------------------- |   |   | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| E4 factory-quality                 | Does PL lane score higher on factory-quality rubric?                                   | Closed. Verdict: PL 10 vs codex 8 on rubric, but E5 treats this as circular (rubric rewards PL's own structure).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `experiments/results/e4-*` batches B01–B06 (commits `9472404`, `0dcab2c`, `9300cc1`, `42c0f78`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | N/A — archived; findings feed E5 protocol.                                                                                                                            |
| E5 maintenance-viability           | Does a second blind lane maintain PL output better than codex output?                  | Pilot scaffolded. Runner live, synthetic-mode only (no real pair run).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Program: `experiments/results/e5-maintenance/program.md`; scorecard template, 7 CRM journeys, 5 change requests, pair runner, blinding verifier under `scripts/experiments/e5/` (commit `4e61066`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | Run B01 pair P01 live once a supported-host Claude/codex auth is available; capture scorecard.                                                                        |
| E6 pure-PL CRM factory             | Prove a CRM factory can be driven entirely by `claude -p project.flow`.                | Shipped scaffold; live smoke **PARTIAL** (2026-04-17). Runtime parsed 38 nodes, advanced n1..n6, stuck at n7 on cold-start `memory:`/`remember` race. Separate bounded evidence now exists from the smaller `factory-runtime-proof` packs: `20260418-083500/codex-medium` is a clean committed bounded completion through PL review, while `20260418-074500/claude-medium` is a terminal-but-not-clean bundle that exposed the old await-integrity gap.                                                                                                                                                                                                                                                                                                                                                                        | `experiments/full-saas-factory/` + commit `02efd45`. Live-smoke bundle `experiments/meta-factory/results/e6-live-1776375508/` + bead `prompt-lm0o.17`. Bounded runtime-backed series: `experiments/results/factory-runtime-proof/` and [`docs/evaluation/2026-04-18-runtime-factory-proof-codex-claude-medium-evidence.md`](../evaluation/2026-04-18-runtime-factory-proof-codex-claude-medium-evidence.md). Use `20260418-083500` for the clean bounded Codex completion and `20260418-074500` for the latest committed Claude runtime evidence. No downstream maintenance metric has been taken.                                                                                                                                                                                                                                                                                                                                                                                    | Fix cold-start hydration in the shipped flow; keep the bounded proof slice as runner evidence only; then feed a clean rerun into E5 B01.                              |
| E7 enterprise CRM factory          | Full-SDLC factory: 18 entities, 6 phases, 37 ship gates, race/foreach-spawn/try-catch. | Shipped scaffold; live smoke **FAIL** (2026-04-17). Parsed 190 nodes with 57 `Unknown keyword "#..."` warnings (comments silently became prompt nodes); runtime advanced n1 only, then silent 82s exit with no `session-state.json`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `experiments/full-saas-factory/e7-enterprise-crm-factory/` + commit `a5bc64d`. Live-smoke bundle `experiments/meta-factory/results/e7-live-1776376120/` + bead `prompt-lm0o.18`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | Fix `#` comment parsing; diagnose silent-exit; re-run live smoke.                                                                                                     |
| Meta-factory                       | Can PL develop PL?                                                                     | M1 scaffolded + ran live once. Claude authored a smoke test; trace-gate failed because `PL_TRACE_DIR` was not populated (no `provenance.jsonl` emitted by the child). **Trace-dir propagation now fixed in current codebase** — `run-meta-experiment.mjs` sets `PL_TRACE`, `PL_TRACE_DIR` in child env; `claude-process-spawner.ts` forwards them. **Bootstrap-envelope preflight is now an enforced live-run gate**: `blocked` refuses execution; `degraded` may run but is not claim-eligible.                                                                                                                                                                                                                                                                                                                               | `experiments/meta-factory/results/meta-1776129598770-407100/report.json` — `success: false, reason: verify-trace-failed, verify.reason: no-provenance`. Scaffold at commits `08c0c08`, `b567405`, `0cccfbd`. Bootstrap-envelope semantics are implemented in `scripts/experiments/meta/bootstrap-envelope.mjs`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Reach `overall: ready` on a supported host, then re-run MF-1 live and capture a claim-eligible bundle.                                                                |
| Trace verification + witness chain | Tamper-resistant trace for meta-runs.                                                  | Dual-recorder Merkle chain shipped; verifier runs; 6 Severity ≥ 3 attack paths documented; AP-5 + AP-6 patched. Phase-2 cross-family enforcement shipped across preflight, reviewer, and verifier surfaces: `verify-trace.mjs` accepts `--expected-reviewer-family`, the bootstrap-envelope distinguishes `blocked` vs `degraded`, and live-run claim eligibility now depends on a `ready` preflight. **AP-9 verifier/signing support is now in code**: `verify-trace.mjs` verifies detached attestations and `scripts/experiments/meta/attest.mjs` can sign bundle payloads. The checked-in signer registry now contains `operator-45ck-2026-05`, and the first signed multi-node Ollama bundle exists, but `verify-trace --require-attestation --require-role operator` still fails on concrete trace/witness/path blockers. | `39a7c75` runtime chain, `80b9ae1` witness shim + verifier + Z-series + strict mode, `0271483` tamper-drill integration test, `60c5c40` canonical-JSON dedup (AP-6). **AP-5 external nonce anchor shipped**: `writeRunNonce`/`readRunNonce` in `run-meta-experiment.mjs` now use a private per-user nonce store, and `verify-trace.mjs` enforces `--expected-run-id`/`--freshness-window-ms`. **Phase-2 cross-family guard shipped**: bootstrap preflight classifies the reviewer setup before execution, and `verify-trace.mjs` enforces `--expected-reviewer-family` against trace-carried `reviewer.family` evidence. **AP-9 verifier/signing surface shipped**: `scripts/eval/verify-trace.mjs`, `scripts/experiments/meta/attest.mjs`, `scripts/experiments/meta/attestation-lib.mjs`, and the signer registries under `docs/security/`. Signed-bundle milestone: `experiments/attested-smoke-2026-05-11/MILESTONE.md`. Attack report: `docs/security/witness-chain-attacks.md`. | Clear the signed-bundle verifier blockers, replace the stub reviewer with a real cross-family review, then produce the first claim-eligible operator-attested bundle. |
| Aider-vs-PL experiment             | Does PL orchestration beat solo aider with local models?                               | Phase-1 informal dev-time comparison (10 hypotheses, N=1 per lane, narrative scoring, not §3a-eligible). Phase-2 is now partially run on local Ollama: H12 tied, H14 favored solo, H15 favored PL. The result is mixed and points toward task-fit flow design rather than blanket PL wrapping.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `experiments/aider-vs-pl/SCORECARD.md` (narrative), per-hypothesis blurbs in `experiments/aider-vs-pl/results/`, and [`experiments/aider-vs-pl/results/2026-04-28-local-model-experiments.md`](../../experiments/aider-vs-pl/results/2026-04-28-local-model-experiments.md). Scrutiny: [`docs/security/aider-vs-pl-scrutiny.md`](../security/aider-vs-pl-scrutiny.md). `prompt-language ci --runner aider` verified end-to-end.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Redesign H14 with explicit imports, field-merge assertions, and oracle-fed repair; then run k>=3 before making a comparative claim.                                   |
| FSCRUD local-model diagnostics     | Can local Ollama own more than bounded semantic choices under PL supervision?          | Active diagnostics through R45. R30-R39 show free-form local implementation, handoff authoring, and senior-plan prose remain brittle. R40-R45 passed when local responsibility was reduced to bounded section selection, decision choices, weighted ranking, short rationale, and risk response, with deterministic PL tooling owning product behavior and artifact rendering.                                                                                                                                                                                                                                                                                                                                                                                                                                                 | [`docs/evaluation/2026-05-06-evidence-snapshot.md`](../evaluation/2026-05-06-evidence-snapshot.md), [`experiments/fullstack-crud-comparison/`](../../experiments/fullstack-crud-comparison/), and `experiments/fullstack-crud-comparison/manifests/experiment-manifest.json`. Current evidence supports bounded local semantic judgment inside deterministic PL control, not autonomous local full-stack software engineering.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | Package the bounded-choice module pattern, then test one tiny executable implementation slice with protected files, hidden verifier, and k>=3 repeats.                |

## 2a. Hybrid-efficiency tracker

Added 2026-05-11. Portfolio-level metric to detect the "hybrid failure mode"
(`docs/strategy/thesis.md` §"What would weaken the thesis" / Kill rule) where a
hybrid arm spends as much or more frontier budget than a frontier-only arm.

**Definition.** For each promoted route, record:

- `route_frontier_calls` — frontier-model API calls consumed by the
  promoted route (local-only, advisor, or hybrid-router) in the most recent
  live runs that pass the route's oracle.
- `frontier_only_frontier_calls` — frontier-model API calls consumed by the
  frontier-only baseline arm on the same fixture.
- `efficiency_ratio = route_frontier_calls / frontier_only_frontier_calls`.

A route is in **hybrid-failure-mode** when `efficiency_ratio ≥ 1.0` and
pass rate is not strictly higher than frontier-only on the same oracle.
A route at `efficiency_ratio = 0.0` with passing oracle is the strongest
possible outcome: local-only has fully displaced frontier on this task
class.

| Route                         | Class               | Route frontier calls        | Frontier-only frontier calls | Ratio | Local pass rate                                    | Status                                                | Source                                                                                                                                         |
| ----------------------------- | ------------------- | --------------------------- | ---------------------------- | ----- | -------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| H11 multi-file refactor       | cross-file refactor | 0 (local-only)              | not measured                 | 0.00  | 4/4 (runs 003–006)                                 | **local-only displaces frontier**                     | [`docs/evaluation/2026-05-09-ha-hr1-h11-live-local.md`](../evaluation/2026-05-09-ha-hr1-h11-live-local.md) (lines 35, 100, 142, 191, 239, 282) |
| H14 implementation-from-tests | TDD implementation  | 0 (local-only)              | 1 (baseline pass)            | 0.00  | 3/3                                                | **local-only displaces frontier**                     | [`docs/evaluation/2026-05-08-ha-hr1-h14-live-local.md`](../evaluation/2026-05-08-ha-hr1-h14-live-local.md) (lines 266, 485)                    |
| H14 API-preserving impl       | API-preserving      | 0 (local-only)              | not measured                 | 0.00  | 3/3                                                | **local-only displaces frontier**; no baseline yet    | [`docs/evaluation/2026-05-08-ha-hr1-h14-live-local.md`](../evaluation/2026-05-08-ha-hr1-h14-live-local.md) line 632                            |
| H14 full TDD ownership        | TDD end-to-end      | 0 (local-only)              | not measured                 | 0.00  | 3/3 + 3/3 fallback                                 | **local-only displaces frontier**; no baseline yet    | [`docs/evaluation/2026-05-08-ha-hr1-h14-live-local.md`](../evaluation/2026-05-08-ha-hr1-h14-live-local.md) lines 1678–1755                     |
| H15 PATCH test-authoring      | test authoring      | 0 (local-only)              | not measured                 | 0.00  | 6/10; 3/3 post-fix                                 | **local-only displaces frontier**; no baseline yet    | [`docs/evaluation/2026-05-08-ha-hr1-h15-hybrid.md`](../evaluation/2026-05-08-ha-hr1-h15-hybrid.md) lines 507–510                               |
| H15 API endpoint              | API endpoint        | 3 (hybrid + 1 failed local) | 1 (clean pass)               | 3.00  | hybrid PASS after frontier repair; local-only FAIL | **hybrid-failure-mode**; route held frontier-baseline | [`docs/evaluation/2026-05-08-ha-hr1-h15-hybrid.md`](../evaluation/2026-05-08-ha-hr1-h15-hybrid.md) lines 152–197                               |
| H15 validation-only           | validation          | 0 (local-only)              | not measured                 | —     | 1/2 (oracle miss, not frontier issue)              | screen-only after failed repeat                       | [`docs/evaluation/2026-05-08-ha-hr1-h15-live-local.md`](../evaluation/2026-05-08-ha-hr1-h15-live-local.md) lines 138–185                       |

**Portfolio verdict (2026-05-11):** 5 of 7 promoted routes show
`efficiency_ratio = 0.00` — local-only fully displaces frontier on H11
multi-file refactor, three H14 TDD subroles, and H15 PATCH test-authoring.
**1 route is in hybrid-failure-mode** (H15 API endpoint, ratio 3.00). 1
route (H15 validation-only) is route-screen pending and unrelated to
frontier cost. The kill rule in `thesis.md` requires ≥3 distinct task
classes in failure mode before VHO is declared dead — current portfolio is
**positive directional evidence** for the local-delegation hypothesis on
the measured task classes, with one well-isolated negative result on the
implementation-shape API endpoint route.

**Scope-divided update (2026-05-11 evening):** Two same-day pilots
outside harness-arena measured the hybrid hypothesis at finer
scopes. Both findings stand:

- **Per-task micro-routing (positive on cost; OVERCLAIMED on novelty —
  see corrections):**
  [`experiments/concord-vho-microtask-cost/2026-05-11/`](../../experiments/concord-vho-microtask-cost/2026-05-11/)
  — 10 single-function tasks (4 novel-spec, 6 utility) routed to
  qwen3-coder:30b. Original headline was **10/10 first-attempt
  pass at k=10 (100/100 individual oracle passes)**, framed as
  novel signal. After adversarial review:
  - **Published-baseline comparison** ([`results/published-baseline-comparison.md`](../../experiments/concord-vho-microtask-cost/2026-05-11/results/published-baseline-comparison.md)):
    qwen2.5-Coder-32B already scores 92.7% HumanEval; Qwen3-Coder-
    30B-A3B-Instruct is positioned stronger. **At baseline p=0.95,
    P(10/10) by chance ≈ 60%.** The 10/10 is the _expected mode_ on
    a 10-task curated battery, not a research finding.
  - **Cross-family adversarial review** ([`cross-family/findings.md`](../../experiments/concord-vho-microtask-cost/2026-05-11/cross-family/findings.md)):
    devstral-small-2:24b (Mistral family) graded qwen's k=1
    solutions on adversarial cases. **Found 1 real qwen bug** that
    the v2 oracle missed: `parseQuery("?a=b=c")` returns
    `{a:"b"}` instead of `{a:"b=c"}` (uses `split('=', 2)` which
    drops the tail). Cross-family coverage was partial (4/10 tasks
    gradable; the rest blocked by devstral writing invalid-JSON
    cases) but the 1 bug found is sufficient to falsify the
    "100/100 = deterministic substitution" narrative.
    Honest revised verdict: the routing pattern works at micro-task
    scope and saves ~40-130 tiktoken-equiv tokens per task vs
    frontier-only when local passes, but the headline "100/100" was
    inflated by oracle weakness (same author wrote prompts and
    tests). Real per-task pass rate against cross-family adversarial
    oracles is ≤8/10. Per-pilot scaffolding still costs ~3500
    frontier tokens; break-even at ~5 reused pilots.
- **Per-app construction (negative):**
  [`experiments/concord-vho-pilot-todo-cli/2026-05-11/`](../../experiments/concord-vho-pilot-todo-cli/2026-05-11/)
  — 7-task TODO CLI build via routing. Headline 5/7 by local was
  overclaimed: independent critic agent showed frontier
  scaffolding (~8-12k tokens) dominates over local savings
  (~1k tokens), making hybrid **2-3× more expensive than
  frontier-only at single-app scope**. Tagged `app-build-orchestration`
  in the kill-rule sense. Did not trigger the kill rule alone
  (1 task class, not ≥3) but is consistent with hybrid-failure-mode
  at this scope.
- **Per-real-software multistage (negative on dollar economics after
  rpncalc correction):**
  [`experiments/concord-vho-realsoftware-tinymd/2026-05-11/`](../../experiments/concord-vho-realsoftware-tinymd/2026-05-11/)
  — Markdown→HTML converter (8 routable functions + frontier
  orchestrator). 6/8 first-attempt local pass. Final assembled CLI
  passes 24/24 oracles (8/8 per-function + 13/13 integration). Two
  real local failure modes characterised: regex backslash error
  and wrong sibling-contract assumption. Per-pilot token cost:
  hybrid ~13.2k frontier scaffolding vs frontier-only ~911. The
  later
  [`rpncalc`](../../experiments/concord-vho-realsoftware-rpncalc/2026-05-11/)
  cross-app reuse pilot measured actual reusable scaffolding at roughly
  42%, not near 100%, and found per-pilot scaffolding still around
  ~5.5k frontier tokens versus ~600 tokens saved locally. Revised
  verdict: **real-software multistage hybrid is not dollar-cost
  justified at current reuse rates**. It can still be justified for
  privacy, latency, quota independence, or offline operation. Tagged
  `real-software-multistage`. Skill updated with new boundary "don't
  use for one-off real-software builds."

**Engineering-readiness synthesis (2026-05-11):** See
[`engineering-readiness-2026-05-11.md`](engineering-readiness-2026-05-11.md)
for the explicit GO / CONDITIONAL / NO verdict per engineering
decision. Headline: narrow-scope routing is shippable today; broad
architectural moves need more evidence (k≥10 stability, cross-app
reuse measurement, adversarial oracle writing, claim-eligible
operator signer).

The hybrid pattern is therefore **scope-dependent in this evidence**:
directionally useful for stateless single-function code generation when prompts
are clear and oracles are cheap; cost-negative for app-build orchestration and
real-software multistage work where frontier scaffolding dominates local
savings. Future §2a entries should tag scope (`harness-arena-route`,
`micro-task`, `app-build`, `real-software-multistage`) so the kill-rule
condition (≥3 distinct task classes in failure mode) is measured within scope,
not across.

Caveat: every "frontier-only baseline not measured" entry above is a real
gap. Without the matched frontier-only number, `efficiency_ratio = 0.00`
shows local-only passes the oracle, but cannot quantify how much frontier
budget the program is actually saving versus a frontier-only equivalent.
Closing those baseline runs is the next high-value measurement after the
HA-HR1 cross-arm fixture lands.

**Reading rule.** A new entry is only valid once the run that produced its
numbers is claim-eligible per §3a (strict trace, ready preflight,
attestation, cross-family reviewer, trusted signer). All entries above are
currently **non-claim-eligible** engineering signal — no run in the repo
satisfies all five §3a gates yet. The trusted-signers registry now has an
operator key, but the signed-bundle verifier blockers and real cross-family
review gap still prevent a claim-eligible result. Treat the portfolio verdict
as directional, not thesis-confirming.

## 3. What has shipped (Q2 2026)

Every item below is a commit on `main`.

- Runtime trace chain — `TraceLogger` port, state-hash chain, `FileTraceLogger`, `TracedPromptTurnRunner` — commit `39a7c75`.
- Witness shim + trace verifier with strict mode, self-test, Z1–Z7 differential smoke tests (only-PL-runtime-passes oracle), and AR–AV coverage-gap cases — commit `80b9ae1`. Z-series and AR–AV are visible in `scripts/eval/smoke-test.mjs`.
- 10-case tamper-drill integration test for the verifier — commit `0271483`.
- AP-6 canonical-JSON duplication eliminated (verifier consumes compiled canonicalization) — commit `60c5c40`.
- Pure-PL CRM factory (E6) — commit `02efd45`.
- Meta-factory scaffold, M1 `PL-writes-smoke-test` flow, acceptance protocol, and synonyms — commit `08c0c08`. Design pack (`architecture.md`, `corpus.md`, `verification.md`, `risks.md`) landed in the same commit; DSL-primitives design handshake (snapshot/rollback/diff-review/self-trace-replay) followed in `0cccfbd`.
- Meta-experiment harness with git-stash pre-flight, manifest pre/post diffing, protected-config rule-weakening guard, wall-clock cap, and stash restoration — commit `b567405` (`scripts/experiments/meta/run-meta-experiment.mjs`).
- E5 maintenance-viability program scaffold — program doc, scorecard template, 7 journeys, 5 change requests, pair runner, journey suite, blinding verifier — commit `4e61066`.
- E7 enterprise CRM factory — 18 entities, 6 SDLC phases, 37 ship gates, race/foreach-spawn/try-catch, 658 lines across 16 files — commit `a5bc64d`.
- Aider-vs-PL phase-1 informal comparison — 10 hypotheses, single run per lane, narrative scoring (see `experiments/aider-vs-pl/SCORECARD.md` and per-hypothesis blurbs in `experiments/aider-vs-pl/results/`). Not claim-eligible per §3a (no pre-declared fixtures for H1-H10, no blinding, no cross-family reviewer, no reproducibility manifest, N=1 per cell). Full scrutiny: [`docs/security/aider-vs-pl-scrutiny.md`](../security/aider-vs-pl-scrutiny.md). `prompt-language ci --runner aider` verified end-to-end. Phase-2 design with pre-declared fixtures, k=3 repetition, and blinded scoring shipped at `experiments/aider-vs-pl/phase2-design.md`; not yet run.
- Aider harness adapter for local-model agentic coding — commits `91c64ae` (feat) and `bcb33d7` (`RunnerName` + `RUNNER_BINARIES` fix).
- Snapshot/rollback DSL primitives (PR1) — `snapshot "name"` + `rollback to "name"`, state-only, 23+ unit tests + smoke test AX.
- Coverage lift: global statements coverage raised to 90% + CLI drift fix — commit `1a53d7b`.
- Tracing + thesis-verification operator docs — commit `5425ab5` (`docs/tracing-and-provenance.md`, `docs/thesis-verification.md`).
- Smoke catalog expansion — `scripts/eval/smoke-test.mjs` currently carries 52 distinct scenario IDs (A–Z, AA–AX, Z1–Z7, AR–AV). CLAUDE.md catalog updated to match.
- E5 HTTP journey probes — `app-boot.mjs` (app detection + health wait) and `documented-endpoint-probe.mjs` (HTTP request verification) shipped in `scripts/experiments/e5/` — commit `0e66d34`.
- Cross-family reviewer design — `docs/strategy/cross-family-reviewer.md` (10-family taxonomy, lane assignment rule, reviewer prompt contract, rollout phases) and `scripts/experiments/meta/cross-family-review.mjs` (review module) — commit `0e66d34`.
- Phase-2 cross-family review schema/verifier support — `experiments/results/e5-maintenance/templates/scorecard.template.json` now carries `admissibility.crossFamilyReviewer`, `scripts/experiments/e5/score-scorecard.mjs` preserves that status for old and new scorecards, and `scripts/eval/verify-trace.mjs` recognises `--expected-reviewer-family`.
- Bootstrap-envelope enforcement for meta-runs — `scripts/experiments/meta/bootstrap-envelope.mjs` now makes `blocked` a live-run stopper while leaving `degraded` as recorded-only, non-claim-eligible evidence.
- AP-9 attestation support — `scripts/eval/verify-trace.mjs` now verifies detached attestations, `scripts/experiments/meta/attest.mjs` signs bundle payloads, `scripts/experiments/meta/attestation-lib.mjs` centralizes payload/registry/signature logic, `scripts/experiments/meta/run-meta-experiment.mjs` can auto-sign and propagate attestation-required verifier flags, signer registries exist at `docs/security/trusted-signers.json` and `docs/security/revoked-signers.json`, and `trusted-signers.json` now contains operator signer `operator-45ck-2026-05`.
- AP-9 **hardening pass** (2026-04-17) — commits `070e0b2`, `290b052`, `fb2ca59`, `60d36ec`. Closes epics `prompt-qwvu` (F1/F2/F3) and `prompt-kv57` (NF1–NF5, G1, G2). Changes: signed payload widened to full 9 fields (runId, commitSha, manifestHash, finalStateHash, pairCount, runtimeFamily, reviewerFamily, createdAt, signedAt), ed25519 signature encoding pinned to raw 64-byte/base64 with explicit length checks, trust-root sha256 now build-time-pinned via `scripts/build/emit-attestation-trust-root.mjs` → `dist/eval/attestation-trust-root.js`, `dev` role removed from accepted enum (moved to `docs/security/trusted-signers.example.json`), harness `PL_META_SIGN=1` refuses to run without explicit `operator|ci` signer, verify step order locked to §6.2(a)–(i), trust-root sha256 surfaced in OK line, `~/.pl-attest-log.jsonl` audit log on sign. 16 attestation tests pass.
- Bootstrap envelope path fix (2026-04-17) — `scripts/experiments/meta/bootstrap-envelope.mjs:106` now checks `dist/domain/state-hash.js` (was `dist/state-hash.js`). Commit `9fe2bb4`. Discovered by L4 MF-1 retry (every fresh build had been marking item #1 `blocked`, preventing `claude -p` launch). Bead `prompt-lm0o.16`.
- Live-smoke falsifier for E6/E7 factories (2026-04-17) — commit `ffd93d7` (evidence bundles) + `85446f4` (beads). E6 = PARTIAL, E7 = FAIL. Five cross-cutting findings filed as sub-issues of `prompt-lm0o.17` / `prompt-lm0o.18`: (1) `memory:` prefetch races `remember` writes on cold start; (2) `verify-trace` rejects entry 0 when `prevEventHash` is not serialized as `null`; (3) Windows PATH picks `claude.exe` over `pl-claude.cmd` (shim bypassed, witness chain single-sided); (4) `#` comment lines silently become prompt nodes; (5) E7 silent 82s exit with no state persisted (stop-hook race suspected).
- Bounded runtime-factory evidence refresh (2026-04-18) — committed proof bundles now include `experiments/results/factory-runtime-proof/20260418-074500/` and `20260418-083500/`. Discovery-only imported factory slice advances on both Codex and Claude with medium effort, persists PL state/audit artifacts, records spawn + await events, and writes discovery docs. `20260418-083500` is the clean bounded Codex completion through review. `20260418-074500` is the latest committed Claude terminal bundle, but it also documents the pre-fix await-integrity gap because the parent completed over failed children.
- Aider adapter security audit (2026-04-17) — commit `910e995`. No Sev≥3 findings. One Sev 2 follow-up (`prompt-xgav.11.1`, open): prompt argv visible to `ps`. Audit doc `docs/security/aider-adapter-audit.md`.
- Aider-vs-PL phase-1 scrutiny (2026-04-17) — commit `9882c56`. Phase-1 "6/0/3" result fails all 5 §3a gates; phase-2 entry criteria filed as `prompt-xgav.10.1`–`.7`. Scrutiny doc `docs/security/aider-vs-pl-scrutiny.md`.
- Aider phase-2 design — `experiments/aider-vs-pl/phase2-design.md` with H11-H20 harder hypotheses, model matrix, E5 CR mapping, scoring rubric — commit `0e66d34`.
- Local-model ladder evidence refresh (2026-04-28) — commits `34bd205` and `de4032d`. Added a fixture-pair harness, curated H12/H14/H15 evidence, and a local-model findings report. Current result: H15 PL win, H12 tie at higher PL cost, H14 solo win due to a weak TDD flow. This is useful engineering evidence but remains non-claim-eligible under §3a.
- Hybrid model routing experiment plan (2026-04-28) — `experiments/harness-arena/hybrid-model-routing.md` and bead `prompt-language-sfd3`. The planned HA-HR1 pilot compares local-only, frontier-only, advisor-only, and dynamic hybrid-router arms.

Provider telemetry is now captured for bounded smoke prompt turns across the
primary runners: Ollama records local token/duration/GPU evidence, Codex parses
JSONL token/timing events where exposed, and Claude parses structured JSON
usage/cost plus metadata-only session-log fallback. Still not measured / not
claimed: thesis-level cost, latency, or pass-rate deltas for H1-H6. No E5 pair
has produced a `maintenanceViabilityIndex` number, and cloud cost claims still
require provider-reported cost or a versioned pricing basis.

The first repeated smoke `A` telemetry slice is now measurable through
`npm run eval:smoke:summary`: Ollama `qwen3:8b` failed `0/3`, Codex `gpt-5.2`
passed the available local artifacts with unknown cost basis, and Claude passed
with provider-reported cost. This is operational measurement evidence, not a
thesis-level local-vs-cloud comparison.

## 3a. Claim-eligibility rule

A run becomes **claim-eligible** only once ALL of the following hold. Today
**zero repo runs satisfy all five**. The checked-in signer registry now contains
`operator-45ck-2026-05`, so gate 5 is satisfiable in principle, but the signed
bundle still fails verifier/witness/path checks and has only a stub
cross-family review.

1. `PL_TRACE=1 PL_TRACE_STRICT=1` was set for the run (strict tracing)
2. Preflight envelope returned `overall: ready` (not `degraded` or `blocked`)
3. An attestation-aware verifier pass exited 0 with `--require-attestation --require-role operator`
4. Cross-family reviewer lane ran AND did not veto (factoryFamily ≠ reviewerFamily)
5. Bundle includes `attestation.json` signed by a non-placeholder operator key in `docs/security/trusted-signers.json`

Any other run is **recorded** (evidence archived, trace verified at whatever
level was possible) but **not counted toward thesis verdicts**.

| Item                         | Shipped?                                                                                                                    | Blocker                                        |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| Strict mode (1)              | yes (commit ec086e8)                                                                                                        | operator must set the env var                  |
| Preflight (2)                | yes; `blocked` refuses live runs and `degraded` remains recorded-only                                                       | operator environment must reach `ready`        |
| Attestation flags (3)        | yes + hardened (2026-04-17); full 9-field payload, raw-64-byte ed25519 pinned, `dev` role removed, verify order §6.2 locked | signed bundle must pass verifier strictness    |
| Cross-family reviewer (4)    | yes; phase-2 enforcement shipped across preflight, review module, scorecard, and verifier                                   | reviewer lane must remain distinct and approve |
| Trusted-signers registry (5) | yes; `operator-45ck-2026-05` is committed as an operator signer                                                             | use it in a verifier-clean bundle              |

## 3b. Verification state of closed beads

Beads are closed when a commit lands. That does not always mean the
outcome was verified by running the experiment. Honest audit:

| Bead                                          | Closed on         | Verification state                                                                                                                                                  |
| --------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| prompt-lenh (coverage → 90)                   | `1a53d7b`         | **Verified**: coverage test ran post-commit; 90.00%                                                                                                                 |
| prompt-y0hv (tamper-drill nightly)            | `0271483`         | **Committed but not run**: 10 unit cases pass locally; nightly job has never fired (workflow pending first schedule trigger)                                        |
| prompt-lm0o.4 (MF-1 original live run)        | `b567405`         | **Run but not claim-eligible**: bundle's own `report.json` says `success: false, reason: verify-trace-failed`. Code Claude authored was reverted per harness rules. |
| prompt-lm0o.1 (meta-factory scaffold)         | `08c0c08`         | **Verified**: parse-tested, dry-run works, 0 warnings                                                                                                               |
| prompt-lm0o.3 (DSL primitives PR1)            | `c28d3c4`         | **Verified**: 2745 unit tests pass including 23 new snapshot/rollback cases; no live smoke-tested                                                                   |
| prompt-axt5.4 (shared blinding regex)         | open              | Not started                                                                                                                                                         |
| prompt-xgav.1 (F4 security review)            | closed (doc-only) | **Verified**: doc shipped, no code change claimed                                                                                                                   |
| prompt-xgav.2 through .8 (G1/G2/G3 hardening) | closed            | **Verified**: T1–T18 + fuzz + shim integration tests pass; no live-run pressure                                                                                     |

## 4. In flight (as of this doc)

"In flight" = an agent or workstream has open work, not that a human is literally watching a terminal. The specific workstream labels the operator referenced (F2, G1, G2, G3, H1, H3 and bead IDs `prompt-axt5`, `prompt-lm0o`) are not grep-findable in `.beads/issues.jsonl` in that form — the beads tracker uses `prompt-language-<slug>` IDs. The concrete in-flight items I can verify from the tracker and the ranked-patch table:

- Verifier hardening track: **AP-9 hardening complete in code** (2026-04-17). Harness auto-sign and attestation-required verifier wiring are shipped in `scripts/experiments/meta/run-meta-experiment.mjs`; `docs/security/trusted-signers.json` now contains `operator-45ck-2026-05`. AP-5 external nonce, AP-1/2 mandatory `--state`, AP-3 shim-binary allow-list, AP-8 `--expected-pair-count`, and AP-9 detached attestation verification (with 9-field payload, pinned ed25519 encoding, build-time trust-root, operator/ci-only roles) are all shipped. Remaining claim-eligibility blockers are the signed-bundle verifier failures documented in `experiments/attested-smoke-2026-05-11/MILESTONE.md`: trace entry 0 `prevEventHash`, missing Ollama witness shim pairing, `--state` path resolution, and replacement of the stub cross-family review with a real reviewer. Ranked in `docs/security/witness-chain-attacks.md` §"Ranked patches". AP-6 already shipped in `60c5c40`.
- Related security beads open in `.beads/issues.jsonl`: `prompt-language-oei8` ("Capture nonce 32-bit insufficient for anti-spoofing"), `prompt-language-36ot` (Fagan inspection audit of 43f3227..70873bc).
- MF-1 re-run workstream: reproduce the live meta-run with `PL_TRACE_DIR` correctly propagated to the child `claude -p` process so `provenance.jsonl` is emitted and `verify-trace` can gate. The main remaining blocker is host readiness (`meta:preflight` must be `ready` for claim eligibility), not missing cross-family/preflight wiring.
- E5 B01 P01 live-pair execution: orchestrator is live (`scripts/experiments/e5/run-pair.mjs`); HTTP probes shipped (`app-boot.mjs` + `documented-endpoint-probe.mjs` in `scripts/experiments/e5/`). Awaiting live pair run with auth on a supported host.
- Bounded factory-runtime proof follow-up: committed bundles now show one clean bounded Codex completion (`20260418-083500`) and one Claude terminal bundle that still closes over failed children (`20260418-074500`). The current tree fixes that await-integrity bug; the next useful rerun is to produce a post-fix Claude bundle that cannot false-complete past failed children.
- Generic runner abstraction: `prompt-language-9uqe.4` — in_progress. Headless prompt-turn seam is shipped (OpenCode, codex, aider); spawned-session runner abstraction remains.
- Codex parity matrix closure: `prompt-language-72a5.6` — open; blocked on supported-host smoke evidence.

## 5. Known unknowns / blockers (by urgency)

1. **Live-agent auth on a supported host.** A blocker between the current scaffolds and a first claim-eligible E5 pair. Native Windows `claude -p` currently exits with an org/access error for the smoke harness (bead `prompt-language-040u`). Codex binary is available on this host.
   1a. **E6/E7 cold-start runtime bugs** (L3, 2026-04-17). Live smoke surfaced 5 issues preventing either factory from running clean: `memory:`/`remember` race, `verify-trace` seq=0 `prevEventHash` serialization, Windows shim bypass, `#` comments becoming prompt nodes, E7 silent 82s exit with missing `session-state.json`. Filed under `prompt-lm0o.17` (E6 PARTIAL) and `prompt-lm0o.18` (E7 FAIL). A narrower 2026-04-18 bounded proof now confirms a clean committed Codex completion for the discovery-only slice, and the current tree also fixes the await-integrity bug exposed by the latest Claude terminal bundle, but those narrower results do not clear the broader E6/E7 blockers.
   1b. **Attested-run verifier blockers.** Operator signer provisioning is done in principle (`operator-45ck-2026-05` is committed), but the first signed bundle is still not claim-eligible. Open blockers: trace entry 0 `prevEventHash` serialization, Ollama witness shim pairing, `verify-trace --state` path resolution, and a real cross-family reviewer lane.
2. ~~**MF-1 trace-dir propagation.**~~ **RESOLVED.** `run-meta-experiment.mjs` already sets `PL_TRACE`, `PL_TRACE_DIR` in child env; `claude-process-spawner.ts` forwards them. Needs re-run confirmation.
3. ~~**Snapshot/rollback PR1.**~~ **RESOLVED.** Runtime primitives shipped: `SnapshotNode`/`RollbackNode` in `flow-node.ts`, parser support, advance-flow handlers, `addSnapshot`/`applySnapshot` in `session-state.ts`, 23+ unit tests, smoke test AX.
4. ~~**Verifier hardening AP-5.**~~ **RESOLVED.** External nonce anchor shipped: `writeRunNonce`/`readRunNonce`/`deleteRunNonce` in `run-meta-experiment.mjs` now use a private per-user nonce store, and `verify-trace.mjs` enforces `--expected-run-id`/`--freshness-window-ms`. **Remaining:** clear the signed-bundle verifier blockers and produce the first verifier-clean operator-attested bundle.
5. ~~**Cross-family reviewer lane (META-5 operator sign-off #3).**~~ **RESOLVED FOR PHASE 2.** Design doc shipped at `docs/strategy/cross-family-reviewer.md`. Review module shipped at `scripts/experiments/meta/cross-family-review.mjs`. Family taxonomy with 10 families defined. Phase-2 verifier/schema support is in place: `verify-trace.mjs` recognises `--expected-reviewer-family`, the E5 scorecard schema carries `admissibility.crossFamilyReviewer`, and bootstrap-envelope preflight now hard-fails live runs on `blocked` reviewer setup while leaving `degraded` runs non-claim-eligible. **Remaining trust gap:** AP-9 attestation and first live claim-eligible evidence.
6. ~~**E5 journey suite HTTP probes.**~~ **RESOLVED.** `app-boot.mjs` and `documented-endpoint-probe.mjs` shipped in `scripts/experiments/e5/` (commit `0e66d34`). Journey suite can now boot apps and probe documented endpoints.

## 6. Operator checklist (next concrete actions)

1. Fix Beads source-of-truth drift so `bd list` works from this workspace and
   the current May 11 beads in `.beads/backup/issues.jsonl` are visible from the
   primary Beads store.

2. Clear the signed-bundle verifier blockers from
   `experiments/attested-smoke-2026-05-11/MILESTONE.md`: trace entry 0
   `prevEventHash`, `verify-trace --state` path resolution, Ollama witness
   pairing decision, and real cross-family review.

3. Produce a tiny claim-eligible attestation smoke bundle before adding more
   thesis claims.

   Required shape: strict trace, `overall: ready` preflight,
   `--require-attestation --require-role operator`, real cross-family review,
   and trusted signer `operator-45ck-2026-05`.

4. Define one claim-verifier profile and one runner capability boundary so
   smoke/eval/meta claim paths use the same strict evidence contract.

5. Run E1 repeated-failure elimination at real scale (`prompt-language-j64j`):
   10 recurring failure patterns, locked oracles, plain-prompt baseline,
   prompt-only checklist arm, and PL recovery arm.

6. Only after the above, replay hybrid/local claims with better controls:
   cross-family microtask oracle replay, HA-HR1 H14 matched baselines, and one
   actual PL-runtime code-substitution path.

7. Re-run MF-1 live only when the claim-eligibility substrate is ready.

   ```
   npm run experiment:meta:live -- experiments/meta-factory/m1-pl-writes-smoke-test/m1.flow
   ```

   Expect a populated `provenance.jsonl` and `verify.json` in the new bundle under `experiments/meta-factory/results/<run-id>/`.

8. Execute the E5 B01 P01 pair end-to-end once supported-host auth is available.

   ```
   node scripts/experiments/e5/run-pair.mjs experiments/results/e5-maintenance/batches/e5-b01-mv-gpt52-pilot/pairs/p01-codex-first.json --run-id e5-b01-p01-live
   ```

9. Verify the runtime quality gate on a clean tree.

   ```
   npm run ci
   npm run eval:smoke
   ```

## 7. Failure modes to watch (from META-5 risk register)

- **MR-3 runtime regression.** Meta-runs and the rest of the test plane share the same `dist/`. A bad meta-change can silently regress smoke/CI. Defence: frozen `dist/` + worktree isolation (MD-1, MD-2). Still a risk whenever we modify CI-authoritative scripts during a meta-run.
- **MR-2 rule weakening.** A meta-run could edit CI thresholds, coverage floors, or verifier strictness to make its own output pass. Defence: `diff.protectedChanged` guard in `scripts/experiments/meta/run-meta-experiment.mjs`; `protectedChanged: []` must stay empty on every bundle.
- **MR-4 self-confirming bias.** PL grading PL, or the same judge model lane reviewing its own output, inflates thesis-supporting signal. E5 addresses this structurally (blinded second lane + change-request suite declared before factory runs); the risk returns the moment anyone short-circuits the blinding step.

---

Sources consulted when writing this doc: `git log --oneline -30`, `docs/strategy/thesis.md`, `experiments/results/e5-maintenance/program.md`, `experiments/meta-factory/README.md`, `experiments/meta-factory/design/*.md`, `experiments/meta-factory/results/meta-1776129598770-407100/report.json`, `experiments/results/factory-runtime-proof/`, `docs/evaluation/2026-04-18-runtime-factory-proof-codex-claude-medium-evidence.md`, `docs/security/witness-chain-attacks.md`, `scripts/experiments/e5/README.md`, `scripts/experiments/meta/README.md`, `scripts/eval/smoke-test.mjs`, `.beads/issues.jsonl`.
