# Experiments

This is the docs entry point for prompt-language evidence and experiments. It keeps
the product README clean: stable usage docs stay in guides/reference, while
measurements, hypotheses, caveats, and raw run artifacts live here or under
[`../experiments/`](../experiments/).

For the current short answer to "what works now?", start with
[What Works Now](evaluation/what-works-now.md). For the current post-FSCRUD evidence
boundary, see
[Evidence Snapshot: 2026-05-06](evaluation/2026-05-06-evidence-snapshot.md). For the
May 12 local-model claim-profile receipt, see
[Local Ollama Smoke With Claim-Profile Classification](evaluation/2026-05-12-local-ollama-smoke-claim-profile.md).
For the governed local/frontier routing sequence after OpenAI Symphony, see the
[Post-Symphony GSLR Research Decision](evaluation/2026-05-12-post-symphony-research-decision.md)
and the
[GSLR-2 Policy Schema Runbook](../experiments/harness-arena/GSLR-2-POLICY-SCHEMA-RUNBOOK.md).
For the post-live route decision, see
[GSLR Route Policy After Live Schema Evidence](evaluation/2026-05-12-gslr-route-policy-after-live.md).
For the next research conclusion and fixture-family sequence, see
[GSLR Next Research Conclusions](evaluation/2026-05-12-gslr-next-research-conclusions.md).
For the first fixture-family rung, see the
[GSLR-3 Policy Manifest Transform Runbook](../experiments/harness-arena/GSLR-3-POLICY-MANIFEST-TRANSFORM-RUNBOOK.md).
For the post-GSLR-3 decision on what to run next and why, see the
[Post-GSLR-3 Research Decision](evaluation/2026-05-12-post-gslr3-research-decision.md).
For the GSLR-3 live result, see
[GSLR-3 Live Policy-Manifest Transform Result](../experiments/harness-arena/results/gslr3-live-2026-05-12/report.md).
For the next scaffolded fixture, see the
[GSLR-4 Two-File Validator Runbook](../experiments/harness-arena/GSLR-4-TWO-FILE-VALIDATOR-RUNBOOK.md).
For the GSLR-4 live result and post-run decision, see
[GSLR-4 Live Two-File Validator Result](../experiments/harness-arena/results/gslr4-live-2026-05-12/report.md)
and the
[Post-GSLR-4 Research Decision](evaluation/2026-05-12-post-gslr4-research-decision.md).
For the cross-experiment claim ledger, see
[Research Synthesis: 2026-05-06](evaluation/2026-05-06-research-synthesis.md). For the
raw research tree, run catalog, and dated receipts, see
[`../experiments/README.md`](../experiments/README.md).
For experiment design templates and methodology, see
[`../experiments/templates/`](../experiments/templates/).

## Boundaries

- Product guarantees belong in [guides/](guides/index.md) and
  [reference/](reference/index.md), not in experiment notes.
- Current evidence and caveats belong in [evaluation/](evaluation/index.md).
- Experiment designs, raw outputs, scorecards, and research work-in-progress belong
  in [`../experiments/`](../experiments/).
- External research synthesis belongs in [research/](research/README.md).

## Experiment catalog

| ID    | Name                         | Domain             | Status     | Summary                                                     |
| ----- | ---------------------------- | ------------------ | ---------- | ----------------------------------------------------------- |
| E1    | Repeated Failure Elimination | Reliability        | Seeded     | Gate enforcement vs prompt-only completion on a 3-case seed |
| E4    | CRM Factory                  | Enterprise SaaS    | Complete   | End-to-end CRM with spawn/await, foreach-spawn, race, retry |
| E5    | Maintenance Viability        | Maintainability    | Designed   | Blind second-lane maintenance protocol, not yet executed    |
| E7    | Enterprise CRM Factory       | Enterprise SaaS    | Scaffolded | Full-SDLC scaffold; 2026-04-17 live smoke failed            |
| E7-MK | Marketing Factory            | Marketing website  | Complete   | PL achieved 30/30 x3 vs solo 28.3/30 average                |
| E8    | Website Factory              | Enterprise website | Run 1 done | 6-phase enterprise build: Factory 3/4 vs Solo 4/4           |

## Comparison and benchmark experiments

| Name                                                          | Domain                      | Status     | Summary                                                                        |
| ------------------------------------------------------------- | --------------------------- | ---------- | ------------------------------------------------------------------------------ |
| [Aider vs PL](../experiments/aider-vs-pl/SCORECARD.md)        | Coding assistant comparison | Historical | Phase-1 directional signal; later local ladder is mixed and non-claim-eligible |
| [Full-stack CRUD](../experiments/fullstack-crud-comparison/)  | Local-model product build   | Active     | R30-R45 local diagnostics: implementation limits, bounded selector/ranker wins |
| [Harness Arena](../experiments/harness-arena/)                | Local/frontier routing      | Active     | GSLR-4 live result routes the two-file validator to frontier-baseline          |
| [Senior Pairing](../experiments/senior-pairing-protocol/)     | Local-model supervision     | Planned    | Senior-engineer supervision flow program                                       |
| [Premature Stop](../experiments/premature-stop-benchmark/)    | Reliability                 | Scaffold   | Repeated-stop and premature-exit comparisons                                   |
| [Bounded Feature](../experiments/bounded-feature-benchmark/)  | Implementation quality      | Scaffold   | Bounded implementation quality benchmarks                                      |
| [Parallel Planning](../experiments/parallel-planning/)        | Coordination                | Scaffold   | Plan quality and coordination experiments                                      |
| [Parallel Modules](../experiments/parallel-isolated-modules/) | Build concurrency           | Scaffold   | Isolated module build concurrency                                              |
| [Self-healing CI](../experiments/self-healing-ci/)            | CI repair                   | Scaffold   | CI repair and auto-fix via retry/try-catch                                     |

The next planned wave is documented in
[Non-Factory Proof Program](evaluation/non-factory-proof-program.md). That note
sequences the repo's next proof work around runtime truth, QA lift, and bounded
outcome lift rather than defaulting to larger factory claims.

The broader experiment-area map is maintained in
[`../experiments/README.md`](../experiments/README.md) and
[`../experiments/EXPERIMENT-AREAS.md`](../experiments/EXPERIMENT-AREAS.md).

## Detailed experiment descriptions

### E1: Repeated Failure

Tests whether verification gates catch failures that prompt-only workflows miss. Located in [experiments/eval/e1-repeated-failure/](../experiments/eval/e1-repeated-failure/) and [experiments/results/e1-repeated-failure/](../experiments/results/e1-repeated-failure/).

### E4: CRM Factory

End-to-end SaaS product factory generating a CRM application with entity generation, CRUD scaffolding, and deployment configuration. Uses spawn/await for parallel module builds. Located in [experiments/full-saas-factory/](../experiments/full-saas-factory/).

### E5: Maintenance Viability

Tests whether software produced under PL supervision is easier for a separate
blind lane to run, extend, and maintain than direct Codex output. The protocol and
templates exist, but no live pair has produced a maintenance viability score yet.
See
[experiments/results/e5-maintenance/](../experiments/results/e5-maintenance/).

### Runtime Trace Verification

Trace verification is product/evaluation infrastructure, not the E5 experiment.
Differential Z-series tests prove the runtime is executing live, not replaying
recorded outputs. Each test depends on fresh UUIDs, real child PIDs,
capture-gated branches, or hash-matched send/receive payloads. See
[docs/thesis-verification.md](thesis-verification.md) and
[docs/tracing-and-provenance.md](tracing-and-provenance.md).

The repo also carries a smaller bounded runtime-proof series under
[experiments/results/factory-runtime-proof/](../experiments/results/factory-runtime-proof/)
that exercises the CRM discovery slice through the same PL runtime-backed
Claude/Codex paths. The current committed evidence is asymmetric: `20260418-083500`
is the clean bounded Codex proof through `await all` and `review`, while
`20260418-074500` is the latest Claude terminal bundle but closes with both
spawned children failed. That series is useful as execution evidence, but it is
not an end-to-end factory completion claim on both hosts. Use `20260418-083500`
when the point is the clean bounded completion datapoint and `20260418-074500`
when the point is the latest provenance-backed Claude runner evidence, including
the historical await-integrity gap it exposed. See
[2026-04-18 Runtime Factory Proof: Codex + Claude, Medium Effort](evaluation/2026-04-18-runtime-factory-proof-codex-claude-medium-evidence.md).

### Non-factory proof program

The repo's next proof layer is intentionally not another giant factory.

The current planned sequence is:

- runtime-truth reruns and falsifiers
- repeated differential proof
- bounded QA and outcome-lift benchmarks

See [Non-Factory Proof Program](evaluation/non-factory-proof-program.md).

### Full-stack CRUD local-model diagnostics

The FSCRUD series is the current best evidence for local-model boundaries. R30-R39
showed that the local 30B lane remained brittle on executable product implementation
and rich senior-plan authoring. R40-R45 then showed a narrower working pattern:
bounded local section selection, decision-matrix choice, rubric-derived choice,
weighted ranking, rationale, and senior risk-response selection under deterministic
validation/rendering. The current conclusion is not "local model builds full-stack
CRUD"; it is "local model can provide bounded semantic choices inside a deterministic
PL control system." See
[Evidence Snapshot: 2026-05-06](evaluation/2026-05-06-evidence-snapshot.md) and
[experiments/fullstack-crud-comparison/](../experiments/fullstack-crud-comparison/).

### E7-MK: Marketing Factory

Marketing website generation comparing PL factory against solo prompting. PL achieved perfect 30/30 scores across three consecutive runs versus a solo average of 28.3/30. Located in [experiments/marketing-factory/](../experiments/marketing-factory/).

### Enterprise CRM factory scaffold

The enterprise CRM factory under
[experiments/full-saas-factory/e7-enterprise-crm-factory/](../experiments/full-saas-factory/e7-enterprise-crm-factory/)
is scaffolded and useful as a stress case for DSL surface area, but it should not
be cited as a clean live completion. The 2026-04-17 live smoke failed; current
evidence from the factory line is strongest on smaller bounded runtime-backed
slices, not the full enterprise factory.

### E8: Website Factory

Enterprise 6-phase website build: discovery, architecture, design system, implementation, QA, release. Uses 22 flow files and 8 reusable libraries with specialized agent assignments. Compares PL factory (Astro, 30 files, 12 docs) against solo (Next.js, 14 files). Run 1 scorecard: [experiments/website-factory/results/run1-scorecard.md](../experiments/website-factory/results/run1-scorecard.md). Located in [experiments/website-factory/](../experiments/website-factory/).

This is structured-prompt/process evidence, not clean PL-runtime evidence. The
reported run shows flow-shaped output and useful artifacts, but runtime
hooks/gates did not reliably drive the key result.

### Aider vs PL

Head-to-head comparison of Aider coding assistant against prompt-language across
10 hypotheses. Treat the phase-1 `6/0/3` result as historical dev-time evidence,
not claim-eligible thesis proof: it used N=1 narrative scoring and the later
local-model ladder produced mixed results. [Scorecard](../experiments/aider-vs-pl/SCORECARD.md).
Located in [experiments/aider-vs-pl/](../experiments/aider-vs-pl/).

### Meta Factory

Meta-level experiments: factories that generate factories. Explores self-hosting and recursive flow generation. Located in [experiments/meta-factory/](../experiments/meta-factory/).

### Full SDLC Factory

Full software development lifecycle factory with QA-heavy variant. Located in [experiments/full-sdlc-factory/](../experiments/full-sdlc-factory/).

## Supporting infrastructure

| Directory                                           | Purpose                                                                          |
| --------------------------------------------------- | -------------------------------------------------------------------------------- |
| [experiments/eval/](../experiments/eval/)           | Shared rubrics, scoring scripts, and evaluation helpers                          |
| [experiments/results/](../experiments/results/)     | Run outputs, scores, and analysis artifacts                                      |
| [experiments/templates/](../experiments/templates/) | Reusable experiment design templates (hypothesis, methodology, metrics, results) |

## Evaluation and evidence

For evaluation methodology, A/B results, and parity matrices, see:

- [Evaluation results](evaluation/eval-analysis.md)
- [What works now](evaluation/what-works-now.md)
- [Research Synthesis: 2026-05-06](evaluation/2026-05-06-research-synthesis.md)
- [Eval parity matrix](evaluation/eval-parity-matrix.md)
- [Thesis](strategy/thesis.md) and [thesis roadmap](strategy/thesis-roadmap.md)
