# E4 Comparative Analysis

Date: `2026-04-12`

## Question

For the bounded CRM factory slice, is `prompt-language` better than direct Codex, and if not, what
is it better or worse at?

## Dataset

Runs considered:

- `A02-crm-http-headless`
- `20260412-0916-a03-core-proof-prebootstrapped`
- `20260412-1005-a04-core-proof-sequential`
- `20260412-1047-a05-core-proof-sequential-patched`
- `20260412-1407-a06-core-proof-paired-clean`
- `a07-e4-b01-s0-clean-gpt52-pilot-p01-s0-clean-codex-first`
- `a08-e4-b02-s0-clean-gpt52-primary-p01-s0-clean-codex-first`
- `a09-e4-b02-s0-clean-gpt52-primary-p02-s0-clean-pl-first`
- `a10-e4-b02-s0-clean-gpt52-primary-p03-s0-clean-codex-first`
- `a11-e4-b02-s0-clean-gpt52-primary-p04-s0-clean-pl-first`
- `a14-e4-b04-fq1-gpt52-pilot-p01-factory-quality-codex-first`
- `a15-e4-b04-fq1-gpt52-pilot-p02-factory-quality-pl-first`
- `20260414-0619-a16-core-proof-paired-clean`
- `20260414-0619-a17-core-proof-paired-clean`
- `20260414-0619-a18-core-proof-paired-clean`
- `20260414-0619-a19-core-proof-paired-clean`

Primary references:

- [research-method.md](./research-method.md)
- [comparison.md](./comparison.md)
- [A03 trace summary](./runs/20260412-0916-a03-core-proof-prebootstrapped/trace-summary.md)
- [A05 trace summary](./runs/20260412-1047-a05-core-proof-sequential-patched/trace-summary.md)
- [A06 trace summary](./runs/20260412-1407-a06-core-proof-paired-clean/trace-summary.md)
- [A07 trace summary](./runs/a07-e4-b01-s0-clean-gpt52-pilot-p01-s0-clean-codex-first/trace-summary.md)
- [A07 batch summary](./batches/e4-b01-s0-clean-gpt52-pilot/summary.md)
- [B02 batch summary](./batches/e4-b02-s0-clean-gpt52-primary/summary.md)
- [B04 batch summary](./batches/e4-b04-fq1-gpt52-pilot/summary.md)
- [B05 batch summary](./batches/e4-b05-fq1-gpt52-primary/summary.md)
- [B06 batch summary](./batches/e4-b06-s2-gpt52-pilot/summary.md)

## Main Findings

### 1. Product capability is at least at parity on the bounded CRM core slice

In `A03`, both `prompt-language` and direct Codex completed the same bounded CRM core slice and
passed `lint`, `typecheck`, and `test`.

That means the current evidence does **not** support the claim that prompt-language blocks or
degrades core product completion for this scope.

### 2. Direct Codex is better on raw throughput in the first claim-eligible clean `S0` batch

In `A03`, the direct Codex baseline ran with less launcher/path friction and cleaner top-level
evidence files. Prompt-language needed Windows-specific launcher fixes, state-root fixes, and had
misleading run-node timeout behavior before the end-state evidence was stable.

`A07` was the directional pilot. `B02` is the first claim-eligible clean throughput batch under the
same tightened `paired-throughput-s0-external-verification` envelope.

In `B02`:

- both lanes succeeded in all four eligible pairs
- the order split was balanced at two `codex-first` pairs and two `pl-first` pairs
- direct Codex reached median `timeToGreenSec` in `719.79s`
- prompt-language reached median `timeToGreenSec` in `1514.34s`
- direct Codex was faster in both order strata:
  - `codex-first`: `673.94s` vs `1372.77s`
  - `pl-first`: `743.49s` vs `1683.37s`

Prompt-language again reached the first relevant workspace write earlier on the secondary
exploratory metric, but that did not translate into faster verified completion.

So for this bounded slice and current harness, direct Codex currently has the advantage in **setup
simplicity** and **raw time-to-green**.

### 3. Prompt-language was better on auditability and experiment structure

Prompt-language preserved:

- persisted session state
- audit trails
- explicit gates
- intervention history
- canonical closure artifacts

That makes it materially better for **controlled software-factory experimentation**, even when the
final built slice is similar to the direct Codex result.

### 4. Early prompt-language failures were mostly runtime confounds, not factory-language failures

`A02` and `A04` should not be read as “prompt-language cannot build the product.” They are better
read as:

- `A02`: state invariant and audit-path setup failure
- `A04`: runner/runtime divergence and orphan-process cleanup failure

`A05` matters because it closes that loop: after the runtime fixes, the sequential prompt-language
lane completed successfully.

## Scientific Conclusion

The most defensible conclusion today is:

1. `prompt-language` is **not better than direct Codex on raw throughput** for the bounded CRM core
   slice under the current `S0` protocol.
2. `prompt-language` **is already better as an auditable, restartable, governed software-factory
   control layer**.
3. The clean-throughput question for this specific setting now has a credible batch-level answer:
   `codex-alone-better`.
4. Any future claim that prompt-language is “better” must now target a different dependent
   variable, specifically `factory-quality` or `recovery`, not the already-answered `S0`
   throughput archive.

Admissibility note:

- `B02` is the current batch-level throughput evidence because it satisfies the clean-pair, order,
  and trace requirements
- `A08` through `A11` are the constituent primary-comparison runs inside that batch
- `A07` remains a useful pilot, but it is no longer the main throughput reference
- `A06` remains useful supporting timing context, but it used the older asymmetric timing envelope
- `A03` remains useful paired evidence, but it is not throughput-admissible because lane-normalized
  timing was not captured
- `A02`, `A04`, and `A05` are supporting context, not primary throughput evidence

That is a credible result, not a weak one. It tells us where the actual advantage is and what still
needs to be tested.

## E4 Hypothesis Status

- `throughput`: answered for the archived bounded `S0` slice, currently `codex-alone-better`
- `factory-quality`: confirmed by `B04` + `B05`, currently `prompt-language-better` (FQ 10 vs 8 in all 4 counterbalanced pairs)
- `recovery`: answered at pilot strength by `B06`, currently `codex-alone-better`

## What Prompt-Language Looks Good For

Based on the runs so far, prompt-language is strongest when the goal is:

- explicit completion gates
- stable artifact requirements
- postmortem-friendly evidence
- multi-agent orchestration that should survive interruptions and restarts
- experiments where closure quality matters as much as end-state code

It is not better for this `S0` setting at:

- fastest path to one bounded code result
- lowest operational overhead
- easiest initial setup on Windows/headless environments

## Recommended Next Experiment

Recommended next step:

- stop spending more cycles trying to prove raw-throughput superiority for prompt-language on this
  exact bounded clean `S0` slice unless the prompt-language control surface changes materially
- move to experiments where prompt-language has a plausible causal advantage:
- trace-first factory-quality comparison
- forced interruption / resume
- multi-agent handoff with explicit gates
- required artifact closure under the same bounded scope
- governed restart after mid-run failure with preserved audit state

## Decision

The next experiment should not ask “can prompt-language build code at all?” That is already
answered.

The next experiment should ask:

> Does prompt-language outperform direct Codex on restartability, governed recovery, and
> multi-agent completion quality for the same bounded factory slice after raw-throughput parity has
> already failed?

## A06 Update

Run: `20260412-1407-a06-core-proof-paired-clean`

- comparative verdict: `mixed`
- throughput admissible: false
- admissibility reason: This is a clean paired timed run on the same common product contract, but throughput superiority remains provisional until order effects are counterbalanced or repeated clean pairs agree.

## A07 Update

Run: `a07-e4-b01-s0-clean-gpt52-pilot-p01-s0-clean-codex-first`

- comparative verdict: `mixed`
- throughput admissible: false
- admissibility reason: This is a clean paired timed run on the same common product contract, but throughput superiority remains provisional until order effects are counterbalanced or repeated clean pairs agree.
- timing envelope: `paired-throughput-s0-external-verification`
- pilot batch summary: `experiments/results/e4-factory/batches/e4-b01-s0-clean-gpt52-pilot/summary.md`

## A08 Update

Run: `a08-e4-b02-s0-clean-gpt52-primary-p01-s0-clean-codex-first`

- comparative verdict: `codex-alone-better`
- throughput admissible: false
- admissibility reason: This is a clean paired timed run on the same common product contract, but throughput superiority remains provisional until order effects are counterbalanced or repeated clean pairs agree.
- timing envelope: `paired-throughput-s0-external-verification`

## A09 Update

Run: `a09-e4-b02-s0-clean-gpt52-primary-p02-s0-clean-pl-first`

- comparative verdict: `codex-alone-better`
- throughput admissible: false
- admissibility reason: This is a clean paired timed run on the same common product contract, but throughput superiority remains provisional until order effects are counterbalanced or repeated clean pairs agree.
- timing envelope: `paired-throughput-s0-external-verification`

## A10 Update

Run: `a10-e4-b02-s0-clean-gpt52-primary-p03-s0-clean-codex-first`

- comparative verdict: `codex-alone-better`
- throughput admissible: false
- admissibility reason: This is a clean paired timed run on the same common product contract, but throughput superiority remains provisional until order effects are counterbalanced or repeated clean pairs agree.
- timing envelope: `paired-throughput-s0-external-verification`

## A11 Update

Run: `a11-e4-b02-s0-clean-gpt52-primary-p04-s0-clean-pl-first`

- comparative verdict: `codex-alone-better`
- throughput admissible: false
- admissibility reason: This is a clean paired timed run on the same common product contract, but throughput superiority remains provisional until order effects are counterbalanced or repeated clean pairs agree.
- timing envelope: `paired-throughput-s0-external-verification`

## B02 Batch Update

Batch: `e4-b02-s0-clean-gpt52-primary`

- throughput claim eligible: `true`
- batch verdict: `codex-alone-better`
- median `timeToGreenSec`: `prompt-language=1514.34s`, `codex-alone=719.79s`
- order-stratified result: direct Codex was faster in both `codex-first` and `pl-first` subsets

## A13 Update

Run: `a13-e4-b03-fq1-gpt52-pilot-p01-factory-quality-codex-first`

- scenario kind: `factory-quality`
- primary claim type: `factory-quality`
- primary endpoint: `factoryQualityOverall`
- comparative verdict: `inconclusive`
- factory-quality claim eligible: false
- prompt-language closure completeness: mixed
- codex-alone closure completeness: mixed
- admissibility reason: Both lanes completed the local factory-quality artifact contract, but the shared Windows `noslop` verification path was confounded, so this pilot is not comparative evidence.
- timing envelope: `paired-factory-quality-s0-external-verification`

## A14 Update

Run: `a14-e4-b04-fq1-gpt52-pilot-p01-factory-quality-codex-first`

- scenario kind: `factory-quality`
- primary claim type: `factory-quality`
- primary endpoint: `factoryQualityOverall`
- comparative verdict: `prompt-language-better`
- factory-quality claim eligible: true
- prompt-language closure completeness: strong
- codex-alone closure completeness: strong
- admissibility reason: This is a trace-first paired factory-quality run on the same common product contract with closure and lane-level evidence strong enough for a governed-factory comparison.
- timing envelope: `paired-factory-quality-s0-external-verification`

## A15 Update

Run: `a15-e4-b04-fq1-gpt52-pilot-p02-factory-quality-pl-first`

- scenario kind: `factory-quality`
- primary claim type: `factory-quality`
- primary endpoint: `factoryQualityOverall`
- comparative verdict: `prompt-language-better`
- factory-quality claim eligible: true
- prompt-language closure completeness: strong
- codex-alone closure completeness: strong
- admissibility reason: This is a trace-first paired factory-quality run on the same common product contract with closure and lane-level evidence strong enough for a governed-factory comparison.
- timing envelope: `paired-factory-quality-s0-external-verification`

## B04 Batch Update

Batch: `e4-b04-fq1-gpt52-pilot`

- factory-quality claim eligible: `true`
- batch verdict: `prompt-language-better`
- median `factoryQualityOverall`: `prompt-language=10`, `codex-alone=8`
- both pairs showed the same pattern: PL stronger on processConformance (2 vs 1) and reuseReadiness (2 vs 1)
- counterbalance satisfied: one codex-first pair, one pl-first pair
- status: pilot — promotion to primary requires additional pairs

## A16 Update

Run: `20260414-0619-a16-core-proof-paired-clean`

- scenario kind: `factory-quality`
- primary claim type: `factory-quality`
- primary endpoint: `factoryQualityOverall`
- comparative verdict: `prompt-language-better`
- factory-quality claim eligible: true
- prompt-language closure completeness: strong
- codex-alone closure completeness: strong
- admissibility reason: This is a trace-first paired factory-quality run on the same common product contract with closure and lane-level evidence strong enough for a governed-factory comparison.
- timing envelope: `paired-factory-quality-s0-external-verification`

Lane-level metrics from scorecard:

- PL: FQ 10 (closureQuality 2, processConformance 2, traceAuthority 2, reuseReadiness 2, claimStrength 2), overall 23, processConformance strong, verificationPassRate 1, timeToGreenSec 2302.13
- Codex: FQ 8 (closureQuality 2, processConformance 1, traceAuthority 2, reuseReadiness 1, claimStrength 2), overall 21, processConformance mixed, verificationPassRate 1, timeToGreenSec 1639.84

## A17 Update

Run: `20260414-0619-a17-core-proof-paired-clean`

- scenario kind: `factory-quality`
- primary claim type: `factory-quality`
- primary endpoint: `factoryQualityOverall`
- comparative verdict: `prompt-language-better`
- factory-quality claim eligible: true
- prompt-language closure completeness: strong
- codex-alone closure completeness: strong
- admissibility reason: This is a trace-first paired factory-quality run on the same common product contract with closure and lane-level evidence strong enough for a governed-factory comparison.
- timing envelope: `paired-factory-quality-s0-external-verification`

Lane-level metrics from scorecard:

- PL: FQ 10 (closureQuality 2, processConformance 2, traceAuthority 2, reuseReadiness 2, claimStrength 2), overall 23, processConformance strong, verificationPassRate 1, timeToGreenSec 1742.01
- Codex: FQ 8 (closureQuality 2, processConformance 1, traceAuthority 2, reuseReadiness 1, claimStrength 2), overall 21, processConformance mixed, verificationPassRate 1, timeToGreenSec 787.08

## A18 Update

Run: `20260414-0619-a18-core-proof-paired-clean`

- scenario kind: `s2-pre-verification`
- primary claim type: `recovery`
- primary endpoint: `resumeToGreenSec`
- comparative verdict: `codex-alone-better`
- recovery claim eligible: false
- admissibility: supporting-context
- admissibility reason: This is a trace-backed S2 governed-recovery pilot on the same bounded CRM contract, but it remains supporting context until repeated predeclared interruption/resume pairs agree.
- timing envelope: `paired-recovery-s2-pre-verification`

Lane-level metrics from scorecard:

- PL: partial, FQ 5, overall 14, processConformance mixed, verificationPassRate 0.67, recoveryQuality mixed, resumeToGreenSec n/a (did not reach green), failureClass product
- Codex: success, FQ 6, overall 17, processConformance mixed, verificationPassRate 1, recoveryQuality strong, resumeToGreenSec 189.75

## A19 Update

Run: `20260414-0619-a19-core-proof-paired-clean`

- scenario kind: `s2-pre-verification`
- primary claim type: `recovery`
- primary endpoint: `resumeToGreenSec`
- comparative verdict: `codex-alone-better`
- recovery claim eligible: false
- admissibility: supporting-context
- admissibility reason: This is a trace-backed S2 governed-recovery pilot on the same bounded CRM contract, but it remains supporting context until repeated predeclared interruption/resume pairs agree.
- timing envelope: `paired-recovery-s2-pre-verification`

Lane-level metrics from scorecard:

- PL: success, FQ 8, overall 19, processConformance strong, verificationPassRate 1, recoveryQuality strong, resumeToGreenSec 1011.81
- Codex: success, FQ 6, overall 17, processConformance mixed, verificationPassRate 1, recoveryQuality strong, resumeToGreenSec 144.28

## B05 Batch Update

Batch: `e4-b05-fq1-gpt52-primary`

- factory-quality claim eligible: `true`
- batch verdict: `prompt-language-better`
- median `factoryQualityOverall`: `prompt-language=10`, `codex-alone=8`
- both pairs showed the same pattern: PL stronger on processConformance (2 vs 1) and reuseReadiness (2 vs 1)
- counterbalance satisfied: one codex-first pair (A16), one pl-first pair (A17)
- status: primary — confirms the B04 pilot result
- combined with B04, there are now 4 clean counterbalanced factory-quality pairs all showing prompt-language-better

## B06 Batch Update

Batch: `e4-b06-s2-gpt52-pilot`

- recovery claim eligible: `false`
- batch verdict: `codex-alone-better`
- codex-alone `resumeToGreenSec`: A18=189.75s, A19=144.28s
- prompt-language `resumeToGreenSec`: A18=n/a (partial), A19=1011.81s
- counterbalance satisfied: one codex-first pair (A18), one pl-first pair (A19)
- status: pilot — first S2 recovery batch, supporting context only
- direct Codex recovered faster in both pairs; prompt-language had a partial failure in A18
