# R1 Phase 3 — scenario-S2 4-SUT gap table (2026-06-11)

> Companion to Phase 2 ([`v0.4-r1-phase-2-baseline1-4sut-gap-2026-06-10.md`](v0.4-r1-phase-2-baseline1-4sut-gap-2026-06-10.md))
> and to the locked pre-registration ([PR #774](https://github.com/Hashevolution/James-RAG-Evol/pull/774),
> `docs/research/r1-phase-3-scenario-s2-preregistration-2026-06-10.md`).
>
> **The single most important reading**: the *gap structure* discovered
> on scenario-S1 reproduces exactly on scenario-S2 — same direction,
> same per-type breakdown, ten times the operation count. That is the
> ⭐⭐⭐ axis the pre-registration committed to. The *separate* RF-cost
> hypothesis (cycle γ's "graph build O(N²)" finding activating at
> larger scale) does **not** clear its sentinel threshold; it stays
> exploratory.

## 4-SUT gap table (RAB SPEC v0.1.1 / scenario-S2)

```
spec                 v0.1.1
scenario             S2 'lifecycle-large'
scenario_sha         1a91c256e9e58a7b89d1b94f605a2f727e373f52aecace9f0af2f4cebb3b7614
sut_version (HEAD)   aeb44c6540f9c0cf0164b9a90e954f3e4cea50c5
runner_env           Python 3.14.4 on Windows 11
ops                  400 (110 INGEST / 40 UPDATE / 30 SUPERSEDE / 20 DELETE / 200 QUERY)
checkpoints          40
```

| SUT | AC | AC INGEST | AC UPDATE | AC SUPERSEDE | AC DELETE | AC ANSWER | RF-exact | RF-graded | PC | log events | citations | traceable | RF cost (s/1k events) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reference (self-verify gate) | **1.000** | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | **1.000** | 1.000 | **1.000** | 800 | 531 | 531 | 0.0051 |
| **JAMES** (audit-native) | **1.000** | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | **1.000** | 1.000 | **1.000** | 800 | 597 | 597 | 0.1081 |
| **Baseline-1** (OTel GenAI bolt-on) | **0.500** | **0.00** | **0.00** | **0.00** | **0.00** | **1.00** | **0.000** | 0.000 | **0.000** | 400 | 597 | 0 | n/a |
| **Baseline-0** (vanilla quickstart + default logging) | **0.275** | **1.00** | 0.00 | 0.00 | 0.00 | 0.00 | **0.000** | 0.000 | **0.000** | 400 | 0 | n/a | n/a |

## S1 ↔ S2 cross-scenario comparison

Same SPEC v0.1.1 scorer, both scenarios use the same canonical
vocabulary; the only thing that changes is the corpus size (×10) and
the supersede-chain shape (avg 3.33 / max 5 vs S1's max 2).

| SUT | AC S1 | AC S2 | PC S1 | PC S2 | RF-exact S1 | RF-exact S2 | shape preserved? |
|---|---|---|---|---|---|---|---|
| Reference  | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ✓ |
| JAMES      | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ✓ |
| Baseline-1 | 0.500 | 0.500 | 0.000 | 0.000 | 0.000 | 0.000 | ✓ |
| Baseline-0 | 0.275 | 0.275 | 0.000 | 0.000 | 0.000 | 0.000 | ✓ |

Every cell is identical to the third decimal place. Same gap, ×10 scale.

## What this scores against the pre-registration

The pre-registration's §5 honest-tier ladder (locked before measurement):

| Pre-registered tier | Required evidence | Met? |
|---|---|---|
| ⭐⭐⭐ cross-scenario confirmed (gate cleared) | 4 SUT S1 pattern reproduced **and** RF-cost finding confirmed | ✗ — only first half met |
| ⭐⭐ cross-scenario partial + scale-axis pending | 4 SUT S1 pattern reproduced, RF-cost finding not confirmed | **✓ — this is the honest landing** |
| Exploratory | one or more SUTs diverge from S1 pattern | n/a |

The pre-registration was explicit that ⭐⭐⭐ requires *both* legs. Only
the first leg cleared. The handover records the second leg's outcome
honestly rather than narrowing the claim post-hoc.

## The two fail-modes reproduce — the paper's substantive headline strengthens

The Phase 2 finding (Baseline-0 and Baseline-1 fail at *different*
canonical event categories, with strictly disjoint catches) replays
unchanged on scenario-S2:

### Fail-mode A — Default logging (Baseline-0)
- AC INGEST = 1.000 (110/110); UPDATE/SUPERSEDE/DELETE/ANSWER = 0.
- "Indexed document …" is the only canonical category the default
  `logger.info` lines map to.
- 0 citations, 0 parent_id → RF = 0, PC = 0.

### Fail-mode B — OTel GenAI bolt-on tracing (Baseline-1)
- AC ANSWER = 1.000 (200/200); INGEST/UPDATE/SUPERSEDE/DELETE = 0.
- The semconv targets runtime LLM-call observability, not record
  keeping. Corpus lifecycle has no vocabulary in the spec.
- 597 citations emitted, 0 traceable → PC = 0; payloads carry no
  document state → RF = 0.

### Audit-native (JAMES, Reference)
- Lifecycle vocabulary + LLM-call vocabulary both present.
- Payloads carry enough to replay state from log alone.
- parent_id chain reaches origin events.
- 1.000 / 1.000 / 1.000 — same as S1.

**Why this strengthens the paper**: the v1.2 revision already lifted
Baseline-1 from Future Work to Main Results on S1 evidence alone.
Reproducing the disjoint-fail structure at 10× scale removes the
"S1 might be small enough to be cherry-picked" objection that
reviewers were going to ask about. Same gap, larger corpus, identical
direction and magnitude.

## RF-cost (the scale-axis hypothesis): exploratory, not confirmed

The pre-registration's §4 hypothesised that cycle γ's "graph build
O(N²)" finding (carried over from `project_cycle_gamma_phase_c2_retrieval_bottleneck`)
would activate visibly on a larger scenario.

Locked decision rule:

| Observed | Interpretation per pre-reg §4 |
|---|---|
| JAMES RF-cost @ S2 ≥ 0.05 s/1k events **and** S1 → S2 ratio ≥ 10× | ⭐⭐ — RF-cost finding confirmed |
| Either condition fails | exploratory; RF-cost column definition kept, claim withheld |
| JAMES RF-cost @ S2 > 1 s/1k events | adapter / `reconstruct_graph_at` implementation concern |

Observed values:

```
                 RF cost (s / 1k events)        S2 / S1 ratio
                 ──────────────────────         ──────────────
Reference        S1 0.0013   S2 0.0051          3.92×
JAMES            S1 0.0387   S2 0.1081          2.79×
Baseline-0/1     n/a (RF = 0 — nothing to fold)
```

JAMES @ S2 = 0.1081 s/1k events clears the first sentinel
(≥ 0.05 s/1k) but the S2 / S1 ratio is 2.79× — well below the locked
5× threshold. Reference's own scaling at 3.92× is in the same
neighborhood, which is a useful signal: the cost gradient JAMES shows
at this scale is closer to "modestly super-linear like the gate" than
to "O(N²) finding clearly active". A single S2 data point cannot
distinguish "linear-ish with constant factor" from "early O(N²) curve
that hasn't compounded yet".

Locked decision applies: **claim withheld; the column definition
remains in SPEC v0.1.1 §2.2**. Promotion would require an additional
scenario data point (S2.1 or S3 at ≥ 4× S2 op count) to fit a curve,
not a re-interpretation of the present number. This is the same
discipline that closed the multi-hop arc (cycle γ): a second
measurement, not a tightened framing of the first.

## Re-verification artifacts (SPEC §4 obligation)

All twelve files committed under `reports/rab/`:

```
reference-S2-20260610T232629Z.{result.json, log.jsonl, mapping.json}
james-S2-20260610T232638Z.{result.json, log.jsonl, mapping.json}
baseline0-S2-20260610T232643Z.{result.json, log.jsonl, mapping.json}
baseline1-S2-20260610T232645Z.{result.json, log.jsonl, mapping.json}
```

Any reviewer with the SPEC + fixture + these triples can re-run
`eval/rab/scorer.py` and reproduce every number above bit-for-bit
(the scorer is pure-functional).

**Note on `scenario_sha`**: the value `1a91c256…` recorded in every
S2 result.json is the SHA of the fixture as Python's text-mode
`read_text()` returns it (universal-newline normalised). The generator
script's announced SHA `604c2084…` is the SHA of the raw bytes the
generator writes. All four SUTs hit the same `read_text()` path, so
all four result.json files agree, which is what scoring requires. A
SPEC §3 normative clarification — "scenario_sha is the SHA of the
fixture file content as read by `Path.read_text(encoding='utf-8')`" —
should land in a future spec bump (v0.1.2 candidate, not in scope for
this PR).

## Honest framing — what this PR does NOT claim

Per pre-registration §5 + the project-wide honest-framing rules:

- ✗ "JAMES wins" — not the headline. The gap structure is the headline;
  JAMES happens to be the audit-native reference and scores accordingly.
- ✗ "RAB certifies EU AI Act compliance" — RAB **operationalises** the
  record-keeping / traceability / provenance concepts of Art. 10 / 12
  / 19. It does not certify compliance.
- ✗ "S1 and S2 scores blended" — each scenario reports its own
  numbers; we never average across scenarios.
- ✗ "RF-cost O(N²) finding confirmed" — claim withheld, see §4
  discussion above.
- ✗ "Mid-June joint piece evidence" — cycle γ closure and the joint
  piece are separate arcs ([`feedback_eval_cycle_vs_collab_arc_separation`](../../../../Users/karu-/.claude/projects/C--Project-James-RAG-Evol-v010/memory/feedback_eval_cycle_vs_collab_arc_separation.md)).
  Phase 3 belongs to the RAB / R1 internal arc.

## What this PR closes and what stays open

**Closed**:
- Pre-registration §8 steps 1–6 (fixture build → unit tests → Reference
  re-run → 4 SUT measurement → §4 RF-cost rule applied → handover).

**Open** (next R1 increments):
- Pre-reg §8 step 7: arXiv preprint v1.3 revision to integrate the S2
  cross-scenario evidence into Results (separate PR, no main-track
  blocker).
- Pre-reg §8 step 8: S3 (cross-lingual) sub-prereg as planned (a
  v0.1.3 candidate; out of scope for this PR).
- SPEC v0.1.2 candidate items: `scenario_sha` normative-form
  clarification; possibly a scenario-size guidance §3 amendment.
- RF-cost N² claim revisit: a third scenario data point at higher op
  count is the only honest path; not on the immediate roadmap.

## Pointers

- Phase 2 handover: [`v0.4-r1-phase-2-baseline1-4sut-gap-2026-06-10.md`](v0.4-r1-phase-2-baseline1-4sut-gap-2026-06-10.md)
- Phase 3 pre-registration (locked): `docs/research/r1-phase-3-scenario-s2-preregistration-2026-06-10.md`
- RAB SPEC v0.1.1: `eval/rab/SPEC-v0.1.md`
- Fixture: `eval/rab/scenarios/s2_lifecycle_large.json` (sha pinned per result.json)
- v0.4.3 software DOI (Zenodo): https://doi.org/10.5281/zenodo.20625533
- Cycle γ Phase D handover (RF-cost O(N²) finding origin): `docs/handovers/v0.4-cycle-gamma-phase-c2-musique-retrieval-bottleneck-2026-06-10.md`
