# Phase 2 — Baseline-1 (OTel GenAI) + 4-SUT gap table (2026-06-10)

> Companion to v0.4.3 ([Zenodo
> 10.5281/zenodo.20625533](https://doi.org/10.5281/zenodo.20625533))
> and to the R1.4 single-SUT-comparison handover
> (`v0.4-r1-4-gap-table-2026-06-10.md`).
>
> **This Phase 2 result upgrades the paper's main contribution**: the
> single audit-native-vs-default-logging gap that v1.1 reports becomes
> a 3-way decomposition that quantifies *which parts* of the gap close
> under bolt-on tracing and *which parts do not* — the substantive
> question reviewers were going to ask.

## 4-SUT gap table (RAB SPEC v0.1.1 / scenario-S1)

| SUT | AC overall | AC INGEST | AC UPDATE | AC SUPERSEDE | AC DELETE | AC ANSWER | RF-exact | RF-graded | PC | log events | citations | traceable |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Reference (self-verify gate) | **1.000** | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | **1.000** | 1.000 | **1.000** | 80 | 40 | 40 |
| **JAMES** (audit-native) | **1.000** | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | **1.000** | 1.000 | **1.000** | 80 | 40 | 40 |
| **Baseline-1** (OTel GenAI bolt-on) | **0.500** | **0.00** | **0.00** | **0.00** | **0.00** | **1.00** | **0.000** | 0.000 | **0.000** | 40 | 49 | 0 |
| **Baseline-0** (vanilla quickstart + default logging) | **0.275** | **1.00** | 0.00 | 0.00 | 0.00 | 0.00 | **0.000** | 0.000 | **0.000** | 40 | 0 | n/a |

## The two fail-modes of non-audit-native instrumentation

The single most important reading of this table:

> **AC = 0.275 (default logging) and AC = 0.500 (bolt-on tracing) are
> not on the same axis. They fail differently, and they fail at
> different canonical-event categories.**

Two distinct fail-modes, both produced by SUT classes that operators
have actually deployed:

### Fail-mode A — Default logging (Baseline-0)
- Catches the obvious thing: `logger.info("Indexed document …")` → AC INGEST = 1.0
- Cannot distinguish ingest from update from supersede → AC UPDATE = 0, SUPERSEDE = 0
- No canonical delete log → AC DELETE = 0
- No canonical answer event in the log (strings, not events) → AC ANSWER = 0
- AC overall = 0.275 (11 of 40 ops)
- No payload to fold → RF = 0, no `parent_id` → PC = 0

### Fail-mode B — Bolt-on OTel GenAI tracing (Baseline-1)
- Catches the LLM-call axis perfectly: `chat` span → AC ANSWER = 1.0
- **Cannot catch corpus lifecycle at all**: OTel GenAI semconv has no
  INGEST / UPDATE / SUPERSEDE / DELETE vocabulary (the source spec
  targets runtime LLM-call observability, not record-keeping)
  → AC INGEST = 0, UPDATE = 0, SUPERSEDE = 0, DELETE = 0
- AC overall = 0.500 (20 of 40 ops)
- Span payloads carry token counts and messages but not document
  state → RF = 0
- ANSWER has a parent-chain back to RETRIEVE, but RETRIEVE has no
  INGEST event to chain to → **49 citations emitted, 0 traceable**

### Audit-native (JAMES, Reference)
- Both classes covered: lifecycle vocabulary + LLM-call vocabulary
- Payloads carry enough to replay state from log alone
- `parent_id` chain reaches origin events
- 1.000 / 1.000 / 1.000 across the board

## Why this is the headline upgrade

The v1.1 paper draft framed the contribution as "the audit-native vs
default-logging gap, quantified". A reviewer's first question against
that framing would have been: *isn't the floor a strawman? What about
real-world tracing (LangSmith, OTel)?* Phase 2 answers that question
**on the same scenario with the same scorer and a published, hash-pinned
mapping table**. The answer is:

* The bolt-on column closes the AC ANSWER half of the AC gap
  (0 → 1.0)
* The bolt-on column **opens a new gap** in AC INGEST (1.0 → 0) —
  the corpus-lifecycle half default logging was actually catching
* On RF and PC, the bolt-on column changes nothing: both remain 0

Net effect on AC overall: Baseline-0 → Baseline-1 is +0.225 (a real
improvement, not negligible), but the trade is structural: bolt-on
trades INGEST observability for ANSWER observability and gains
nothing on the replay or provenance axes.

This is what the gap-as-headline reporting protocol (Spec §6.5)
exists for: the number that matters is the per-canonical-type and
per-axis decomposition, not the single AC overall column.

## Per-row honest commentary

### Reference vs JAMES
Identical on S1, as the pre-registration (`8a35dee`) declared
expected and as Spec §6.5 explicitly disclaims as not-a-finding. The
purpose of including both is that the Reference adapter pins the
driver and scorer's correctness; JAMES then independently reproduces
those numbers using production code paths
(`core.lifecycle.replay_audit.emit_lifecycle_event` +
`core.lifecycle.replay_graph.reconstruct_graph_at`) under
workspace isolation. The cross-verification tests pin the bridge.

### Baseline-1 emits RETRIEVE log events the AC scorer cannot use
The Baseline-1 SUT correctly emits `retrieval` spans that map to the
RAB canonical RETRIEVE event type — but scenario-S1's driver
ground-truth (`e_exec`) does not contain RETRIEVE ops, only mutation
ops + ANSWER (per the `_OP_TO_CANONICAL` mapping in `driver.py`).
The Baseline-1 RETRIEVE events therefore appear in the exported log
but score nothing in AC. The honest report:
- AC scoring is over the driver's ground-truth canonical set,
  per Spec §2.1
- Future scenarios that add RETRIEVE / RERANK / SYNTH as
  driver-side ground truth would let those canonicals contribute
  to AC; Spec v0.1.1 deliberately keeps S1 to the
  highest-stakes-for-audit subset (lifecycle + answer)
- The Baseline-1 result is therefore exactly the AC the source spec
  (OTel GenAI semconv) lets it earn against the AI Act
  record-keeping ground truth — not a lower number than the SUT
  deserves on a richer ground truth

### Baseline-1's 49 citations, 0 traceable
The chat span carries `gen_ai.response.citations`, and the
mapping-table adapter surfaces them as the SPEC `payload.citations`
field, so PC's denominator is 49 (not 0 — the SUT *does* emit
citations). But PC's numerator is 0: no INGEST event exists in the
log, so the parent-chain `ANSWER → SYNTH → RETRIEVE →
(INGEST|SUPERSEDE)` dies at RETRIEVE. This is the clearest
single-number demonstration in v0.1.1 of *why* corpus-lifecycle
vocabulary matters for provenance.

## Honest tier

⭐⭐ scenario-S1 4-SUT gap structure confirmed (deterministic,
re-runnable from artifacts under `reports/rab/`). Same ceiling as
the 3-SUT result; the upgrade is in the *shape* of the gap, not in
its dimensionality. The ⭐⭐⭐ cross-scenario ceiling remains
ungated; this Phase 2 result strengthens the case that the
single-scenario evidence is not a strawman, but does not by itself
satisfy the cross-scenario gate.

## Artifacts (committed, SPEC §4 re-verification triple)

```
reports/rab/baseline1-S1-20260610T114611Z.result.json
reports/rab/baseline1-S1-20260610T114611Z.log.jsonl
reports/rab/baseline1-S1-20260610T114611Z.mapping.json
```

Mapping table is the published source-of-truth file
`eval/rab/mappings/otel_genai_to_rab.json` —
`mapping_table_sha = 88b5c8…` in the result.json pins it.
Re-running `python scripts/research/rab_run.py --sut baseline1` on
the same git tree reproduces the AC / RF / PC numbers bit-for-bit;
timestamps differ.

## Test pin (42/42 green)

| File | Tests | Role |
|---|---|---|
| `tests/test_rab_benchmark.py` | 14 | Reference adapter + driver + scorer + 3 fault injections |
| `tests/test_rab_baseline0.py` | 8 | Default-logging floor pinned |
| `tests/test_rab_baseline1.py` | **11 (new)** | Bolt-on OTel tier pinned: ANSWER 1.0, lifecycle 0, RF 0, PC 0, mapping table audited, emitted RETRIEVE not in ground-truth set |
| `tests/test_rab_james_adapter.py` | 9 | Audit-native + workspace isolation + lifecycle bridge + reconstruct_graph_at agreement |

## Future Work re-mapping in light of Phase 2

| Future-work track (paper v1.1 §Future Work) | After Phase 2 |
|---|---|
| Baseline-1 OTel/LangSmith adapter | ✅ Done — Baseline-1 result is in this handover |
| Scenario-S2+ (larger graph + cross-lingual + different vocabulary basis) | Still queued; now with stronger motivation (Phase 2 isolates *which* canonical types need lifting to ground truth to credit bolt-on more fully) |
| Replication invites (ActiveGraph, DFAH, …) | Still queued (separate collab arc, 4-question gate) |
| Mutation-site wiring (v0.4.2 carry-over) | Still queued |

## Suggested paper revision (v1.2 draft after arXiv submit)

This Phase 2 result was named in the v1.1 paper's Future Work as the
"highest single-step lift in scientific value". Now that it exists,
the natural paper revision is:

* Replace the v1.1 3-row gap table with the 4-row table above
* Add a §Analysis subsection enumerating the two fail-modes
  (default-logging-misses-LLM-events vs bolt-on-misses-corpus-lifecycle)
* Move "Baseline-1" from Future Work into Results
* Bump the SPEC release version note in §Reproducibility to refer
  to the same v0.1.1 spec but now with three SUT-class baselines
  measured

Whether to do this revision **before** the arXiv preprint posts or
**as v2 after the initial post** is an operator decision. The v1.1
draft is honest as-is (it names Baseline-1 as the single highest-
value future-work item); revising to include Phase 2 makes the paper
stronger but delays submission. Both choices are defensible.
