# R1 — Replayable-Audit Benchmark: design memo (외부 객관 인정 설계)

**Date**: 2026-06-10
**Status**: DESIGN VERIFIED — prior-art check complete (§2, arXiv API
2026-06-10): the benchmark does not exist and the gap is survey-certified
(arXiv 2606.04990 names "realistic execution-trace benchmarks" an open
challenge). **Spec freeze (R1.1) unblocked.** Remaining pre-freeze item:
EU AI Act Art. 10/12 verbatim clause verification (eur-lex).
**Origin**: 2026-06-09 project-direction review R1 — "측정 에너지가
모트가 아닌 축에 쓰이고, 모트(replayable audit)는 측정되지 않고 있다."
User constraint (2026-06-10): **"외부에서도 객관적으로 인정받을 수 있는
기준"** — the benchmark must not be a self-eval trap.

---

## 0. The self-eval-trap problem, stated honestly

JAMES defining a benchmark for JAMES's own moat is exactly the "엄마
100점" pattern (`feedback_self_evaluation_trap`) unless the design
**externalises every source of authority**:

| Authority | Self-eval trap | This design |
|---|---|---|
| What is measured (criteria) | JAMES decides | **External standards decide** — EU AI Act articles, W3C PROV, NIST AI RMF (§1) |
| How it is scored | LLM judge / JAMES oracle | **Deterministic only** — byte-diff, event counts. No LLM judge anywhere (§3) |
| Who can run it | only JAMES | **Generic spec** — any system that emits an audit log can be scored; baselines included (§4) |
| Who validates it | JAMES | **Publication + external replication** — spec/scorer/fixtures public (Zenodo DOI), third-party can re-run (§5) |

AbstentionBench (facebookresearch 2025) is the worked precedent: a lab
defined a benchmark for an under-measured axis and earned recognition by
publishing data + deterministic scorers + measuring *many* systems, not
just its own.

---

## 1. External anchoring — the criteria come from outside

**VERIFIED 2026-06-10** against Regulation (EU) 2024/1689 (Official
Journal version of 13 June 2024), fetched via
`scripts/research/_r1_eurlex_art12.py` (`reports/_r1_eurlex.txt`).
**Entry into force for these high-risk requirements: 2 August 2026**
(per Article 113) — 7 weeks from this design's date. The timing window
the 2026-06-09 review flagged is now clause-verified.

| Metric | EU AI Act anchor (verbatim) | Other anchors |
|---|---|---|
| **Audit Completeness** | **Art. 12(1)**: high-risk AI systems "shall **technically allow for the automatic recording of events (logs) over the lifetime of the system**"; **Art. 12(2)**: logging shall ensure "a level of **traceability of the functioning** … appropriate to the intended purpose". **Art. 19 (Automatically Generated Logs)**: providers keep Art. 12(1) logs ≥ 6 months | NIST AI RMF "Measure" (accountability); ISO/IEC 42001 audit-trail requirement |
| **Replay Fidelity** | **Art. 12(2)(b)**: logs must facilitate "the **post-market monitoring** referred to in Article 72" — i.e. the log must support post-hoc reconstruction of operation; RF scores exactly how reconstructable the state is from logs alone | Event-sourcing literature (deterministic fold); "The Log is the Agent" determinism contract (arXiv 2605.21997); ML reproducibility checklists |
| **Provenance Coverage** | **Art. 10(2)(b)**: data governance shall cover "data collection processes and the **origin of data**"; **Art. 10(2)(c)**: data-preparation lineage (annotation, cleaning, enrichment, aggregation); Art. 11 + Annex IV (technical documentation) | **W3C PROV-DM** (`wasDerivedFrom` chains); "From Agent Traces to Trust" provenance taxonomy (arXiv 2606.04990) |

The pitch to an external reviewer is therefore: *"these are the
regulation's own requirements — quoted, not paraphrased —
operationalised into deterministic, runnable metrics"* — not "JAMES's
idea of quality". Honest scope note (§7): RAB operationalises these
concepts; it does NOT certify Art. 12 compliance.

## 2. Prior art — VERIFIED 2026-06-10 (arXiv API survey) → spec freeze GO

Verification run: `scripts/research/_r1_prior_art_search.py` +
`_r1_prior_art_detail.py` (arXiv API, 6 queries + 5 full abstracts,
results in `reports/_r1_prior_art{,_detail}.txt`).

**Verdict: the benchmark R1 defines does NOT exist — and the field has
formally named the gap.**

| Prior art | What it is | What it means for R1 |
|---|---|---|
| **"The Log is the Agent" (ActiveGraph)**, arXiv 2605.21997 (2026-05-21) | Event-sourced runtime: append-only log = source of truth, graph = deterministic projection, **deterministic replay + log-only reconstruction + determinism contract** | ⚠️ **JAMES's replayable architecture is NOT novel** — independent co-invention of the same design. ActiveGraph proposes NO benchmark/metric ("without claiming to demonstrate"). → honest framing: JAMES = independent implementation of an emerging pattern; **the benchmark is the contribution, not the architecture**. ActiveGraph becomes an ideal *second audit-native measurement subject*. |
| **"From Agent Traces to Trust"** (survey), arXiv 2606.04990 (2026-06-03) | Survey + taxonomy of evidence tracing / execution provenance; maps existing benchmarks to provenance capabilities | ★ **Names "realistic execution-trace benchmarks" an explicit OPEN CHALLENGE** — the gap R1 fills is academically certified, not self-declared. The survey's taxonomy (trace sources / provenance relations / granularity / trust functions) joins §1 as an external anchor. |
| **PCAA**, arXiv 2606.04104 (2026-06-02) | Runtime governance via action certificates; "replay-ready proof" concept | Adjacent (governance); cite for the replay-proof framing; not a benchmark. |
| **OpenClawBench**, arXiv 2605.29253 | Process-anomaly dataset over agent trajectories | Adjacent (process-side evaluation ≠ replay/audit quality). |
| **AuditBench**, arXiv 2606.10281 | LLM-as-investigator over security logs | Name collision only — measures LLMs reading logs, not systems producing replayable logs. **Avoid the name.** |
| GraphRAG provenance, arXiv 2605.15109 | Citation faithfulness as trajectory-level provenance | Supports PC's direction (citation chains insufficient w/o trajectory provenance — noted as v2 extension). |

**Three consequences for the spec:**
1. **GO** — no existing benchmark to adopt; R1 proceeds as a response to
   a survey-certified open challenge (cite 2606.04990 §open-challenges
   in the spec's motivation).
2. **Honest-framing correction**: drop any "JAMES invented event-sourced
   replay" implication. Position: *two independent audit-native runtimes
   now exist (JAMES, ActiveGraph); no way to compare them or quantify
   their gap vs bolt-on tracing — R1 is that instrument.*
3. **Naming**: avoid "AuditBench" (taken, different meaning). Working
   name: **RAB (Replayable-Audit Benchmark)** until release naming.

Remaining verification: EU AI Act Art. 10/12 exact clause text (eur-lex
fetch) before the §1 mapping table is cited verbatim in the frozen spec.

## 3. The three metrics — deterministic definitions

### 3.1 Audit Completeness (AC)
> Of the decision-bearing events the system actually executed, what
> fraction emitted an audit row sufficient to identify the event?

- Ground truth: an **instrumented run** — hooks at each pipeline stage
  record the events that actually happened (retrieve, rerank, synth,
  verify, lifecycle mutations…). The hook list is published in the spec.
- Score = |audited events ∩ executed events| / |executed events|.
- Deterministic: event-id set intersection. No judgement calls.
- JAMES surface: `audit_bridge` rows (§5.7.2 trace schema) vs pipeline
  hooks.

### 3.2 Replay Fidelity (RF)
> Given ONLY the audit log, how exactly can a third party reconstruct
> the system's state and decision path at any past time t?

- Protocol: run a published **event scenario** (a fixed sequence of
  ingests / updates / supersedes / deletes / queries). At each step k,
  snapshot the live state (ground truth S_k). Then, *from the audit log
  alone*, reconstruct R_k = `reconstruct_graph_at(t_k)`.
- Score = fraction of checkpoints where R_k ≡ S_k (canonical-form
  byte-identical), plus a graded edge/node overlap (Jaccard) for
  partial credit.
- Deterministic: canonical serialisation + diff.
- JAMES surface: `core/lifecycle/replay_graph.py` (already implemented +
  112 tests); the benchmark turns the existing invariant into a *score*.

### 3.3 Provenance Coverage (PC)
> Of the claims in the system's answers, what fraction is traceable to
> a source through an unbroken machine-readable chain?

- Each answer's cited sources must chain back (answer → synth context →
  retrieved doc → ingested artifact) expressible as W3C PROV
  `wasDerivedFrom` edges from the logs.
- Score = traceable-citation count / citation count (deterministic:
  follow ids through the log; a break = untraceable).
- Caution: "claims" segmentation can drift toward judgement — v1 scopes
  to **cited sources only** (ids in the answer), which is fully
  deterministic. Claim-level coverage is a v2 extension with its own
  honesty review.

### Scale axis (from cycle γ's finding)
Graph build O(N²) (2026-06-10 finding) means replay **cost** matters at
scale. Report RF with a **cost column** (events/sec folded, wall-time
per 1k events) so the benchmark captures moat-at-scale, not just
correctness on toy logs.

## 4. Generic + comparable — not a JAMES-only yardstick

The spec defines an **abstract audit-log interface** (event row: id,
ts, type, inputs-hash, outputs-summary, parent) and scores ANY system
that can export it:

- **Baseline 0**: vanilla RAG (LangChain/LlamaIndex quickstart, default
  logging) — expected to score near-zero on RF (no event-sourced state)
  and low on AC. This is the honest contrast that quantifies the moat.
- **Baseline 1**: same stack + LangSmith/OTel tracing — measures how far
  *tracing tools* get you (likely: AC mid, RF still near-zero — traces
  ≠ replayable state).
- **JAMES**: full pipeline.

Publishing scores for systems *we don't own* is what makes the bench a
benchmark rather than a brochure.

## 5. External-recognition path (the user's constraint)

1. **Spec + scorer + fixtures public** (GitHub + Zenodo DOI) — anyone
   can re-run; scoring is deterministic so re-runs reproduce.
2. **Standards mapping table** (§1) published in the spec — reviewers
   check the mapping against the regulation text, not against JAMES.
3. **Third-party replication invite** — Robin/Ali (existing
   collaborators) run the scorer on their stacks; their numbers appear
   in the release. (Collab-arc rule: this is an *invitation with its own
   scope*, NOT folded into the mid-June joint piece —
   `feedback_eval_cycle_vs_collab_arc_separation`.)
4. **Honest tier**: the benchmark itself is the contribution
   (methodology). JAMES scoring well on its own bench is expected and
   NOT the headline; the headline is the *measurable gap* between
   audit-native and bolt-on-tracing architectures, reproducible by
   anyone.

## 6. Work plan (each step pre-registered before its measurement)

```
R1.0  prior-art online verification (§2)            — blocks spec freeze
R1.1  spec freeze: 3 metric definitions + standards mapping + abstract
      log interface (this memo → versioned spec doc)
R1.2  fixture: published event scenario (ingest/update/supersede/delete/
      query sequence) — synthetic but public + deterministic
R1.3  scorer implementation (deterministic; no LLM) + unit tests
R1.4  pre-register thresholds → measure JAMES + Baseline 0 (+1)
R1.5  release: spec + scorer + fixtures + all scores (Zenodo DOI)
R1.6  replication invites (separate collab scope)
```

Estimated: R1.1–R1.3 ≈ 1–2 days; R1.4 ≈ hours (deterministic, no LLM
calls in scoring); R1.5–R1.6 operator-paced.

## 7. What this is NOT (scope guards)

- NOT a MuSiQue/quality bench — quality axes are covered elsewhere.
- NOT a conformity-assessment substitute — it *operationalises* Art.
  10/12 concepts; it does not certify compliance (say so explicitly in
  the spec to stay honest).
- NOT joint-piece content (separate arc; 4-question check applies).
- NOT frozen until §2 prior-art verification runs online.
