# Reproduction — Roy-1a llm-primary on held-out fixture

**Companion:** [16_roy_1a.md](../16_roy_1a.md)
**Iteration spec:** [scenarios/roy/roy_1a_iteration.yaml](../../../scenarios/roy/roy_1a_iteration.yaml)
**Held-out fixture:** [scenarios/roy/roy_1_holdout.yaml](../../../scenarios/roy/roy_1_holdout.yaml)

Roy-1a is the first "real" iteration on the persona-convergence harness after the Roy-0 smoke test. Two methodology axes differ from Roy-0:

1. **`aut_mode: llm-primary` at test time.** Roy-0 ran substrate-primary throughout; its cold-start regime produced a `sense_food_source` cluster monoculture (6 cluster ids, one tool). llm-primary gives the AUT the validated proposer so we can isolate the open question: *does substrate-only priming produce LLM-readable signal at test time?*
2. **Held-out test fixture.** Roy-0 reused the priming fixture (`scenarios/cradle/warmup.yaml`, 3 percepts); arms could only be measured on rehearsal, not generalisation. Roy-1a uses [`roy_1_holdout.yaml`](../../../scenarios/roy/roy_1_holdout.yaml) — 10 percepts split across three classes (matching / novel / unrelated) per [persona_convergence_crucible.md §"Test phase"](../../plans/persona_convergence_crucible.md).

## Prerequisites

- Maxim checkout at or after the Roy-1a merge commit (or this branch: `feat/roy-1a-llm-primary-holdout`).
- `pip install -e .` (or `PYTHONPATH=src`).
- A working LLM provider for the `large` lane. Either:
  - A reachable leader at `MAXIM_LANE_LARGE_REMOTE_URL` (or `~/.config/maxim/peer.yml`) serving `/v1/chat/completions`, OR
  - An exported cloud API key (`ANTHROPIC_API_KEY` is cheapest), OR
  - A local llama.cpp lane (`maxim doctor` reports `large` healthy).
- Writable `~/.maxim/` (or `MAXIM_DATA_HOME=<path>`).

## A. Pre-flight (3 min)

Before burning ~25 min of LLM time, validate the leader + the spec:

```bash
# 1. Spec parses?
maxim roy run scenarios/roy/roy_1a_iteration.yaml --dry-run

# 2. Leader healthy? (HTTP 200 or 401 is alive; anything else is a failure)
curl -si --max-time 10 \
  -H "Authorization: Bearer $(yq .api_key ~/.config/maxim/peer.yml)" \
  "$(yq .url ~/.config/maxim/peer.yml)/models" | head -5

# 3. Fixture activates? — a 2-turn ad-hoc sim against local mistral
maxim --sim scenarios/roy/roy_1_holdout.yaml --interactive false \
  --sim-max-turns 2 --aut-mode substrate-primary \
  --embodiment bodies/infant_humanoid 2>&1 | head -40
```

Pre-flight passes when (1) prints `OK: roy-1a — ...`, (2) returns HTTP 200/401, and (3) the fixture's first percept (`heat blooms across your fingertips.`) reaches `agent_loop user_input_received`.

## B. Roy-1a (≈20-25 min wall)

```bash
# Kill any stale sims first (per CLAUDE.md "Kill stale sims before running tests").
pkill -f "maxim.*sim" 2>/dev/null

# Run, capturing both the human-readable log and the JSONL trace.
MAXIM_LOG_FILE=/tmp/roy_1a_live.jsonl MAXIM_BACKEND_TRACE=1 \
  maxim roy run scenarios/roy/roy_1a_iteration.yaml 2>&1 | tee /tmp/roy_1a_run.log
```

The JSONL trace is what makes failures debuggable — without it, `dispatch_exhausted` warnings only hit stderr and the orchestrator's narrator silently falls back to "The story continues to unfold around you..." with no useful signal about WHY.

## Expected output

Numbers below are from the recorded run on 2026-05-12 against the leader at `https://maxim.dennyschaedig.com/v1` (qwen2.5-14b-instruct via cloudflared). Treat them as rough thresholds, not point estimates — operating point depends on leader latency, seed, and cold-load timing.

### Priming (50 turns, substrate-primary, ~12-15 min)

Same shape as Roy-0 priming — the priming-side configuration is identical between Roy-0 and Roy-1a so the per-stage shape is a single-variable comparison.

Five stages of `cradle_prelinguistic` × 10 turns:

- `act1_neonatal_a`
- `act1_neonatal_b`
- `act2_primary_circular_a`
- `act2_primary_circular_b`
- `act3_secondary_circular`

`resume_session` should chain through all 5 stages. `final_session_id` is forwarded to arm A.

### Arms (10 turns each at llm-primary, ~3 min each)

| Arm | Substrate | system_prompt | turns | finish_reason |
|---|---|---|---|---|
| a | from_priming | neutral | 10 | cancel |
| b | blank | You are a hungry infant | 10 | cancel |
| c | blank | neutral | 10 | cancel |

`turns=10, finish_reason=cancel` reflects the 10-percept fixture being exhausted — not a failure. With llm-primary AUT and 10 test percepts each, expect 10-30 LLM calls per arm.

### Divergence thresholds (recorded run; refresh once you have seed variation)

See [16_roy_1a.md § Result](../16_roy_1a.md) for the headline table. The numeric values are auto-generated by `maxim roy log roy-1a`; this protocol just describes how to reproduce them.

Quick read of the headline metric:

```bash
jq '.pairwise_diffs.a_vs_b.nac.cluster_reward_bias' ~/.maxim/roy/roy-1a/result.json
jq '.pairwise_diffs.a_vs_b.hippocampus' ~/.maxim/roy/roy-1a/result.json
jq '.pairwise_diffs.a_vs_b.nac | {reward_bias_l2, causal_link_count_delta, goal_reward_bias_l2}' \
   ~/.maxim/roy/roy-1a/result.json
```

## What to do if it fails

**Priming aborts on stage 1 within ~30s** — the leader is unreachable or the local-LLM lane is broken. Re-probe with `curl` (per pre-flight (A) above). `result.preflight` carries the diagnostic; the runner aborts at preflight if `outcome != "ok"` and `outcome != "auth_rejected"`.

**Priming completes but arm A's substrate diff against C shows ~0 across the board** — the `resume_session` handoff broke. Inspect `result.priming.final_session_id` and verify arm A's `resumed_from` matches it. Then check that `~/.maxim/sim_reports/<priming-final>/aut_nac.json` is > 0 bytes — substrate snapshots write at session-end only.

**Every arm shows `turns=0, finish_reason=error`** — the AUT loop crashed or never booted. Check `MAXIM_LOG_FILE` for `Traceback` / `ERROR` events.

**`cluster_reward_bias_l2 < Roy-0's 2.46 on a_vs_b`** — under llm-primary at test time, arm A may not invoke the same sense_food_source loop substrate-primary did. The post-run analysis should compare arm A's test-time actions.jsonl tool distribution against the priming session's; if llm-primary explored more broadly (or stalled on memory_recall), the cluster bias signal is *expected* to decrease.

**Valence KS = 0 across every pair** — under llm-primary, valence annotation should fire on outcomes that trigger pain/reward. If KS=0, either (a) no test-time actions produced bio-system reactions (then the question is whether the held-out fixture is engaging enough), or (b) the valence pipeline regressed since Roy-0.

## What changed vs Roy-0

Single-variable change at the test-time AUT mode. Priming is identical (same arc, same turns, same aut_mode=substrate-primary). The test fixture is different (10 percepts vs 3) and the test-time AUT proposer is different (llm-primary vs substrate-primary).

Useful diffs from `result.json`:

```bash
# Compare priming substrate carryover (should be similar across iterations)
diff <(jq '.priming | {stages: .stages | length, final_session_id}' \
        ~/.maxim/roy/roy-0-smoke/result.json) \
     <(jq '.priming | {stages: .stages | length, final_session_id}' \
        ~/.maxim/roy/roy-1a/result.json)

# Compare a_vs_b cluster_reward_bias
diff <(jq '.pairwise_diffs.a_vs_b.nac.cluster_reward_bias' \
        ~/.maxim/roy/roy-0-smoke/result.json) \
     <(jq '.pairwise_diffs.a_vs_b.nac.cluster_reward_bias' \
        ~/.maxim/roy/roy-1a/result.json)
```

## Optional: tail the JSONL during the live run

```bash
# In another terminal while the iteration runs:
tail -f /tmp/roy_1a_live.jsonl | jq -c 'select(.event=="peer_backend_call") | {ts: .ts, lane: .lane, agent: .agent_id, latency_ms: .latency_ms}'
```

Expected: ~30-60 `peer_backend_call` events during the arms phase (~10 per arm × 3 arms). If counts are dramatically off, the AUT proposer is either over-generating (loop) or under-generating (stalled).

## Cleanup

```bash
unset MAXIM_LOG_FILE MAXIM_BACKEND_TRACE
# Roy artifacts persist in ~/.maxim/roy/ and ~/.maxim/sim_reports/ — keep them for diff analysis.
```

## Related docs

- [`16_roy_1a.md`](../16_roy_1a.md) — outcome doc with the full result table + interpretation
- [`persona_convergence_crucible.md`](../../plans/persona_convergence_crucible.md) — three-arm methodology + Roy-1a iteration log entry
- [`15_g4_cluster_reward_wire.md`](../15_g4_cluster_reward_wire.md) — the substrate-primary cluster wire Roy-0 validated (Roy-1a inherits it; test-time learning still routes through llm-primary)
- [`roy_0_smoke.md`](roy_0_smoke.md) — Roy-0 protocol Roy-1a builds on
