# Prompt Caching for Cloud Backends

**Status:** DRAFT 2026-06-08 (post-Sonnet/Haiku rate-limit cascade)
**Author:** Denny + Claude
**Motivates:** Exp 37 cross-model characterization (Sonnet replication blocked by
Anthropic tier-1 ITPM cascade on 2026-06-08); future cloud-LLM experiments.
**Out of scope:** New backend types; substrate / bio-system changes; LLM router
provider-fallback redesign.

## Why this exists

2026-06-08 Sonnet replication attempt cascaded into Anthropic tier-1 rate-limit
exhaustion after just 2 arms. Each cradle sub-sim consumed ~120K input tokens
across 12 turns (~30K ITPM sustained), right at the tier-1 ceiling. When Arm B
pushed over the limit, the bucket exhausted and every subsequent sub-sim
launched into 429-throttled API state and never recovered.

The root cause is architectural, not provider-specific: **the cradle scenario
re-sends a ~100K-token system prompt (entity manifests, scene state, bio-system
descriptions, sense_tools registry) every turn, identical across all 12 turns of
each sub-sim.** That's the textbook prompt-caching use case — large stable
prefix, varying suffix (the user's turn).

Anthropic's [rate-limits documentation](https://platform.claude.com/docs/en/api/rate-limits)
explicitly states that `cache_read_input_tokens` do **NOT** count toward ITPM on
Sonnet 4.x, Haiku 4.5, and Opus 4.x. With a single `cache_control: {"type":
"ephemeral"}` breakpoint at the end of the system prompt, the first turn writes
the cache (~100K tokens count toward ITPM at 1.25× billing), and turns 2-12 read
from cache (0 tokens count toward ITPM, billed at 0.1× input price).

**Effective ITPM reduction per sub-sim: ~120K → ~20K (6× lower).**
**Effective input cost reduction per sub-sim: ~75% on cached portion.**

This solves the tier-1 cascade structurally without requiring a tier upgrade,
and makes future cloud-LLM experiments dramatically cheaper across the board.

## What already exists in main

Surprise from the 2026-06-08 audit: the cache_control wiring is **already
implemented** in [`_AnthropicBackend._build_system_blocks`](../../src/maxim/models/language/anthropic_backend.py).
The infrastructure is shipped:

- `_build_system_blocks(system)` builds a `[{"type": "text", "text": system,
  "cache_control": {"type": "ephemeral"}}]` block when caching is enabled,
  falls back to `[{"type": "text", "text": system}]` otherwise.
- `_prompt_cache_enabled()` reads `cfg.get("prompt_cache")` from the per-provider
  config; supports both bool shorthand and `{"enabled": true}` dict form.
- Usage tracking already reads `cache_read_input_tokens` and
  `cache_creation_input_tokens` for billing (lines 285, 359).

**The gap is that no profile in [`profiles.yml`](../../src/maxim/_data/profiles/) sets
`prompt_cache.enabled: true`, so the flag is off-by-default and the wiring
never fires in production.** That's the 2026-06-08 incident's structural cause.

What does NOT exist:
- Same wiring in `_OpenAIBackend` (OpenAI also benefits from explicit
  `cache_control` markers for prompts >1024 tokens; their automatic caching is
  separate and runs on top of explicit markers).
- Cache-hit instrumentation in router-level logs (you can see token-counts in
  the backend's billing accumulator, but there's no per-request structured
  event saying "cache hit: 95K read / 5K new" for ops visibility).
- An audit of [`PromptBuilder`](../../src/maxim/agents/prompt_builder.py) for
  silent cache invalidators (timestamps, UUIDs, session IDs in the stable
  prefix would defeat caching even with the flag on).
- A CLAUDE.md invariant making the "stable prefix MUST stay stable" rule
  explicit so future PromptBuilder changes don't silently break caching.

## The prefix-invariant rule (load-bearing)

Per [Anthropic's prompt caching docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching),
the cache is a **prefix match** — any byte change anywhere in the prefix
invalidates everything after it. Specifically:

> A single byte difference at position N — a timestamp, a reordered JSON key,
> a different tool in the list — invalidates the cache for all breakpoints at
> positions ≥ N.

This means the wiring fix is necessary but not sufficient. If `PromptBuilder`
interpolates anything dynamic into the stable portion of the system prompt, the
cache will silently fail every request (showing `cache_read_input_tokens: 0`
across repeated requests with notionally-identical-looking prompts).

**The audit MUST happen before the flag flip.** Otherwise we ship a "fix" that
silently does nothing and we waste another night debugging it.

Known suspects from the codebase (need verification):
- Drive states in system prompt (per-tick values change every call)
- Bio-state annotations (Wire-A cluster bias, Wire-1 variance) — these are
  literally per-turn substrate values that vary
- Tool descriptions (sense_tools registry changes as scene state evolves)
- Acting Coach modulation (drive guidance varies per tick)
- Timestamps via `build_datetime_section`
- Workspace manifest scanning (`scan_workspace_entries` could enumerate in
  non-deterministic order)

**This is GOOD news for the audit** — most of these are deliberately
per-turn, which means the architectural pattern is correct: put dynamic stuff
LATE in the prompt (in the messages/user turn) and keep the system prompt
stable. The bad news is some of them may have been silently bleeding into the
system prompt for years and nobody noticed because no caching was on to
expose them.

## Three-phase plan

### Phase 0 — Audit PromptBuilder for silent invalidators (READ-ONLY)

Goal: identify every dynamic field that flows into the system prompt and
classify each as (a) genuinely-stable-belongs-in-system, (b)
dynamic-must-move-out-of-system, or (c) cacheable-with-care (changes only at
session boundaries, not turn boundaries).

Method:
1. Read [`PromptBuilder.assemble`](../../src/maxim/agents/prompt_builder.py)
   and every `build_*_section` function. Trace what flows into the `system`
   string vs the `messages` array.
2. Hash-test: for two consecutive turns of a single cradle sub-sim, compute the
   SHA256 of the system prompt at each call. Different hash = cache miss.
3. Bisect: if hashes differ, binary-search the section list to find the
   culprit(s).
4. Classify each finding per the above rubric.
5. Write the audit report into this plan doc's Phase 0 results section.

Output: a list of `(section_name, current_placement, recommended_placement,
reasoning)` rows. Each row is independently actionable in Phase 1.

Wall time: ~2-3 hours of code reading + one cradle smoke-run with print
debugging on PromptBuilder.assemble. Zero LLM cost (read-only audit).

**Acceptance:** every dynamic field in the system prompt is named, classified,
and has a placement recommendation. No silent invalidators remain undocumented.

#### Phase 0 — RESULTS (completed 2026-06-08, read-only audit)

**Decisive architectural finding — the plan's premise does not match the code.**

The mental model behind this plan was "a ~100K-token system prompt is re-sent
every turn; mark its end with one `cache_control` breakpoint." The code does
**not** work that way:

1. `PromptBuilder.build_prompt` assembles **every** section (identity, tools,
   observations, memories, drive states, datetime, bio-annotations, conversation
   history — all of it) into a **single string** returned as
   `TOOL_PROMPT|<everything>` ([prompt_builder.py:940](../../src/maxim/agents/prompt_builder.py)).
2. The router's `_generate_tool_response`
   ([router.py:1873-1890](../../src/maxim/models/language/router.py)) routes that
   entire string into the **`user`** message and sets **`system`** to the tiny
   static constant `SYSTEM_TOOL_RESPONSE` (~145 tokens,
   [cloud_dispatch.py:49](../../src/maxim/models/language/cloud_dispatch.py)).
3. `_AnthropicBackend.complete_with_usage` builds
   `messages = [{"role": "user", "content": user}]` (one user message, no
   multi-turn array — conversation history is rendered as text *inside* the
   payload) and applies the `cache_control` marker via `_build_system_blocks`
   **to the `system` parameter only** — i.e. to the ~145-token constant.

**Consequence: flipping `prompt_cache: true` today caches the ~145-token static
system constant and nothing else. ~0% of the per-turn ITPM is recoverable.** The
"wiring already exists" surprise from the original audit is real but the wiring
points at the wrong message.

Two distinct problems, both must be solved for any benefit:

- **P1 (placement):** the large stable content lives in the `user` message, but
  `cache_control` only marks the `system` message. A cacheable prefix must be
  moved into `system` (or the breakpoint moved onto the user message at the
  stable/dynamic boundary).
- **P2 (stability):** even within the `user` payload there is **no byte-stable
  prefix**. Dynamic content begins at position 2 (the user-request/percept) and
  is interleaved throughout — datetime to minute precision, per-tick drive
  states (`body_state`, CRITICAL priority), Acting Coach bio-modulation,
  Wire-A/W1 substrate annotations, observations, accumulating memories. The
  `PromptBudgeter` re-emits in insertion order ([prompt_budgeter.py:152](../../src/maxim/agents/prompt_budgeter.py)),
  and budget-driven truncation/drop of any earlier section shifts every byte
  after it.

**Hash-test artifact:** [tests/integration/test_prompt_builder_audit.py](../../tests/integration/test_prompt_builder_audit.py).
- `test_system_block_carries_negligible_tokens_vs_user_payload` — PASSES;
  documents that the cacheable system block is a small minority of tokens (real
  cradle ratio ≈ 145 system vs ~120K user).
- `test_baseline_user_payload_is_not_byte_stable_across_turns` — PASSES;
  documents 12 distinct payload hashes across 12 turns (cache miss every turn).
- `test_cacheable_prefix_is_byte_stable_across_turns` — `xfail(strict=True)`;
  this is the target invariant. It stays xfail until Phase 1 introduces a
  byte-stable prefix, at which point it XPASSes and (under `strict`) fails,
  forcing removal of the marker. This is the Phase 1/3 acceptance gate and the
  Phase 3 `test_system_prompt_byte_stable_across_turns` in embryo.

**Per-section audit.** "Current placement" is the `user` message for *every*
section (the `TOOL_PROMPT` single-string architecture). Classification rubric:
**(a)** genuinely-stable → belongs in `system`, cacheable; **(b)**
dynamic-must-move-out → keep in `user` turn; **(c)** cacheable-with-care →
session/phase-stable, can go in `system`, invalidates only at session/phase
boundaries (which we don't cache across anyway).

| section_name | inserted at | class | recommended placement | reason |
|---|---|---|---|---|
| instructions | 1 | (c) | system | static per autonomy level; session-stable |
| user_request | 2 | (b) | user turn | the percept/goal — varies per turn |
| planning_banner | 3 | (a) | system | static; empty in autonomous cradle |
| modification | 4 | (b) | user turn | only on user-revise; dynamic when present |
| identity | 5 | (c) | system | mode/goal/sim/interactive flags fixed per session (date/time params unused in body) |
| tools | 6 | (c) | system | scene roster stable **within a narrative phase**; changes at act boundary (and per-tick if imagination/sense_tools mutate roster — see caveat) |
| failed_tools | 7 | (b)/(a) | user turn | accumulates hallucinated names; empty by default (`MAXIM_TOOL_FAILURE_HINTS` off) |
| workspace_manifest | 8 | (b) | user turn | filesystem scan; empty for embodied cradle, dynamic when files written |
| realtime_hint | 9 | (b) | user turn | derived from question keywords |
| pfc_preamble | 10 | (a) | system | `PFC_PREAMBLE` static constant |
| acting_coach | 11 | (b) | user turn | bio-modulated by `body_state`/`causal_context`/`motor_programs` — varies per tick (large in cradle) |
| entity_context | 12 | (c) | system | composed from session-fixed `entity_spec` |
| cluster_bias_annotations | 13 | (b) | user turn | Wire-A NAc bias, updates per tick |
| grayscale_tools | 14 | (b) | user turn | W1 substrate annotation, per tick |
| tool_guidance_core | 15 | (a) | system | static given `mode_name`+`is_embodied` |
| tool_guidance_extended | 16 | (a) | system | static (empty when embodied) |
| datetime | 17 | (b) | user turn | minute-precision timestamp — classic silent invalidator |
| budget_context | 18 | (b) | user turn | cloud spend totals, per request |
| conversation | 19 | (b) | user turn | grows every turn (ideally a real message array) |
| context_pool | 20 | (b) | user turn | accumulating observations |
| active_protocols | 21 | (c)/(b) | user turn | "re-injected each submission"; treat as dynamic unless proven stable |
| reasoning_carryover | 22 | (b) | user turn | prior-cycle reasoning, grows |
| prefetch_context | 23 | (b) | user turn | speculative prefetch (not in cradle) |
| coding_guidelines | 24 | (b) | user turn | derived from question text |
| foundational | 25 | (a) | system | `_load_foundational_context` is session-cached (Constitution/AGENTS) |
| mode_context | 26 | (c) | system | `mode.context_prompt`, session-fixed |
| observation | 27 | (b) | user turn | current percept |
| speech | 28 | (b) | user turn | detected speech |
| agent_states | 29 | (b) | user turn | multi-agent states |
| recent_outcomes | 30 | (b) | user turn | last N tool outcomes |
| body_state | 31 | (b) | user turn | **drive states drift every tick** — CRITICAL priority, large in cradle |
| bio_enrichment | 32 | (b) | user turn | per-percept bio associations |
| auto_sense | 33 | (b) | user turn | passive perception sweep |
| deliberation_transcript | 34 | (b) | user turn | accumulating multi-cycle reasoning |
| working_memory_thoughts | 35 | (b) | user turn | prior-cycle THOUGHT entries |
| relevant_memories | 36 | (b) | user turn | recalled episodes, accumulate |
| concept_context | 37 | (b) | user turn | active ATL concepts |
| knowledge_context | 38 | (b) | user turn | semantic knowledge recalls |
| causal_context | 39 | (b) | user turn | learned causal predictions |
| valence_context | 40 | (b) | user turn | learned valence associations |
| motor_programs | 41 | (b) | user turn | learned motor programs |
| statistical_patterns | 42 | (b) | user turn | active statistical patterns |

**Stable-prefix candidates (class a/c → `system`):** instructions,
planning_banner, identity, tools (phase-scoped), pfc_preamble, entity_context,
tool_guidance_core, tool_guidance_extended, foundational, mode_context. These
are the only sections that can form a cacheable prefix. **Everything else — and
notably the largest cradle contributors (acting_coach, body_state, memories,
conversation, observation, bio annotations) — is genuinely per-tick dynamic.**

**Caveat on `tools` (the most important class-(c) section).** The tool roster is
stable *within* a cradle narrative phase but (i) changes at act boundaries when
`world_entities` activate, and (ii) may change per-tick if imagination registers
ephemeral entities or `sense_tools` reshapes the active window. If the roster
churns per-tick in practice, `tools` collapses from (c) to (b) and the cacheable
prefix shrinks further. **This must be measured on a real cradle run before
relying on it** (the synthetic hash test holds the roster constant, so it does
not exercise this).

**Scope verdict — Standing Risk #1 is REALIZED, and larger than anticipated.**
The risk anticipated "half the system prompt is per-turn dynamic." The reality
is stronger: there is effectively *no* system prompt today (it's a 145-token
constant), and realizing caching requires a **structural split** of the single
`TOOL_PROMPT` user-string into (1) a deterministic, byte-stable `system` prefix
assembled only from class-(a)/(c) sections and (2) a `user` turn carrying all
class-(b) content — plus making the budgeter emit the stable prefix
deterministically (truncation of dynamic sections must not perturb the prefix).
This is a multi-day refactor touching `PromptBuilder.build_prompt`, the
`PromptBudgeter` ordering/segmentation contract, the router's
`_generate_tool_response` system/user split, and a swath of prompt-builder
tests — **not a flag flip plus a few section moves.** Phase 1a as originally
scoped ("move dynamic content out of system") is inverted: the work is "lift
stable content *into* a newly-real system prefix."

**Recommendation:** checkpoint with the user before starting Phase 1 (per the
plan's own instruction and Standing Risk #1). Decision needed: (A) proceed with
the structural prompt-architecture split, or (B) defer caching and unblock Exp 37
cloud fires another way (e.g. tier upgrade, or pivot to higher-ITPM providers
DeepSeek/GPT-4o per Sequencing Option B). Estimated effort for (A):
~2–4 days including the smoke validation, dominated by the budgeter
segmentation refactor and test churn.

### Phase 1 — Fix invalidators + enable cache + add instrumentation

Goal: make `prompt_cache.enabled: true` actually deliver cache hits in
production.

Substeps:

**1a. Move dynamic content out of system prompt (per Phase 0 findings).**

For each "dynamic-must-move-out-of-system" item:
- Move from `build_*_section` into either the user turn or the assistant
  context. Most natural placement: append to the last user message as a
  `<turn_context>` block.
- Update the test that pins section ordering if it exists.

For each "cacheable-with-care" (session-level dynamic) item:
- Leave in system prompt. These are session-stable (e.g., the agent's name,
  the goal string set at session start) so they're cacheable across turns
  within a session and only invalidate at session boundaries — which we don't
  cache across anyway.

**1b. Enable `prompt_cache: true` on cloud profiles.**

Edit [`profiles.yml`](../../src/maxim/_data/profiles/) for all cloud profiles
in `CLOUD_MODEL_PREFIXES`. Add `prompt_cache: true` (or `prompt_cache:
{enabled: true}`) per profile:

```yaml
claude-sonnet-4-6:
  provider: anthropic
  model: claude-sonnet-4-6
  prompt_cache: true       # ← new
claude-haiku-4-5:
  provider: anthropic
  model: claude-haiku-4-5
  prompt_cache: true       # ← new
```

Local-model profiles stay unset (llama-cpp doesn't have an equivalent;
self-hosted peer routes go through `_MaximPeerBackend` which has no system-
prompt caching path).

**1c. Add cache-hit instrumentation.**

In `_AnthropicBackend.complete_with_usage`, after parsing `usage`, emit a
structured log event:

```python
logger.info(
    "anthropic_cache",
    extra={
        "event": "anthropic_cache",
        "cache_read_tokens": cache_read,
        "cache_write_tokens": cache_creation,
        "input_tokens_uncached": input_tokens,
        "cache_hit_ratio": cache_read / max(1, cache_read + cache_creation + input_tokens),
        "request_id": getattr(usage, "_request_id", None),
    }
)
```

Pair with `MAXIM_LOG_FILE=/tmp/maxim.jsonl` for downstream analysis. The first
turn of a sub-sim should show `cache_write_tokens > 0, cache_read_tokens =
0`. Turns 2-12 should show `cache_read_tokens > 0, cache_write_tokens = 0`
(or rare re-writes on TTL expiry).

**1d. Verify on a single Sonnet smoke trial.**

Single Arm A sub-sim with the new wiring. Read the JSONL log, sum the cache
metrics:
- Turn 1: cache_creation should be ~100K (the system prompt size)
- Turns 2-12: cache_read should be ~100K each, cache_creation should be 0
- Total ITPM consumption across the 12-turn sub-sim should drop from ~120K
  to ~20K-30K (the user-turn deltas only)

If cache_read stays at 0 across turns 2-12, that's a silent invalidator we
missed in Phase 0 → bisect and re-fix before going further.

**Acceptance:** Sonnet smoke trial shows >80% cache hit ratio on turns 2-12;
total ITPM consumption per sub-sim drops by >5×; rate-limit cascade does not
occur on a single-trial fire.

### Phase 2 — Extend to OpenAI backend (parallel ship)

Goal: same wiring for `_OpenAIBackend` so GPT-4o (and any other OpenAI-
compatible cloud provider) gets the same benefit.

OpenAI's caching is somewhat different mechanically — their automatic caching
kicks in at prompts >1024 tokens without explicit markers. But explicit
`cache_control` markers (using their compatible API shape) still help on
deterministically-routed providers and don't hurt on automatic-caching
providers.

Substeps:
1. Add `_build_system_blocks` parallel to Anthropic's (same shape:
   `cache_control: {"type": "ephemeral"}` at end of system block).
2. Add `_prompt_cache_enabled()` reading from the same config field.
3. Wire `prompt_cache: true` into GPT-4o profile in profiles.yml.
4. Add cache instrumentation matching Anthropic's.
5. Smoke trial: single Arm A run with `--model gpt-4o`.

**Acceptance:** GPT-4o smoke trial shows comparable cache hit ratio + ITPM
reduction.

### Phase 3 — CLAUDE.md invariant + ship

Goal: make the prefix-invariant rule load-bearing so future PromptBuilder
changes can't silently break caching.

Add to CLAUDE.md "Architectural invariants" section:

> **[engineering] System prompt content MUST be byte-stable across turns
> within a session** (prompt-caching prerequisite). The cradle and other
> long-context scenarios route through `_AnthropicBackend` /
> `_OpenAIBackend` with `prompt_cache: true`; any byte change in the system
> prompt between turns invalidates the cache and re-spends ITPM on the full
> prefix. Dynamic content (drive states, bio-substrate annotations, tool
> registries that change per-tick, timestamps, request IDs) belongs in the
> user/assistant message stream, NOT the system prompt. Test for invalidators
> by hashing the system prompt across consecutive turns of a single sub-sim
> and confirming bit-identity. Regression guard:
> `tests/integration/test_prompt_caching.py::test_system_prompt_byte_stable_across_turns`
> (forthcoming in Phase 1).

Add regression test as named. Test fixture: spawn a cradle sub-sim with mock
LLM that records every system prompt it receives, assert all received system
prompts are byte-identical to each other.

**Acceptance:** CLAUDE.md updated; regression test green; Phase 1 + Phase 2
PRs link to this plan.

## Tradeoffs and honest concerns

**1. Phase 0 audit might reveal a lot of in-flight invalidators.** Wire-A
annotations, Wire-1 variance, Acting Coach drive modulation — all of these
were designed as substrate-derived prompt augmentations and might be in the
system prompt by current architecture. If they are, moving them out is a
deeper refactor than "flip one flag." The plan accounts for this — Phase 1a
is explicitly the "move dynamic stuff out" step — but if half the system
prompt turns out to be per-turn dynamic, Phase 1a becomes the big work item
rather than a footnote.

**2. Cache writes ARE billable.** The 1.25× write premium means a one-turn
sub-sim with no cache reuse costs MORE than no caching at all. For sub-sims
that complete in 1-2 turns (or get killed early), caching is a slight cost
loss. For sub-sims that go to 12 turns (cradle target), it's a 6× win. Most
cradle sub-sims complete the full 12 turns, so net-positive on average.
Quantify in Phase 1d.

**3. TTL is 5 minutes by default.** If sub-sims are slow (e.g., Sonnet under
load with high TTFT), the cache might expire mid-sub-sim, forcing a re-write.
This shouldn't happen at Sonnet's normal latency (~5s per turn × 12 turns = 60s
total, well under the 5-min TTL) but is worth watching in Phase 1d. Switching
to `ttl: "1h"` (at 2× write premium) is a backup if 5-min TTL bites.

**4. Provider-specific shape differences.** Anthropic uses `cache_control`
markers on individual blocks. OpenAI uses automatic caching above a threshold.
Future cloud providers (DeepSeek, Mistral, etc.) may have yet another shape.
Phase 2 ships the OpenAI-specific wiring; subsequent providers are added
incrementally per the established pattern (`_build_system_blocks` +
`_prompt_cache_enabled` in each backend, profile flag in profiles.yml).

**5. Local models don't benefit.** Qwen14B/32B/Mistral24B all route through
llama-cpp-server which has no prompt-cache mechanism (though KV-cache on the
local model partially compensates). This means the Exp 37 cross-model
comparison still has an apples-to-oranges shape: cloud sub-sims have prompt
caching savings, local sub-sims don't. The headline LLM-prior-dominance
finding is unaffected (we measure behavior, not cost), but cost comparisons
will lopside-favor cloud.

**6. Phase 0 first is non-negotiable.** Skipping the audit and flipping the
flag would silently fail (cache miss every request, no measurable benefit,
hours of debugging). The audit is cheap relative to the alternative.

## Sequencing relative to Exp 37 cross-model fires

Currently running on the leader: Qwen32B fire (~26 records of 60 in, ~18 more
hours). Independent of this plan.

Currently waiting on the peer: Sonnet replication (killed 2026-06-08), GPT-4o
(queued), DeepSeek-V3 (queued).

Two options for sequencing:

**Option A — Pause cloud replications, ship Phases 0-3, then resume.**
Pause: Sonnet, GPT-4o, DeepSeek (no work loss; nothing's started). Ship
prompt caching. Resume cloud replications with caching active. Wall time: +
~1-2 days (audit + implementation + smoke). Closes the rate-limit blocker
structurally before re-firing.

**Option B — Pivot to DeepSeek + GPT-4o now, ship prompt caching after.**
DeepSeek has dramatically higher tier-1 limits and is cheap. GPT-4o tier 1
might have higher ITPM than Anthropic's 30K. Fire them now without caching.
Add Sonnet later once prompt caching is shipped. Wall time: 0 delay. Less
methodologically clean (Anthropic comparison gets the caching speedup that
other models don't).

**Recommendation: Option A.** The prompt caching work is architecturally
valuable independent of Exp 37 (every future cloud experiment benefits). The
1-2 day delay is small relative to the ~30 hours we're already waiting on
Qwen32B. Doing the audit + ship before resuming cloud experiments means we
get a clean Anthropic comparison AND we ship a real engineering improvement.

## Out of scope (explicitly)

- Caching for self-hosted peer routes (`_MaximPeerBackend`). Self-hosted
  llama-cpp via tunnel has KV-cache locally; adding HTTP-layer prompt
  caching adds complexity without clear benefit since the peer-to-leader
  link is already the bottleneck, not LLM compute.
- Substrate / bio-system architecture changes. This plan moves dynamic stuff
  OUT of the system prompt INTO the message stream; it does not change WHAT
  goes into prompts.
- Replacing PromptBuilder. The audit may surface architectural cleanups
  (e.g., a clearer split between "stable session context" and "per-tick
  state"), but those are follow-up work, not blocking 1.0.
- Provider-specific cost-optimization beyond caching (batch API, lower-cost
  routing, etc.). Separate plan.

## Cross-references

- 2026-06-08 Sonnet/Haiku rate-limit cascade evidence in this conversation's
  results.jsonl for the cradle sub-sims that failed.
- [docs/plans/exp37_cross_model_characterization.md](exp37_cross_model_characterization.md)
  — the cross-model study this plan unblocks for Anthropic comparison.
- [docs/plans/cloud_dispatch_debug.md](cloud_dispatch_debug.md) — the prior
  debug plan whose Phase 1 validated the dispatch path (2026-06-07).
- [Anthropic rate limits](https://platform.claude.com/docs/en/api/rate-limits)
  — authoritative tier table + cache-aware ITPM rule.
- [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
  — mechanics, breakpoint placement, silent invalidator audit checklist.
- [src/maxim/models/language/anthropic_backend.py](../../src/maxim/models/language/anthropic_backend.py)
  — existing cache_control wiring (lines 118-176) that this plan enables in
  production.
- [src/maxim/agents/prompt_builder.py](../../src/maxim/agents/prompt_builder.py)
  — the PromptBuilder that Phase 0 audits.
- [src/maxim/models/language/openai_backend.py](../../src/maxim/models/language/openai_backend.py)
  — the OpenAI backend that Phase 2 extends.

## Results

### Phase 1 — implemented (commit `d5cd543`, 2026-06-08)

The Phase 0 audit invalidated the plan's premise (see Phase 0 RESULTS above), so
Phase 1 became the structural refactor it described as the worst case:

- **`PromptBudgeter`** gained a `cacheable` flag + `build_segmented()` — stable
  sections are budgeted first, independent of dynamic content (capped at 70% of
  budget so dynamic MANDATORY can't starve), guaranteeing a byte-stable prefix.
- **`PromptBuilder`** tags 10 class-(a)/(c) sections cacheable and emits
  `TOOL_PROMPT|<stable>\x1e<dynamic>`.
- **Router** `_generate_tool_response` splits on the delimiter → stable prefix
  into `system` (after `SYSTEM_TOOL_RESPONSE`, so the whole system block ends at
  the cache breakpoint), dynamic into `user`. No-delimiter payloads unchanged.
- **Profiles/config**: `prompt_cache: true` on the 3 Claude profiles; new
  `LLMConfig.prompt_cache` threaded profile → synthesized provider → backend.
- **Instrumentation**: `anthropic_cache` structured event per call.

Regression coverage: budgeter (6), router split (4), backend wiring +
instrumentation (8), audit byte-stability gate flipped green (4). Full fast
suite: 7791 passed.

### Phase 1d — Sonnet smoke (2026-06-08, 1 trial, fire_pit Arm A, $0.30)

Command: `benchmark_cross_session.py --scenario fire_pit --arms A --trials 1
--seed-base 600 --model claude-sonnet --sim-max-turns 12` with
`MAXIM_LOG_FILE` capturing `anthropic_cache` events.

**Caching works and the prefix is byte-stable** (proven by 43K cache reads — a
non-stable prefix would read ~0). Measured across the 44 LLM calls of the
12-turn sub-sim:

| metric | value |
|---|---|
| total input tokens (sub-sim) | 112,368 |
| cache_read (not ITPM-counted) | 43,016 |
| cache_write (1.25× billed) | 8,645 (2 writes) |
| uncached input | 60,707 |
| **ITPM-counted input (write+uncached)** | **69,352** |
| **ITPM reduction vs no-cache** | **38.3%** |
| calls with prompt ≥1024 tok (Anthropic min cacheable) | 24 / 44 |
| per-call prompt size (min/median/max) | 383 / 1,147 / 7,685 |

**Key corrections to the plan's assumptions:**

1. **The "~120K input tokens" is the SUM across the sub-sim's 44 calls, not a
   per-turn 120K system prompt.** Per-call median is only ~1.1K tokens.
2. **The plan's acceptance bar (>80% hit ratio, >5× reduction) is NOT met and
   was never achievable** — it assumed a large *stable* prefix re-sent every
   turn. Reality: the stable prefix is ~4.2-4.4K tokens; the rest is genuinely
   dynamic (memories, drives, conversation) or sub-1024 calls that can't cache.
3. **The real win — ~38% ITPM reduction — still clears the tier-1 ceiling**
   that caused the 2026-06-08 cascade (~30K → ~18.6K ITPM sustained). This is
   the meaningful success: it unblocks Sonnet/Haiku Exp 37 fires at tier 1.
4. **The `tools` phase-stability caveat manifested benignly:** one prefix
   re-write at a narrative-act boundary (4,218 → 4,427 tokens, +209 when new
   `world_entities` activated), then reuse resumed. NOT per-tick churn — `tools`
   was correctly classified phase-stable. The default 5-min TTL held across the
   ~6-min run; no need to bump to 1h.

**Why only 24/44 calls are cache-eligible:** the cradle sub-sim makes several
LLM call *types* (AUT tool-prompt, narrator, answer-only, imagination). Only the
AUT tool-prompt calls (~one per turn) carry the large stable prefix and exceed
Anthropic's 1024-token minimum; the rest are small and cannot cache. The ~38%
is therefore close to the practical ceiling for this scenario without making
more of the per-call content stable (out of scope — that content is genuinely
dynamic).

### Phase 2 — OpenAI caching (commit `0610d2d`, 2026-06-08)

OpenAI Chat Completions has **no `cache_control` marker** — caching is automatic
and server-side for prompts >1024 tokens, keyed on the stable message prefix
that Phase 1's system/user split already provides. So Phase 2 added measurement,
not a marker: `_OpenAIBackend._prompt_cache_enabled()` + `_log_cache_usage()`
emitting an `openai_cache` event (buffered + streaming paths), `prompt_cache:
true` on the gpt-4o profile (gates the event), and `cached_tokens` capture in the
streaming path (previously hardcoded 0). 8 unit tests.

### Phase 2d — gpt-4o smoke (2026-06-08, 1 trial, fire_pit Arm A, $0.27)

| metric | value |
|---|---|
| LLM calls | 45 |
| total prompt tokens | 99,165 |
| cache_read | 30,720 (10 calls) |
| uncached | 68,445 |
| cache served | 31.0% of input tokens |
| calls with prompt ≥1024 tok | 23 / 45 |
| per-call prompt size (min/median/max) | 353 / 1,028 / 6,758 |

OpenAI caching works on the same stable prefix (31% of input served from cache).
**Important difference from Anthropic:** OpenAI cached tokens get a ~50% *cost*
discount but, unlike Anthropic's ITPM exemption, still count toward TPM rate
limits. So for OpenAI the benefit is **cost (~15% on input), not rate-limit
headroom**. OpenAI tier limits are generally higher than Anthropic's anyway, so
this is a cost optimization rather than a blocker-clearing fix.

### Phase 2e — DeepSeek (deepseek-chat) smoke (2026-06-08, 1 trial, $0.03)

DeepSeek routes through the OpenAI-compatible backend. The smoke surfaced — and
this work fixed — a **pre-existing gap: DeepSeek cloud dispatch was never wired
on the default (no user-configured providers) path.** Two missing pieces:

1. **No pricing entry** → the cloud cost-gate excluded it (`attempts: []`). Added
   `deepseek-chat` / `deepseek-reasoner` to `DEFAULT_PRICING`.
2. **`base_url` / `api_key_env` were dropped.** The synthesized provider entry
   lives in `router._providers`, but backends read `cfg.providers` (empty on the
   default path) — so DeepSeek's `api.deepseek.com` endpoint and
   `DEEPSEEK_API_KEY` never reached the backend, which silently fell back to
   `api.openai.com` / `OPENAI_API_KEY` → `empty_response`. (claude/gpt-4o were
   unaffected because their defaults match.) Fixed by threading `base_url` +
   `api_key_env` through `LLMConfig` → synthesized provider entry, plus a
   cfg-level fallback in `_OpenAIBackend._get_api_key` / `_get_base_url` and
   `_AnthropicBackend._get_api_key` — the same fallback shape already used for
   `model` and `prompt_cache`.

| metric | value |
|---|---|
| LLM calls | 47 |
| total prompt tokens | 105,532 |
| cache_read | 49,024 |
| uncached | 56,508 |
| **cache served** | **46.5% of input tokens** (32/47 calls) |
| total sub-sim cost | $0.03 |

DeepSeek's automatic caching hit *more* of the workload than Anthropic/OpenAI
(46.5%, and on smaller calls too). DeepSeek bills cached tokens at ~74% off
(~$0.07 vs $0.27 per 1M) but, like OpenAI, does not exempt them from rate limits
— so the benefit is **cost** (DeepSeek's limits are very high regardless). The
`openai_cache` event captures DeepSeek's `prompt_cache_hit_tokens` field (which
differs from OpenAI's `prompt_tokens_details.cached_tokens`).

## Summary

Caching is live for Anthropic (Sonnet/Haiku/Opus), OpenAI (gpt-4o), and DeepSeek
(deepseek-chat), validated end-to-end. The plan's headline "6× / 75% ITPM reduction" was **not achievable**
(it rested on a false premise of a large *stable* per-turn prefix). The real,
measured outcome:

- **Anthropic: ~38% ITPM reduction** — clears the tier-1 ceiling that blocked
  the 2026-06-08 Exp 37 Sonnet/Haiku replication. **This is the blocker-clearing
  result.**
- **OpenAI gpt-4o: ~31% of input cached → ~15% input-cost reduction** (no
  rate-limit benefit by OpenAI's design).
- **DeepSeek deepseek-chat: ~46.5% of input cached → cost reduction** (~74% off
  cached tokens); also fixed DeepSeek/OpenAI-compatible cloud dispatch on the
  default path (pricing + base_url/api_key_env threading).
- Byte-stable system prefix enforced structurally + by regression test, so future
  PromptBuilder changes can't silently break caching.

Out of scope / not pursued: caching for `_MaximPeerBackend` (local llama-cpp has
KV-cache); DeepSeek/other providers (same `prompt_cache` pattern applies when
needed); raising the cache-eligible call fraction (the non-AUT calls are small
and the dynamic content is genuinely per-turn).
