Zaxy 2.3 Promotion Pass — Evidence Plan

Date: 2026-06-11. Author: benchmarks research pass (read-only against /home/cheapseatsecon/Projects/Personal/zaxy, master @ post-2.2). Scratch artifacts: /tmp/zaxy-23-research/ (int8_bench.py, int8_bench_results.json, int8_bench_e.py, int8_bench_e_results.json).

Three deferred promotions from the 2.2 plan (docs/superpowers/specs/2026-06-11-zaxy-2-2-ann-engineering-plan.md): int8 quantization default, ENCODING_GATE_ENABLED flip, FoK organic calibration.

---

1. int8 latency cliff — diagnosis and prototype verdict

Diagnosis (profiled, deterministic seed 20260611, numpy 2.4.4 / OpenBLAS,

Gaussian unit corpora matching the lane's gaussian distribution, dim 1536, limit 10, oversample 4)

The current quantized selector (_quantized_candidate_entity_indexes, src/zaxy/embedded_graph_store.py:2499):

integer_scores = group.matrix.astype(np.int32) @ quantized_query.astype(np.int32)

Component breakdown per query:

The cliff is fully explained: ~72% is the per-query full-matrix int8→int32 cast (a fresh 4nd allocation + page faults every query), the rest is numpy's integer matmul, which never touches BLAS. Nothing else matters.

Prototype matrix (select + exact float64 rerank, p50 over 48 timings,

recall vs float64 ground truth; resident bytes vs exact 8nd = 614.4 MB at 50k)

Recall notes: c1/c2/e compute bit-identical integer scores to the current path — recall identical by construction. The f32 variants (a/b/b2/d/e) were measured: 1.0 strict and tie-aware at both sizes (int8 values are exactly representable in f32; accumulation rounding never flipped a 40-candidate boundary on this corpus; the exact float64 rerank owns final ordering regardless). Host had background load (three codex-capture processes at ~72% CPU); p50s are robust, p95s noisy. Our measured exact (21.6/25.0 ms) and current-int8 (182 ms) bracket the lane's 22.4/223.7 ms — same machine class, lane includes full query-path machinery.

Verdict

No approach clears the bar while preserving int8's memory posture. The only paths that beat exact latency (12 ms vs 17–22 ms) are the precast-f32 family, which carries 0.50–0.63x of exact bytes — that is a float32 mode, not int8, and it defeats int8's reason to exist: at 50k x 1536 the 256 MB VECTOR_INDEX_CACHE_MAX_BYTES budget admits int8 (77 MB) but not f32 (307 MB) or f32+int8 (384 MB). The best int8-resident numpy play (c1, block-wise int32) halves the cliff (182 → 96 ms) but still loses 4.4x to exact; variant (e) shows the int8→f32 cast compute itself (~45 ms at 50k) is irreducible in pure numpy, so no int8-resident numpy kernel can reach 22 ms.

Recommendation: do not promote int8 to default in 2.3. Three concrete actions instead:

1. Land (c1) block-wise accumulation as a repair to the opt-in path — free 1.9x latency improvement (182 → 96 ms at 50k; 39 → 35 ms at 10k), zero memory change, recall identical by construction. Honest docs update: int8 remains the memory-tightest option, latency ~4x exact at high-dim scale.
2. Evaluate float32 selection (b2) as a new lane-gated mode, not as int8 promotion: 1.8x faster than exact, 2x fewer bytes, measured recall 1.0 with the float64 rerank absorbing the f32 tie-flip class the 2.2 diagnostics identified. It is the only measured mode that wins latency AND bytes simultaneously at 50k x 1536.
3. Fold the "int8 fast kernel" question into the 2.3 ANN-backend agenda (LadybugDB / alternative embedded ANN): a real int8 GEMV kernel (SIMD dot-product instructions) is a dependency decision, which 2.2 already deferred to 2.3 strategy.

---

2. Encoding-gate lane spec (ENCODING_GATE_ENABLED flip) — spec only

What the gate does when enabled (read from core.py:1188-1220,

salience.py:752-789)

Classification is tag-only — every event is still appended and hash-chained — but enabled-state consequences are: (i) payloads carry the encoding tag; (ii) redundant appends emit a weak reinforcement toward duplicate_of (the old memory); (iii) interference detection runs only for novel classifications. Thresholds (salience.py): redundant at token-Jaccard content_overlap >= 0.9; reinforcing at >= 0.6, or >= 0.4 with entity_overlap >= 1.0; else novel.

The dangerous error is false-redundant: genuinely novel content classified redundant ⇒ reinforcement credit misdirected to the old memory, interference detection silently skipped, and the new fact treated as a duplicate under the cognitive profile.

Corpus construction (deterministic, no LLM)

Reuse the fok_calibration_lane.py word-table pattern (disjoint adjective/noun/domain tables, hash embeddings, embedded backend, fixed session). Base corpus: N decision.made appends (sizes 200 / 1000). For a base chunk of T tokens, replacing r tokens yields Jaccard J = (T-r)/(T+r), so r = T(1-J)/(1+J) gives controlled overlap levels. Probe families, each with a construction-time intent label and the measured Jaccard recorded per probe:

The genuinely-novel family is the heart of the lane: token-level Jaccard is blind to negation and value changes ("adopt the amber anchor pipeline rollout" vs "abandon the amber anchor pipeline rollout" shares ~5/6 tokens, J ≈ 0.71 — and a 20-token version with one flipped token sits at 0.905, above the redundant threshold). The lane must measure exactly how often that happens.

Metrics

• false_redundant_rate (dangerous): fraction of novel-by-construction probes classified redundant, overall and per Jaccard band.
• false_novel_rate: fraction of exact/near-duplicate probes classified novel (cost: missed dedup, salience inflation — mild).
• Full per-family confusion matrix + threshold-margin histogram (distance of measured J from the nearest threshold, so boundary fragility is visible).
• duplicate_of accuracy: redundant classifications whose citation resolves to the true constructed source chunk.
• End-to-end recall impact: two fabrics over the identical mixed stream, gate on vs gate off, cognitive profile (ZAXY_RETRIEVAL_PROFILE=cognitive); query set targets the novel-by-construction payloads (the flipped facts); report recall@10 gate-on minus gate-off, plus interference-detection counts (novel probes must still trigger detection when gated).

Deterministic by construction (fixed tables, hash embeddings, no wall clock); "validation": "internal".

Exit criteria for flipping the default

1. false_redundant_rate = 0 on the genuinely-novel family at every size, on two consecutive lane runs (zero tolerance: the harm is silent suppression of new facts; this mirrors the lane-evidence rule).
2. false_novel_rate ≤ 0.05 on exact + near-duplicate families.
3. End-to-end recall@10 for novel-fact queries, gate on ≥ gate off (no regression), and interference detections for novel probes unchanged.
4. duplicate_of accuracy ≥ 0.95 on true duplicates.
5. Boundary report published: classification behavior within ±0.02 of each threshold documented (informational, not gated).

If criterion 1 fails (likely for the negation sub-band at J ≥ 0.9), the lane has earned a threshold/feature change (e.g. content-token vs filler-token weighting or a negation guard) before any flip — defaults move on lane evidence, not on assertion.

---

3. FoK promotion — data status, time-to-data, join design

Marker count: zero, everywhere

metacognition.fok.predicted is emitted only by the MCP memory_feeling_of_knowing handler (mcp_server.py:2788-2816; shipped 2.1.0, deployed zaxy 2.2.0 is serving). Read-only scan of all local Eventloom stores (~/Projects/Personal/*/.eventloom — 24 workspaces including zaxy, the-hollow-lodge, codex-game, jedi-devproject-0): 0 markers. The string "fok" appears in zaxy-default.jsonl only inside captured agent-transcript content, not as events.

Meanwhile checkout volume is healthy — memory.checkout.completed by workspace: zaxy 644 (last 7 days: 17/45/24/47/64/97/54 ≈ 50–100/day), the-hollow-lodge 140 (bursts to 88/day), codex-game 24, jedi-devproject-0 3. Agents check out constantly; no agent ever calls the FoK tool first. At the current organic rate (0/day), time-to-calibration is infinite. This is an instrumentation problem, not a patience problem.

Power assumption and N

Target: a Brier-score CI that distinguishes FoK skill from the base-rate predictor (the existing lane's exit criterion, now on organic data). Paired per-query squared-error differences have sd ≈ 0.25 (scores in [0,1], outcome rate ~0.5–0.7); to detect a Brier improvement of 0.05 at 95% confidence: N ≈ (1.96 × 0.25 / 0.05)² ≈ 100–200 joined pairs. Add per- verdict bucket calibration (≥ 30 pairs in each of likely/possible/unlikely) and a realistic join-success rate of ~0.7 ⇒ target ~400 emitted predictions.

Time-to-data: if 2.3 auto-emits a FoK prediction at every memory_checkout (prediction computed against the pre-retrieval index, then the same checkout provides the outcome — a natural prediction/resolution pair), the zaxy workspace alone produces ~60/day ⇒ ~400 predictions in roughly 7 days; a meaningful calibration join after ~1–2 weeks of normal use. Without that change, never.

Join design

• Prediction side: metacognition.fok.predicted — payload {query_hash: sha256(query), verdict, score, authority_status}, keys thread (= session_id) and seq.
• Outcome side: memory.checkout.completed — payload carries the

plaintext query plus token_efficiency {prompt_tokens, current_fact_count, evidence_count, facts_per_1k_prompt_tokens}.

• Match: same thread; sha256(checkout.payload.query) == marker.query_hash; checkout.seq > marker.seq within a window (next 50 events or 15 minutes); nearest-following wins.
• Outcome label: positive iff token_efficiency.evidence_count >= 1 (proxy for "retrieval surfaced something"); upgrade with current_fact_count >= 1 as a stricter variant and with memory.feedback.recorded / memory.reinforced events in the window as ground-truth-quality labels when present (currently rare: ~1–46 per workspace). Report both labelings — the proxy's optimism is itself a finding.
• Known gap + 2.3 fix: agents paraphrase between a check and the checkout, breaking exact hash equality. Two cheap remedies: (i) record query_hash in the checkout payload too, (ii) allow a sequence-adjacency fallback join (nearest following checkout in-window, flagged weak_match) reported separately. The auto-emit-at-checkout design makes the join trivial (same event pair, same query string by construction) and is the recommended path.

---

Promotion-pass sequencing recommendation for 2.3

1. First increment — FoK instrumentation (smallest change, calendar-time bound): auto-emit metacognition.fok.predicted inside handle_memory_checkout before retrieval (plus query_hash on checkout payloads). Every day this isn't deployed is a day of calibration data lost; landing it first lets organic data accrue during the whole cycle.
2. Encoding-gate lane (cheap, deterministic, spec above is buildable in one increment): build, run, and either flip ENCODING_GATE_ENABLED on a double pass or harvest the threshold findings. This is the most likely promotion to actually land in 2.3.
3. int8: demote from promotion candidate to repair + reframe: land the block-wise accumulation fix on the opt-in path (free ~2x), add the float32-selection mode to the vector-scale lane as the new latency+memory candidate, and hand the "true int8 kernel" question to the already-scheduled ANN-backend evaluation. The promotion-pass verdict on int8-as-default is no — supported by the prototype matrix above.
4. End of cycle — FoK calibration join lane over the accumulated organic markers (~2+ weeks of data by then), evaluated against the Brier-vs-base-rate criterion with both outcome labelings.