# Ralph Progress Log
Started: Wed Apr 15 07:55:56 PM PDT 2026
---

## Codebase Patterns
- `rivetkit` package-level `vitest run` only discovers `*.test.*` and `*.spec.*` files. `src/driver-test-suite/tests/*.ts` coverage lives outside that glob, so validate those stories through the driver-suite harness or another explicit entrypoint instead of assuming a direct file filter will run them.
- `rivetkit-sqlite-native` reopen tests can hit RocksDB `LOCK: No locks available` when they run alongside other heavy Rust suites, so rerun those checks in isolated `cargo test -p rivetkit-sqlite-native <filter> -- --test-threads=1` invocations before calling the branch broken.
- `wrapJsNativeDatabase(...)` must forward new native SQLite introspection hooks like `getSqliteVfsMetrics()`, or `/inspector/metrics` will quietly report zero VFS commit timings even when Rust recorded them.
- `pegboard-envoy` SQLite websocket handlers should validate page numbers, page sizes, and duplicate dirty pages at the websocket trust boundary and downgrade unexpected failures to `SqliteErrorResponse` so one bad actor request cannot tear down the shared envoy connection.
- `sqlite-native` v2 should poison the VFS inside `flush_dirty_pages()` and `commit_atomic_write()` for non-fence commit failures; callback wrappers should only translate fence mismatches into SQLite I/O return codes.
- `sqlite-native` v2 must treat `head_txid` and `db_size_pages` as connection-local authority. `get_pages(...)` can refresh `max_delta_bytes`, but only commits and local truncate/write paths should mutate those fields.
- RivetKit sleep shutdown should wait for in-flight HTTP action work and pending disconnect callbacks before running `onSleep`, but it should not treat open hibernatable connections alone as a blocker because existing connection actions may still finish during the shutdown window.
- `sqlite-storage` owns UniversalDB value chunking in `src/udb.rs`, so `pegboard-envoy` should call `SqliteEngine` directly instead of reintroducing a separate `UdbStore` layer.
- Actor KV prefix probes should build ranges with `ListKeyWrapper` semantics instead of exact-key packing. SQLite startup now uses a single prefix-`0x08` scan via `pegboard::actor_kv::sqlite_v1_data_exists(...)` to distinguish legacy v1 data.
- `sqlite-native` v2 edge-case coverage should prefer the direct `SqliteEngine` + RocksDB harness in `src/v2/vfs.rs`; keep `MockProtocol` tests for transport-unit behavior, but use the direct harness for cache-miss, compaction, reopen, and staged-commit regressions.
- `sqlite-native` v2 slow-path commits should queue `commit_stage` requests fire-and-forget and only await `commit_finalize`; if you need per-stage response assertions, keep them in the direct-engine test transport instead of the real envoy path.
- Baseline sqlite-native VFS tests belong in `rivetkit-typescript/packages/sqlite-native/src/vfs.rs` and should use `open_database(...)` with a test-local `SqliteKv` implementation instead of mocking SQLite behavior.
- Keep `sqlite-storage` acceptance coverage inline in the module test blocks and back it with temp RocksDB UniversalDB instances from `test_db()` so commit, takeover, and compaction assertions exercise the real engine paths.
- `sqlite-storage` crash-recovery tests should capture a RocksDB checkpoint and reopen it in a fresh `SqliteEngine` rather than faking restart state in memory.
- Envoy-protocol VBARE version bumps can deserialize old payloads straight into the new generated type only if old union variant tags stay in place, so add new variants at the end and explicitly reject v2-only variants on v1 links.
- If a versioned envoy payload changes a nested command shape like `CommandStartActor`, update both `ToEnvoy` and `ActorCommandKeyData` migrations instead of relying on the same-bytes shortcut.
- Fresh worktrees may need `pnpm build -F rivetkit` before example `tsc` runs can resolve workspace `rivetkit` declarations.
- New engine Rust crates should use workspace package metadata plus `*.workspace = true` dependencies, and any missing shared dependency must be added to the root `Cargo.toml` before the crate can build cleanly.
- SQLite VFS v2 key builders should keep ASCII path segments under the `0x02` prefix and encode numeric suffixes in big-endian so store scans preserve numeric ordering.
- `sqlite-storage` callers that need a prefix scan should use a dedicated prefix helper like `pidx_delta_prefix()` instead of truncating a full key at the call site.
- `sqlite-storage` PIDX entries use the PIDX key prefix plus a big-endian `u32` page number, and store the referenced delta txid as a raw big-endian `u64` value.
- In `sqlite-storage` failure-injection tests, use `MemoryStore::snapshot()` for assertions after the first injected error because further store ops still consume the `fail_after_ops` budget.
- `sqlite-storage` LTX V3 blobs should sort pages by `pgno`, terminate the page section with a zeroed 6-byte page-header sentinel, and record page-index offsets and sizes against the full on-wire page frame.
- `sqlite-storage` LTX decoders should cross-check the footer page index against the actual page-frame layout instead of trusting offsets and sizes blindly.
- `sqlite-storage` takeover should delete orphan DELTA/STAGE/PIDX entries in the same `atomic_write` that bumps META, then evict the actor's cached PIDX so later reads reload the cleaned index.
- `sqlite-storage` `get_pages(...)` should resolve requested pages to unique DELTA or SHARD blobs first, issue one `batch_get`, then decode each blob once and map pages back into request order.
- `sqlite-storage` fast-path commits should update an already-cached PIDX after `atomic_write`, but should not trigger a fresh PIDX load just to mutate the cache because that burns the 1-RTT fast path.
- `sqlite-storage` staged commits reserve a txid with `commit_stage_begin`, write encoded LTX chunks directly under `delta_chunk_key(...)`, and rely on the `head_txid` META flip plus takeover cleanup of `txid > head_txid` orphans instead of `/STAGE` keys.
- `sqlite-storage` coordinator tests should inject a worker future and drive it with explicit notifiers so dedup and restart behavior can be verified without the real compaction worker.
- `sqlite-storage` shard compaction should derive candidate shards from the live PIDX scan and delete DELTA blobs only after comparing global remaining PIDX refs, which keeps multi-shard and overwritten deltas alive until every page ref is folded.
- `sqlite-storage` compaction must re-read META inside its write transaction and fence on `generation` plus `head_txid` before updating `materialized_txid` or quota fields, so takeover and commits cannot rewind the head.
- `sqlite-storage` metrics should record compaction pass duration and totals in `compaction/worker.rs`, while shard outcome metrics like folded pages, deleted deltas, delta gauge updates, and lag stay in `compaction/shard.rs` to avoid double counting.
- `sqlite-storage` quota accounting should count only META, SHARD, DELTA, and PIDX keys, and META usage must be recomputed with a fixed-point encode because the serialized head includes `sqlite_storage_used`.
- UniversalDB low-level `Transaction::get`, `set`, `clear`, and `get_ranges_keyvalues` ignore the transaction subspace, so sqlite-storage helpers must pack subspace bytes manually for exact-key reads/writes and prefix scans.
- `UDB_SIMULATED_LATENCY_MS` is cached once via `OnceLock` in `Database::txn(...)`, so set it before starting a benchmark process if you want simulated RTT on every UDB transaction.
- `sqlite-storage` latency tests that depend on `UDB_SIMULATED_LATENCY_MS` should live in a dedicated integration test binary, because UniversalDB caches that env var once per process with `OnceLock`.
- `PegboardEnvoyWs::new(...)` is per websocket request, so shared sqlite dispatch state belongs in a process-wide `OnceCell`; otherwise each connection spins its own `SqliteEngine` cache and compaction worker.
- `sqlite-storage` fast-path commit eligibility should use raw dirty-page bytes, while slow-path finalize must accept larger encoded DELTA blobs because UniversalDB chunks logical values under the hood.
- `KvVfs::register(...)` now always takes a startup preload vector, so v1 callers that do not have actor-start preload data should pass `Vec::new()`.
- `rivetkit-sqlite-native::vfs::open_database(...)` now performs a startup batch-atomic probe and fails open if `COMMIT_ATOMIC_WRITE` never increments the VFS metric.
- Native sqlite startup state should stay cached on the Rust `JsEnvoyHandle`, and `open_database_from_envoy(...)` should dispatch on `sqliteSchemaVersion` there. Schema version `2` must fail closed if startup data is missing instead of inferring v2 from `SqliteStartupData` presence.
- `sqlite-native` v2 tests that drive a real `SqliteEngine` through the VFS need a multithread Tokio runtime; `current_thread` is only reliable for mock transport tests.
- `sqlite-native` batch-atomic callbacks must treat empty atomic-write commits as a no-op, because SQLite can issue zero-dirty-page `COMMIT_ATOMIC_WRITE` cycles during startup PRAGMA setup.

## Completed Stories (Archive)

One-line summary per story. See git log + archived-stories.json for full titles; see this file history for full learnings. Specific reusable learnings have been distilled into the Codebase Patterns section above.

- `2026-04-15` **US-001** — Added a test-local `MemoryKv` for `SqliteKv` and five end-to-end baseline VFS tests covering create/insert/select, multi-row insert, update, delete, and multi-table schema flows...
- `2026-04-15` **US-002** — Added a repeatable v1 baseline benchmark driver in `rivetkit-sqlite-native`, wired `examples/sqlite-raw` to run it, and captured the measured workload latencies plus KV round-trip...
- `2026-04-15` **US-003** — Created the `engine/packages/sqlite-storage` crate skeleton, wired it into the root workspace, added the required shared dependency entry for `parking_lot`, and added placeholder...
- `2026-04-15` **US-004** — Replaced the sqlite-storage type and key stubs with concrete `DBHead`, `DirtyPage`, `FetchedPage`, and `SqliteMeta` structs, added spec-default helpers and `serde_bare` round-trip...
- `2026-04-15` **US-005** — Added the `SqliteStore` trait plus `Mutation` helpers, then built a reusable `MemoryStore` test backend with latency simulation, operation logging, failure injection,...
- `2026-04-15` **US-006** — Replaced the sqlite-storage LTX stub with a real V3 encoder that writes the 100-byte header, block-compressed page frames with size prefixes, a sorted varint page index, and a...
- `2026-04-15` **US-007** — Added an LTX V3 decoder with header parsing, varint page-index decoding, page-frame validation, LZ4 decompression, and random-access helpers, then covered it with round-trip and...
- `2026-04-15` **US-008** — Added a real `DeltaPageIndex` backed by `scc::HashMap<u32, u64>`, including store loading through `scan_prefix`, sorted range queries, and unit plus MemoryStore-backed integration...
- `2026-04-15` **US-009** — Added the initial `SqliteEngine<S>` with `Arc<S>` store ownership, per-actor PIDX cache storage, compaction channel construction, a lazy `get_or_load_pidx(...)` helper, and unit...
- `2026-04-15` **US-010** — Added `SqliteEngine::takeover(...)` with META creation and generation bumping, orphan DELTA/STAGE/PIDX recovery, page-1-first preload handling with optional hints and ranges, and...
- `2026-04-15` **US-011** — Added `SqliteEngine::get_pages(...)` with META generation fencing, page-0 rejection, one-shot blob batching across DELTA and SHARD sources, LTX decoding, shard fallback, and...
- `2026-04-15` **US-012** — Added the fast-path `SqliteEngine::commit(...)` handler with generation and head-txid fencing, LTX delta encoding, max-delta enforcement, one-shot `atomic_write` for DELTA plus...
- `2026-04-15` **US-013** — Added slow-path `commit_stage(...)` and `commit_finalize(...)`, including staged chunk serialization, generation and head-txid fencing, atomic promotion into DELTA plus PIDX plus...
- `2026-04-15` **US-014** — Added `CompactionCoordinator` with actor-id queue ownership, per-actor worker deduping, periodic finished-worker reaping, a tokio-spawnable `run(...)` entry point, and unit...
- `2026-04-15` **US-015** — Added the real sqlite-storage compaction path with a default worker, shard-pass folding into SHARD blobs, global DELTA deletion based on remaining PIDX refs, cache cleanup for...
- `2026-04-15` **US-016** — Added sqlite-storage quota helpers plus persistent `sqlite_storage_used` and `sqlite_max_storage` fields, enforced the quota in commit and finalize paths, updated takeover and...
- `2026-04-15` **US-017** — Added the full sqlite-storage Prometheus metric set from the spec, then wired commit, read, takeover, compaction worker, and shard compaction paths to update the counters,...
- `2026-04-16` **US-017b** — Replaced the `SqliteStore`/`MemoryStore` layer with direct UniversalDB access, added a chunking-aware `udb.rs` helper for logical values, rewired sqlite-storage engine handlers...
- `2026-04-16` **US-026** — Added `envoy-protocol` schema `v2` with SQLite request/response wire types, startup data, and top-level SQLite protocol messages; bumped the Rust and TypeScript protocol SDKs to...
- `2026-04-16` **US-028** — Added real sqlite websocket dispatch in `pegboard-envoy` for `sqlite_get_pages`, `sqlite_commit`, `sqlite_commit_stage`, and `sqlite_commit_finalize`; introduced a process-wide...
- `2026-04-16` **US-029** — Extended the actor start command with optional `sqliteStartupData`, populated it in `pegboard-envoy` by reusing internal takeover/preload before actor start, added explicit v1/v2...
- `2026-04-16` **US-029b** — Ported the UniversalDB simulated-latency hook and added the `sqlite-storage` RTT benchmark example, then updated the benchmark output to report direct actor round trips separately...
- `2026-04-16` **US-028b** — Switched `sqlite-storage` fast-path commit gating to raw dirty-page bytes, collapsed the fast path into a single UniversalDB transaction, removed the slow-path finalize...
- `2026-04-16` **US-025b** — Added a startup batch-atomic probe to `open_database(...)` that performs a tiny write transaction, checks `commit_atomic_count`, logs the configured error message, and aborts...
- `2026-04-16` **US-030** — Added real sqlite request/response plumbing to `rivet-envoy-client`, replaced the v2 VFS protocol trait with direct envoy-handle transport calls, and taught...
- `2026-04-16` **US-032** — Added explicit `sqliteSchemaVersion` to envoy actor-start commands, threaded it through pegboard actor creation plus the Rust and JavaScript envoy bridges, defaulted fresh actor2...
- `2026-04-16` **US-018** — Added the missing sqlite-storage integration coverage for direct commit/read cases, multi-actor isolation, explicit preload and orphan cleanup checks, and multi-shard plus...
- `2026-04-16` **US-045** — Expanded `sqlite-native` v2 coverage with direct-engine RocksDB tests for stale-head cache-miss reads, batch-atomic startup probing, real slow-path staged commits, transport-error...
- `2026-04-16` **US-021** — Added sqlite-storage quota and failure-path coverage for within-quota commits with unrelated KV data, atomic rollback on injected fast-commit failures, clean compaction retry...
- `2026-04-16` **US-023** — Collapsed `sqlite-storage` `get_pages(...)` into a single UniversalDB transaction, added stale-PIDX-to-SHARD fallback so reads stay correct during compaction, and added real...
- `2026-04-16` **US-042** — Added a test-only direct `SqliteEngine` transport for the v2 VFS, wired `sqlite-native` to real RocksDB-backed `sqlite-storage` in tests, and covered create/insert/select,...
- `2026-04-16` **US-041** — Removed creation-time SQLite schema selection from pegboard config and actor workflow state, then moved v1-vs-v2 dispatch to actor startup by probing the actor KV subspace for...
- `2026-04-16` **US-027** — Verified that `US-017b` already eliminated the `SqliteStore` abstraction and moved UniversalDB chunking into `engine/packages/sqlite-storage/src/udb.rs`, so `US-027` is satisfied...
- `2026-04-16` **US-034** — Fixed the remaining v2 E2E regressions in the bare/static driver suites by recovering `get_pages(...)` from stale PIDX and missing source blobs, serializing v2 VFS commit/flush...
- `2026-04-16` **US-046** — Stopped v2 `get_pages(...)` reads from overwriting VFS-owned `head_txid` and `db_size_pages`, limited read-side meta refreshes to `max_delta_bytes`, removed the unnecessary...
- `2026-04-16` **US-036** — Fenced shard compaction META writes by re-reading META inside the write transaction, comparing `generation` plus `head_txid`, and recomputing the updated META from the live head...

---

## Recent Story Details (last 3)

## 2026-04-16 09:43:52 PDT - US-037
- What was implemented: Hardened SQLite websocket handling in `pegboard-envoy` so actor validation failures, bad dirty-page payloads, and unexpected `sqlite-storage` errors return typed protocol responses instead of bubbling through the shared connection task. Replaced string-parsed fence/size/stage detection with typed `sqlite-storage` errors, added a shared `SqliteErrorResponse` wire variant, and updated the native v2 VFS plus direct transport harness to understand the new response path.
- Files changed: `engine/CLAUDE.md`, `engine/packages/pegboard-envoy/src/ws_to_tunnel_task.rs`, `engine/packages/sqlite-storage/{Cargo.toml,src/commit.rs,src/error.rs,src/lib.rs,src/read.rs,src/takeover.rs}`, `engine/sdks/schemas/envoy-protocol/v2.bare`, `engine/sdks/typescript/envoy-protocol/src/index.ts`, `scripts/ralph/prd.json`, `scripts/ralph/progress.txt`, `rivetkit-typescript/packages/sqlite-native/src/v2/vfs.rs`
- **Learnings for future iterations:**
  - `sqlite-storage` fence, missing-meta, oversized-commit, stage-missing, and concurrent-takeover cases should stay as typed errors so envoy and direct VFS harnesses can classify them without brittle string parsing.
  - `pegboard-envoy` should validate SQLite dirty-page shape before dispatch. `pgno == 0`, wrong page byte length, and duplicate page numbers are trust-boundary errors, not storage concerns.
  - Any shared-connection SQLite failure path needs a protocol error payload fallback. Letting a handler `?` out of `ws_to_tunnel_task` kills unrelated actors on the same envoy connection.
---
## 2026-04-16 09:50:37 PDT - US-038
- What was implemented: Moved sqlite v2 non-fence commit failure poisoning into `flush_dirty_pages()` and `commit_atomic_write()` themselves, kept callback wrappers focused on fence-mismatch translation, and added direct regressions for flush failure, atomic-write failure, and the startup batch-atomic probe.
- Files changed: `rivetkit-typescript/CLAUDE.md`, `rivetkit-typescript/packages/sqlite-native/src/v2/vfs.rs`, `scripts/ralph/prd.json`, `scripts/ralph/progress.txt`
- **Learnings for future iterations:**
  - `flush_dirty_pages()` and `commit_atomic_write()` need to own fatal transport/staging cleanup directly. Leaving that responsibility in outer sqlite callback wrappers makes direct callers and future refactors easy to get wrong.
  - Batch-atomic startup verification is worth keeping as a real open-path test. If `SQLITE_ENABLE_BATCH_ATOMIC_WRITE` disappears, v2 should fail fast instead of quietly pretending journal fallback is acceptable.
  - Fence mismatches are a separate path from ambiguous transport failures. The VFS should still surface them cleanly, but the "poison this connection" side effect for non-fence failures belongs at the commit helper layer.
---

## 2026-04-16 09:57:20 PDT - US-039
- What was implemented: Added an envoy-client fire-and-forget `sqlite_commit_stage` send path, switched sqlite-native v2 slow-path commits to queue stage uploads without awaiting per-chunk responses, and tightened the mock transport regression to prove only `commit_finalize` is awaited.
- Files changed: `engine/sdks/rust/envoy-client/src/handle.rs`, `rivetkit-typescript/packages/sqlite-native/src/v2/vfs.rs`, `scripts/ralph/prd.json`, `scripts/ralph/progress.txt`
- **Learnings for future iterations:**
  - Slow-path sqlite v2 commits should enqueue `commit_stage` messages immediately and rely on FIFO transport ordering, then surface any staged-write rejection through the final `commit_finalize` response.
  - `MockProtocol` is the right place to prove transport behavior like "queued versus awaited" stage requests; the direct-engine transport should stay conservative because it bypasses websocket ordering semantics.
  - `EnvoyHandle` fire-and-forget SQLite sends can safely drop the oneshot receiver after enqueueing, because the envoy side still tracks and clears the in-flight request when the response arrives.
---
## 2026-04-16 14:55:53 PDT - US-059
- What was implemented: Added phase-level SQLite commit observability across all three surfaces from the story: engine-side Prometheus histograms for fast, stage, and finalize phases plus commit payload sizes; envoy-side dispatch and response histograms plus debug spans around commit handling; and sqlite-native v2 VFS phase counters wired through `rivetkit-native` into RivetKit inspector metrics as `sqlite_commit_phases`. Added coverage for engine metric registration, native VFS counters, and `/inspector/metrics`, plus internal metric docs.
- Files changed: `docs-internal/engine/SQLITE_METRICS.md`, `engine/packages/pegboard-envoy/src/{metrics.rs,ws_to_tunnel_task.rs}`, `engine/packages/sqlite-storage/src/{commit.rs,metrics.rs}`, `rivetkit-typescript/CLAUDE.md`, `rivetkit-typescript/packages/rivetkit-native/{index.d.ts,src/database.rs}`, `rivetkit-typescript/packages/rivetkit/src/{actor/metrics.ts,db/config.ts,db/drizzle/mod.ts,db/mod.ts,db/native-database.ts,driver-test-suite/tests/actor-inspector.ts}`, `rivetkit-typescript/packages/sqlite-native/src/v2/vfs.rs`, `scripts/ralph/prd.json`, `scripts/ralph/progress.txt`
- **Learnings for future iterations:**
  - `rivetkit-native` prebuilt `.node` artifacts can hide Rust-side SQLite changes during TypeScript tests. If inspector metrics still look stale after a Rust change, run `pnpm -C rivetkit-typescript/packages/rivetkit-native build -- --force` before chasing ghosts.
  - New native SQLite getters are not enough on their own. The wrapper in `rivetkit-typescript/packages/rivetkit/src/db/native-database.ts` must forward them, and the DB open path in `src/db/mod.ts` or `src/db/drizzle/mod.ts` must register them with `ActorMetrics`.
  - Prometheus scrape-text assertions should check metric family names and label fragments, not a single exact serialized label order, because exposition order is not stable enough for brittle tests.
---
## 2026-04-16 15:02:53 PDT - US-059
- What was implemented: Re-validated the US-059 instrumentation surfaces that were already in the tree, then synced the Ralph bookkeeping by marking the story complete in `prd.json`.
- Files changed: `scripts/ralph/prd.json`, `scripts/ralph/progress.txt`
- **Learnings for future iterations:**
  - `cargo test -p sqlite-storage commit_registers_phase_metrics` and `cargo test -p rivetkit-sqlite-native vfs_records_commit_phase_durations` are the fastest story-specific smoke checks for the engine and native VFS halves of US-059.
  - Direct `pnpm test ...` invocation from the `rivetkit` package will not discover `src/driver-test-suite/tests/*.ts` files, so those inspector assertions need the driver-suite harness rather than a naive Vitest file filter.
  - If `progress.txt` says a story landed but `prd.json` still has `passes: false`, fix the bookkeeping immediately or Ralph will waste the next iteration rediscovering the same damn story.

### Baseline metrics (captured 2026-04-16)

Bench harness: `examples/kitchen-sink/scripts/bench.ts --filter 'Large TX insert 5MB'`
Environment: local RocksDB engine at `http://localhost:6420`, kitchen-sink serverless on `:3001`, namespace `fix2`, native sqlite v2 VFS. All runs use the single-commit fast path (`path="fast"`).

Bench result (five iterations):
  - `RUST_LOG=debug` (engine-rocksdb.sh default): 4 runs captured 1120.8ms, 1140.9ms, 1133.9ms, 1139.3ms. Median 1139.3 ms, throughput ~4.39 MB/s.
  - `RUST_LOG=info`: 4 runs captured 717.3ms, 740.4ms, 691.8ms, 700.5ms. Median 717.3 ms, throughput ~6.97 MB/s.
  - Per-op (insert) ~0.9 ms, baseline RTT ~13 ms, server-time ~1124 ms at debug level and ~700 ms at info.

Flag: `RUST_LOG=debug` vs `RUST_LOG=info` swings 5 MB commit throughput by ~37% (well above the 5% threshold). This reflects the pre-existing global engine debug firehose (`pegboard`, `gasoline`, `guard`, envoy ping debug spam seen in `/tmp/rivet-engine.log`), not the US-059 spans themselves; the new spans at US-059 only fire once per commit and are dwarfed by the envoy-wide `ToRivetPong` / workflow debug logs. Keep `RUST_LOG=info` for any future perf baselines so the instrumentation under US-048 does not get misattributed.

Engine `/metrics` scrape (port 6430, info run, 42 commits across the info runs):
```
# HELP rivet_sqlite_commit_phase_duration_seconds Phase duration for sqlite commit requests.
# TYPE rivet_sqlite_commit_phase_duration_seconds histogram
rivet_sqlite_commit_phase_duration_seconds_sum{path="fast",phase="decode_request"}  0.035501673
rivet_sqlite_commit_phase_duration_seconds_count{path="fast",phase="decode_request"} 42
rivet_sqlite_commit_phase_duration_seconds_sum{path="fast",phase="meta_read"}        0.004575840
rivet_sqlite_commit_phase_duration_seconds_count{path="fast",phase="meta_read"}     42
rivet_sqlite_commit_phase_duration_seconds_sum{path="fast",phase="pidx_read"}        0.390066439
rivet_sqlite_commit_phase_duration_seconds_count{path="fast",phase="pidx_read"}     42
rivet_sqlite_commit_phase_duration_seconds_sum{path="fast",phase="ltx_encode"}       0.152942212
rivet_sqlite_commit_phase_duration_seconds_count{path="fast",phase="ltx_encode"}    42
rivet_sqlite_commit_phase_duration_seconds_sum{path="fast",phase="udb_write"}        0.491419846
rivet_sqlite_commit_phase_duration_seconds_count{path="fast",phase="udb_write"}     42
rivet_sqlite_commit_phase_duration_seconds_sum{path="fast",phase="response_build"}   0.000048406
rivet_sqlite_commit_phase_duration_seconds_count{path="fast",phase="response_build"} 42

# HELP rivet_sqlite_commit_envoy_dispatch_duration_seconds Duration from sqlite commit frame arrival until sqlite-storage dispatch.
# TYPE rivet_sqlite_commit_envoy_dispatch_duration_seconds histogram
rivet_sqlite_commit_envoy_dispatch_duration_seconds_sum    0.035501673
rivet_sqlite_commit_envoy_dispatch_duration_seconds_count  42

# HELP rivet_sqlite_commit_envoy_response_duration_seconds Duration from sqlite-storage commit return until the websocket response frame is sent.
# TYPE rivet_sqlite_commit_envoy_response_duration_seconds histogram
rivet_sqlite_commit_envoy_response_duration_seconds_sum    0.002669989
rivet_sqlite_commit_envoy_response_duration_seconds_count  42

# HELP rivet_sqlite_commit_dirty_page_count Number of dirty pages written per sqlite commit path.
rivet_sqlite_commit_dirty_page_count_sum{path="fast"}    5852
rivet_sqlite_commit_dirty_page_count_count{path="fast"}  42
# HELP rivet_sqlite_commit_dirty_bytes Raw dirty-page bytes written per sqlite commit path.
rivet_sqlite_commit_dirty_bytes_sum{path="fast"}         23969792
rivet_sqlite_commit_dirty_bytes_count{path="fast"}       42
# HELP rivet_sqlite_udb_ops_per_commit UniversalDB operations per sqlite commit path.
rivet_sqlite_udb_ops_per_commit_sum{path="fast"}         42
rivet_sqlite_udb_ops_per_commit_count{path="fast"}       42
```

Actor `/inspector/metrics` scrape (Authorization: Bearer <actor.inspectorToken>, 10-commit slice on one actor):
```
"sqlite_commit_phases": {
  "type": "labeled_timing",
  "help": "SQLite VFS commit phase totals captured by the native VFS",
  "values": {
    "request_build":  { "calls": 10, "totalMs":   2.762393, "keys": 0 },
    "serialize":      { "calls": 10, "totalMs":   2.556633, "keys": 0 },
    "transport":      { "calls": 10, "totalMs": 607.534296, "keys": 0 },
    "state_update":   { "calls": 10, "totalMs":   6.369320, "keys": 0 }
  }
}
```

Ratio of each phase's average to total commit (engine fast path, sum-over-count):
  - decode_request:   0.85 ms /  25.58 ms =  3.3% (trust-boundary validation)
  - meta_read:        0.11 ms /  25.58 ms =  0.4%
  - pidx_read:        9.29 ms /  25.58 ms = 36.3% (dominant READ cost)
  - ltx_encode:       3.64 ms /  25.58 ms = 14.2%
  - udb_write:       11.70 ms /  25.58 ms = 45.7% (dominant WRITE cost)
  - response_build:  <0.01 ms /  25.58 ms = ~0%
  - envoy dispatch:   0.85 ms (envoy trust-boundary decode accounts for ~all of decode_request)
  - envoy response:   0.06 ms

VFS-side ratio (native counters, 10-commit actor slice):
  - transport   60.75 ms = 98.5% of per-commit wall time (waiting on envoy RTT)
  - state_update  0.64 ms =  1.0%
  - request_build 0.28 ms =  0.4%
  - serialize     0.26 ms =  0.4%

So the bench is bottlenecked on `transport` (native-to-envoy round trip) and, on the engine side, on `udb_write` + `pidx_read`. This matches US-048's expected attack surface: commit pipelining + PIDX cache will show up as a drop in both `transport` (VFS side) and `pidx_read` (engine side) without moving `udb_write` much.

Raw captures retained at `/tmp/us-059-metrics-full.txt` (engine /metrics, all families), `/tmp/us-059-metrics-info.txt` (filtered US-059 only), `/tmp/us-059-inspector-info.json` (full inspector snapshot), and `/tmp/us-059-bench-baseline.log` (one bench run stdout).
---
## 2026-04-16 15:31:34 PDT - US-048
- What was implemented: Finished the per-txid DELTA chunk rewrite by fixing staged-commit reads to fall back to historical DELTA scans when no PIDX rows exist yet, updating sqlite-storage takeover/finalize tests to the new orphan-chunk model, and syncing sqlite-native mock slow-path tests with the new `commit_stage_begin` RPC plus byte-chunk staging.
- Files changed: `AGENTS.md`, `engine/packages/sqlite-storage/src/{commit.rs,read.rs,takeover.rs,compaction/shard.rs}`, `rivetkit-typescript/packages/sqlite-native/src/v2/vfs.rs`, `scripts/ralph/{prd.json,progress.txt}`
- **Learnings for future iterations:**
  - Slow-path SQLite v2 commits do not materialize `/STAGE` keys anymore. Recovery and tests must treat `delta_chunk_key(actor_id, txid, chunk_idx)` with `txid > head_txid` as the orphaned state.
  - `get_pages(...)` still has to recover committed staged data when PIDX is absent, so the read path cannot early-return zero-filled pages just because no shard or PIDX source was found in the first pass.
  - sqlite-native mock slow-path tests cannot assume a fixed stage-request count anymore. The VFS chunks the fully encoded LTX bytes, not the original dirty-page list.
  - Quality checks run clean with `cargo test -p sqlite-storage` and `cargo test -p rivetkit-sqlite-native`.
---
## 2026-04-16 15:42:22 PDT - US-048
- What was implemented: Corrected the stale Ralph bookkeeping for the already-landed US-048 branch commit by marking the story passing in `prd.json` and re-verifying the critical staged-commit, takeover-recovery, read-path, and reopen tests in isolation.
- Files changed: `scripts/ralph/{prd.json,progress.txt}`
- **Learnings for future iterations:**
  - `prd.json` can drift behind the actual branch state. If `git log` already contains `feat: [US-048] - [...]` but `passes` is still false, fix the bookkeeping before Ralph burns another cycle re-implementing the same story.
  - `cargo test -p sqlite-storage` and `cargo test -p rivetkit-sqlite-native` run cleaner as isolated story-focused filters here; a concurrent full-package run produced a hanging compaction test and native RocksDB lock noise that did not reproduce in isolated checks.
---

### 1 MiB shape experiment (captured 2026-04-16 ~15:50 PDT)

Question: for the original 5 MB bench (`largeTxInsert5MB`), engine-side commit work summed to ~233 ms but total E2E was 1128 ms, leaving ~900 ms unaccounted for. Hypothesis was that per-statement / NAPI overhead dominates the gap. To test, three new bench variants commit the same 1 MiB payload shaped three ways (different statement counts).

Environment: local RocksDB engine on `:6420` with US-048 and US-059 both landed, kitchen-sink `--prod dist/server.js` on `:3001`, namespace `fix2`, fresh actor per run (new key), `RUST_LOG=info` (via `scripts/run/engine-rocksdb.sh` default).

| Variant | Rows × payload | NAPI crossings | E2E | Server | Per-op |
|---------|----------------|---------------|------|--------|--------|
| Tiny     | 4096 × 256 B  | 4096           | 334.6 ms | 311.8 ms | 0.1 ms |
| Medium   | 256 × 4 KiB   | 256            | 158.2 ms | 141.2 ms | 0.6 ms |
| One row  | 1 × 1 MiB     | 1              | 132.6 ms | 114.0 ms | 114.0 ms |

All three commit 1 MiB total. The floor (one-row, 1 NAPI crossing) is **132.6 ms**. Adding statements scales the time linearly:
- Tiny vs one-row: +202 ms over +4095 crossings ≈ **49 µs per extra statement**.
- Medium vs one-row: +25.6 ms over +255 crossings ≈ **100 µs per extra statement**.

Interpretation: **per-statement cost (NAPI + SQLite prepare/bind/step/finalize + arg marshaling) is the primary source of the 5 MB bench's unexplained ~900 ms.** The 5 MB bench fires 1280 INSERTs. At ~50 µs/statement (warm cache, small args) that's ~64 ms; the 5 MB bench probably has higher per-statement cost because `randomblob(4096)` produces larger bound args and dirties more pages per statement, pushing per-statement cost into the 500-700 µs range. 1280 × 600 µs ≈ 770 ms, a plausible match for the observed ~900 ms gap.

Follow-up levers (NOT part of US-048 or US-055):
- **Batched INSERT** — the existing `insertBatch` action shape (one multi-VALUES INSERT) would collapse 1280 NAPI crossings to 1. Try adding a 5 MB variant that uses batched insert to confirm.
- **Prepared statement cache** — the native VFS could cache `sqlite3_stmt` for identical SQL text across execute calls to avoid re-prepare costs.
- **JS-side payload batching** — the db.execute() API could accept an array of `[sql, args]` pairs and do N calls in one NAPI round trip.

Per-variant engine-side commit phase histograms could not be cleanly attributed because the `/metrics` histogram has been accumulating across the full engine run (88 commits total in the current window, most from earlier work). For a clean per-variant Prometheus attribution, scrape `/metrics` before and after each run.
---

### Debug vs release build comparison (captured 2026-04-16 ~16:35 PDT)

Both the engine and rivetkit-native's `.node` default to DEBUG builds. Re-ran the exact same bench variants against release builds (`./target/release/rivet-engine start` + `pnpm build:force:release` for rivetkit-native).

| Variant | Debug E2E | Release E2E | Release server | Speedup |
|---------|-----------|-------------|----------------|---------|
| Tiny 1 MiB (4096 × 256 B) | 334.6 ms | 97.1 ms  | 92.8 ms  | 3.4x |
| Medium 1 MiB (256 × 4 KiB) | 158.2 ms | 27.1 ms  | 22.5 ms  | 5.8x |
| One row 1 MiB (1 × 1 MiB) | 132.6 ms | 20.7 ms  | 16.7 ms  | 6.4x |
| 5 MiB (1280 × 4 KiB) | 706-1128 ms | 112.9 ms | 107.7 ms | 6.3-10x |
| Baseline RTT (noop) | 14 ms | 2.5 ms | - | 5.6x |

Engine release per-phase speedup (debug avg / release avg, from `/metrics` sum/count):
- `decode_request`: 5.7x
- `meta_read`:      7.3x
- `ltx_encode`:    **19x**  (CPU-heavy Rust work)
- `pidx_read`:     **21x**  (tight FDB-read loop in Rust)
- `udb_write`:      6.1x

Key conclusions:
- Release builds deliver 3.4-10x across the board. The earlier 30-50% estimate was low by an order of magnitude.
- `ltx_encode` and `pidx_read` see the biggest gains because they run Rust-heavy loops that the Rust debug profile punishes most.
- Per-statement NAPI overhead shrinks from ~50-100 µs (debug) to ~18-22 µs (release). Still a real cost proportional to statement count, but much smaller.
- **A 5 MiB transactional commit now takes ~113 ms E2E on release**, production-viable. Debug numbers made the system look much worse than it is.

IMPORTANT: run all perf baselines and Ralph-level benches against release binaries. Debug numbers will mislead future decisions on where optimization work is warranted. Consider updating `scripts/run/engine-rocksdb.sh` to default to `cargo run --release` when `RIVET_RELEASE=1` is set, or adding a `scripts/run/engine-rocksdb-release.sh` variant.

Also confirmed (earlier assumption corrected): the 5 MB bench does NOT do mid-transaction spills. One `BEGIN...COMMIT` block produces ONE big commit (`le=4096` dirty-page bucket). The 9 extra commits observed in the metrics window are unrelated actor/lifecycle writes (noop warmup, migrations, metadata). SQLite's xSync-at-COMMIT behavior holds.
---
