# Throughput bench — 2-core LXC + trader

**Date:** 2026-05-20
**Engine:** v0.7.17 (Phase 4.3 shipped @ commit 374318e — confirmed via cached source)
**Server binary:** yantrikdb-server v0.8.16 (engine pin v0.7.16, sha 335ffb2, built 2026-05-17)
**Bench target:** CT 132 on PVE node2 (fresh, 2-core / 4GB / Debian 13), and trader CT 168 (4-core / 4GB, prod with lane-b@algo co-tenant)
**Harness:** `crates/yantrikdb-core/examples/wedge_repro.rs` (engine-direct) + `benchmarks/throughput_2core_lxc/http_bench.py` (HTTP)

## Goal

Pranab: *"verify how much we can do in a 2 core lxc, need to push as much as possible since raw sqlite supports 50k+ per sec so we need to push the boundary."*

Previous targets (RFC `decoupled_write_path_rfc.md` post-Phase-4.3):
- Writers=32: write tput ≥ 4000/s, read tput ≥ 180/s, read p99 ≤ 200ms.

## Engine ceiling — fresh 2-core LXC, no HTTP, no embedder

`wedge_repro` calls `db.record()` directly with pre-computed embeddings (no embedder cost, no HTTP overhead). Warmup 2000 records, duration 30s, dim=384, readers=4.

| Writers | Write tput | Read tput | W p50 | W p99 | R p50 | R p99 |
|---|---|---|---|---|---|---|
| 1 | 374/s | 110/s | 0.4ms | 31ms | 14ms | 356ms |
| 4 | 758/s | 87/s | 1.6ms | 53ms | 18ms | 384ms |
| 8 | 837/s | 75/s | 5.5ms | 75ms | 21ms | 412ms |
| 16 | 968/s | 58/s | 11ms | 100ms | 29ms | 426ms |
| 32 | **1115/s** | **39/s** | 19ms | 165ms | 46ms | 485ms |

### Comparison vs pre-Phase-4.3 baseline (May 7 sweep, commit 36ba7da)

| Writers | May-7 W tput | May-20 (2-core) W tput | Ratio |
|---|---|---|---|
| 1 | 196/s | 374/s | 1.9× |
| 4 | 317/s | 758/s | 2.4× |
| 8 | 454/s | 837/s | 1.8× |
| 16 | 536/s | 968/s | 1.8× |
| 32 | 587/s | 1115/s | 1.9× |

Phase 4.3 doubled write throughput. Note: the May-7 numbers were taken on Pranab's Windows dev box (8+ cores); the May-20 numbers are on a **2-core LXC**. So the per-core gain is even larger than the table suggests.

### Engine vs RFC targets

| Writers | RFC target W tput | Actual (2-core) | Status |
|---|---|---|---|
| 32 | ≥ 4000/s | 1115/s | **3.6× below target** |

| Writers | RFC target Read tput | Actual (2-core) | Status |
|---|---|---|---|
| 32 | ≥ 180/s | 39/s | **4.6× below target** |
| 32 | RFC Read p99 ≤ 200ms | 485ms | **2.4× over budget** |

Phase 4.3 helped writes substantially but did NOT achieve the "reads stay flat as writers increase" promise. Reads still degrade under write pressure on 2 cores.

Likely contributors to reader-writer contention that survive Phase 4.3:
- HNSW search reads share `vec_index` lock with delta-tier appends.
- SQLite WAL fsync stalls on heavy write commits.
- 2 cores means writers + readers + materializer all share the same hardware threads.

## HTTP ceiling — trader CT 168 (production)

`http_bench.py` from CT 132 → trader at 192.168.4.13:7438 (network path), pre-computed embedding (skips embedder). Token from `/etc/lane-b.env`.

| Concurrency | Duration | Tput | p50 | p95 | p99 | 503s |
|---|---|---|---|---|---|---|
| 2 | 15s | **1/s** | 2539ms | 6258ms | 6258ms | 0 |
| 8 | 10s (+timeout) | **1/s** | 6574ms | 8887ms | 8887ms | 0 |

Recall (serial, 10 calls): **2/sec, 496ms each.**

Trader at bench time:
- load average 3.67/4.24/4.30 (4-core box → ~100% utilized)
- yantrikdb proc: **110% CPU**, 19.6% mem
- Swap: 423.8 MiB used of 1024 MiB — **memory pressure**
- iowait: 6.6%
- lane-b idle (0% CPU at the moment but its substrate has 8417 entities, 798 consolidated, 41560+ operations cumulative)

### The 1000× gap

Engine ceiling on fresh 2-core: 374 w/s at writers=1.
Trader HTTP at c=2: 1 w/s, p50 2.5s.

**Three-orders-of-magnitude HTTP-path overhead.** Candidate causes (not yet diagnosed):
1. **Memory pressure** — 423MB swap is being touched; each write may hit a paged-out page.
2. **Openraft state-machine apply** — single-node still runs the full raft log + apply pipeline.
3. **Embedder serial contention** — even with pre-computed embedding in the body, the server may still tokenize for FTS / metadata indexing.
4. **Build state** — trader has run 7h+ continuously with lane-b's substrate; vec_index is large, materializer queue may be in degraded state.

## Tuned env-var sweep — `DELTA_MAX=64 MAX_DIRTY_AGE_SECS=10`

After core flagged the read-collapse as DeltaIndex tier-balance (single compactor falls behind on 2 cores, recall scans the 256-entry delta linearly), re-ran the engine sweep with tightened env vars. Same harness, same hardware.

| Writers | Base W/s | Tuned W/s | ΔW | Base R/s | Tuned R/s | ΔR | Tuned R p99 |
|---|---|---|---|---|---|---|---|
| 1 | 374 | 487 | +30% | 110 | 137 | +25% | 301ms |
| 4 | 758 | 798 | +5% | 87 | 174 | +100% | 260ms |
| 8 | 837 | 1023 | +22% | 75 | 150 | +100% | 264ms |
| 16 | 968 | 1283 | +33% | 58 | 107 | +84% | 304ms |
| 32 | 1115 | **1566** | **+40%** | 39 | **67** | **+72%** | 322ms |

Reads recovered 72% at writers=32 — close to but below core's predicted 80-120/s p99 250ms floor. Writes also got 40% faster, presumably because tighter delta keeps HNSW insertion paths shorter and the cold-tier scan during recall stays bounded. Engine pressure (delta_len / delta_max) is 100% at writers=8/16/32 → compactor is now the bottleneck on 2 cores.

Implication: a default lower than 256 (or a `num_cpus`-based auto-tune) would help small deployments. The structural lift is parallel compactor (engine saga task 18).

## Anomaly: fresh CT 132 HTTP-server wedge

On a clean DB on CT 132, the HTTP server **accepts exactly 256 writes then permanently wedges** with "ingest queue full (256 pending ops, max=256)". State stays at 256 indefinitely. `oplog.applied=1` for all rows, `entities=0`, `edges=0` — the in-memory materializer queue does not drain even though "background workers started worker_count=6" was logged.

Trader does NOT exhibit this; it has 8417 entities + 181 edges + 41546+ operations.

Hypothesis: the materializer's first-run path on a fresh DB has a stall, possibly waiting on a model load that never completes, or a config-driven thread pool that isn't engaged in single-node mode. Worth a sibling investigation issue.

### Update from gdb on CT 132 wedged state (pid 8173)

All 8 server threads captured via `gdb thread apply all bt`. **No thread is in any engine code path.** Tokio multi-thread workers parked, rayon pool sleeping, ONNX runtime cond_wait, deadlock detector periodic sleep, main runtime block_on idle. The workers think there's nothing to do.

SQLite direct dump on `/root/yantrikdb-data/bench/yantrik.db`:
- `memories=3244`, `oplog_total=768`, **`oplog_applied_1=768`**, **`oplog_applied_0=0`**
- `entities=0`, `edges=0`, `conflicts=0`

The oplog is fully drained (every op applied=1). Yet `/v1/remember` returns `"ingest queue full (256 pending ops, max=256)"` **synchronously** (no wait). This is a **counter-leak in the in-memory MAX_PENDING_OPS accounting**, not a worker hang — the workers drained the channel but the counter never decremented. Likely cause: `apply_materialize_record_post` Err path on fresh-install state (empty graph_index, empty entities) that short-circuits without releasing the permit. Trader doesn't trip this because its accumulated state means the extraction path returns Ok.

`/v1/recall` returns correct results on the same wedged server instantly. Only the write path is wedged.

Diagnostics shipped to yantrikdb-core via swarm msg `278ab045`.

## Answer to Pranab's question

> "In your opinion should we be able to handle this load easily?"
> "We should be able to support 1k+ writes/reads per sec."

**Writes — engine layer YES (1115/s ceiling at writers=32 on 2-core). Server/HTTP layer NO right now (1/s on trader, 256-cap wedge on fresh installs).**

**Reads — engine layer NO under write pressure (39/s at writers=32). Server layer NO (2/s serial on trader).**

The 1k+ writes target is achievable at the engine layer today — Phase 4.3 already gets us there on 2 cores. The HTTP/server stack is the gap that needs work, plus the reader-writer contention that survives Phase 4.3.

## Recommended next moves (priority order)

1. **Investigate trader memory pressure** — bump CT 168 from 4GB → 8GB (similar treatment to CT 167 per rid 019e31c6) to eliminate swap.
2. **Lift `MAX_INFLIGHT`** server-side from 256 → 4096 (or make config-driven). Small PR. Buys burst tolerance.
3. **Diagnose the CT 132 materializer wedge** — fresh-install HTTP-mode servers shouldn't accept exactly 256 writes then die. File issue.
4. **Switch lane-b @ algo / brain / DMN / salience to `/v1/remember/batch`** for client-side amortization (per core's reply).
5. **Engine PRs from core's reply** (if still below target after the above):
    1. Combine two `log_op` INSERTs (~30% conn.lock reduction)
    2. Multi-row INSERT VALUES in `record_batch` (5-10× under wide batches)
    3. Group-commit primitive (concurrent `record()` coalesce into one SAVEPOINT — brainstorm-first; RAII SyncWriteGuard contract makes it tricky)
6. **RFC for reader-writer contention beyond Phase 4.3** — reads should not collapse under write load. The original `decoupled_write_path_rfc.md` post-fix target of "read tput stays flat" is not met. May require splitting `vec_index` read/write locks per tier.

## Files

- Engine sweep harness: `crates/yantrikdb-core/examples/wedge_repro.rs` (in engine repo @ commit 374318e)
- HTTP harness: `benchmarks/throughput_2core_lxc/http_bench.py`
- Probe shell: `benchmarks/throughput_2core_lxc/probe_drain.sh`
- Raw engine sweep logs: CT 132 at `/root/sweep_results/wedge_w{1,4,8,16,32}.log`
- CT 132 (bench): PVE node2, 192.168.4.70, 2-core / 4GB Debian 13
- Trader CT 168: PVE node1, 192.168.4.13:7438
