Embedded MCP Single-Owner Enforcement + Orphan Reaping — Investigation & Design

Status: investigation complete; design proposed (agree before Phase 2 code) Date: 2026-06-16 Owner: Zaxy core

1. Incident

For an entire working session, every memory_checkout appeared to "hang" (~18-22s+ then the MCP server disconnected). After misattributing it to checkout performance (real but unrelated work shipped as #81/#82/#83) and a stale install (server was on 2.2.0), the true cause was found: the embedded LadybugDB projection was corrupted — opening it throws RuntimeError: Assertion failed ... wal_record.cpp ... UNREACHABLE_CODE — and four zaxy serve processes (three orphans from prior sessions + the live one), all with EVENTLOOM_PATH=.../zaxy/.eventloom, were associated with that one store. Manual recovery: kill the this-workspace servers + move the corrupt store aside + reconnect. See memory incident-mcp-orphan-projection-corruption.

This note root-causes why the safeguards let that happen and proposes the durable fix. The embedded backend takes an exclusive write lock; concurrent writers corrupt the WAL. The design intends one owner + N proxies per workspace (EmbeddedMcpRuntimeCoordinator: "Coordinates a single embedded MCP owner per Eventloom directory"; non-owners run mcp_server.proxy_main over the owner's unix socket).

2. Method

Static trace of cli/serving.py:serve (the owner/proxy decision) and mcp_runtime.py (the coordinator), plus a reproduction of the coordinator's claim behavior.

3. Findings (each confirmed by code and/or reproduction)

F1 — The coordinator only guards the serve stdio path; every other store-open bypasses it

Owner claim happens only here: cli/serving.py:1242 — if transport == "stdio" and projection_backend == "embedded": coordinator.try_claim_owner(). The CLI commands that open the embedded store directly — memory checkout (its projection path / #83 lock-fallback), reproject, doctor, dashboard — build a MemoryFabric/store and connect() with no owner claim. So any non-serve store access is uncoordinated: it can open the store as a second writer alongside a serve-owner (or another CLI), or hit the lock and fall back.

F2 — The owner lock and the store are keyed independently; divergence defeats single-owner

These are resolved separately. When they diverge — mismatched env, cwd-relative settings, or an explicit --embedded-graph-path that doesn't match the eventloom — single-owner is enforced on the wrong key. Reproduced: two coordinators on different eventloom paths both successfully claim owner; if both then open one shared store, that's two writers → corruption. (Same-key correctly refuses the second claimant.)

F3 — No orphan reaping; recovery only handles dead owners, not live-but-broken ones

try_claim_owner uses fcntl.flock(LOCK_EX|LOCK_NB) held for process lifetime — correct for live coordination. But:

free (owner dead). If the lock is held but the owner socket is unhealthy it returns a warning telling the human to "Fully exit stale processes" — i.e. exactly the manual step we performed. No automatic recovery from a live-broken owner, and no reaping of accumulated orphans.

F4 — Unclean owner death leaves a dirty WAL; the next open crashes instead of self-healing

If an owner is SIGKILLed or crashes mid-write (or a CLI store-write is timeout-killed — F1), LadybugDB's WAL is left uncheckpointed/dirty. The next open replays it and can hit UNREACHABLE_CODE. EmbeddedGraphStore.connect only handles the pre-fork-Kuzu format-incompatibility case (moves it aside); it re-raises everything else, so WAL corruption propagates and crashes the checkout rather than quarantining the (derived) store and rebuilding.

4. Why the incident happened (synthesis)

The four orphans shared one EVENTLOOM_PATH, so F2 alone wouldn't multi-own them. The corruption is best explained by F1 + F4 + F3 compounding: uncoordinated direct store-opens (CLI checkouts, including timeout-killed ones mid-write) and/or an uncleanly-killed owner left a dirty WAL (F4); orphaned servers accumulated and were never reaped (F3); and once the store was corrupt, every owner/proxy that opened it crashed (F4) — the perpetual hang+disconnect. The exact historical sequence can't be fully reconstructed (processes gone), but all four gaps are confirmed and each independently enables this corruption class.

5. Design — the durable fix (phased; agree before code)

Invariants (hard): multiple clients in one repo (e.g. two Claude instances) must work as exactly one owner + N proxies; the store is opened by at most one process; the Eventloom log stays authority and the projection is rebuildable; reaping never touches live or cross-workspace servers.

6. Done-when

Root cause documented (this note); a concurrent two-instance start against one eventloom yields exactly one owner + proxies with zero store corruption (a concurrency/stress test); divergent eventloom/store keys can no longer produce two owners of one store; orphaned dead-session servers are reaped or cannot accumulate; a corrupt store self-heals rather than crashing checkout; ruff + mypy + full suite green after each phase.

7. Open questions

  1. Lock key: derive strictly from the resolved store path, or keep the eventloom key but assert dirname(store) == eventloom/projections? (Leaning: key on the store path — it's what actually needs protecting.)
  2. CLI-while-no-owner: should a lone CLI checkout become a transient owner (claim → open → release) so it's still coordinated, or always graph-degrade? (Leaning: transient owner-claim so a solo CLI still gets full graph + can't race a second CLI.)
  3. Reaping trigger: startup self-heal only, an explicit doctor --repair, or both? Liveness signal for "orphan" (no client) vs "legitimate idle owner".

References