# Node Identity Matcher — Save-Time vs Load-Time Boundary

## Context

OpenWriter persists every document as a plain `.md` file with minified-JSON frontmatter between `---` delimiters. Every save round-trips an in-memory TipTap tree through markdown text. Identity tracking (node IDs that survive edits) lives in the `nodes` field of frontmatter as **positional slim tuples** — `[id, sentences[], marks?]` for paragraphs, `[id, type, ...]` for everything else. The slim shape gets resolved to a rich in-memory `Fingerprint` at load time via `resolvePreviousNodes` (positional re-fingerprint from the body for fields that are trivially recomputable). The matcher never sees slim tuples.

The matcher reconstructs identity by comparing two snapshots of the same document — the pre-edit graph (last save's `nodes`) against the post-edit tree (current TipTap). When fingerprints align, IDs flow forward; when content changes shape, the matcher's mutation rules decide whether it's an edit (preserve ID), an insert (mint new), a delete (move to graveyard), a type-change (preserve ID across type swap), or a paste-back (restore ID from graveyard).

For full design context: `docs/node-identity.md`.

## Current invariants

- **Disk is the canonical identity store.** `nodes` and `graveyard` in frontmatter are the only durable record of node identity. No parallel in-memory cache mirrors them.
- **The matcher reads `previousNodes` + `graveyard` from disk at save time.** Specifically: `writeToDisk` in `server/state.ts` calls `readPersistedIdentity(state.filePath)` immediately before serializing, parses the existing file's frontmatter, runs `matchNodes(previousNodes, newBlocks, { graveyard })`, applies pinned IDs back onto the in-memory tree via `applyIdsToTiptap`, and writes the resulting `nextGraveyard` into the new frontmatter.
- **The matcher also runs at load time** (`server/markdown-parse.ts` → `applyMatcher`) so the in-memory tree starts each session with the same IDs the disk frontmatter encodes.
- **External docs** (anything outside `~/.openwriter/profiles/`) bypass the matcher entirely. They keep their frontmatter verbatim via `state.originalFrontmatter`; we never inject `nodes` / `graveyard` into a user's other markdown files.
- **`tiptapToMarkdownChecked` is the serializer of record for internal docs.** It re-parses the just-written markdown and logs a sync report if the round-trip changes block shape — drift never blocks the save, but it shouts in the console.
- **`PadState` does NOT cache `lastSavedNodes` or `graveyard`.** If you find yourself reaching for `state.lastSavedNodes`, stop — that's the anti-pattern this ADR exists to prevent.
- **Container fingerprints are content-empty.** Containers (`blockquote`, `bulletList`, `orderedList`, `taskList`) have `charCount: 0, sentences: []` because their content lives in child nodes. As a side effect, the graveyard-restore rule will match any container of the right type at the right slot — deleting a blockquote and inserting a different blockquote at a similar slot will restore the original container ID (children get fresh IDs because their own fingerprints differ). This is intentional: container identity is structural, not content-based.
- **Slot-continuity wins over operation-intent.** If a single `write_to_pad` batch contains both `rewrite(X)` and `insert(above X)`, the matcher's slot-continuity rule pins X's ID to its original slot position (now holding the inserted content) and mints a fresh ID for the rewritten content at its new slot. The matcher sees fingerprints + positions, not operation tuples. Callers that depend on `rewrite` being identity-preserving should not combine it with adjacent inserts in the same save batch — issue them as separate saves if ID continuity for the rewritten block matters.
- **TipTap inserts a trailing empty paragraph on every load.** After a `markdownToTiptap` round-trip, the parsed tree gains an empty paragraph at the end so the editor cursor has a landing spot. The matcher mints a fresh ID for it (no fingerprint match in `previousNodes`). This shows up as a one-time `+1` to active node count on the first reload after a save. Not drift, not a bug — an editor UX artifact that's invisible in the rendered body (`<!-- -->` marker).
- **`restoreVersion` reverts content, but identity flows through the save-time matcher.** Restoring an old snapshot returns the parsed snapshot doc with the snapshot's IDs assigned. But the moment the restored doc is saved, the matcher's `previousNodes` is the CURRENT disk frontmatter (the post-snapshot state), not the snapshot's frontmatter. So the restored content's fingerprints get matched against the post-snapshot state's node graph. Most IDs flow back via exact-match or slot-continuity, but blocks that share slot positions with post-snapshot orphans may be slot-continuity-paired with those orphans rather than graveyard-restored to their snapshot IDs. For backlink fidelity across restores, the right pattern is to re-extract forward links from the restored content and let `updateBacklinksForSource` reconcile.
- **Backlinks ride on matcher ID stability.** `extractForwardLinks` produces `{from_doc, from_node, to_doc, to_node?}` entries where `from_node` is the source block's ID and `to_node` is the target block's ID. `updateBacklinksForSource` writes these into target docs' frontmatter `backlinks:` after every save. Because the matcher preserves IDs through edits, type-changes, and slot-continuity, backlink references stay valid across content changes. Delete-then-paste-back at either end uses graveyard-restore to recover the original ID, keeping backlinks intact even across a delete cycle. The `doc:DOCID#NODEID` href format requires both IDs to be 8-char lowercase hex (`[a-f0-9]{8}`) — non-hex IDs are silently dropped by `parseDocHref` and won't produce a backlink entry.
- **Pending state is keyed by ordinal position + text fingerprint, not by node ID.** The `pending: {n: {s, o, t}}` frontmatter map records pending-review state per leaf block at its ordinal position. On load, `rehydratePendingState` reassigns these entries to blocks by position with text-fingerprint fallback for shifts. This makes pending state matcher-independent — the matcher mutates `attrs.id` without touching `attrs.pendingStatus`, so pending state travels with the block through every matcher operation (edit, type-change, slot-continuity, graveyard-restore). Deleting a pending block removes its entry; inserting a new block above a pending block shifts its ordinal position correctly.
- **`applyIdsToTiptap` and `tiptapToBlocks` use identical descent rules.** Both walk the tree in pre-order, both emit one position per block, and both treat tables as opaque (don't descend into rows / cells / cell paragraphs). Without this alignment, position counters diverge: `tiptapToBlocks` skips past a table at flat-position N, the matcher pins ID X to flat-position N+1 (the next top-level block), and `applyIdsToTiptap` walks INTO the table — arrives at descended-position N+1 inside the table — assigns X to a table internal node. The top-level node never gets X. Both walkers descend ONLY into `bulletList`, `orderedList`, `taskList`, `listItem`, `taskItem`, `blockquote`. The two functions are siblings; any change to one must mirror in the other.
- **`findNode` treats tables as opaque, mirroring the matcher's walker.** Top-level write resolution (`findNode` in `server/state.ts`) does not descend into `table`, `tableRow`, `tableCell`, or `tableHeader`. Table-internal IDs are ephemeral (regenerated every `markdownToTiptap` parse via `parseTableTokens`) and untracked by the matcher; `compactNodes` never exposes them to the agent. Any agent ID that happens to match a node inside a table is therefore a collision, not a legitimate target — descending would silently corrupt the table cell. Every layer (matcher walker, `compactNodes`, `parseTableTokens` regeneration, `findNode`) now agrees: table internals are not addressable from outside.
- **Save-time matcher broadcasts ID rewrites to all clients.** `writeToDisk` captures the pre-matcher `attrs.id` for every block (via `beforeIds` from `tiptapToBlocks` output), runs the matcher, then computes the set of positions where the matcher's assigned ID differs from the pre-matcher ID. Each `{oldId, newId}` pair fires the `idRewriteListeners` listeners (registered via `onIdRewrites`), which in turn drive `ws.ts` to send an `id-rewrites` WebSocket message to every connected client. Browser clients walk their in-memory TipTap doc and call `setNodeMarkup` for each match in a single atomic ProseMirror transaction. The matcher is the authority on block identity; this is how that authority propagates to clients. Without this broadcast, the browser holds stale IDs after any matcher reassignment (transient editor-minted IDs at slots with previousNodes orphans, doc-update reconciliation, etc.), subsequent server→browser `node-changes` targeting the new IDs silently fail at `bridge.ts`'s anchor lookup, and the browser's autosave clobbers fresh server state with the stale view. The empty `oldId` case (genuinely new blocks the editor never named) doesn't fire — only positions where both old and new IDs exist and differ count as rewrites.
- **External-write reload refreshes disk frontmatter.** `reloadActiveDocFromDisk` calls `writeToDisk` after applying the external body to in-memory state. The matcher's edit rule pins IDs and emits fresh per-block fingerprints, so disk frontmatter stays coherent with disk body. Without this, a paragraph edited externally would leave the disk frontmatter pointing at the previous save's fingerprints — a subsequent in-memory cut would graveyard the block with that stale fingerprint, and paste-back's fingerprint (matching the current body) would never trigger graveyard-restore. fs.watch self-suppression via `state.loadedMtime === diskMtime` prevents a reload→save→reload loop.
- **`tiptapToBlocks` carries `node.attrs?.id` into `Block.id`, and the matcher honors it.** `applyInsertRule` preserves `candidate.block.id` instead of always minting fresh, and claims the preserved ID from `claimedPrevIds` / `claimedGraveIds` so the same ID never ends up in both `nodes:` and `graveyard:`. `applySlotContinuityRule` skips candidates whose ID is already in `previousNodes` or `graveyard` — those IDs are real (agent-assigned via `applyChangesToDocument`, restored from a snapshot, or graveyard paste-back) and slot-continuity's positional guessing must not override them. Without this, the matcher would silently rewrite IDs the browser had just received over WebSocket, the browser's `findNodeByIdInDoc` would return null for subsequent updates, and the browser's autosave would eventually overwrite fresh server state with stale content. Transient IDs (not in either set — e.g. a fresh ID minted by `@tiptap/extension-unique-id` when the user replaces a block in place) still flow through slot-continuity per the "slot is innocent" principle. This is the v0.14.1 hotfix invariant.
- **Agent marks tolerate stale nodeId references by design.** Marks live in sidecar files at `DATA_DIR/_marks/{filename}.json` with `nodeId` + optional `nodeIds[]` references into the doc body. `marks.ts` exports a `pruneStaleMarks(filename, validNodeIds)` helper but it is intentionally NOT called from the save-time matcher path. The reason: when a block is deleted and later pasted back, the matcher graveyard-restores its original ID, and any stale mark references to that ID become valid again automatically — no code intervention needed. The rendering layer (`src/decorations/marks-plugin.ts`) has a Pass-3 text fallback that resolves marks by content when nodeIds go stale, so user-visible marks always anchor correctly even before the paste-back recovery. Auto-pruning would destroy this paste-back recoverability; the latent staleness in the sidecar is the price of that recovery.

## Decision log (append-only)

### 2026-05-16 — Option B: save-time matcher reads from disk, no in-memory identity cache

- **Trigger.** Live flood-test on `Matcher Integration Test.md` revealed that type-change (Op 7) and graveyard-restore (Op 8) never fired within a session. Root cause: the matcher only ran on load. The editor minted fresh IDs at every insert, and by the time a save fired, the in-memory tree had drifted from the disk-saved graph with no path to reconcile.
- **First attempt (rejected).** Added `state.lastSavedNodes` and `state.graveyard` to `PadState`, refreshed via a `computeLastSavedNodes(doc)` helper after every write. The matcher ran at save time using those cached entries as `previousNodes`. Worked, but introduced a parallel state store mirroring disk — every cache invalidation path (`updateCacheEntry`, `setActiveDocument`, `clearAllCaches`, `CachedDoc`) had to know about it.
- **User pushback.** Travis: *"save really doesn't make sense to me, isn't markdown just always LIVE, all changes present? Why do we have 'States' for this?"* Then: *"an oversight this obvious seems like we need to really understand what we're trying to do."* The frame: markdown is the source of truth; memory is an ephemeral working copy. A parallel identity cache violates that frame.
- **Option B chosen.** Drop `state.lastSavedNodes` and `state.graveyard`. At save time, `readPersistedIdentity(state.filePath)` opens the existing file, runs `matter()` over the frontmatter, normalizes `data.nodes` and `data.graveyard` into `NodeEntry[]`, and hands them to the matcher. One extra `readFileSync` per save — negligible cost for documents of any realistic size, and disk stays the single source of identity truth.
- **Why not Option A (save-time matcher + broadcast).** Would also have required a change-listener path to push corrected IDs to the browser. Memory becomes the truth, disk follows, and broadcast races become possible.
- **Why not Option C (load-time only).** Type-change and graveyard-restore stay broken in-session. Operations would only re-pin after a doc switch / reload — invisible, surprising failure mode.
- **Change made.** `server/state.ts:writeToDisk` reads identity from disk before serializing; `server/state.ts` drops the `lastSavedNodes` / `graveyard` fields from `PadState` and `CachedDoc`; `server/documents.ts` drops the trailing two args from all 7 `setActiveDocument` call sites. Verified with `scripts/test-save-time-matcher.mjs` (22/22 pass) plus the existing `scripts/test-sync-check.mjs` (9/9), `scripts/test-nodeid-roundtrip.mjs` (22/22), `scripts/test-backlinks.mjs` (20/20), `scripts/test-link-href.mjs` (26/26). Commit hash: 2d5ec07.
- **Invariant for future edits.** Do not add an in-memory identity cache to `PadState` or `CachedDoc` to avoid the disk read. The disk read is the design, not a performance bug. If save-time identity becomes a hot path, the right fix is to cache the parsed frontmatter, not to introduce a parallel state store.

### 2026-05-16 — End-to-end integration test landed (commit 6ab1662)

- `scripts/test-state-integration.mjs` drives the real `state.ts` production path (setActiveDocument → save → writeToDisk) against an isolated test profile rather than mirroring writeToDisk's logic. 52 assertions across 12 scenarios: first save, text edit, insert, delete, paste-back, type-change, first-save-edge-case, external-doc bypass, reload roundtrip, multi-op burst, sync observer, slot-continuity.
- Combined with the existing matcher unit tests, total project test count is 151 assertions across 6 scripts.
- Why a separate file from `test-save-time-matcher.mjs`: that one mirrors writeToDisk's logic; this one calls the real exported function. Both are kept — the mirror is faster to debug, the real one catches plumbing bugs the mirror would miss.

### 2026-05-16 — Live MCP flood test (77 operations, 0 failures)

- Drove a rich 16-block doc (headings, paragraphs, lists, code, blockquote, marks) through 77 live MCP operations across 11 phases: 8 edits / 7 deletes / 8 inserts / 8 type-changes / 7 graveyard cycles / 17 multi-op burst changes / 5 slot-continuity replacements / 6 reorders / switch_document roundtrip / 5 edge cases (marks, special chars, 362-char paragraph, wikilinks, hash characters) / external file modification + reload_from_disk.
- After each phase, frontmatter was read directly from disk and asserted against expected ID state. Final integrity audit confirmed: all 29 active IDs unique, all 15 graveyard IDs unique, no overlap between active and graveyard, every fingerprint complete, graveyard within cap (15 ≤ 50), body markdown clean (no embedded `^id` anchors), frontmatter keys clean.
- 11/12 baseline IDs survived all 77 operations (1 intentionally deleted in Phase F).
- **New invariants surfaced and added above:** container fingerprints are content-empty (graveyard-restore by type matches blindly across containers); slot-continuity wins over operation-intent when `rewrite` and `insert` collide in the same batch; TipTap inserts a trailing empty paragraph on every load.
- Cumulative test surface across this ADR's three logged actions: 151 unit/integration assertions + 77 live MCP operations + 7 final integrity checks = **235 verifications, 0 failures**.

### 2026-05-16 — Phase K/L/M: backlinks, pending, versions integration tests

- **Phase K — backlinks (`scripts/test-backlinks-integration.mjs`, 23 assertions).** Two cross-linked docs (A target + B source linking to A). Verifies the backlinks pipeline rides on the matcher's ID stability: editing the target preserves `to_node`; type-changing the target preserves `to_node`; editing the source preserves `from_node`; deleting the source prunes the backlink entry from A; paste-back of the source restores the backlink via graveyard; delete-then-paste-back of the target restores `to_node` via graveyard. All 23 assertions pass. Surfaced gotcha: `parseDocHref` requires 8-char lowercase hex IDs; non-hex chars in IDs are silently dropped from backlink entries (test originally used `at000001` with `'t'` — fixed to `a7000001`).
- **Phase L — pending changes (`scripts/test-pending-integration.mjs`, 20 assertions).** Verifies that the `pending:` frontmatter map (keyed by ordinal position with text fingerprint backup) survives every matcher operation: edits to unrelated blocks don't disturb pending state; edits to a pending block keep its pending status while updating the text fingerprint; inserts shift pending positions correctly; deletes remove the corresponding pending entry; type-changes preserve pending state on the converted block. Confirms pending state is matcher-independent because it lives on the block's `attrs.pending*` fields, which the matcher doesn't touch.
- **Phase M — versions (`scripts/test-versions-integration.mjs`, 22 assertions).** Verifies snapshot creation after writeToDisk, snapshot content captures the matcher's nodes graph, listVersions/restoreVersion round-trip correctly. Surfaced the new invariant added above: `restoreVersion` reverts content but the save-time matcher applies its own rules against the CURRENT disk's previousNodes, not the snapshot's. The test was rewritten to assert content-level restoration (not ID-level), with notes for callers that need backlink fidelity across restores.
- **Cumulative test surface after Phase K/L/M:** 216 unit/integration assertions across 9 test scripts + 77 live MCP operations + 7 final integrity checks = **300 verifications, 0 failures**.

### 2026-05-16 — Phase N: agent marks integration test

- **Phase N — agent marks (`scripts/test-marks-integration.mjs`, 23 assertions).** Marks live in sidecar files at `DATA_DIR/_marks/{filename}.json`, not frontmatter. Test covers: editing a marked block (nodeId preserved → mark anchored), type-changing a marked block (nodeId preserved → mark anchored), deleting a marked block (nodeId becomes stale in sidecar but the mark's *other* nodeIds in multi-node marks stay valid), paste-back of a deleted marked block (matcher graveyard-restores the ID and the stale sidecar reference automatically revives), inserting above a marked block (nodeId stable), resolveMarks (sidecar entry removed).
- **Surfaced and documented above:** `marks.ts` exports `pruneStaleMarks` but it is intentionally not wired into the save-time matcher path. Auto-pruning would break the paste-back recovery: when a deleted block returns via graveyard-restore, any stale mark references to its ID become valid again automatically. The rendering layer's Pass-3 text fallback handles the user-visible anchoring while the sidecar stays stale.
- **Cumulative test surface after Phase N:** 239 unit/integration assertions across 10 test scripts + 77 live MCP operations + 7 final integrity checks = **323 verifications, 0 failures**.

### 2026-05-16 — v0.14.1 deeper root cause: `applyIdsToTiptap` descended into tables (position counter divergence)

- **Trigger.** Live verification of the `findNode` fix on the real Beat Sheet doc surfaced the actual underlying corruption mechanism. Restored Beat Sheet to the pre-corruption snapshot (`1778983115250.md`), ran a 3-op `write_to_pad` batch (table rewrite + paragraph rewrite + h2 rewrite). The disk markdown looked clean. But the next `read_pad` showed `[tr:3141ee2a]` — the target-total paragraph's stable ID — INSIDE the table as the first row's id. The top-level paragraph had been given a fresh `f831c9a4` id.
- **Honest second-pass walk.** The `findNode`-opaque-on-tables fix from the prior commit closes the corruption channel (a stolen ID inside the table can't be reached from outside anymore), but it doesn't prevent the theft in the first place. The theft itself happens at save time, in `applyIdsToTiptap`. `tiptapToBlocks` skips tables (one Block per table, no descent). `applyIdsToTiptap`'s `CONTAINER_TYPES` set included `table`, `tableRow`, `tableCell`, `tableHeader` — so it DID descend, incrementing position for every table-internal node. The matcher returns pins keyed by `tiptapToBlocks`'s flat positions; `applyIdsToTiptap` looked them up at DESCENDED positions, which don't match. The pin for a top-level paragraph landed on a table row instead.
- **Fix.** `applyIdsToTiptap`'s descent set now mirrors `walkNodes`/`tiptapToBlocks` exactly: descends only into `bulletList`, `orderedList`, `taskList`, `listItem`, `taskItem`, `blockquote`. Tables, code blocks, horizontal rules, images stay opaque. Position counters align.
- **Verification.** `scripts/test-applyids-position-alignment.mjs` (16 assertions, 3 cases). Synthetic doc with `[h1, p, p, h2, table, p, h2]` (same shape as Beat Sheet). Pin id `targetID` to flat-position 5, run `applyIdsToTiptap`, assert position 5 is the top-level paragraph and got `targetID`, and no table row stole it. Full matcher round-trip case asserts the same end-to-end. Uniqueness case asserts no id ever appears twice in the doc post-matcher. Plus live verification: restored Beat Sheet, ran the brief's exact corruption batch (then escalated to an 8-op batch with 6 chapter rewrites), confirmed table intact on disk + accurate `appliedCount` + zero ID drift. Full suite: 16 test scripts, 0 failures.
- **Invariant added (above).** `applyIdsToTiptap` and `tiptapToBlocks` are siblings. Their descent rules must stay identical, forever.

### 2026-05-16 — v0.14.1 root-cause fix: `findNode` opaque on tables (multi-op write_to_pad corruption)

- **Trigger.** TM-Book brief `2026-05-17-write-to-pad-multi-op-batches-unreliable.md` documenting Beat Sheet table corruption from a multi-op `write_to_pad` session: a rewrite reported `appliedCount: 3` success but the intended target paragraph (`9e5768ca`) stayed untouched, and an empty single-cell row appeared inside the chapter-mapping table. Diff between `1778983115250.md` (clean) and `1778983532861.md` (corrupted) on the Beat Sheet doc snapshot directory confirmed the corruption: header row + separator gone, body rows replaced with `|  |` single-cell rows.
- **Honest root cause walk.** Initial response framed the issue as a follow-on to the matcher hotfix (claimed v0.14.1 already addressed it). That was wrong. The matcher only runs at save time; the brief described failures within a single `write_to_pad` call, no save in between. Travis pushed back on the certainty; on re-examination, `findNode` (`server/state.ts:668`) was the actual root cause. It does a depth-first recursive descent and returns the first ID match anywhere in the tree. Table-internal node IDs are regenerated on every `markdownToTiptap` parse via `parseTableTokens`, never tracked by the matcher, never exposed by `compactNodes` — so they're ephemeral and unaddressable. Any agent ID that happened to match a table-internal node (random 8-hex collision across many saves, or a stale ID after a prior re-parse) silently routed the splice into the wrong slot. `processed.push` reported success because the splice DID happen — just not where the agent intended.
- **Fix.** `OPAQUE_CONTAINER_TYPES = new Set(['table', 'tableRow', 'tableCell', 'tableHeader'])`. `findNode` continues past any node whose type is in that set instead of descending. Mirrors the matcher's existing opaque-table behavior in `node-blocks.ts`. Three lines of real change.
- **Verification.** `scripts/test-findnode-opaque-tables.mjs` (20 assertions, 3 cases). Test deliberately constructs a doc where a top-level paragraph and a table-cell paragraph share an ID, then asserts: (1) rewrite-by-colliding-ID hits the top-level paragraph and leaves the table intact; (2) delete-by-colliding-ID removes the top-level paragraph and leaves the table intact; (3) rewrite of a table-only ID is a no-op (the agent can't reach into table internals at all). Full suite: 15 test scripts, 0 failures.
- **Invariant added (above).** Every layer of the system now agrees that table internals are not addressable from outside: the matcher walker doesn't track them, `compactNodes` doesn't expose them, `parseTableTokens` regenerates them on every parse, and `findNode` no longer descends into them.

### 2026-05-16 — v0.14.1 architectural fix: matcher broadcasts ID rewrites to keep browser+server in sync

- **Trigger.** Post-hotfix review: the hotfix narrowed when the matcher reassigns block IDs but couldn't eliminate all reassignment paths (transient editor-minted IDs at slots with previousNodes orphans still slot-continuity-pair, by design per Test 12 in `test-state-integration.mjs`). The residual cases meant the same browser↔server ID divergence pattern was latent — just less frequent. Travis pushed: *"I want the full architectural fix that is elegant and works with our system. Why the hesitation? Why only shipping workarounds?"*
- **Decision.** Make the matcher's identity authority propagate to all connected clients via a new WebSocket message type. Whenever the save-time matcher reassigns a block ID, every browser must update its in-memory TipTap doc to match. This closes the divergence loop at the protocol layer, not just at the matcher heuristic layer.
- **Implementation.** Three layers, ~80 lines.
  - `server/state.ts`: new `IdRewriteListener` type, `onIdRewrites(listener)` subscribe API mirroring `onChanges`. `writeToDisk` captures `beforeIds` (the `attrs.id` values present in `tiptapToBlocks(state.document)` output before the matcher runs), diffs them against the matcher's `pinnedByPosition`, and fires `{oldId, newId}[]` to all listeners when any position changed. Empty-`oldId` positions (genuinely new blocks the editor never named) don't count — only real ID rewrites fire.
  - `server/ws.ts`: subscribes to `onIdRewrites` and broadcasts `{type: 'id-rewrites', rewrites}` to every connected WebSocket client.
  - `src/decorations/bridge.ts`: new `applyIdRewritesToEditor(editor, rewrites)` export. Walks the editor's TipTap doc with `descendants`, collects positions where `node.attrs.id` is in the rewrite map, calls `setNodeMarkup` for each in a single atomic ProseMirror transaction. Other attrs (`pendingStatus`, etc.) are preserved via `{ ...node.attrs, id: newId }`.
  - `src/ws/client.ts`: `onIdRewrites` callback option, message handler that invokes it on `id-rewrites` arrival.
  - `src/App.tsx`: wires `onIdRewrites` to apply rewrites to every editor in `allEditorsRef` (multi-editor / tweet-thread mode safe — `applyIdRewritesToEditor` only touches nodes whose `attrs.id` is in the rewrite map, so applying to all editors is a no-op for editors that don't hold the affected blocks).
- **Verification.** `scripts/test-id-rewrite-broadcast.mjs` exercises the full pipeline through 10 assertions across 6 cases: first-save fires no rewrite (no previousNodes); in-place edit with preserved id fires no rewrite; transient-id replacement triggers slot-continuity rewrite with correct `{oldId, newId}` mapping that matches the frontmatter; agent-stamped end-of-doc insert fires no rewrite (the hotfix matcher guard handles this); idempotent re-save fires nothing; closed-loop simulation that applies the captured rewrites to a simulated browser-state copy and asserts the server's in-memory IDs match the rewritten browser IDs (convergence proof). Full suite: 14 test scripts, 0 failures. New invariant added above.
- **Why now, not v0.14.2.** The hotfix and the architectural fix are conceptually one piece of work — narrowing matcher rewrites and closing the loop when they do happen. Shipping them in a single v0.14.1 release tells the upgrade story cleanly: 0.14.0 had silent data loss, 0.14.1 fixes it both at the matcher and at the protocol.

### 2026-05-16 — v0.14.1 hotfix: matcher must preserve existing block IDs to avoid browser↔server divergence

- **Trigger.** TM-Book brief (`/.claude/inbox/2026-05-16-mcp-writes-clobbered-by-stale-browser-autosave.md`) reported silent data loss when MCP `write_to_pad` writes hit a doc that was open in the browser. Symptom: MCP returned success and version history confirmed the write, then ~45 s – 2 min later the document reverted. Repro: v5 (2,766 words) → v4 (1,797-word revert) → v3 (2,986 words) → v2 (1,834-word revert) → tab closed → v1 (3,955 words, persists).
- **Root cause.** The save-time matcher's `applyInsertRule` (`server/node-matcher.ts:611`) unconditionally called `generateNodeId()` for every unmatched block — the new block at `afterNodeId: "end"` always got a fresh ID, overwriting the ID that `applyChangesToDocument` had stamped onto the in-memory tree and broadcast to the browser milliseconds earlier. The server's `state.document` and the on-disk frontmatter ended up with one ID; the browser's TipTap doc still had the broadcast ID. Subsequent `node-changes` targeting the rewritten ID hit `findNodeByIdInDoc(tr.doc, change.afterNodeId)` → null → silent `continue` in `src/decorations/bridge.ts:111`. The browser's debounced autosave then shipped its stale TipTap state via `doc-update`; the server accepted it (agent lock expired, version matched, destructive-size threshold not tripped); the matcher graveyarded the missing content. Every MCP observable returned success — the agent had zero signal that the write would be undone.
- **Fix (primary).** Carry `attrs.id` through `tiptapToBlocks` into `Block.id`. `applyInsertRule` now uses `candidate.block.id || generateNodeId()` and claims the preserved ID from `claimedPrevIds` / `claimedGraveIds` so it doesn't end up in both `nodes:` and `graveyard:` of the output frontmatter. `applySlotContinuityRule` skips candidates whose ID is already in `previousNodes` or `graveyard` — without that skip, a snapshot-restored block carrying its old ID would be absorbed by slot-continuity into whatever ID happens to sit at the same slot, and `applyGraveyardRestoreRule` would never get its chance.
- **Fix (defense-in-depth).** `src/ws/client.ts:124` now bumps `docVersionRef.current = msg.version` AFTER `onNodeChangesRef.current?.(msg.changes)` returns, so a thrown apply leaves the version at the previous value (next browser autosave is rejected as stale instead of accepted as authoritative). `src/decorations/bridge.ts` now logs `console.warn` at every `findNodeByIdInDoc(...) || continue` silent-skip site so future divergence surfaces in the browser console instead of vanishing.
- **Files changed.** `server/node-fingerprint.ts` (added `id?: string` to `Block`), `server/node-blocks.ts` (capture `node.attrs?.id` at every push site), `server/node-matcher.ts` (`applyInsertRule` preserves + claims; `applySlotContinuityRule` skips known IDs), `src/ws/client.ts` (version bump order), `src/decorations/bridge.ts` (warn-on-skip).
- **Verification.** New regression test `scripts/test-matcher-preserves-ids.mjs` — 16 assertions across 7 cases including: insert with pre-stamped ID survives matcher; no ID appears in both active and graveyard; explicit ID known to graveyard wins over slot-continuity; full markdown round-trip stable. Full suite: 13 test scripts, all 0 failures. Released as v0.14.1.
- **Invariant added (above).** The matcher must respect `block.id` from upstream (agent / load-time matcher / snapshot restore) when the ID is meaningful (already in `previousNodes` or `graveyard`). Transient IDs (not in either set) still flow through slot-continuity per the "slot is innocent until proven changed" principle — Test 12 in `scripts/test-state-integration.mjs` documents this case.

### 2026-05-18 — v0.15 compact fingerprint format (sentence content hash replaces word arrays)

- **Trigger.** TM-Book brief `2026-05-18-frontmatter-fingerprint-bloat-15x-overhead.md`: Architecture.md (4,218-word body) on disk was 140 KB total, 116 KB of that frontmatter (4.79× body in bytes, ~11–15× in tokens because JSON is token-expensive). Frontmatter exceeded the Read tool's 25k-token limit by itself, breaking the agent-orientation contract that frontmatter must be loadable as a routing layer without descending into the body.
- **Honest root cause.** During v0.14 build, the matcher hit fingerprint collisions on the test corpus and the implementation cascaded defenses: added 3-char prefix/suffix per sentence (`f`/`l`), then a word-length sequence (`wls`), then the full word array (`w`), then block-level `firstWords[5]`/`lastWords[5]`. None of these layers turned out to be load-bearing against the test corpus — the matcher's rules consume `same-or-not-same` and `prefix/suffix-of-list`, never `how-similar`. The cascade was paranoia tax that never paid out, but the storage cost shipped with every save. Original design called for ~150-250 bytes per block; v0.14 emitted 3-5 KB per block.
- **Architectural framing.** The matcher's on-disk signal should be the MINIMUM needed for its rules, not the maximum it can afford. A single 8-byte content hash per sentence uniquely identifies that sentence's text; it subsumes the role of word arrays, word-length arrays, and prefix/suffix windows simultaneously. The matcher's correctness (exact-match, split, merge, edit, type-change, paste-back, slot-continuity) is preserved because all rules either compare full equality or compare prefix/suffix of a sentence list — both work on hashes.
- **Change.**
  - `SentenceTuple` shrinks from `{c, f, l, t, wls, w}` (6 fields) to `{c, h, t}` (3 fields) where `h = simpleHash(sentence.text)`.
  - `Fingerprint` drops `firstWords[]` and `lastWords[]` (block-level word arrays never consumed by any matcher rule — verified by code audit).
  - `sentenceTuplesEqual` compares c, h, t. `isExactMatch` and `isSameContent` unchanged in structure (they call `sentenceArraysEqual` which calls `sentenceTuplesEqual`).
  - `sentenceSignalOverlapScore` (slot-continuity disambiguator) becomes: +1 same terminator, +2 same char count, +10 same hash. Hash dominates; c/t break ties for partial-match scenarios.
- **Migration.**
  - `isLegacyFingerprint(fp)` detects v0.14 entries by presence of `w`/`wls`/`f`/`l` fields in any sentence tuple.
  - `migrateLegacyEntries(prev, freshBlocks)` re-fingerprints positionally from freshly-parsed body. Called from `markdown-parse.ts:applyMatcher` (load-time) and `state.ts:writeToDisk` (save-time) before invoking `matchNodes`.
  - `dropLegacyGraveyard(grave)` filters out legacy entries. Deleted blocks have no body to recompute from; best-effort hashing from `w[]` would produce hashes that don't match fresh paste-back content. Paste-back recovery for pre-migration deletes is the one-save migration tax. Acceptable.
  - Migration is opportunistic on every load/save — no bulk script. Each doc converts on first interaction post-upgrade.
- **Verification.**
  - **Unit + integration:** 33 test scripts, 675+ assertions, 0 failures. New `test-fingerprint-migration.mjs` (27 assertions) explicitly covers legacy detection, positional re-fingerprinting, graveyard drop, and end-to-end legacy → v0.15 file round-trip with ID preservation.
  - **Matcher stress test (242 ops):** unchanged — 34 assertions still pass under the new format. Original heading still pinned through 242 operations.
  - **Live integration:** new doc creation emits pure v0.15 format (sentence tuples `{c, h, t}` only, no firstWords/lastWords); edit detection preserves IDs through paragraph edits; delete populates graveyard; paste-back triggers graveyard-restore with original ID returned; external file edit + `reload_from_disk` reconciles correctly with ID continuity.
  - **Production migration on Argument Arc (`c6d56b96`, 4,218 words):** 140,681 → 78,504 bytes (**44.2% reduction**). Frontmatter 116,388 → 54,211 bytes (**53% shrink**). Body bit-identical. All sampled IDs preserved through migration. File now within Read-tool 25k-token limit; agents can read it directly.
- **Invariant added.** The matcher's on-disk signal must be the minimum needed for its rules, not the maximum it can afford. Future disambiguators that bloat the per-sentence tuple must be justified against a concrete collision case in the test corpus, not against a hypothetical edge case. Add such cases to the corpus first; only ship the disambiguator if the corpus failed without it.
- **Files.** `server/node-fingerprint.ts` (format + migration helpers), `server/node-matcher.ts` (scoring), `server/state.ts` + `server/markdown-parse.ts` (migration wiring), `scripts/test-fingerprint-migration.mjs` (new), `scripts/test-versions-integration.mjs` (assertion fix — `firstWords` removed). Commit: `75d65c3`.

### 2026-05-18 — Ultra-lean tuple format: slim disk shape, rich in-memory shape

- **Trigger.** Follow-up to the v0.15 fingerprint shrink. Travis pushed: *"I need leaner. I need maximally lean. We shouldn't store things that are trivial on demand."* Then: *"are you missing innovated engineering methods we can use to make it even leaner?"* v0.15 halved frontmatter bytes but still emitted JSON object entries with explicit field names (`{id, fp:{type, position, charCount, sentences:[{c, h, t}], ...}}`). The keys themselves dominated the bytes — every `"type":"paragraph"` is 18 chars for one bit of signal.
- **Architectural framing.** Disk and memory should not share a shape. On disk: positional array tuples, omit anything trivially recomputable on load (position from array index, sentence text from body, structureSig from sentence terminators). In memory: keep the rich `Fingerprint` shape the matcher's rules already consume. A serialize/enrich boundary at the save/load edge swaps between the two.
- **Slim tuple shapes.** All entries are positional arrays:
  - paragraph (default, no type tag): `[id, sentences[], marks?]`
  - empty paragraph: `[id]`
  - heading: `[id, "h1".."h6", sentences[], marks?]`
  - codeBlock: `[id, "code", language, contentHash]`
  - atomic blocks (hr, img, tbl): `[id, "hr"|"img"|"tbl"]`
  - containers (ul, ol, tl, bq, li, ti): `[id, "ul"|..., childTypes[]]`
  - sentences: bare hash strings (terminator implied by trailing punctuation in body; charCount recomputed on enrich)
  - marks: compact object `{b?, i?, l?, c?}` only when present
- **Hash collisions.** Per-sentence `simpleHash` uses `>>> 0` (unsigned 32-bit, 1-8 hex chars). Includes terminator so `"foo."` and `"foo!"` hash differently. Sentences with identical text get identical hashes — matcher's exact-match works either way; slot-continuity ties broken by position.
- **Resolver functions.** `resolvePreviousNodes(rawNodes, freshBlocks)` and `resolveGraveyard(rawGraveyard)` in `markdown-parse.ts` are the load-time edge. They accept both slim tuples (current) and verbose objects (v0.14 / v0.15 legacy). For legacy, they positionally re-fingerprint from the freshly-parsed body. The matcher never sees slim tuples — it consumes rich `NodeEntry` only.
- **gray-matter cache footgun.** `gray-matter` caches its parsed `data` object by raw string content within a process. Test wrappers that mutated `data.nodes` / `data.graveyard` after parsing leaked mutations into subsequent calls — same raw string → same cached object → corrupted graveyard arrived at the matcher. Two-part fix: (1) `readPersistedIdentity` bypasses gray-matter and parses the JSON frontmatter directly (`/^---\r?\n([\s\S]*?)\r?\n---/` + `JSON.parse`) — honest about the format since we serialize via `JSON.stringify(meta)` between `---` delimiters anyway, and skips YAML overhead in the save hot path. (2) Test `readFrontmatter` helpers now project to a new object (`{ ...data, nodes: ..., graveyard: ... }`) instead of reassigning fields — re-establishes the invariant "gray-matter's parsed data is read-only."
- **Verification.**
  - **Unit + integration:** 31 test scripts, 700+ assertions, 0 failures across the full suite. New stress test (`test-matcher-stress.mjs`) survives 242 mixed ops on a 50-block doc with the original heading's ID preserved end-to-end.
  - **Byte savings across 141 real docs** (production profile, in-place migration via parse → re-serialize): old frontmatter total 2,636,324 bytes → new 335,750 bytes. **87.3% reduction.** Average per doc: 18,697 → 2,381 bytes.
  - **Live MCP migration on Ch 1 — Beats** (1,428-word doc, 60,739 → 11,717 bytes total, frontmatter 52,253 → 3,231 bytes — 93.8% shrink, body bit-identical). All 60+ IDs preserved through the legacy → slim migration via `resolvePreviousNodes` positional re-fingerprint.
- **Invariant for future edits.** The on-disk format is positional tuples, NOT JSON objects. Any new field must justify the per-doc cost across the corpus. If a piece of data can be recomputed from the body on load (position, sentence text, structural signature), it does not go on disk. The resolver functions are the only path that produces a `NodeEntry` for the matcher — never call the matcher on raw disk data directly.
- **Files.** `server/node-fingerprint.ts` (slim/rich shapes, `slimEntry`/`enrichEntry`), `server/node-matcher.ts` (consumes rich `NodeEntry`, no slim awareness), `server/markdown-serialize.ts` (slim emission), `server/markdown-parse.ts` (`resolvePreviousNodes`/`resolveGraveyard`), `server/state.ts` (`readPersistedIdentity` bypasses gray-matter cache), `scripts/measure-frontmatter-bytes.mjs` (corpus migration measurement). All `*-integration.mjs` test wrappers updated to project (not mutate). Commit: `847009b`.

### 2026-05-18 — External-write reload must refresh disk frontmatter

- **Trigger.** Live /openwriter-testing on the ultra-lean refactor surfaced a paste-back graveyard-restore failure for a paragraph that had been externally edited mid-session. Diagnosis: phantom sentence hash `125a02cd` was the fingerprint of `"BROWSERMIGRATIONMARKER" + "-"` — text that had been typed into the paragraph via the browser, autosaved into disk frontmatter, then removed externally via the `Edit` tool. After the external removal, `reloadActiveDocFromDisk` updated in-memory body to the clean state, but disk frontmatter still held the 4-sentence fingerprint from the previous browser-autosave. Cutting the paragraph in that stale window graveyarded it with the stale 4-sentence fingerprint; paste-back fingerprinted as 3 sentences (matching the current body); exact-match failed; graveyard-restore silently missed.
- **Architectural framing.** Disk frontmatter and disk body must stay coherent. The external-write reload pathway refreshed in-memory body but skipped refreshing the on-disk fingerprints — opening a window where graveyard-bound entries could capture obsolete content. The matcher's `previousNodes` is read from disk at every save; if disk lies, the graveyard inherits the lie. The lie has to be closed at the reload boundary.
- **Decision.** `reloadActiveDocFromDisk` triggers a `writeToDisk()` at the end. The matcher's edit rule pins IDs and emits fresh fingerprints for blocks still present in the body. Disk frontmatter is then guaranteed coherent with disk body. fs.watch self-suppression via `state.loadedMtime` (handleWatcherEvent: `if (diskMtime === state.loadedMtime) return`) prevents a reload→save→reload loop. External editors see a brief mtime tick after their write but body content is identical — VSCode handles gracefully, scripts don't poll.
- **Invariant added (above).** Disk fingerprints reflect the body as of the last server write OR external-write reload — never an earlier state. Reload-pathway changes must preserve this; do not add code paths that mutate in-memory body without subsequently re-emitting frontmatter.
- **Verification.** `scripts/test-reload-refreshes-frontmatter.mjs` — 11 assertions across 4 phases: external write extends paragraph → reload refreshes fm to new fingerprint; external write shrinks paragraph → reload refreshes again; in-memory delete graveyards with the post-reload fingerprint; paste-back with matching content triggers graveyard-restore (original ID returns, graveyard clears). Full suite: 32 test scripts, 0 failures.
- **Files.** `server/state.ts:reloadActiveDocFromDisk` (~10 lines added: `try { writeToDisk(); } catch { ... }` after `setOverlayFromEntries`). New test `scripts/test-reload-refreshes-frontmatter.mjs`. Commit: `53cfe27`.

### 2026-05-16 — Phase O: scale stress test + final orthogonal-system audit

- **Scheduler / Workspaces / Tasks subsystems audited.** All three are doc-level only (filenames or platform-assigned IDs); none reference block-level nodeIds. Confirmed via codebase audit: scheduler proxies to platform API keyed by `doc_id` (post-sync.ts:57), workspaces store `{type, file}` items and container UUIDs in `_workspaces/*.json` (workspace-types.ts:12-29), tasks are profile-global free-form text in `tasks.json` with no doc/block coupling. **Matcher risk: ZERO for all three.**
- **Stress test (`scripts/test-matcher-stress.mjs`, 34 assertions over 242 ops).** Built a 50-block doc, ran 100 sequential edits + 50 inserts + 50 deletes + 50 type-changes, verifying integrity invariants after each batch (uniqueness, no overlap, complete fingerprints, graveyard ≤ 50). Final state: 50 active nodes, 50 in graveyard (cap reached and oldest entries pruning correctly). Performance: ~8.5 ms per save (242 ops in 2.05 seconds). The original heading `sh000001` survived all 242 operations via fingerprint match — proving the matcher reliably preserves untouched blocks through arbitrary churn.
- **Cumulative test surface — final:** 273 unit/integration assertions across 11 test scripts + 77 live MCP operations + 7 final integrity checks = **357 verifications, 0 failures**. Matcher system is fully tested across all dependent subsystems and at scale.
