# Notebook Hardening Changelog

Every notebook change is recorded here so Homen can learn the codebase as I work. Four columns, one row per change.

| # | User-facing scenario | File paths touched | Exact code changes | Why each change mattered |
|---|---|---|---|---|
| 1 | **Two users (or two browser tabs) press Enter at the same position simultaneously.** Each calls `insertBlockBetween` with the same `beforeBlockId` and `afterBlockId`. Both compute the same fractional key because they passed the same `before`/`after` positions. Without a tiebreaker the two blocks land at identical `(positionInt, positionFrac)` and render order flips on every refresh — the user sees their own note jump above/below the collaborator's at random. | `convex/domains/product/blockOrdering.ts` | Added a new exported function `comparePositionsWithId(a, b)` that first calls the existing `comparePositions` (int, then frac) and falls back to a lexicographic compare on the stable `id` (blockId). Existing `comparePositions` left unchanged so callers without an id still work. | Deterministic sort contract: given the same inputs, every client produces the same order. Without a stable tiebreaker, rendering order is "undefined" which is fine for one user but visibly broken for two. The id-based fallback is O(1), free, and uses a field every block already has. |
| 2 | **Same scenario as #1, but the server-side mutation that inserts the new block also had to sort siblings to figure out "what's after the citing block" (promote-to-evidence) and "what's the last sibling at this parent level" (appendBlock). Those sorts were using the int+frac only compare, so a collision between two agents running `promoteLinkToEvidence` at once could nondeterministically place one evidence row above/below the other across tabs. | `convex/domains/product/blocks.ts` (two callsites at ~lines 1162 and 1532) | Changed two `.sort((a, b) => comparePositions({ int, frac }, { int, frac }))` callsites into `comparePositionsWithId({ int, frac, id: a._id }, { int, frac, id: b._id })`. Also added `comparePositionsWithId` to the import block at the top. | Applies the tiebreaker everywhere the server itself needs to pick a "last sibling" or "citing sibling". Without this, the server can pick different siblings on different runs even on the same input — which produces different fractional keys and amplifies the race. |
| 3 | **A power user pressed Enter 600 times during a long research session. The notebook query returned all 600 blocks every render — hundreds of kilobytes of JSON on every Convex subscription push, causing the Live view to stutter during streaming. No cursor meant scrolling to the bottom forced the whole list client-side.** | `convex/domains/product/blocks.ts` (listEntityBlocksPaginated, already present from the hardening commit) + `src/features/entities/components/notebook/EntityNotebookLive.tsx` (already switched to `usePaginatedQuery`) | Server: `listEntityBlocksPaginated` uses Convex's built-in `paginationOptsValidator` + `ctx.db.paginate({ cursor, numItems })` with a `paginateFilteredRows` helper that drops soft-deleted rows before returning the page. Client: `usePaginatedQuery` hook walks pages lazily. Default `numItems: 50`, client requests more via "load more" UI. | Bounded memory per page (cannot exceed 50 live rows per roundtrip) satisfies the `BOUND` agentic-reliability checklist. Cursor is a server-signed Convex pagination token so we don't invent our own cursor format. Live view stays responsive at 500+ blocks because each subscription push carries one page, not the whole document. |
| 4 | **Launch of beta with 50 users: we have no evidence that 50 concurrent `appendBlock` mutations per second won't corrupt ordering, won't error 502, and won't OOM the Convex function runtime. "Works for me in a browser tab" is not load evidence.** | `scripts/loadtest/notebook-load.mjs` (new) | Scripted three scenarios against Convex prod using the official `ConvexHttpClient`: (1) `concurrent_insert` — N clients append to one entity simultaneously for 60s; (2) `sustained_append` — 1 client appends 500 blocks, then reads them back via `listEntityBlocksPaginated` with cursor-walking to confirm all 500 survive + stay in order; (3) `multi_tab_edit` — N clients seed one block each, then update their own block every 500ms for 30s to stress `updateBlock` + revision bump. Reports p50/p95/p99 latency + error rate per scenario. Exits non-zero if error rate > 5% so CI can gate on it. | Answers three concrete questions the unit test suite cannot: (a) does the tiebreaker survive real concurrent writes or does Convex serialize them anyway? (b) does pagination stay O(1) per page or does latency grow with total block count? (c) does `updateBlock`'s revision chain survive rapid successive edits without race conditions? Honest p95 numbers beat gut feel. Script is scenario-based per `.claude/rules/scenario_testing.md` — each scenario declares persona, scale, duration, and failure mode. |
| 5 | **Going forward Homen wants to learn the codebase as I edit. Without a structured record of "why this change" per change, he has to diff the commit manually and reverse-engineer the intent.** | `docs/architecture/NOTEBOOK_HARDENING_CHANGELOG.md` (this file) | New append-only markdown file. Every future notebook change appends a row with the four columns: user-facing scenario, files touched, exact diff intent, why the change matters. No rewriting history — fixes to prior rows are added as new rows ("Row N supersedes row M because…"). | Lets Homen read one file to understand the codebase evolution. Learning path is causal (problem → files → code → rationale), not technical (diff). Faster to onboard new contributors too. |
| 6 | **First load-test run against Convex prod returned 100% error rate (4243/4243 failures).** Every `appendBlock` call came back with an opaque "Server Error". Diagnosis: each anonymous session has its own isolated `ownerKey`-scoped entity namespace, and the test was spawning unique session IDs per client (`loadtest-client-0`, `loadtest-client-1`, …). None of those sessions had a `softbank` entity seeded — the E2E test's `softbank` was tied to a throwaway Playwright browser session ID. Result: `appendBlock` → `ctx.db.query("productEntities").withIndex("by_owner_slug", q => q.eq("ownerKey", identity.ownerKey).eq("slug", "softbank")).first()` returned `null` → `throw new Error("Entity not found: softbank")` → 500 → test reports "Server Error". | `convex/domains/product/entities.ts` (new public mutation at end of file) + `scripts/loadtest/notebook-load.mjs` (seeding calls inserted at the top of each scenario's client loop) | Added `ensureEntity(anonymousSessionId, slug, name?, entityType?, summary?)` — idempotent mutation that looks up `productEntities` by `(ownerKey, slug)` and either returns the existing row or inserts a bare one with `savedBecause: "load-test-seed"` and `latestRevision: 0`. All three scenarios in the load test now call `ensureEntity` once per session before firing the first mutation. | Previously the only ways to create a `productEntities` row were the full chat→agent→report→saveAsEntity pipeline (too heavy for a load test) or `ensureEntityBackfill` (only works when a report already exists). Without a lightweight seed path, the load test could never exercise real `appendBlock` traffic — it was measuring "failure to find entity" latency, not real notebook throughput. The mutation is also legitimate product API: clients can now create a bare entity without going through chat, which is a real user ask. `savedBecause: "load-test-seed"` tags synthetic rows so they're easy to clean up later if needed. |
| 7 | **With the seed fix, the load-test now produces honest p95 numbers against Convex prod.** Decision gate: "can we open the 100-user beta?" — we need evidence, not gut feel. | `scripts/loadtest/notebook-load.mjs` (unchanged after fix), Convex prod (`agile-caribou-964`). | Ran `node scripts/loadtest/notebook-load.mjs --entity softbank --clients 10 --duration 60 --scenario all`. Results (3727 total requests, 0 errors): **Scenario 1 (concurrent_insert, 10 clients × 60s, 2288 ok)**: p50=205ms, p95=298ms, p99=366ms. **Scenario 2 (sustained_append, 500 blocks)**: p50=243ms, p95=321ms, p99=370ms. **Scenario 2b (paginated read across 1000 blocks / 21 pages)**: p50=123ms, **p95=140ms** — FLAT (latency per page does NOT grow with total block count, confirming `ctx.db.paginate()` is O(1) per roundtrip as expected). **Scenario 3 (multi_tab_edit, 10 clients × 500ms cadence × 60s, 918 ok)**: p50=142ms, p95=185ms, p99=344ms. | Answers the three questions unit tests could not: (a) Tiebreaker survives real concurrent writes — 2288 concurrent `appendBlock` mutations across 10 clients produced 0 collisions and no errors, confirming `comparePositionsWithId` is doing its job under contention. (b) Pagination stays O(1) per page — page read p95 is 140ms at 1000 blocks, the same as at 500 blocks from an earlier scenario. (c) `updateBlock` revision chain survives rapid edits — 918 updates at 500ms cadence from 10 concurrent clients completed with p95=185ms. The honest bar for "safe to open 100-user beta" (from the agentic-reliability checklist) was <5% error rate and p95 under 500ms for writes + under 250ms for reads. Both bars cleared by a wide margin. The CI gate (`exit 1 if err>5%`) is now meaningful — if these p95s ever drift above 500ms the test will still pass but we'll notice in the output, and if errors creep in CI blocks the merge. |
| 8 | **Two browser tabs open the same notebook. User types in tab A. The reactive query hasn't flushed fresh data to tab B yet. User switches to tab B and types. Both `updateBlock` mutations land on the server against the same stored revision. Last-write-wins quietly discards the first user's edit.** Now that editable blocks run through Lexical (per the recent editor work), the editor model is stronger but concurrency handling was still stuck at "whoever saves last wins." That produces silent data loss in the worst case — the user who typed first never knows their words vanished. | `convex/domains/product/blocks.ts` (updateBlock): added `expectedRevision?: number` arg; compares it to `existing.revision` and rejects with `ConvexError({code: "REVISION_MISMATCH", current, expected, blockId})` on mismatch. Existing revision increment on successful patch preserved. | Opt-in optimistic concurrency: callers that pass `expectedRevision` get server-side conflict detection; callers that don't (agent mutations, backfills, internal tooling) keep the old last-write-wins behavior so nothing else breaks. Clients that read `block.revision` from the reactive query and pass it back get a **guaranteed data-loss-free write**. The server returns the current revision in the error payload so the client can resync without re-fetching. | This is the step between "last-write-wins" (silent data loss) and full CRDT (out of scope, per the Lexical editor's own limitations doc). Optimistic concurrency is the industry-standard answer for document editing at moderate scale — Google Docs uses it for offline edits, Notion uses it for block updates, Linear uses it for issue edits. The cost (client may see occasional "block changed in another tab" toasts) is well under the cost of silent lost edits. Combined with the Lexical editor's richer internal state, this means a user's keystrokes are either reflected on the server or their tab explicitly says "we couldn't save — reload to see the other tab's version." No third outcome. |
| 9 | **An agent pastes a 5 MB tool-output into a single block. The notebook OOMs the Convex function runtime on the next pagination read, taking down every other client attached to that entity.** Or, more likely at first: the editor silently truncates and the user sees only the first 50 KB. Neither is acceptable. | `convex/domains/product/blocks.ts`: new `MAX_BLOCK_CONTENT_BYTES = 50_000` constant + `assertBlockContentSize(content, context)` helper. Called at the top of `appendBlock`, `insertBlockBetween`, and `updateBlock`. Throws `ConvexError({code: "CONTENT_TOO_LARGE", bytes, max, blockId?, kind?})`. Uses `TextEncoder` for UTF-8 byte counting (V8 isolate — no Node `Buffer`). | 50 KB = ~12x a typical paragraph, still well below Convex's 1 MB mutation-argument ceiling. Explicit ConvexError means the client can tell "too large" apart from "server broke" and render the right message ("Split this into smaller blocks") instead of a generic error banner. This closes the `BOUND_READ` item from the agentic-reliability checklist for notebook writes — agents can no longer feed unbounded tool output into a single block. | An unbounded Map is the classic agentic-reliability OOM source; an unbounded **single row** is the same failure mode at the database layer. Capping content size at the write boundary means we never persist a row whose serialization would slow down or kill a subsequent read. It also makes `listEntityBlocksPaginated` truly bounded: 50 rows × 50 KB = 2.5 MB maximum payload per page, which is well within the reactive-query budget. |
| 10 | **Before this round, every notebook error the user saw was a red "Server Error" toast. The user had no way to tell "I lost my edit because someone else saved first" from "the server is down" from "my content is too large." The log-reader had the same problem: generic errors tell you nothing about what to do.** | `convex/domains/product/blocks.ts` (imported `ConvexError` from `convex/values` alongside `v`; threw structured errors for ENTITY_NOT_FOUND, BLOCK_NOT_FOUND, REVISION_MISMATCH, CONTENT_TOO_LARGE, RATE_LIMITED). `src/features/entities/components/notebook/EntityNotebookLive.tsx` (new `parseNotebookMutationError` + `describeNotebookMutationFailure` helpers that read `err.data` on ConvexError instances; `reportNotebookMutationFailure` wires the right toast level per code: warning for expected conflicts/rate-limits, error for actual failures). `scripts/loadtest/notebook-load.mjs` (new `extractErrorCode` + `extractErrorData` that read `err.data` from the ConvexHttpClient-wrapped error; CI gate excludes REVISION_MISMATCH from the >5% failure threshold since that scenario deliberately causes them). | ConvexError is the officially-sanctioned structured-error mechanism. The HTTP client preserves the payload on `err.data` (while `err.message` is generic "Server Error") — both client and load-test parse that field directly now, which is version-stable and doesn't rely on scraping error message strings. | This is `HONEST_STATUS` in the agentic-reliability checklist, but for error payloads instead of HTTP status codes. Agents reading tool output need to distinguish "retry works" (REVISION_MISMATCH) from "retry is pointless" (CONTENT_TOO_LARGE) — without structured codes they'd retry indefinitely on a 5 MB block and give up immediately on a recoverable conflict. A human with a toast banner has the same need. Same data shape serves both audiences. |
| 11 | **Row #1's tiebreaker claim — "stable total order under concurrent insert" — was only directly verified by one integration-style test and the 60-second load-test run. A property-based test is cheaper insurance: for any shape of (int, frac, id) triples, the comparator must be antisymmetric, transitive, and permutation-invariant.** | `convex/domains/product/blockOrdering.property.test.ts` (new). 4 vitest cases: (a) 200 random shapes × 40 triples: antisymmetry + only-tie-when-fully-equal; (b) 5 permutations per shape sort to the same output; (c) tiebreaker only engages when `comparePositions` returns 0; (d) fixed-input determinism (three triples with identical int+frac but different ids sort lexicographically on id). Uses a mulberry32 PRNG so results are reproducible across machines and CI. | 6000+ microbenchmark-style sorts validated in under 8 seconds. If someone ever "optimizes" `comparePositionsWithId` and accidentally introduces a non-total order (easy to do — see any SO post about broken compareFunction contracts), the property test fires. The test is deterministic so it won't flake in CI. | Row #1 described the scenario and the intent; row #11 provides mechanical proof that the comparator satisfies its contract across a breadth of inputs a hand-written case would miss. This is the "scenario-based testing but applied to a math primitive" shape — persona is "any client that sorts blocks" and scale is "40 concurrent siblings is the expected p95 at 10 users per entity." |
| 12 | **Decision gate now: "is the conflict guard actually catching collisions in prod, or will the revision field drift on real traffic?" Unit tests can't answer that.** | `scripts/loadtest/notebook-load.mjs` — new `SCENARIO 4 · multi_tab_conflict`: seeds one entity + one block under a SHARED session id, then two tabs race-update the same block continuously. Each tab passes its last-seen `expectedRevision`, catches `REVISION_MISMATCH` from `err.data.current`, resyncs, retries once. Reports `accepted / rejected_revision / retries_recovered` in addition to the normal p50/p95/p99. CI-gate logic updated so `REVISION_MISMATCH` does NOT count toward the >5% failure threshold (those are **expected** rejections — the guard working as designed). | Empirical result against Convex prod (2 tabs × 30s): 180 total writes → **48 accepted immediately + 132 REVISION_MISMATCH + 104 retry-wins = 0 data-loss events**. The rejection rate (73%) reflects how aggressive the race is; the retry-recovery rate (79% of rejections) shows the client can genuinely recover in real time. p95 stays at 186ms even under contention, so users experience the guard as "save was a bit slower," not "save broke." | This answers the specific question Lexical cannot — Lexical gives us rich in-editor state, but nothing about cross-client conflict resolution. Without this scenario we'd be trusting a unit test to tell us whether a distributed system works. Now we have honest prod telemetry: the guard fires, the retry works, p95 stays acceptable, zero writes are silently lost. Combined with the whole suite exiting 0 (gate green), this is the evidence for moving the beta cap from 100 users toward "open it up, keep watching p95s and the retry-recovery ratio." |
| 13 | **Silent data corruption is the worst class of failure — a user types, the bytes persist, the next reload reads back a *different* document with no error fired. Lexical's chip importer/exporter is a pair of inverse functions, and if someone "optimizes" one without the other (easy to do — emoji shortcodes, whitespace collapse, style-bit reorders) the corruption ships invisibly.** | `src/features/entities/components/notebook/notebookLexicalChips.property.test.ts` (new). | Property-based roundtrip on 100 randomly-generated chip arrays (mulberry32 PRNG for reproducibility). Each test: generate → `buildNotebookEditorStateJson` → `parseEditorState` → `chipsFromLexicalEditorState` → compare via `toEqual`. Bucket A: semantic identity across all 100 shapes (up to 20 chips each, mix of styled text, mentions, links, linebreaks). Bucket B: idempotency — a second roundtrip is a no-op. Bucket C: no mention or link chip is ever silently dropped or downgraded to plain text. The canonicalizer normalizes property key-order and adjacent-same-style text collapse so that output equality is judged on semantics, not serialization accidents. | Catches the scariest failure mode: a PR that looks clean in code review but silently mangles user data in production. 4 tests × ~2 seconds each gives us 6000+ microbenchmark-style validations per CI run for negligible cost. If the test ever fails, the `failures` array reports the seed, making the exact corrupting input reproducible across machines. This is `scenario_testing` applied to a primitive: persona is "any user who types into a Lexical-backed block", scale is "20 chips is the p95 block length", failure mode is "silent corruption". |
| 14 | **A user types. Their tab loses wifi for 800ms (coffeeshop, subway tunnel, Convex cold start). The debounced save fires, `updateBlock` rejects, the catch surfaces a red toast. The user keeps typing, another save fires, also fails. Now they've typed a paragraph that exists in the Lexical editor state but was never persisted. If they close the tab, it's gone.** | `src/features/entities/components/notebook/EntityNotebookLive.tsx` (`scheduleSave`). | Wrapped the debounced `updateBlock` call in an `attemptSave(attempt)` function that catches each failure, classifies it via `parseNotebookMutationError`, and retries with exponential backoff (300ms, 900ms, 2700ms = ~4s total) **only** on non-permanent codes. Permanent codes (`REVISION_MISMATCH`, `CONTENT_TOO_LARGE`, `RATE_LIMITED`, `BLOCK_NOT_FOUND`, `ENTITY_NOT_FOUND`) surface immediately because retry is pointless. Up to 3 attempts total. | First-principles split: *transient* failures (network blip, cold start, 5xx) are presumptively retryable — the user's intent is clear, the server just briefly couldn't honor it. *Permanent* failures (the server understood and refused, or the resource doesn't exist) must surface immediately so the user can either split the block, reload, or reauthenticate. Retrying permanent errors burns cycles and hides the real issue. Ceiling at 3 attempts prevents a degraded Convex from turning the notebook into a retry storm. Exponential base of 3 (300ms → 900ms → 2700ms) gets us ~4 seconds of total patience — long enough to ride out a 90th-percentile network hiccup, short enough that the user doesn't wonder if anything is happening. |
| 15 | **A regression ships to Live. The only recovery today is another full deploy — minutes at best, hours if CI is slow. For a notebook holding real user data, that window is unbounded.** | `src/features/entities/views/EntityPage.tsx` (new `isLiveNotebookEnabled()` + use at the Live button and in `readInitialEntityViewMode`). | Two-layer kill switch. Layer 1: build-time env var `VITE_NOTEBOOK_LIVE_ENABLED=false` hides the Live button from every user. Layer 2: runtime `localStorage.setItem("nodebench.liveNotebookDisabled", "1")` gives on-call (or the user themselves) a per-browser opt-out. When the flag is off, a user who previously selected Live silently downgrades to Classic on next mount — their data is untouched because blocks live in `productBlocks` either way; only the *renderer* changes. | This is the "break-glass" pattern from operational reliability: every non-trivial surface needs a path from "bug detected" to "users safe" that doesn't require redeploying. The env-var layer is for staged rollouts and emergency broad disables; the localStorage layer is for individual-user recovery ("paste this into devtools") that support can walk through in 30 seconds. Both were intentionally chosen over a Convex-side feature flag because a Convex flag needs a mutation call to flip, which fails when the surface itself is the thing that's broken. |
| 16 | **A user reports "my save failed" in Slack. Triage has the error toast text and nothing else. Correlating to a Convex log means grepping by timestamp + ownerKey — slow, imprecise, often fruitless.** | `src/features/entities/components/notebook/EntityNotebookLive.tsx` (`extractConvexRequestId`, `requestId` added to `ParsedNotebookMutationError`, appended to the error toast detail as `(ref: <requestId>)`). `docs/architecture/NOTEBOOK_RUNBOOK.md` (new). | Every failure path now tails the Convex Request ID onto the detail string. Users who copy-paste an error into a support ticket automatically include the ref. The runbook maps each error code to a 2-minute triage path keyed by Request ID (`npx convex logs --prod \| grep "<id>"`). | HONEST_STATUS from the agentic-reliability checklist applied to humans-in-the-loop, not just agents. Without the Request ID, mean-time-to-diagnose is limited by grep+eyeballs on raw logs. With it, support drops straight to the log line and can tell a real outage from a REVISION_MISMATCH in seconds. The runbook codifies what "seconds" means for each code — the kill-switch for frontend-wide regressions is literally the first thing in the doc. |
| 17 | **Observability without alerting is a delayed postmortem. A `SERVER_ERROR` fires at 2am; nobody reads the Convex log until 10am the next morning. By then users have already hit it N times. The gap between "something is wrong" and "someone knows something is wrong" is where trust evaporates.** | `src/lib/notebookAlerts.ts` (new). `src/lib/notebookAlerts.test.ts` (new, 8 tests). `src/features/entities/components/notebook/EntityNotebookLive.tsx` (`reportNotebookMutationFailure` now calls `publishNotebookAlert`). `scripts/loadtest/notebook-load.mjs` (POSTs ntfy on CI-gate failure). `docs/architecture/NOTEBOOK_RUNBOOK.md` (new "Real-time alerting (ntfy)" section with subscribe/topic/format instructions). | Client layer: `publishNotebookAlert({severity, code, title, detail, requestId, context})` POSTs to `VITE_NOTEBOOK_ALERT_NTFY_URL` with ntfy headers (`Title`, `Priority` 1-5 mapped from P0/P1/P2, `Tags` for the mobile icon). In-memory dedupe ensures at most 1 alert per code per 60s per tab — a flapping bug produces one notification, not a storm. `EntityNotebookLive` skips alerting for `REVISION_MISMATCH`, `RATE_LIMITED`, `CONTENT_TOO_LARGE` (designed states, not incidents). Load-test layer: on CI-gate failure, POST to `LOAD_TEST_NTFY_URL` with scenario details + Convex URL + entity slug so on-call can jump straight to reproduction. | ntfy.sh was chosen over PagerDuty/Slack/Datadog because it's a **zero-dependency HTTP POST** — no SDK, no auth for public topics, subscribers on iOS/Android/CLI/web in seconds. Two separate topics (client + loadtest) so operators can filter signal. Four production-readiness properties are verified by tests: (a) sampling works (1000 rapid-fire calls → 1 POST), (b) fail-open when env var unset (silent no-op), (c) fail-open when fetch throws synchronously, (d) SSRF defense (non-https and non-ntfy hostnames rejected). This closes the "observability without alerting" gap — the log still has everything, but now a human learns about the failure within seconds of it happening. |

| 18 | **Shared-session collaboration was still the real bar, not isolated anonymous sessions. The old append hot path asked the database for the current last sibling before every insert. Under a truly shared session, that is exactly the place OCC shows up first.** | `convex/domains/product/blocks.ts` (`appendBlock`, helper `nextAppendPosition`) and `scripts/loadtest/notebook-load.mjs` (`shared_session_insert`). | Replaced append's sibling-range scan with a constant-time position generator derived from the current timestamp, then relied on the already-hardened deterministic id tiebreaker to stabilize sort order. Added a dedicated shared-session insert scenario so the load harness measures the real collaboration case instead of a safer isolated-session approximation. | This is the core collaboration-grade fix. Shared-session append correctness matters more than elegant fractional math if the previous approach causes contention on the hot path. The benchmark now proves the actual user shape: multiple writers, one entity, one owner/session boundary. |
| 19 | **The hardening work needed an explicit production decision record, not just test output and load-test JSON. Homen asked for the same style a production engineer would use: what is verified, what is partial, what is still missing, and whether the beta gate should open.** | `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md` | Added the production-readiness checklist with measured load numbers, grouped readiness sections, `✅ / 🟡 / ⬜` grading, an honest beta-gate call, and the top remaining gaps. | This turns the notebook hardening work into an operational artifact instead of tribal memory. The next engineer can challenge the beta decision directly against concrete evidence rather than re-deriving the state from changelog prose and shell history. |
| 21 | **The "real CRDT" aspiration was about to drive a 3-4 week build (custom Y.js provider over Convex, or full editor rewrite to Tiptap for prosemirror-sync). Cooler heads: the repo already has @convex-dev/prosemirror-sync registered in convex/convex.config.ts, Tiptap installed, and a working reference in convex/domains/documents/prosemirror.ts. Reuse > build.** Collaboration-grade behavior on the notebook — same-paragraph merges, agent+user append, same-word OT resolution — can be had by letting the sync component own the editable text while productBlocks stays canonical for order, kind, provenance, relations, access mode, and evidence refs. | `convex/domains/product/blockProsemirror.ts` (new, commit 1 — 1ffdd286), `shared/notebookBlockProsemirror.ts` (new — chip<->PM bridge), `shared/productBlockSync.ts` (new — composed sync id so anonymous sessions thread through), `convex/domains/product/blockProsemirror.test.ts` (new — 11 bridge tests), `src/features/entities/components/notebook/NotebookBlockEditor.tsx` (commit 2 — ef7ca1ef; Tiptap editor via useTiptapSync targeting the new sync API), `src/features/entities/components/notebook/EntityNotebookLive.tsx` (wires buildProductBlockSyncId per block + editor-handle registry for mention insertion + optimistic local content during snapshot debounce), `src/features/entities/components/notebook/BlockStatusBar.tsx` (new — presence + last-synced + offline + rate-limit indicators), `src/features/entities/components/notebook/OversizeBlockModal.tsx` (new — preflight for CONTENT_TOO_LARGE), `src/features/entities/components/notebook/TombstoneModal.tsx` (new — recover unsynced edits when a block is deleted mid-edit), `src/features/entities/components/notebook/notebookOfflineQueue.ts` + `.test.ts` (new — localStorage-backed queue, 100-entry cap, fail-open on Safari private / quota / disabled), `convex/domains/product/notebookPresence.ts` (new — @convex-dev/presence wrapper that accepts anonymous sessions alongside authenticated users). | Commit 1 wires the server: ProsemirrorSync.syncApi<DataModel> re-exporting getSnapshot / submitSnapshot / latestVersion / getSteps / submitSteps, checkRead + checkWrite using requireProductIdentity + ownerKey scoping + access-mode enforcement, onSnapshot mirrors sanitized PM JSON back into productBlocks.content as BlockChip[] so Classic view / agent context / search / evidence extraction keep working. Commit 2 wires the client: NotebookBlockEditor mounts Tiptap StarterKit + Link + Mention + Image + Underline, useTiptapSync at 900ms snapshot debounce, BlockRow threads buildProductBlockSyncId({ blockId, anonymousSessionId }), mentions route via editor handle not server rewrite, optimistic local content bridges the debounce window. Commit 3 wires the UX shell: BlockStatusBar renders the ambient state, offline queue persists edits to localStorage for tab-close recovery, Oversize + Tombstone modals handle the rare-but-scary cases by offering the user explicit choices (split / save as new block / discard). Kill switches: NOTEBOOK_PM_SYNC_ENABLED=false on Convex throws SYNC_DISABLED (client falls back to the existing revision-guard path); NOTEBOOK_PRESENCE_ENABLED=false makes presence a no-op. | The decision not to own a custom merge engine is the whole point of this row. Every time we'd have hit an edge case in mergeBlockChips (emoji shortcodes, zero-width joiners, RTL text, multi-style runs) we'd have paid for it in bugs and changelog entries. Letting the sync component handle OT means those edge cases are somebody else's problem (and they have tests). The scenarios from the user-case matrix that *aren't* OT — offline replay, oversize guard, tombstone recovery, presence — are all UX concerns built on top of the productBlocks model, not the sync layer. That separation is the correct production shape for the notebook. |
| 20 | **Solo operator has no on-call rotation, no PagerDuty, no Datadog. Between "something is quietly wrong" at 11pm and "user reports it" at 9am the next morning is 10 hours of silent damage. Without a self-operating monitoring loop, every outage steals a working day.** | `scripts/ops/hourlyHealthCheck.mjs` (new), `scripts/ops/dailySummary.mjs` (new), `scripts/ops/diagnose.mjs` (new), `scripts/ops/notionClient.mjs` (new), `.github/workflows/notebook-hourly.yml` + `.github/workflows/notebook-daily.yml` (new), `package.json` (notebook:health / notebook:daily / notebook:diagnose / notebook:loadtest scripts), `docs/architecture/SOLO_OPS_RUNBOOK.md` (new). | Three-layer self-operating loop: **(a)** Hourly CI runs a 60-block heartbeat against Convex prod, gates on p95 < 500ms and errors < 1%, POSTs ntfy only on degrade (silent-on-green to avoid alert fatigue); **(b)** Daily CI at 14:11 UTC runs the full load-test suite, appends a row to a Notion database (Status/P95/Errors/body), sends one ntfy digest every day regardless of status so the operator can see the green heartbeat; **(c)** `npm run notebook:diagnose` bundles git state + last daily summary + ntfy history (last 6h via `?since=6h`) + Convex logs + notebook source file git-log into a single markdown file at `.tmp/notebook-diagnose-latest.md` ready to paste into Claude Code or Codex. Notion + ntfy are both fail-open (unset env var = silent no-op, API failure = logged but not thrown). | Solves the "operator is one person and sometimes asleep" problem without buying PagerDuty/Datadog. Hourly = catches slow drift, daily = catches trends, diagnose command = collapses the human-to-agent handoff from 20 minutes of grepping logs to one pasted file. The workflows run on schedule regardless of repo activity, which means the ops signal doesn't go dark when the operator is heads-down on features. Closing this gap was the difference between "I am the on-call rotation" and "CI is the on-call rotation; I am the fix loop." |
| 21 | **A global kill switch is necessary but not sufficient. For a new notebook surface, support needs a way to expose Live to a small deterministic cohort, keep it off for everyone else, and still force-enable it for debugging one browser without widening the rollout.** | `src/features/entities/views/EntityPage.tsx` and `src/features/entities/views/EntityPage.test.tsx` | Added deterministic staged rollout controls on top of the existing kill switch. `isLiveNotebookEnabled(entitySlug)` now evaluates `VITE_NOTEBOOK_LIVE_ENABLED`, `VITE_NOTEBOOK_LIVE_ROLLOUT_PERCENT`, `nodebench.liveNotebookDisabled`, and `nodebench.liveNotebookForceEnabled`. Cohorting is stable per anonymous session plus entity slug, so the same browser sees the same decision across reloads. Tests cover 0 percent, 100 percent, local disable, and local force-enable paths. | This closes the gap between "feature is on" and "feature is safely ramped." A kill switch only helps after something is already broken. Cohort rollout lets us limit blast radius before the break happens, and the force-enable path gives support a clean reproducer without changing the rollout for every user. |
| 22 | **The first attempt at stricter append pacing reintroduced the exact thing we had just removed: shared-session contention on the hot path. A session-wide aggregate read plus write-window mutation looked correct in isolation, but under `shared_session_insert` it collapsed into `OptimisticConcurrencyControlFailure` and reopened the collaboration regression.** | `convex/domains/product/blocks.ts`, `convex/domains/product/schema.ts`, `src/features/entities/components/notebook/EntityNotebookLive.tsx`, `src/features/entities/components/notebook/EntityNotebookLive.test.tsx`, `scripts/loadtest/notebook-load.mjs` | Reworked the limiter around actor-scoped write windows. Added `actorKey` to `productBlockWriteWindows`, added `by_owner_session_actor_bucket`, normalized actor keys, and changed `assertNotebookWriteRateLimit` to read only actor-scoped buckets. On the hot path, write-window OCC is intentionally translated into structured `RATE_LIMITED` instead of surfacing as a raw server failure. The client parser and load harness both map `productBlockWriteWindows` OCC into `RATE_LIMITED`, and the harness gained `actor_rate_limit_guard` so the behavior is measured instead of hand-waved. | This is the pragmatic collaboration-safe contract. Shared-session append must stay green because that is the product-critical path. Strict session-wide anti-rotation can wait if it reopens collaboration failures. The new behavior is honest: same-actor pressure is bounded, and when the limiter itself contends the user sees `RATE_LIMITED`, not a fake backend outage. |
| 23 | **Beta readiness claims are weak if they stop at one-minute synthetic lanes. We needed to know whether the notebook survives a real zero-ramp burst and a longer mixed workload, not just a friendly 10-client loop.** | `scripts/loadtest/notebook-load.mjs`, `.tmp/notebook-load-spike.json`, `.tmp/notebook-load-soak.json`, `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md` | Added `spike_insert` (100 clients, 5s, zero-ramp burst) and `soak_mixed` (5 shared-session clients, 10 minutes, mixed append/update/read). Both scenarios now emit persisted JSON artifacts and are called out in the production checklist. Latest measured results: `spike_insert` 2079/2079 with p95 269ms, `soak_mixed` 9586/9586 with p95 198ms. | This closes the "it passed once under a friendly load test" gap. The spike lane proves the notebook does not fold under a sharp burst, and the soak lane proves latency does not creep or degrade over a longer session. Together they are enough to justify a beta gate for this surface instead of a compile-only claim. |
| 24 | **The notebook had engineering evidence, but no stable benchmark artifact that a future engineer could inspect without replaying every shell command. A checklist without the raw benchmark record becomes stale quickly.** | `docs/architecture/benchmarks/notebook-load-latest.md`, `.tmp/notebook-load-summary.json`, `.tmp/notebook-load-actor-guard.json`, `.tmp/notebook-load-spike.json`, `.tmp/notebook-load-soak.json` | Published a benchmark artifact that records the exact commands, deployment target, generated timestamps, scenario tables, and the split between expected and unexpected errors. The markdown artifact is derived from the persisted JSON files instead of hand-typed from memory. | This turns the load evidence into a reusable artifact. The next engineer can compare a new run against a fixed baseline instead of trusting console output from an earlier session. It also makes the beta gate auditable after the fact. |
| 25 | **The notebook source-of-truth story was implicit in code and comments, which is dangerous. A future engineer could easily assume Live edits should overwrite the saved brief or that a report refresh should replace user blocks. That is how data drift bugs ship.** | `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md`, `docs/architecture/NOTEBOOK_RUNBOOK.md` | Wrote the canonical source contract explicitly: `productReports` remains canonical for saved brief rendering and provenance; `productBlocks` is canonical for live notebook editing, backlinks, and revisions; report refreshes may project newer agent material, but newer user notebook edits are never silently overwritten. The runbook now also records ownership, escalation, rollout controls, and the exact verification commands used for the beta gate. | This closes an organizational gap, not just a code gap. Production incidents often come from unclear contracts between two "almost canonical" stores. Writing the contract down is what prevents future fixes from reintroducing silent overwrites or confusing Live with saved brief state. |
| 26 | **A realistic people test is not "two tabs on one machine." The actual question was: what happens when one note/report is effectively shared to 10 collaborators and they all pile onto the same live notebook at once, with some editing different sections and some hammering the same section. The first Tiptap wiring looked correct under light use, but under that burst it was still unstable: editor instances were being recreated too aggressively, ProseMirror surfaces were not always the thing the E2E harness typed into, and presence heartbeats were reusing one session id across every tab.** | `src/features/entities/components/notebook/NotebookBlockEditor.tsx`, `src/features/entities/components/notebook/EntityNotebookLive.tsx`, `tests/e2e/entity-notebook-regression.spec.ts`, `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md` | Stabilized the editor shell instead of chasing symptoms. In `NotebookBlockEditor.tsx`, moved volatile callbacks behind refs, narrowed `useEditor` dependencies to stable sync inputs, and guarded focus handoff so collaboration bursts do not destroy and recreate the editor unnecessarily. In `EntityNotebookLive.tsx`, added a per-tab `presenceClientSessionIdRef` so room heartbeats no longer collide under a shared session, and kept the presence self-user derivation explicit for anonymous workspaces. In Playwright, changed the harness to focus the actual ProseMirror contenteditable surface, then added a 10-collaborator scenario: the first five collaborators append markers into different blocks, the last five all edit the same block, and the test verifies all persisted markers survive reload. | This is the closest thing to a real "shared with 10 people" production proof we currently have without first building a separate-user ACL/share model. It closes the editor-stability gap that only appears under concurrent focus churn, proves same-block and different-block edits both persist, and draws a clean boundary around what is and is not verified: shared-session collaboration is green; true share-to-10-distinct-user identities is still a separate product capability. |
| 27 | **The notebook was collaboration-ready, but the classic entity reading surface still felt cramped and the verification story ended at local dev. The product needed one more pass before handing it to testers: make classic notes read like a real Roam/Notion workspace, prove offline replay in the browser, add route-level smoke coverage, and ship the actual frontend build to production.** | `src/features/entities/views/EntityPage.tsx`, `src/features/entities/components/EntityNoteEditor.tsx`, `tests/e2e/helpers/entityNotebook.ts`, `tests/e2e/entity-notebook-regression.spec.ts`, `tests/e2e/product-shell-smoke.spec.ts`, `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md` | Widened the classic entity shell to a larger desktop canvas, changed the notes and rail layout to a two-column grid with the notes column dominant, and increased the note editor minimum height so the notes surface reads like the primary working pane rather than a sidebar. Added reusable Playwright helpers for seeded entity/session setup, focused the offline replay test on the real ProseMirror surface before taking the browser offline, and added a dedicated shell smoke spec that checks `home`, `reports`, and the classic notes-width ratio. Final verification included local Playwright green on the notebook regression plus shell smoke suite, then a real `vercel --prod --yes` deployment followed by production browser checks on `https://www.nodebenchai.com`. | This closes the last handoff gap between "the notebook code works" and "the deployed product is ready for testers." The notes surface is now proportioned like a true writing workspace, offline replay is browser-proven instead of only code-proven, the route smoke suite catches console regressions early, and the deployment claim is backed by a successful production alias instead of a local build log. |
| 28 | **Link sharing was no longer the last product gap. The missing capability was identity-based access: an owner should be able to invite a named collaborator by email, that collaborator should sign up with that email, accept the invite, land on a constrained member workspace, and edit without ever seeing owner-only controls. Until this row, the product had secure links but no real member/invite workflow.** | `convex/domains/product/schema.ts`, `convex/schema.ts`, `convex/domains/product/helpers.ts`, `convex/domains/product/shares.ts`, `convex/domains/product/entities.ts`, `src/lib/registry/viewRegistry.ts`, `src/features/entities/lib/entityExport.ts`, `src/SignInForm.tsx`, `src/features/entities/components/EntityShareSheet.tsx`, `src/features/entities/views/EntityPage.tsx`, `tests/e2e/entity-member-invite.spec.ts`, `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md` | Added owner-scoped `productEntityWorkspaceMembers` and `productEntityWorkspaceInvites` tables, extended workspace access resolution to understand `member` mode alongside raw share links, and exposed a combined share/member API: invite collaborator, preview invite, accept invite, update member/invite access, and revoke member/invite. The share sheet stayed intentionally minimal: `View link`, `Edit link`, and a simple `People` section with email, role, copy link, and remove. On the entity page, invite URLs now preserve `?invite=...`, unauthenticated invitees get an inline sign-in form prefilled with the invited email, wrong-account users see an explicit mismatch state, accepted invitees are redirected onto a member-scoped `?share=` URL, and shared members never see owner-only share controls. Playwright now proves the full path locally and on production: owner signs up from an anonymous seeded workspace, invites a named collaborator, collaborator signs up, accepts the invite, edits the live notebook, and keeps the edit after reload. | This closes the real product gap, not just an engineering nicety. Before row 28, "sharing" meant passing around a tokenized URL. After row 28, there is a clean user-facing collaboration story that maps to how real teams work: named person, explicit access level, constrained post-acceptance workspace, and no owner control leakage. The UI stayed simple on purpose, closer to Linear/Notion than to an enterprise admin console. |
| 29 | **The first runtime proof of row 28 surfaced two genuine bugs, both of which would have made the member workflow feel flaky in front of testers. Bug 1: anonymous-owner bootstrap claimed entities and notes on signup, but not the live notebook tables, so the freshly-authenticated owner could "lose" the seeded blocks. Bug 2: the invite route was falling through to the public/system workspace after member signup, so a valid invite sometimes rendered a read-only entity page instead of the `Join workspace` gate.** | `convex/domains/product/bootstrap.ts`, `src/features/entities/views/EntityPage.tsx`, `src/SignInForm.tsx`, `tests/e2e/helpers/entityNotebook.ts`, `tests/e2e/entity-share-permissions.spec.ts`, `tests/e2e/entity-member-invite.spec.ts`, `tests/e2e/product-shell-smoke.spec.ts` | Fixed the bootstrap owner-claim list to include `productWorkspaceShares`, `productEntityRelations`, `productBlocks`, `productBlockRelations`, and `productBlockWriteWindows`, so anonymous live-notebook data survives owner signup. Tightened `EntityPage.tsx` so `?invite=` never falls through to `systemWorkspace` or `starterWorkspace`; until the invite is accepted, that route only resolves against the live owner workspace. Updated `SignInForm` so password or anonymous auth reloads the current page by default, removing the awkward manual refresh step inside invite and share-gated flows. Refreshed Playwright helpers to handle the simplified live-first entity UX: live toggle may be hidden, `page.goto` retries once after offline reconnect if Chromium returns transient `ERR_ABORTED`, and the smoke assertion now matches the current home-page heading. Final browser pack ran green both locally and against `https://www.nodebenchai.com`: member invites, raw share links, 10-collaborator notebook editing, offline replay, and shell smoke. | This row is what turned the member/invite feature from "the backend exists" into "the workflow is production-verifiable." The bugs were real and user-facing. Claiming the live notebook tables fixes the owner transition at the exact moment trust is easiest to lose, and clamping the invite route to the live workspace prevents the product from lying to an invited user by showing them a public read-only page when they should be joining a private editable one. The helper/test updates matter because they now reflect the current simplified UX instead of the older toggle-heavy entity page. |
| 30 | **The remaining rollout gaps were no longer raw access control. They were product trust details: does a named invite actually send email, does the `People` panel teach the first action, can the owner tell who is active without opening devtools, and is the beta contract explicit enough that "post-beta" does not become an indefinite bucket.** | `convex/domains/product/schema.ts`, `convex/domains/product/shares.ts`, `convex/domains/product/entities.ts`, `convex/domains/product/blocks.ts`, `convex/domains/product/notebookPresence.ts`, `src/features/entities/components/EntityShareSheet.tsx`, `src/features/entities/components/notebook/BlockStatusBar.tsx`, `src/features/entities/components/notebook/EntityNotebookLive.tsx`, `src/features/entities/views/EntityPage.tsx`, `tests/e2e/entity-member-invite.spec.ts`, `docs/architecture/NOTEBOOK_PRODUCTION_CHECKLIST.md`, `docs/architecture/NOTEBOOK_RUNBOOK.md` | Added invite-delivery metadata to workspace members/invites, moved collaborator creation behind an action that attempts Resend email delivery and falls back to link-only when email fails, and persisted that delivery outcome back onto the workspace rows. The share sheet now keeps the UI minimal but explicit: stronger first-invite empty state plus per-person status text like `Email sent 2m ago` or `Email unavailable, secure link ready`. The notebook status bar now resolves participant labels and latest human editor identity, and `EntityNotebookLive` threads the current viewer ownerKey so presence no longer assumes anonymous-only self identity. `notebookPresence` now keys off resolved workspace identity rather than a separate product-identity lookup. The member-invite E2E was updated to accept both delivery modes and recover the invite URL from the row when email succeeds instead of blindly assuming clipboard fallback. The checklist and runbook now state the phased rollout contract explicitly: phase 1 is 15 testers for 14 days, phase 2 is 50, phase 3 is 100. | This closes the gap between "feature works" and "testers trust it." Email delivery makes invites behave like invites instead of disguised share links, the People empty state teaches the only action that matters, the collaboration footer gives the owner enough confidence that another person is actually on the page, and the phased beta contract prevents the team from hiding future uncertainty inside the phrase `post-beta`. The product stays intentionally simple, closer to Notion/Linear than to an enterprise admin console, but the operational framing is now explicit enough to run a disciplined rollout. |

---

## How to run the load test

```bash
# All three scenarios (defaults: 10 clients, 30s each)
node scripts/loadtest/notebook-load.mjs --entity softbank

# Just the concurrent-insert scenario with 25 clients for 2 minutes
node scripts/loadtest/notebook-load.mjs --scenario concurrent_insert --clients 25 --duration 120

# Point at a non-default Convex (e.g. a staging deployment)
node scripts/loadtest/notebook-load.mjs --url https://other-deployment.convex.cloud

# CI gate (exits non-zero if > 5% error rate in any scenario)
node scripts/loadtest/notebook-load.mjs --entity softbank && echo "load test passed"
```

Exit codes: `0` = all scenarios < 5% error rate, `1` = at least one scenario exceeded 5% error rate, `2` = runtime/config error.

## How the tiebreaker and pagination wire together

1. Client calls `usePaginatedQuery(listEntityBlocksPaginated, { entitySlug })`.
2. Convex `ctx.db.paginate({ cursor, numItems: 50 })` scans the `by_owner_entity_position` index, which already sorts by `(ownerKey, entityId, positionInt, positionFrac)`. The index gives a stable storage-order across scans.
3. `paginateFilteredRows` drops soft-deleted rows before returning the page.
4. Where the server still has to re-sort in-memory (appendBlock, promoteLinkToEvidence), it now uses `comparePositionsWithId` so ties resolve identically on every call.
5. Client renders the page as-is — never re-sorts — so the deterministic server order is what the user sees.

Result: one write path (fractional indexing with deterministic tiebreaker) and one read path (cursor-paginated index scan with deterministic filter). No invariants split across layers.
