---
id: 11640
title: 'Phase 4B — Manifest Reconciliation Daemon: Tenant-State vs Chroma-Actual Sync'
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
assignees:
  - neo-opus-ada
createdAt: '2026-05-19T11:57:21Z'
updatedAt: '2026-06-07T17:13:39Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11640'
author: neo-opus-ada
commentsCount: 2
parentIssue: 11628
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy:
  - '[x] 11639 Phase 4A — Per-Tenant Ingestion Observability Daemon (KBRecorderService Extension)'
blocking:
  - '[x] 11641 Phase 4C — Stale-Chunk Garbage Collection Daemon: Orphan Detection + Retention Enforcement'
closedAt: '2026-05-21T09:33:01Z'
---
# Phase 4B — Manifest Reconciliation Daemon: Tenant-State vs Chroma-Actual Sync

## Context

Sub of Phase 4 Epic [#11628](https://github.com/neomjs/neo/issues/11628) (meta-Epic [#11624](https://github.com/neomjs/neo/issues/11624)).

**Periodic state-reconciliation daemon** — catches missed tombstones, handles force-push detection beyond per-push payload.

## The Problem

Phase 0/1A defines tombstone / manifest / revision-boundary deletion-signaling, but production failure modes exist:

- Client hooks fail mid-push (some files pushed, deletes lost)
- Force-push history rewrite (per-push revision-boundary insufficient for branch-rewrite scenarios)
- Tenant config changes invalidate previously-pushed chunks
- Network errors cause partial-push state

Without periodic reconciliation, tenant Chroma state drifts from tenant's actual repo state silently.

## The Fix

New daemon: `ai/scripts/kb-reconciliation-daemon.mjs` (sibling to existing daemons).

Per scheduled tick (configurable; default hourly):

1. **For each active tenant**:
   - Fetch tenant's claimed-state manifest (from `KnowledgeBaseTenantConfig` Phase 2E + last-received revision boundary)
   - Fetch Chroma's actual-state chunks for the tenant (via `where: {tenantId}` filter from Phase 0/1D)
   - Diff: chunks in Chroma but not in claimed-state → orphaned (queue for GC or auto-tombstone)
   - Diff: claimed-state paths not represented in Chroma → re-ingestion candidates (notify tenant via observability surface)
2. **Reconciliation actions**:
   - Auto-tombstone orphans older than retention threshold (configurable per-tenant)
   - Emit observability events via Phase 4A
   - Alert operator if drift exceeds threshold (Phase 4D)

## Acceptance Criteria

- [ ] `ai/scripts/kb-reconciliation-daemon.mjs` exists; follows existing daemon pattern
- [ ] Per-tenant reconciliation logic implemented (claim vs actual diff)
- [ ] Configurable tick interval (`aiConfig.knowledgeBase.reconciliationIntervalMs`; default 3600000 = 1h)
- [ ] Configurable per-tenant retention threshold (orphan auto-tombstone after N days)
- [ ] Reconciliation events emitted via Phase 4A observability daemon
- [ ] Alerts emitted via Phase 4D when drift exceeds threshold
- [ ] Unit tests: diff logic, auto-tombstone, retention threshold
- [ ] Integration test: simulate tenant force-push → reconciliation removes orphans

## Out of Scope

- Initial observability daemon → Phase 4A
- Operator alerting infrastructure → Phase 4D
- Per-chunk garbage collection scheduling → Phase 4C (separate ticket for GC-specific concerns)

## Contract Ledger

> Added at intake by @neo-opus-ada (Claude Code) 2026-05-21 — satisfies the ticket-intake §7 Contract Completeness readiness gate (intake comment: https://github.com/neomjs/neo/issues/11640#issuecomment-4504572192). The original-author session is inactive; per ticket-intake §7 the claiming maintainer authors the missing ledger. Tier target: **T3 (Explicit Matrix)**. The ledger is the binding contract; the loose Acceptance Criteria above are refined by these rows.
>
> **V1 scope — substrate-grounded.** A fresh sweep of the merged Phase 2 code shows the ticket's "claim-vs-actual path-manifest diff" envisions a *persisted claimed-state manifest* that Phase 2 does not store: `KnowledgeBaseTenantConfig` carries no path manifest, and revision boundaries (`baseRevision` / `headRevision`) are per-push payload parameters, not persisted state. A periodic daemon with no repo access and no stored manifest therefore cannot detect arbitrary drift. The substrate-real V1 reconciliation signal is **config-invalidation reconciliation** — the `tenantConfigVersion` chunk stamp (#11637 / `VectorService.resolveTenantStamp`) compared against the tenant's current `getTenantConfig().version`. This delivers the ticket's failure-mode #3 ("tenant config changes invalidate previously-pushed chunks") completely; failure-modes #1/#2/#4 (force-push, mid-push, partial-push) are V1.x-deferred — Row 6.
>
> **Refined 2026-05-21** per @neo-gpt's #11640 pre-PR peer review: V1 Phase 4D integration is drift-*presence/frequency* alerting via `reconcileEvents` (Row 5) — drift-*volume* threshold alerting needs a `chunks_total` rollup metric and is V1.x (Row 6).

| Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
|---|---|---|---|---|---|
| `aiConfig.knowledgeBase` reconciliation config block — 4 new keys | #11628 Phase 4B; this ticket AC; #11642's `aiConfig.knowledgeBase` precedent | `reconciliationEnabled` (Boolean, default `false`) — master opt-in; the daemon exits early when false. `reconciliationIntervalMs` (Number, default `3600000` = 1h) — poll-tick interval. `reconciliationAutoTombstone` (Boolean, default `false`) — opt-in for the destructive auto-tombstone action; default-off ⇒ the daemon detects + reports only. `reconciliationOrphanVersionGap` (Number, default `2`) — a config-stale chunk is auto-tombstone-eligible when `currentVersion − chunk.tenantConfigVersion ≥ this`. | A stale gitignored `config.mjs` predating #11640 has no `knowledgeBase` key (or lacks the reconciliation keys) → every key is read defensively against its default (mirrors #11642's defensive read). | Yes — `ai/config.template.mjs` block + JSDoc | Unit: config-defaulting + the daemon's opt-in gate |
| Config-stale orphan detection — `KbReconciliationEngine` pure core | #11637 `tenantConfigVersion` stamp; `getTenantConfig().version` | The pure, dependency-free diff engine (mirrors #11642's `KbAlertRuleEngine`). `diffTenantChunks({rows, currentVersion, orphanVersionGap})` → a chunk is a **config-stale orphan** when `typeof metadata.tenantConfigVersion === 'number' && metadata.tenantConfigVersion < currentVersion`; its `versionGap = currentVersion − tenantConfigVersion`; it is **actionable** when `versionGap ≥ orphanVersionGap`. Returns `{staleOrphans, staleCount, actionableIds, actionableCount}`. No I/O, no clock. | `currentVersion === 0` (tenant on the yaml/default config tier — no graph node) → no chunk can be stale (`v < 0` is never true); zero orphans, no special-case. A chunk with a missing / non-numeric `tenantConfigVersion` (pre-#11637 ingest) → **not** flagged (fail-safe: never auto-action a chunk we cannot classify). | Yes — JSDoc | Unit: stale-detection, version-gap partition, `currentVersion: 0` no-op, missing-stamp skip |
| The auto-tombstone reconciliation action — `knowledge-base` Chroma collection delete | this ticket AC ("auto-tombstone orphans"); the destructive-action conservatism principle | When `reconciliationAutoTombstone` is `true`, the daemon deletes a tenant's `actionableIds` via `collection.delete({ids})`. The delete is **tenant-scoped** — `actionableIds` derive only from rows fetched with `where: {tenantId}` (the `getTenantRows` batched-`collection.get` pattern); tenant A's reconciliation never touches tenant B's chunks. | `reconciliationAutoTombstone` is `false` (the default) → **no delete is ever issued**; the daemon detects + emits telemetry only. A `collection.delete` throw → `logger.error`, best-effort; the daemon continues to the next tenant. | Yes — daemon JSDoc | Unit: delete gated by the opt-in flag; tenant-scoped id set; delete-throw tolerance |
| Phase 4A telemetry emission — `KBRecorderService.recordIngestionMetric` | #11639 / #11665 Phase 4A; `recordIngestionMetric`'s `'reconcile'` event type | When a tenant has ≥ 1 config-stale orphan, the daemon emits exactly one `recordIngestionMetric({tenantId, repoSlug, eventType: 'reconcile', chunksTotal: staleCount, chunksDeleted: <count tombstoned this tick — 0 when auto-tombstone is off>, detail: {staleCount, actionableCount, currentVersion, autoTombstone}})`. A clean tenant (zero orphans) emits nothing — so `reconcileEvents > 0` genuinely means "drift was found". | `recordIngestionMetric` is already best-effort (never throws into the caller). The recorder unavailable → the metric is silently dropped; reconciliation still runs. | Yes — JSDoc | Unit: a `reconcile` metric is emitted for a drifting tenant, suppressed for a clean one |
| Phase 4D alerting integration — telemetry-only seam | #11642 Phase 4D; `KbAlertRuleEngine.KNOWN_METRICS` | The ticket AC "alerts emitted via Phase 4D when drift exceeds threshold" is satisfied **with no #11642 code change** as **drift-presence/frequency** alerting. The daemon emits one `reconcile` event per tenant-with-drift per tick; `reconcileEvents` is already a `KbAlertRuleEngine.KNOWN_METRICS` field, so an operator's `aiConfig.knowledgeBase.alertRules` entry on `reconcileEvents` fires when a tenant shows persistent / frequent drift across the alert window. `chunksDeleted` is also `KNOWN_METRICS`-covered — alertable as *action volume*, but non-zero only when `reconciliationAutoTombstone` is on (`0` in the default detect-only posture). The existing #11642 `KbAlertingService` rolls up the events this daemon emits and fires; the reconciliation daemon does **not** dispatch alerts itself — no duplication of #11642's channel logic. | Drift-*volume* thresholding (alert when the stale-chunk count exceeds N) is **not** available in V1: `getTenantIngestionRollup` does not aggregate `chunks_total`, and `chunksTotal` / `staleCount` are not in `KNOWN_METRICS`. The daemon still records `chunksTotal: staleCount` + a `detail` payload (raw-row and `detail`-visible), so the data is captured — but rollup-aggregated volume alerting needs a `chunks_total` rollup metric + `KNOWN_METRICS` coverage → Row 6 (V1.x). No `alertRules` configured → telemetry is still recorded; only alert *dispatch* is absent. | Yes — JSDoc cross-ref | Verified by inspection (@neo-gpt #11640 pre-PR peer review, 2026-05-21): `reconcileEvents` ∈ `KNOWN_METRICS`; `getTenantIngestionRollup` SQL has no `chunks_total` aggregation |
| V1 scope boundary — manifest / force-push detection + per-tenant threshold override + drift-volume alerting | this ticket "The Fix" (failure-modes #1/#2/#4) + AC; ticket-intake "challenge prescribed fixes"; @neo-gpt #11640 pre-PR peer review | **V1.x-deferred, documented.** (a) Manifest / force-push / partial-push orphan detection requires a *persisted claimed-state manifest* or a *persisted last-received revision boundary* — neither exists in Phase 2 substrate. (b) A per-tenant `orphanVersionGap` override requires extending `getTenantConfig`'s fixed 8-field projection — a #11637-surface change. (c) Drift-*volume* threshold alerting requires a `chunks_total` rollup metric in `getTenantIngestionRollup` + `chunksTotal` in `KbAlertRuleEngine.KNOWN_METRICS` — a #11639 / #11642-surface change. V1 ships drift-*presence/frequency* alerting via `reconcileEvents` (Row 5) and touches zero merged Phase 2 code (purely additive: 3 new files + the `aiConfig` template block). | V1.x is a separate follow-up ticket: it adds the persisted-manifest substrate (or a daemon-side repo walk), the per-tenant override, and the `chunks_total` rollup metric. The PR body files it. | Yes — PR body "Deltas" + a V1.x follow-up ticket | N/A — explicit scope boundary |

## Prior Art / Backup-Restore Substrate Cross-References

Substrate-correct V-B-A calibration 2026-05-19 post-PR-#11647 merge: the backup/restore/defrag substrate I underspecified during Phase 0/1A scoping is load-bearing for this daemon's design. Per [#10129](https://github.com/neomjs/neo/issues/10129) Phase 3 peer architecture:

- **`buildScripts/ai/backup.mjs`** — canonical atomic-bundle orchestrator. Layout `.neo-ai-data/backups/backup-<ISO-ts>/{kb,mc,graph,concepts,trajectories}/`. Reconciliation daemon should COEXIST with the existing bundle-restore lifecycle, NOT replace it.
- **`buildScripts/ai/restore.mjs`** — merge-mode preserve-live semantics already shipped. Graph SQLite uses `INSERT OR IGNORE` (post-[#11141](https://github.com/neomjs/neo/issues/11141) fix; pre-fix used silently-broken `INSERT OR REPLACE`). Chroma side uses `collection.upsert()`; [#11144](https://github.com/neomjs/neo/issues/11144) tracks preserve-live parity follow-up. **The daemon's tenant-state reconciliation operates ABOVE this layer** — it diffs tenant-claimed vs Chroma-actual; restore.mjs operates on the bundle vs collection axis. Both can coexist.
- **`buildScripts/ai/defragChromaDB.mjs`** — peer of backup.mjs (NOT a delegate). 5-step nuke-and-pave with private pre-nuke physical-copy snapshots at `dist/chromadb-backups/<target>/`. Operators chain `ai:defrag-kb && ai:backup` for compacted backups. Phase 4C (#11641 GC daemon) is the closer substrate sibling.
- **Existing test substrate to extend, not duplicate**: `test/playwright/unit/ai/buildScripts/restore.spec.mjs`, `restore-hardening.spec.mjs`, `restore-filters.spec.mjs`, `backup.spec.mjs`.

Operator framing (2026-05-19): *"we have more complex restoration scripts, since our live db got wiped 2x already, and we can now merge backups and new live DB data."* — merge-mode capability IS prior art, not Phase 4 new substrate. This daemon adds tenant-state reconciliation on top.

## Related

- **Parent:** [#11628](https://github.com/neomjs/neo/issues/11628)
- **Blocked-by:** Phase 4A (observability event emission), Phase 2 Epic (ingestion pipeline must exist)
- **Related substrate (cross-references):** [#10129](https://github.com/neomjs/neo/issues/10129) atomic-bundle backup orchestrator, [#11141](https://github.com/neomjs/neo/issues/11141) graph preserve-live fix, [#11144](https://github.com/neomjs/neo/issues/11144) Chroma preserve-live parity follow-up
- **Sibling backup primitives**: `buildScripts/ai/{backup,restore,defragChromaDB}.mjs`; tests under `test/playwright/unit/ai/buildScripts/`
- **Daemon pattern precedent:** `ai/scripts/orchestrator-daemon.mjs`
- **Tombstone contract:** Phase 0/1A

## Origin Session ID

`7360e917-1733-4cdd-a6f3-5ac51c34b838`

## Handoff Retrieval Hints

- `orchestrator-daemon.mjs` scheduling pattern is the architectural reference
- Reconciliation = claim-vs-actual diff; the substrate-correct sibling pattern to look at is git's index-vs-working-tree-diff (conceptual reference, not code reuse)

## Timeline

- 2026-05-19T11:57:22Z @neo-opus-ada added the `enhancement` label
- 2026-05-19T11:57:22Z @neo-opus-ada added the `ai` label
- 2026-05-19T11:57:22Z @neo-opus-ada added the `architecture` label
- 2026-05-19T11:59:16Z @neo-opus-ada added parent issue #11628
- 2026-05-19T11:59:44Z @neo-opus-ada marked this issue as being blocked by #11639
- 2026-05-19T11:59:48Z @neo-opus-ada marked this issue as blocking #11641
- 2026-05-19T12:01:25Z @neo-opus-ada cross-referenced by #11624
- 2026-05-19T13:52:55Z @neo-opus-ada cross-referenced by #11629
- 2026-05-19T14:40:31Z @neo-opus-ada cross-referenced by #11628
- 2026-05-19T14:44:07Z @neo-opus-ada cross-referenced by #11649
- 2026-05-19T15:05:57Z @neo-opus-ada cross-referenced by #11650
- 2026-05-19T15:10:32Z @neo-opus-ada cross-referenced by PR #11651
- 2026-05-19T15:25:20Z @tobiu referenced in commit `617c712` - "fix(memory-core): add .claude and .codex to FileSystemIngestor ignorePatterns (#11650) (#11651)

Operator V-B-A surfaced 2026-05-19 during graph-backup analysis: latest
atomic-bundle contains 3,168,310 graph elements vs. the ~50k baseline from
~10 days ago. Statistical sampling showed FILE + DIRECTORY nodes + CONTAINS
edges account for ~98% of the corpus.

Root cause: FileSystemIngestor.mjs:31 ignorePatterns_ catches `.gemini`,
`.idea`, `.agents`, `.neo-ai-data`, `node_modules` at the root level — but
NOT `.claude` (Claude Code agent worktrees) or `.codex` (Codex agent state).
Each Claude Code session creates `.claude/worktrees/<NAME>/` carrying its
own multi-GB node_modules AND its own .neo-ai-data (recursive substrate
amplification). Indexer crawled them all because the path doesn't start
with `node_modules/` — it starts with `.claude/`, which isn't in the list.

The path-prefix matcher at line 112 (`relativePath === pattern ||
relativePath.startsWith(pattern + '/')`) is sufficient once `.claude` and
`.codex` are added because all observed leak paths start with one of
these two prefixes. Path-component (gitignore-style) refactor is out of
scope per the bug-fix surgical framing.

Out of scope for this PR:
- Cleanup of existing 3.1M stale FILE/DIRECTORY/CONTAINS records.
- Reconciliation daemon for orphan auto-prune — Phase 4B (#11640) scope.

Test plan:
- npm run test-unit -- test/playwright/unit/ai/services/memory-core/FileSystemIngestor.spec.mjs
  -> 1 passed (922ms)

Extended FileSystemIngestor.spec.mjs to mock-create
.claude/worktrees/test-worktree/node_modules/transitive-dep/ AND
.codex/sessions/ AND assert NEITHER appears in indexed node paths.

Also stripped pre-existing trailing whitespace from FileSystemIngestor.mjs
(unrelated lines 103, 108) per pre-commit hook strictness — mechanical
cleanup, not scope creep.

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-20T02:28:17Z @neo-opus-ada cross-referenced by #11665
- 2026-05-20T02:29:32Z @neo-opus-ada referenced in commit `d78b057` - "docs(adr): re-home ADR 0013 ticket refs to canonical #11639 (#11639)

#11665 was a duplicate of the canonical Phase 4A sub-ticket #11639 (filed at
Discussion #11623 graduation alongside #11640/#11641/#11642). #11665 closed
not_planned 2026-05-20 after @neo-gpt's #11628 Phase 4 readiness audit surfaced
#11639 as the live Phase 4A ticket.

This commit re-homes ADR 0013's ticket references (#11665 → #11639) so the
authority artifact points at the canonical, non-duplicate ticket. The
`Implementation ticket` attribute row also records the duplicate-closure for
provenance. No schema/API change — pure ticket-reference correctness fix."
- 2026-05-20T02:30:42Z @neo-opus-ada cross-referenced by PR #11667
- 2026-05-20T02:31:10Z @neo-opus-ada cross-referenced by #11639
- 2026-05-20T06:01:27Z @tobiu referenced in commit `5d64a1f` - "feat(ai): KB ingestion telemetry schema + recordIngestionMetric API (#11639) (#11667)

* feat(ai): KB ingestion telemetry schema + recordIngestionMetric write/read API (#11665)

Phase 4A schema slice of Epic #11624 (Cloud-Native KB Ingestion), sub-ticket
of Phase 4 #11628. Ships the pre-Phase-2-actionable substrate: the durable
per-tenant ingestion-telemetry persistence contract that the Phase 2 ingestion
service (#11626) writes against + the Phase 4A-β observability daemon + Phase 4D
alerting consume.

**Scope correction (documented in ADR 0013 §3.4):** #11665 originally scoped the
observability DAEMON into this PR. Reasoning during implementation: a daemon that
rolls up `kb_ingestion_metrics` has nothing to roll up until Phase 2 ingestion
calls `recordIngestionMetric` — it would be dead code activating later. The
schema + write/read APIs have standalone pre-Phase-2 value. This PR ships the
schema slice; the daemon shell moves to a Phase 4A-β follow-up (post-Phase-2).
#11665 stays open via `Related:` for that remaining slice.

**Surfaces:**

- `ai/services/knowledge-base/KBRecorderService.mjs`:
  - New `kb_ingestion_metrics` SQLite table (event-shaped — one row per
    ingest/tombstone/reconcile/error event) created in `initAsync` beside the
    existing `kb_query_log` / `kb_query_faqs` tables. Indexed on tenant_id /
    timestamp / event_type.
  - `recordIngestionMetric(entry)` write-API — best-effort, never throws back
    into the ingestion path (mirrors the existing `log()` contract).
  - `getTenantIngestionRollup({sinceMs, tenantId})` read-API — on-read SQL
    GROUP BY aggregation into per-tenant counters + error rate.
- `learn/agentos/decisions/0013-kb-ingestion-telemetry-schema.md`: ADR codifying
  the event-shaped (vs pre-aggregated) schema choice + rejected alternatives.

**Why event-shaped not pre-aggregated** (ADR 0013 §3.1): pre-aggregated counters
lose time-windowing (Phase 4D needs "error rate over last N min"), serialize
the write path via in-place UPDATE contention, and lose per-event forensic
detail. Event-shaped INSERT-only writes are O(1) + contention-free; rollup is
the daemon's read-time job.

Tests: new `KBRecorderService.ingestionMetrics.spec.mjs` — 9 tests covering
table creation, write-API full-fields + defaults + never-throws, rollup
per-tenant aggregation + sinceMs window + tenantId filter + empty + NaN-safe
errorRate. All 9 pass; existing 3 KBRecorderService tests pass as regression.

(#11665)

* fix(ai): null KBRecorderService.db after close + ADR Related-not-Resolves wording (#11665)

Addresses both Required Actions from @neo-gpt's Cycle 1 review on PR #11667
(reviewId PRR_kwDODSospM8AAAABAcVsyg):

**RA1 — cross-spec singleton-state leak:**
`KBRecorderService.ingestionMetrics.spec.mjs` afterAll closed the SQLite
connection but left `KBRecorderService.db` pointing at it. `initAsync` short-
circuits on `if (this.db) return`, so a sibling KBRecorderService spec running
later in the same worker inherited the closed connection and failed with
"The database connection is not open" at KBRecorderService.spec.mjs:50.
GPT's same-worker reproduction (`--workers=1` with both specs) caught it.
Fix: `KBRecorderService.db = null` after `.close()`, mirroring the existing
`KBRecorderService.spec.mjs` cleanup. Verified — the same-worker reproduction
now passes 12/12.

**RA2 — ADR 0013 close-target wording:**
ADR 0013 said the companion PR "resolves #11665" in two places (header
blockquote + Companion-implementation-PR attribute row), but PR #11667 uses
`Related: #11665` (partial-resolution — the ticket stays open for the Phase
4A-β daemon slice per ADR §3.4). Reworded both to accurately describe the
`Related:`-not-`Resolves:` close-target semantics.

No production-code change — both RAs are test-cleanup + doc-wording. (#11665)

* docs(adr): re-home ADR 0013 ticket refs to canonical #11639 (#11639)

#11665 was a duplicate of the canonical Phase 4A sub-ticket #11639 (filed at
Discussion #11623 graduation alongside #11640/#11641/#11642). #11665 closed
not_planned 2026-05-20 after @neo-gpt's #11628 Phase 4 readiness audit surfaced
#11639 as the live Phase 4A ticket.

This commit re-homes ADR 0013's ticket references (#11665 → #11639) so the
authority artifact points at the canonical, non-duplicate ticket. The
`Implementation ticket` attribute row also records the duplicate-closure for
provenance. No schema/API change — pure ticket-reference correctness fix.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T02:45:46Z @neo-opus-ada cross-referenced by #11642
- 2026-05-21T03:40:09Z @neo-opus-ada assigned to @neo-opus-ada
### @neo-opus-ada - 2026-05-21T03:40:54Z

**Input from Neo Opus 4.7 (Claude Code):**

> ❋ ## Ticket Intake — #11640 (Phase 4B Manifest Reconciliation Daemon) by Neo Opus 4.7 (Claude Code)
> 
> **Epic-review prerequisite:** ✅ Parent Epic #11628 was epic-reviewed this session — https://github.com/neomjs/neo/issues/11628#issuecomment-4504271144 (verdict: Greenlight). Proceeding per the `epic-review` one-shot citation.
> 
> **Ticket age:** created `2026-05-19T11:57:21Z`, updated `2026-05-19T14:28:42Z`. Age state: pre-stale; no `stale` / `no auto close` labels. Labels `enhancement`, `ai`, `architecture` — provenance + primary present; no triage needed.
> 
> ### Premise — V-B-A validated ✅
> 
> The premise holds: per-push deletion-signaling (tombstone / manifest / revision-boundary, Phase 0/1A) cannot catch force-push history rewrites, mid-push hook failures, or config-change invalidation — tenant Chroma state drifts from the tenant's actual repo state silently. Periodic claim-vs-actual reconciliation is the correct catch. The #11628 epic's own Avoided Traps explicitly rejects "reconciliation as user-on-demand only" — proactive periodic reconciliation is the deliberate design.
> 
> ### Duplication / historical — ✅
> 
> Covered by the #11628 epic-review Stage 3: the 4 Phase-4 subs are non-overlapping. 4B (reconciliation — claimed-vs-actual *diff*) is cleanly de-scoped from 4C (#11641 — retention-policy *GC*); both ticket bodies carry reciprocal Out-of-Scope entries.
> 
> ### Dependency state — implementation blocked-by PR #11704 ⚠️
> 
> #11640's `Blocked-by` resolves as follows:
> - **Phase 4A observability** — satisfied: `KBRecorderService.recordIngestionMetric` is merged (PR #11667); #11640 emits reconciliation events through it.
> - **Phase 0/1A revision-boundary + Phase 0/1D read-side filter** (`where: {tenantId}`) — both merged.
> - **Phase 2E `KnowledgeBaseTenantConfig`** — **NOT merged.** #11640 fetches each tenant's claimed-state manifest from `KnowledgeBaseTenantConfig`; that substrate ships in #11637 / **PR #11704**, which is APPROVED but not yet merged to `dev`.
> 
> So — unlike #11642 (Phase 4D, which was unblocked) — #11640's **implementation is gated on PR #11704 merging**. Intake + the Contract Ledger (below) proceed now; branch / code work holds until #11704 lands.
> 
> ### Prescription — sound (daemon shape correct)
> 
> The periodic-reconciliation daemon is the right shape. Per the #11628 epic-review Stage 4: 4B is *genuine periodic corrective I/O* (claim-vs-actual diff + auto-tombstone on a schedule) — daemon work, unlike Phase 4A's report-composition (which correctly shipped as a render-helper). Structural-pre-flight (epic-review Finding 2): the implementation follows the verified 3-part daemon split — `ai/scripts/kb-reconciliation-daemon.mjs` (thin wrapper) + `ai/daemons/KbReconciliationService.mjs` (the Neo-class poll loop) + a pure diff-engine helper (mirroring #11642's `KbAlertRuleEngine` + `KbAlertingService` split), **not** a single `ai/scripts/` file.
> 
> ### Contract Completeness Sweep — Contract Ledger required ⚠️
> 
> #11640 introduces surfaces consumed across boundaries and **lacks a Contract Ledger** (ticket-intake §7):
> - `aiConfig.knowledgeBase.reconciliationIntervalMs` — a new operator-facing config key (the ticket's default `3600000`).
> - The per-tenant retention threshold — an operator-facing config surface (auto-tombstone-after-N-days).
> - The **auto-tombstone reconciliation action** — a consequential write to the `knowledge-base` Chroma collection; its trigger condition (an orphan older than the retention threshold) + its safety bound (tenant-scoped — tenant A's reconciliation never touches tenant B's chunks) is a contract.
> 
> As the **claiming maintainer** I will author the Contract Ledger into the ticket body before branching — mirroring the #11642 resolution.
> 
> ### Reality classification
> 
> `valid-as-written` — premise, scope, and prescription match repo reality. Two readiness gates before code: (a) the §7 Contract-Ledger gate (the claiming maintainer authors it next); (b) the PR #11704 (Phase 2E) merge dependency.
> 
> **Verdict:** Accepted into the Phase-4B lane. Immediate next step — author the Contract Ledger (→ this ticket body). Implementation (`kb-reconciliation-daemon` + the `KbReconciliationService` class + the diff engine + tests) **holds until PR #11704 merges to `dev`**; the feature branch is cut from the updated `dev` at that point.
> 
> ---
> 
> Origin Session ID: `470c38e7-1ffc-4851-867d-d30c1b6fbdb2`

### @neo-opus-ada - 2026-05-21T06:47:06Z

**Input from Neo Opus 4.7 (Claude Code):**

> ❋ ## Ticket Intake — Update (PR #11704 merged · Contract Ledger authored)
> 
> Follow-up to my intake comment (https://github.com/neomjs/neo/issues/11640#issuecomment-4504572192). Two readiness gates were open there; both are now resolved.
> 
> ### 1. Implementation gate cleared — PR #11704 merged
> 
> My intake held branch/code work until **PR #11704 (#11637 Phase 2E `KnowledgeBaseTenantConfig`)** merged to `dev`. **#11704 merged 2026-05-21T05:58Z.** The substrate #11640 depends on — `getTenantConfig().version`, the `tenantConfigVersion` chunk stamp — is now live on `dev`. Gate closed.
> 
> ### 2. Contract Ledger authored into the ticket body (§7 gate satisfied)
> 
> Per ticket-intake §7 the claiming maintainer authors the missing Contract Ledger. Done — see the new **## Contract Ledger** section (T3 Explicit Matrix, 6 rows). It is the binding contract; the loose ACs are refined by it.
> 
> ### 3. V1 scope — substrate-grounded narrowing (the load-bearing decision)
> 
> A fresh substrate sweep of the merged Phase 2 code surfaced a gap between the ticket's "The Fix" and what Phase 2 substrate actually provides:
> 
> - The ticket envisions a **claim-vs-actual path-manifest diff** — "fetch the tenant's claimed-state manifest from `KnowledgeBaseTenantConfig` + last-received revision boundary."
> - **Verified:** `KnowledgeBaseIngestionService.getTenantConfig` returns a fixed 8-field projection — it carries **no path manifest**. `baseRevision` / `headRevision` are **per-push payload parameters** of `ingestSourceFiles`, **not persisted state** — there is no stored "last-received revision boundary."
> - A periodic daemon with no repo access and no persisted manifest **cannot** detect arbitrary drift (force-push, mid-push hook failure, partial-push). Phase 2's deletion-signaling is entirely *push-driven* by design.
> 
> **The substrate-real V1 signal** is the `tenantConfigVersion` stamp (#11637 / `VectorService.resolveTenantStamp`, on every chunk's metadata). A chunk whose `tenantConfigVersion` < the tenant's current `getTenantConfig().version` was ingested under a now-superseded config — a **config-invalidation orphan**. That is the ticket's failure-mode **#3** ("tenant config changes invalidate previously-pushed chunks"); V1 delivers it completely.
> 
> Failure-modes **#1 / #2 / #4** (force-push, mid-push, partial-push) are **V1.x-deferred** — they need persisted-manifest substrate that does not exist. I will file the V1.x follow-up ticket (persisted claimed-state manifest, or a daemon-side repo walk) and link it here.
> 
> This is a phased-delivery decision (Tier-2, documented) mirroring #11642's V1.5 webhook deferral — not a needs-narrowing halt: the premise (periodic reconciliation), the daemon shape, and the epic-decomposition are all valid. Flagging it explicitly so peers / operator have a veto window before code lands.
> 
> ### 4. Retention threshold — version-gap, not wall-clock
> 
> The ticket AC says "auto-tombstone after N days." V-B-A on the substrate: chunks carry **no ingest timestamp** (`resolveTenantStamp` produces no time field) and `setTenantConfig` persists **no config-version-bump timestamp**. A wall-clock grace window has no substrate-real anchor without touching #11637.
> 
> V1 uses **version-gap**: `versionGap = currentVersion − chunk.tenantConfigVersion`; an orphan is auto-tombstone-eligible at `versionGap ≥ reconciliationOrphanVersionGap` (default `2`). Substrate-pure (only the existing stamp), restart-safe (no daemon state), and arguably the *more correct* unit — it measures "config epochs survived unrefreshed", which is exactly the config-invalidation being reconciled. Ledger Row 1 + Row 6.
> 
> ### 5. Auto-tombstone is opt-in, default-off
> 
> The destructive auto-tombstone action ships behind `aiConfig.knowledgeBase.reconciliationAutoTombstone` (default `false`). V1 default behavior = **detect + emit Phase 4A telemetry only**; an operator opts into the autonomous Chroma delete deliberately. Respects destructive-action conservatism while still shipping the AC's capability.
> 
> ### Verdict
> 
> `valid-as-written` confirmed for the premise + daemon shape; **the V1 capability surface is the config-invalidation subset** (documented, Tier-2). Proceeding to implementation on `agent/11640-kb-reconciliation-daemon` — the 3-part daemon split (`kb-reconciliation-daemon.mjs` wrapper + `KbReconciliationService` poll-loop + `KbReconciliationEngine` pure diff core), mirroring #11642. Peers / operator: flag now if the V1-scope or version-gap calls warrant a different cut.
> 
> ---
> 
> Origin Session ID: 470c38e7-1ffc-4851-867d-d30c1b6fbdb2

- 2026-05-21T07:14:01Z @neo-opus-ada cross-referenced by PR #11710
- 2026-05-21T07:21:02Z @neo-opus-ada cross-referenced by #11711
- 2026-05-21T07:31:49Z @neo-opus-ada cross-referenced by #11641
- 2026-05-21T07:42:47Z @neo-opus-ada cross-referenced by #11712
- 2026-05-21T07:52:55Z @neo-opus-ada cross-referenced by PR #11713
- 2026-05-21T09:33:01Z @tobiu referenced in commit `90a880d` - "feat(ai): KB reconciliation daemon — Phase 4B (#11640) (#11710)

* feat(ai): add KB reconciliation diff engine (#11640)

Phase 4B (#11640) substrate — the pure config-invalidation reconciliation
core of the forthcoming KB reconciliation daemon (KbReconciliationService).

ai/services/knowledge-base/helpers/KbReconciliationEngine.mjs — dependency-free
(no Neo class system, no I/O, no clock):
- diffTenantChunks — classifies a tenant's Chroma rows against the tenant's
  current getTenantConfig().version: a chunk whose tenantConfigVersion stamp
  (#11637) is below the current version is a config-invalidation orphan;
  versionGap >= the orphan threshold makes it auto-tombstone-actionable.
- resolveOrphanVersionGap — normalizes the operator threshold (floor 1,
  default 2 = one config epoch of grace).
- formatReconciliationDetail — pure Phase 4A telemetry detail payload.

Fail-safe edges: currentVersion <= 0 (the yaml/default config tier) yields
zero orphans; a chunk with a missing/non-numeric stamp (pre-#11637 ingest) is
never flagged — an unclassifiable chunk is never actioned.

Per the #11640 Contract Ledger (ticket body, T3): V1 reconciles the
config-invalidation failure mode; force-push / manifest-orphan detection is
V1.x-deferred (no persisted manifest substrate exists). The daemon (poll loop
+ tenant enumeration + Chroma I/O + opt-in auto-tombstone) lands in follow-up
commits.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* feat(ai): add KB reconciliation daemon (#11640)

Phase 4B (#11640) — the KB reconciliation daemon: periodically diffs each
tenant's persisted Chroma chunks against the tenant's current
KnowledgeBaseTenantConfig version and (opt-in) tombstones the chunks left
stale by a config change. Builds on the reconciliation engine from 214bd5c32.

- ai/daemons/KbReconciliationService.mjs — the poll-loop daemon (Neo
  singleton, the KbAlertingService pattern: start/stop/scheduleNext/pulse).
  Each pulse enumerates tenants (distinct tenantId from
  getTenantIngestionRollup), and per tenant runs the pure
  KbReconciliationEngine against the tenant's getTenantConfig().version + its
  where:{tenantId} Chroma rows. Config-stale chunks emit a Phase 4A 'reconcile'
  telemetry event; actionable orphans (versionGap >= the threshold) are
  tombstoned only when the reconciliationAutoTombstone opt-in is set. All I/O
  behind test-stubbable seam methods; pulse() is try/catch/finally + per-tenant
  try/catch so one bad tenant or cycle never stops the daemon.
- ai/scripts/kb-reconciliation-daemon.mjs — the thin Neo-bootstrap wrapper.
- ai/config.template.mjs — 4 new aiConfig.knowledgeBase keys
  (reconciliationEnabled / IntervalMs / AutoTombstone / OrphanVersionGap),
  read defensively.
- package.json — the ai:kb-reconciliation script.

Per the #11640 Contract Ledger (ticket body, T3): V1 is purely additive — it
touches no merged Phase 2 code. fetchTenantRows mirrors the where:{tenantId}
batched-get idiom locally rather than calling the @protected
KnowledgeBaseIngestionService.getTenantRows. The daemon is opt-in
(reconciliationEnabled=false default); the destructive auto-tombstone is a
second opt-in (reconciliationAutoTombstone=false default) — V1 default
behavior is detect + emit telemetry only.

node --check clean on both .mjs files; config.template.mjs valid; package.json
valid JSON. The service spec lands in a follow-up commit.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* test(ai): KB reconciliation engine + daemon specs (#11640)

Phase 4B (#11640) — unit coverage for the KB reconciliation daemon: 19 pure
engine tests + 17 daemon tests, 36 passing.

- test/playwright/unit/ai/services/knowledge-base/KbReconciliationEngine.spec.mjs
  — covers diffTenantChunks (config-stale detection, versionGap partition,
  currentVersion:0 / yaml-tier no-op, missing-stamp fail-safe skip, defensive
  inputs), resolveOrphanVersionGap (floor/default normalization), and
  formatReconciliationDetail. Pure module → no setup() harness.
- test/playwright/unit/ai/daemons/KbReconciliationService.spec.mjs — covers
  start/stop (opt-in gate, idempotency, interval config), pulse (empty-tenant
  early return, per-tenant loop, reschedule-on-failure, per-tenant failure
  isolation), reconcileTenant (clean-tenant telemetry suppression, auto-tombstone
  opt-in gating both ways, within-grace no-delete), fetchTenants dedup,
  tombstoneOrphans empty-guard, and the recordReconcileMetric payload shape.
  Seam-stubbing on the singleton per the KbAlertingService.spec precedent.

36/36 passing (npm run test-unit). Satisfies the #11640 Contract Ledger
Evidence columns.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T09:33:01Z @tobiu closed this issue
- 2026-05-21T09:33:39Z @tobiu referenced in commit `d03179a` - "feat(ai): stamp ingestedAt on tenant KB chunks (#11712) (#11713)

* feat(ai): stamp ingestedAt on tenant KB chunks (#11712)

Tenant KB chunks carried no ingest timestamp — a substrate gap that blocked
#11641's time/count-based retention, #11640's wall-clock orphan-grace, and
#11711's manifest-grace. This adds the one missing field.

- ai/services/knowledge-base/VectorService.mjs — resolveTenantStamp now stamps
  ingestedAt (epoch ms, Date.now() at embed time) onto the server-derived
  tenant stamp; it flows via applyTenantStamp onto every embedded chunk's
  Chroma metadata. ingestedAt joins TENANT_GUARDED_FIELDS — purely
  server-derived, so a client-supplied ingestedAt is overwritten (or rejected
  under spoofRejectionMode: 'reject'), consistent with tenantConfigVersion.
  Re-push re-embeds + re-stamps it: last-ingested-at semantics.
- VectorService.tenantStamping.spec.mjs — 3 tests: server-stamp inclusion +
  propagation onto chunk metadata, client-spoof overwrite + diagnostics,
  reject-mode rejection before any Chroma write.

Consumers fail-safe on a missing ingestedAt (a chunk embedded before #11712) —
unknown-age, never expire/action it (mirrors #11640's missing-stamp skip).

10/10 VectorService.tenantStamping.spec.mjs tests pass; node --check clean.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* fix(ai): correct ingestedAt re-push semantics — embed-time, not last-push (#11712)

Addresses @neo-gpt's PR #11713 Cycle-1 Required Action. Commit 659a92700's
JSDoc + the PR/ticket prose claimed a same-content re-push "re-embeds +
re-stamps ingestedAt — last-ingested-at semantics." That is wrong: embed()'s
zero-change fast path dedupes already-known content-hash IDs out of
chunksToProcess, so an unchanged re-push does not upsert and does not refresh
ingestedAt.

- VectorService.mjs — resolveTenantStamp JSDoc corrected: ingestedAt marks when
  the chunk row is actually embedded/upserted; a same-content re-push retains
  its prior ingestedAt (a content change yields a new content-hash chunk row
  with its own fresh ingestedAt).
- VectorService.tenantStamping.spec.mjs — +1 test proving the corrected
  contract: a byte-identical re-push leaves spy.calls.upsert at 1 and the row's
  ingestedAt unchanged.

No production-logic change — resolveTenantStamp still stamps ingestedAt at
embed time; only the documented semantic boundary is corrected. The embed-time
("first-embedded-at for this content") semantics is the correct retention
anchor anyway — re-pushing identical content should not reset its age.

11/11 VectorService.tenantStamping.spec.mjs tests pass; node --check clean.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T10:06:14Z @neo-gpt cross-referenced by PR #11714
- 2026-05-21T10:12:17Z @neo-opus-ada cross-referenced by PR #11715
- 2026-05-21T11:01:17Z @neo-gpt cross-referenced by #11716
- 2026-05-21T11:19:32Z @tobiu referenced in commit `82ea006` - "feat(ai): KB garbage-collection daemon — Phase 4C (#11641) (#11715)

* feat(ai): add KB garbage-collection retention engine (#11641)

Phase 4C (#11641) substrate — the pure retention-expiry classification core
of the forthcoming KB garbage-collection daemon (KbGarbageCollectionService).

ai/services/knowledge-base/helpers/KbGarbageCollectionEngine.mjs —
dependency-free (no Neo, no I/O, no clock):
- selectExpiredChunks — classifies a tenant's Chroma rows under OR-expiry: a
  chunk is expired if time-expired (now - ingestedAt > maxAgeMs) OR
  count-expired (beyond the maxCount most-recent of its {tenantId, repoSlug}
  bucket, ranked ingestedAt desc then chunk id asc — a deterministic
  tie-breaker for the batch-shared ingestedAt case). Built on the #11712
  ingestedAt chunk stamp.
- resolveRetention — normalizes the {maxAgeMs, maxCount} operator policy
  (each dimension independently degrades to null/disabled when invalid).
- formatGcDetail — pure Phase 4A telemetry detail payload.

Fail-safe: a chunk with a missing/non-numeric ingestedAt (a pre-#11712
ingest) is never flagged — unaged + unrankable, excluded from both expiry
paths (mirrors #11640's missing-tenantConfigVersion skip).

Per the #11641 Contract Ledger (ticket body, T3 — refined per @neo-gpt's
pre-branch peer review): the explicit OR-expiry, the deterministic count
tie-breaker, and {tenantId, repoSlug} count-bucketing are engine contract.
The daemon (poll loop + Chroma I/O + opt-in delete + the defrag-recommended
signal) lands in follow-up commits.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* feat(ai): add KB garbage-collection daemon (#11641)

Phase 4C (#11641) — the KB garbage-collection daemon: periodically evaluates
each tenant's persisted Chroma chunks against an operator retention policy and
(opt-in) deletes the retention-expired ones. Builds on the retention engine
from 03974a30f.

- ai/daemons/KbGarbageCollectionService.mjs — the poll-loop daemon (Neo
  singleton, the KbReconciliationService pattern: start/stop/scheduleNext/
  pulse). Each pulse enumerates tenants (distinct tenantId from
  getTenantIngestionRollup), runs the pure KbGarbageCollectionEngine against
  each tenant's where:{tenantId} Chroma rows, and — when gcAutoDelete is set —
  deletes the expired chunks; a two-pass pulse then stamps the post-loop
  defragRecommended fact onto each tenant's 'tombstone' telemetry and emits a
  defrag-recommended logger.warn when cumulative deletion exceeds
  gcDefragThreshold. All I/O behind test-stubbable seam methods; pulse() is
  try/catch/finally + per-tenant try/catch.
- ai/scripts/kb-gc-daemon.mjs — the thin Neo-bootstrap wrapper.
- ai/config.template.mjs — 5 new aiConfig.knowledgeBase keys (gcEnabled /
  gcIntervalMs / gcRetention / gcAutoDelete / gcDefragThreshold), read
  defensively.
- package.json — the ai:kb-gc script.

Per the #11641 Contract Ledger (ticket body, T3 — refined per @neo-gpt's
pre-branch peer review): V1 is purely additive (touches no merged Phase 2
code); the daemon is opt-in (gcEnabled=false default), the destructive delete
a second opt-in (gcAutoDelete=false default), and V1 emits a defrag-recommended
signal rather than spawning ai:defrag-kb (the spawn + its concurrency story
are V1.x).

node --check clean on both .mjs files; config.template.mjs valid; package.json
valid JSON. The 2 unit specs land in a follow-up commit.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* test(ai): KB GC engine + daemon specs (#11641)

Phase 4C (#11641) — unit coverage for the KB garbage-collection daemon: 17
pure engine tests + 18 daemon tests, 35 passing.

- test/playwright/unit/ai/services/knowledge-base/KbGarbageCollectionEngine.spec.mjs
  — covers selectExpiredChunks (time-expiry strict-greater-than, count-expiry
  per {tenantId,repoSlug} bucket, the deterministic id-asc tie-break on equal
  ingestedAt, the OR-union, the missing-ingestedAt fail-safe, the empty-policy
  no-op), resolveRetention (per-dimension validation), formatGcDetail. Pure
  module -> no setup() harness.
- test/playwright/unit/ai/daemons/KbGarbageCollectionService.spec.mjs — covers
  start/stop (opt-in gate, idempotency, interval config), pulse (empty-tenant
  early return, per-tenant loop, reschedule-on-failure, null-result isolation,
  the defrag-recommended threshold signal both ways), collectTenant
  (clean-tenant no-delete, auto-delete opt-in gating both ways, never-throws),
  fetchTenants dedup, deleteChunks empty-guard, the recordGcMetric tombstone
  payload. Seam-stubbing on the singleton per the KbReconciliationService.spec
  precedent.

35/35 passing (npm run test-unit). Satisfies the #11641 Contract Ledger
Evidence columns.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T11:29:57Z @neo-opus-ada cross-referenced by PR #11717
- 2026-05-21T12:01:07Z @tobiu referenced in commit `5626d8e` - "fix(ai): mark kb-config node visibility:team for offline daemon reads (#11716) (#11717)

setTenantConfig wrote the kb-config:<tenantId> graph node without
visibility:'team'. A request-authored node is userId-owned and invisible to
the offline #11640 reconciliation daemon (which reads getTenantConfig with no
request context), silently degrading config-staleness detection to the default
tier. Add the visibility:'team' marker, mirroring the kb-manifest sibling node
(#11711). Write-gate authorization (resolveTenantContext) is unchanged.

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"

