---
id: 11641
title: 'Phase 4C — Stale-Chunk Garbage Collection Daemon: Orphan Detection + Retention Enforcement'
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
assignees:
  - neo-opus-ada
createdAt: '2026-05-19T11:57:33Z'
updatedAt: '2026-06-07T17:13:40Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11641'
author: neo-opus-ada
commentsCount: 3
parentIssue: 11628
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy:
  - '[x] 11712 Add a server-stamped ingestedAt timestamp to tenant KB chunk metadata'
  - '[x] 11640 Phase 4B — Manifest Reconciliation Daemon: Tenant-State vs Chroma-Actual Sync'
blocking: []
closedAt: '2026-05-21T11:19:32Z'
---
# Phase 4C — Stale-Chunk Garbage Collection Daemon: Orphan Detection + Retention Enforcement

## Context

Sub of Phase 4 Epic [#11628](https://github.com/neomjs/neo/issues/11628) (meta-Epic [#11624](https://github.com/neomjs/neo/issues/11624)).

**Stale-chunk garbage collection** — distinct from reconciliation (Phase 4B). Reconciliation diffs claimed-vs-actual; GC enforces RETENTION POLICY (time-based, count-based, or version-based expiration).

## The Problem

Without active GC:

- Tombstoned chunks accumulate Chroma collection size (Chroma soft-deletes; physical reclaim needs explicit GC)
- Source-config changes (parser swap, source-type removed) orphan chunks that no longer match tenant config
- Per-tenant retention policies (e.g., "retain only last 90 days of historical chunks") need enforcement
- ChromaDB defrag patterns (`ai:defrag-kb` precedent: 10s wall / 321→189MB / 41% reduction on ~10k chunks) — but defrag is operator-triggered, not automatic per-tenant

## The Fix

New daemon: `ai/scripts/kb-gc-daemon.mjs` (sibling to existing daemons).

Per scheduled tick (configurable; default daily):

1. **For each active tenant**:
   - Identify retention-expired chunks (per `aiConfig.knowledgeBase.tenantRetention` policy)
   - Identify config-orphaned chunks (source-config change leaves chunks unmatched)
   - Identify reconciliation-tombstoned chunks past grace period (Phase 4B integration)
2. **Garbage collection actions**:
   - Delete from Chroma (via `VectorService.delete({ids})`)
   - Emit observability events (Phase 4A)
   - Trigger `ai:defrag-kb` if cumulative deletion > threshold (10% collection size)
3. **Cross-tenant safety**: tenant A's GC cannot touch tenant B's chunks (RLS via Phase 0/1D read-side filter)

## Acceptance Criteria

- [ ] `ai/scripts/kb-gc-daemon.mjs` exists; follows existing daemon pattern
- [ ] Retention policy structure: per-tenant config (time-based / count-based / version-based)
- [ ] Config-orphan detection: chunks whose source-config no longer matches tenant config
- [ ] Tombstone-grace-period integration with Phase 4B
- [ ] Defrag trigger when cumulative deletion > threshold (default 10% collection size)
- [ ] Cross-tenant safety: GC respects RLS; tenant A cannot affect tenant B
- [ ] Configurable tick interval (`aiConfig.knowledgeBase.gcIntervalMs`; default 86400000 = 24h)
- [ ] Unit tests: retention enforcement, orphan detection, RLS isolation
- [ ] Integration test: tenant pushes content, retention expires, GC removes, defrag triggers

## Out of Scope

- Reconciliation logic → Phase 4B
- Observability event collection → Phase 4A
- Initial defrag substrate (`ai:defrag-kb` already exists; this daemon TRIGGERS it conditionally)

## Contract Ledger

> Added at intake by @neo-opus-ada (Claude Code) 2026-05-21 — satisfies the ticket-intake §7 Contract Completeness gate. The original-author session is inactive; per §7 the claiming maintainer authors the ledger. Tier target: **T3 (Explicit Matrix)**. The ledger is the binding contract; the loose Acceptance Criteria above are refined by these rows.
>
> **V1 scope.** V1 = retention-policy chunk expiration + physical-reclaim signalling. The intake scope-refinement dropped config-orphan detection — it overlaps #11640's config-invalidation reconciliation (the same `tenantConfigVersion` signal). Time- and count-based retention are substrate-backed by the `ingestedAt` chunk stamp (#11712, merged to `dev`). Full design rationale: the intake-update comment https://github.com/neomjs/neo/issues/11641#issuecomment-4506820763.
>
> **Refined 2026-05-21** per @neo-gpt's #11641 pre-branch peer review: explicit OR-expiry semantics, a deterministic count-ordering tie-breaker, `{tenantId, repoSlug}` count-bucketing, and V1 emits a `defrag-recommended` signal rather than spawning `ai:defrag-kb` (Rows 2 + 4 + 6).

| Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
|---|---|---|---|---|---|
| `aiConfig.knowledgeBase` GC config block — 5 new keys | #11628 Phase 4C; this ticket AC; #11640 / #11642's `aiConfig.knowledgeBase` precedent | `gcEnabled` (Boolean, default `false`) — master opt-in; the daemon exits early when false. `gcIntervalMs` (Number, default `86400000` = 24h) — poll-tick interval. `gcRetention` (Object, default `{}`) — `{maxAgeMs?, maxCount?}` retention policy. `gcAutoDelete` (Boolean, default `false`) — opt-in for the destructive Chroma delete; default-off ⇒ detect + emit telemetry only. `gcDefragThreshold` (Number, default `0.10`) — the cumulative-deletion fraction above which the daemon emits a `defrag-recommended` signal; `0` disables the signal. | A stale gitignored `config.mjs` predating #11641 lacks the `gc*` keys → each read defensively against its default (the #11640 / #11642 defensive-read pattern). An empty `gcRetention {}` ⇒ no chunk is ever retention-expired (conservative — the ticket's "default conservative" handoff hint). | Yes — `ai/config.template.mjs` block + JSDoc | Unit: config-defaulting + the daemon opt-in gate |
| Retention-expiry classification — `KbGarbageCollectionEngine` pure core | #11712 `ingestedAt` chunk stamp; this ticket's retention AC; @neo-gpt #11641 peer review | The pure, dependency-free classifier (mirrors #11640's `KbReconciliationEngine`). `selectExpiredChunks({rows, retention, now})` → a chunk is retention-expired under **OR-expiry**: expired if **time-expired OR count-expired** (the union — the broader set). **Time-expiry:** `typeof metadata.ingestedAt === 'number' && now − ingestedAt > maxAgeMs`. **Count-expiry:** rows are bucketed by `{tenantId, repoSlug}`, each bucket sorted **`ingestedAt` desc, then chunk `id` asc** (a deterministic tie-breaker — batch-ingested chunks share an `ingestedAt`); a chunk ranked at or beyond `maxCount` within its bucket is count-expired. Returns `{expiredIds, expiredCount, evaluatedCount}`. No I/O, no clock — the caller passes `now`. | A chunk with a missing / non-numeric `ingestedAt` (a pre-#11712 ingest) is **never** flagged — fail-safe for *both* time (age uncomputable) and count (unrankable → excluded from the expired set), mirroring #11640's missing-`tenantConfigVersion` skip. An empty / absent `retention` policy → empty result. | Yes — JSDoc | Unit: time-expiry, count-expiry per `{tenantId, repoSlug}` bucket, the OR-union, the deterministic tie-break on equal `ingestedAt`, missing-`ingestedAt` skip, empty-policy no-op |
| The destructive GC delete — `knowledge-base` Chroma collection delete | this ticket AC ("GC removes"); the destructive-action conservatism principle | When `gcAutoDelete` is `true`, the daemon deletes a tenant's `expiredIds` via `collection.delete({ids})`. **Tenant-scoped** — `expiredIds` derive only from rows fetched with `where: {tenantId}` (the `getTenantRows` batched-`collection.get` pattern); tenant A's GC never touches tenant B's chunks (the ticket's RLS-safety AC). | `gcAutoDelete` is `false` (the default) → **no delete is ever issued**; the daemon detects + emits telemetry only. A `collection.delete` throw → `logger.error`, best-effort; the daemon continues to the next tenant. | Yes — daemon JSDoc | Unit: delete gated by the opt-in flag; tenant-scoped id set; delete-throw tolerance |
| Defrag-recommended signal — physical-reclaim observability | this ticket AC ("trigger defrag"); @neo-gpt #11641 peer review | When a tick's cumulative deletion (summed across tenants) exceeds `gcDefragThreshold` of the collection's chunk count, the daemon emits a **`defrag-recommended`** signal — a `logger.warn` plus a telemetry `detail` flag — surfacing that an operator should run `ai:defrag-kb`. **V1 does not spawn `ai:defrag-kb`** — see Row 6. | `gcDefragThreshold` is `0` → no signal. The daemon spawns no subprocess in V1 → there is no defrag-vs-ingest concurrency surface. | Yes — daemon JSDoc | Unit: the signal fires when cumulative deletion exceeds the threshold, stays silent below it |
| Phase 4A telemetry emission — `KBRecorderService.recordIngestionMetric` | #11639 Phase 4A; `recordIngestionMetric`'s `'tombstone'` event type | When a tenant has ≥ 1 retention-expired chunk, the daemon emits one `recordIngestionMetric({tenantId, repoSlug, eventType: 'tombstone', chunksTotal: expiredCount, chunksDeleted: <count deleted this tick — 0 when gcAutoDelete is off>, detail: {expiredCount, deletedCount, retention, gcAutoDelete, defragRecommended}})`. A clean tenant (zero expired) emits nothing. | `recordIngestionMetric` is best-effort (never throws into the caller). A GC delete is a `'tombstone'`-class event — `recordIngestionMetric`'s taxonomy has no dedicated `'gc'` type; `'tombstone'` is the honest fit (a logical deletion). | Yes — JSDoc | Unit: a `tombstone` metric is emitted for a tenant with expired chunks, suppressed for a clean one |
| V1 scope boundary — config-orphan detection · per-tenant retention override · auto-defrag spawn | this ticket "The Fix" / ACs; the intake scope-refinement; @neo-gpt #11641 peer review | **Documented V1 deltas.** (a) **Config-orphan detection** is dropped — #11640's `KbReconciliationService` already detects + opt-in-tombstones config-stale chunks; 4C re-detecting them is double-handling (the intake de-dup). (b) **Per-tenant retention override** — V1 applies one global `gcRetention` per tenant; a per-tenant *override* needs extending `getTenantConfig`'s fixed projection (#11637 surface) → V1.x. (c) **Auto-defrag spawn** — V1 emits a `defrag-recommended` *signal* only (Row 4); the automated `ai:defrag-kb` *spawn* + its defrag-vs-ingest concurrency-coordination story are V1.x (auto-spawning a whole-collection nuke-and-pave from a poll-loop daemon is a separable, concurrency-sensitive design). | V1.x is a separate follow-up ticket; the PR body "Deltas" documents each. | Yes — PR body "Deltas" + a V1.x follow-up | N/A — explicit scope boundary |

## Prior Art / Defrag-Backup Substrate Cross-References

Substrate-correct V-B-A calibration 2026-05-19: per [#10129](https://github.com/neomjs/neo/issues/10129) Phase 3 peer architecture, `defragChromaDB.mjs` and `backup.mjs` are peer scripts with orthogonal responsibilities, NOT delegates. Phase 4C extends/triggers existing defrag substrate:

- **`buildScripts/ai/defragChromaDB.mjs`** — 5-step "Nuke and Pave": (1) Pre-Nuke Snapshot via `fs.copy()` to `dist/chromadb-backups/<target>/backup-<numeric-ts>/` (HNSW state preserved); (2) Extract all collections to in-memory; (3) Nuke collections via API; (4) Load (recreate + reinsert; forces HNSW rebuild); (5) Cleanup orphan UUID directories. Existing retention: keep last 3, delete others older than 7 days.
- **`buildScripts/ai/backup.mjs`** — JSONL bundle peer. Operators chain `ai:defrag-kb && ai:backup` for compacted bundles. Phase 4C daemon triggers defrag automatically based on cumulative-deletion threshold.
- **Tenant-aware GC requires read-side filter** — Phase 0/1D's `where: {tenantId}` filter on `QueryService` ensures cross-tenant safety; the GC daemon enumerates orphans WITHIN a tenant's scope, never cross-tenant.
- **Existing test substrate to extend, not duplicate**: `test/playwright/unit/ai/buildScripts/backup.spec.mjs` (defrag-trigger logic; retention pattern); KB `DatabaseService.backup.spec.mjs` (export/import lifecycle).

## Related

- **Parent:** [#11628](https://github.com/neomjs/neo/issues/11628)
- **Blocked-by:** Phase 4A (observability), Phase 4B (reconciliation tombstones)
- **Existing defrag substrate:** `npm run ai:defrag-kb` (memory anchor: 10s wall / 321→189MB / 41% reduction on ~10k chunks); peer script `buildScripts/ai/defragChromaDB.mjs`
- **Related substrate**: [#10129](https://github.com/neomjs/neo/issues/10129) atomic-bundle backup peer architecture
- **RLS substrate:** Phase 0/1D read-side filter (cross-tenant GC safety)

## Origin Session ID

`7360e917-1733-4cdd-a6f3-5ac51c34b838`

## Handoff Retrieval Hints

- `ai:defrag-kb` script is the existing defrag substrate to integrate with
- Retention policies are deployment-specific; default conservative (long retention) acceptable for V1

## Timeline

- 2026-05-19T11:57:34Z @neo-opus-ada added the `enhancement` label
- 2026-05-19T11:57:35Z @neo-opus-ada added the `ai` label
- 2026-05-19T11:57:35Z @neo-opus-ada added the `architecture` label
- 2026-05-19T11:59:20Z @neo-opus-ada added parent issue #11628
- 2026-05-19T11:59:48Z @neo-opus-ada marked this issue as being blocked by #11640
- 2026-05-19T12:01:25Z @neo-opus-ada cross-referenced by #11624
- 2026-05-19T14:28:43Z @neo-opus-ada cross-referenced by #11640
- 2026-05-19T14:40:31Z @neo-opus-ada cross-referenced by #11628
- 2026-05-19T14:44:07Z @neo-opus-ada cross-referenced by #11649
- 2026-05-20T02:28:17Z @neo-opus-ada cross-referenced by #11665
- 2026-05-20T02:29:32Z @neo-opus-ada referenced in commit `d78b057` - "docs(adr): re-home ADR 0013 ticket refs to canonical #11639 (#11639)

#11665 was a duplicate of the canonical Phase 4A sub-ticket #11639 (filed at
Discussion #11623 graduation alongside #11640/#11641/#11642). #11665 closed
not_planned 2026-05-20 after @neo-gpt's #11628 Phase 4 readiness audit surfaced
#11639 as the live Phase 4A ticket.

This commit re-homes ADR 0013's ticket references (#11665 → #11639) so the
authority artifact points at the canonical, non-duplicate ticket. The
`Implementation ticket` attribute row also records the duplicate-closure for
provenance. No schema/API change — pure ticket-reference correctness fix."
- 2026-05-20T02:30:42Z @neo-opus-ada cross-referenced by PR #11667
- 2026-05-20T02:31:10Z @neo-opus-ada cross-referenced by #11639
- 2026-05-20T06:01:27Z @tobiu referenced in commit `5d64a1f` - "feat(ai): KB ingestion telemetry schema + recordIngestionMetric API (#11639) (#11667)

* feat(ai): KB ingestion telemetry schema + recordIngestionMetric write/read API (#11665)

Phase 4A schema slice of Epic #11624 (Cloud-Native KB Ingestion), sub-ticket
of Phase 4 #11628. Ships the pre-Phase-2-actionable substrate: the durable
per-tenant ingestion-telemetry persistence contract that the Phase 2 ingestion
service (#11626) writes against + the Phase 4A-β observability daemon + Phase 4D
alerting consume.

**Scope correction (documented in ADR 0013 §3.4):** #11665 originally scoped the
observability DAEMON into this PR. Reasoning during implementation: a daemon that
rolls up `kb_ingestion_metrics` has nothing to roll up until Phase 2 ingestion
calls `recordIngestionMetric` — it would be dead code activating later. The
schema + write/read APIs have standalone pre-Phase-2 value. This PR ships the
schema slice; the daemon shell moves to a Phase 4A-β follow-up (post-Phase-2).
#11665 stays open via `Related:` for that remaining slice.

**Surfaces:**

- `ai/services/knowledge-base/KBRecorderService.mjs`:
  - New `kb_ingestion_metrics` SQLite table (event-shaped — one row per
    ingest/tombstone/reconcile/error event) created in `initAsync` beside the
    existing `kb_query_log` / `kb_query_faqs` tables. Indexed on tenant_id /
    timestamp / event_type.
  - `recordIngestionMetric(entry)` write-API — best-effort, never throws back
    into the ingestion path (mirrors the existing `log()` contract).
  - `getTenantIngestionRollup({sinceMs, tenantId})` read-API — on-read SQL
    GROUP BY aggregation into per-tenant counters + error rate.
- `learn/agentos/decisions/0013-kb-ingestion-telemetry-schema.md`: ADR codifying
  the event-shaped (vs pre-aggregated) schema choice + rejected alternatives.

**Why event-shaped not pre-aggregated** (ADR 0013 §3.1): pre-aggregated counters
lose time-windowing (Phase 4D needs "error rate over last N min"), serialize
the write path via in-place UPDATE contention, and lose per-event forensic
detail. Event-shaped INSERT-only writes are O(1) + contention-free; rollup is
the daemon's read-time job.

Tests: new `KBRecorderService.ingestionMetrics.spec.mjs` — 9 tests covering
table creation, write-API full-fields + defaults + never-throws, rollup
per-tenant aggregation + sinceMs window + tenantId filter + empty + NaN-safe
errorRate. All 9 pass; existing 3 KBRecorderService tests pass as regression.

(#11665)

* fix(ai): null KBRecorderService.db after close + ADR Related-not-Resolves wording (#11665)

Addresses both Required Actions from @neo-gpt's Cycle 1 review on PR #11667
(reviewId PRR_kwDODSospM8AAAABAcVsyg):

**RA1 — cross-spec singleton-state leak:**
`KBRecorderService.ingestionMetrics.spec.mjs` afterAll closed the SQLite
connection but left `KBRecorderService.db` pointing at it. `initAsync` short-
circuits on `if (this.db) return`, so a sibling KBRecorderService spec running
later in the same worker inherited the closed connection and failed with
"The database connection is not open" at KBRecorderService.spec.mjs:50.
GPT's same-worker reproduction (`--workers=1` with both specs) caught it.
Fix: `KBRecorderService.db = null` after `.close()`, mirroring the existing
`KBRecorderService.spec.mjs` cleanup. Verified — the same-worker reproduction
now passes 12/12.

**RA2 — ADR 0013 close-target wording:**
ADR 0013 said the companion PR "resolves #11665" in two places (header
blockquote + Companion-implementation-PR attribute row), but PR #11667 uses
`Related: #11665` (partial-resolution — the ticket stays open for the Phase
4A-β daemon slice per ADR §3.4). Reworded both to accurately describe the
`Related:`-not-`Resolves:` close-target semantics.

No production-code change — both RAs are test-cleanup + doc-wording. (#11665)

* docs(adr): re-home ADR 0013 ticket refs to canonical #11639 (#11639)

#11665 was a duplicate of the canonical Phase 4A sub-ticket #11639 (filed at
Discussion #11623 graduation alongside #11640/#11641/#11642). #11665 closed
not_planned 2026-05-20 after @neo-gpt's #11628 Phase 4 readiness audit surfaced
#11639 as the live Phase 4A ticket.

This commit re-homes ADR 0013's ticket references (#11665 → #11639) so the
authority artifact points at the canonical, non-duplicate ticket. The
`Implementation ticket` attribute row also records the duplicate-closure for
provenance. No schema/API change — pure ticket-reference correctness fix.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T07:21:02Z @neo-opus-ada cross-referenced by #11711
- 2026-05-21T07:30:28Z @neo-opus-ada assigned to @neo-opus-ada
### @neo-opus-ada - 2026-05-21T07:31:47Z

**Input from Neo Opus 4.7 (Claude Code):**

> ❋ ## Ticket Intake — #11641 (Phase 4C Stale-Chunk GC Daemon)
> 
> **Epic-review prerequisite:** ✅ Parent Epic #11628 was epic-reviewed this session — https://github.com/neomjs/neo/issues/11628#issuecomment-4504271144 (verdict: Greenlight). Proceeding per the `epic-review` one-shot citation.
> 
> **Ticket age / labels:** created 2026-05-19; pre-stale, no `stale` / `no auto close` labels. Labels `enhancement`, `ai` — primary + provenance present; adequate for intake.
> 
> ### Premise — V-B-A validated ✅ (greenfield)
> 
> The premise holds: Chroma soft-deletes (physical reclaim needs explicit GC), per-tenant retention policies need enforcement, and `ai:defrag-kb` is operator-triggered, not automatic. A periodic per-tenant GC daemon is the right catch. V-B-A on the substrate — `grep` for `tenantRetention` / `retentionPolicy` / `retentionDays` across `ai/` → **no matches**: the retention-policy substrate is genuinely greenfield, not a duplicate of anything merged.
> 
> ### Duplication / historical — ✅ (one near-miss cleared)
> 
> #11663 ("KB Ingestion Phase 4: Configurable bundle retention policy") is **CLOSED** and is a **near-miss, not an overlap**: #11663 governs *backup-bundle* retention (`aiConfig.backupRetention` — how many backup bundles `backup.mjs` keeps). #11641 governs *per-tenant KB-chunk* retention (chunk expiration in Chroma). Different substrate, different config key. No duplication.
> 
> ### The load-bearing finding — #11641 vs #11640-as-shipped ⚠️
> 
> #11641's premise references "Phase 4B reconciliation diffs claimed-vs-actual." **Phase 4B (#11640, PR #11710) shipped a narrower V1**: config-invalidation reconciliation via the `tenantConfigVersion` stamp. The claimed-vs-actual path-manifest diff was substrate-gated → V1.x-deferred to **#11711**. Two #11641 bullets drift from 4B-as-shipped:
> 
> - **Orphan-class #2 — "config-orphaned chunks (source-config change leaves chunks unmatched)"** overlaps #11640 V1. `KbReconciliationService` already *detects* config-stale chunks (a chunk whose `tenantConfigVersion` is below the tenant's current `getTenantConfig().version`) and, opt-in, tombstones them. A 4C daemon independently re-detecting config-orphans = double-handling the same signal.
> - **Orphan-class #3 — "reconciliation-tombstoned chunks past grace period (Phase 4B integration)"** assumes a *post-tombstone* grace window in 4B. 4B-as-shipped has none — its grace is the *pre-tombstone* `reconciliationOrphanVersionGap`; once 4B tombstones via `collection.delete`, the chunk is a Chroma soft-delete like any other.
> 
> ### Scope refinement (Tier-2, documented)
> 
> Reconciling #11641 against #11640-as-shipped — a sibling-ticket de-dup resolvable with full context (I authored #11640):
> 
> - **4C V1 = retention-policy chunk expiration + physical-reclaim / defrag-trigger** — the two non-overlapping concerns.
> - **Drop orphan-class #2 (config-orphan detection)** — it is #11640's domain (detection + opt-in tombstone). 4C does not re-detect config-orphans.
> - **Reframe orphan-class #3** — 4C's tombstone concern is *physical reclaim of Chroma soft-deleted space* from **any** tombstone source (push-time `applyDeletionSignals`, #11640 reconciliation, retention expiration itself) via the conditional `ai:defrag-kb` trigger — not "chunks past a 4B grace period." This is the genuine 4B↔4C seam: 4B / push *tombstone* (logical delete); 4C *reclaims* (physical defrag).
> 
> Net: 4C stays a coherent, non-overlapping daemon — retention enforcement + space reclamation. The config-orphan lifecycle is wholly #11640's.
> 
> ### Contract Completeness Sweep — Contract Ledger required ⚠️
> 
> #11641 introduces operator-facing + destructive surfaces — `aiConfig.knowledgeBase.tenantRetention` (the per-tenant retention-policy schema), `gcIntervalMs`, the defrag-trigger threshold, and a **destructive Chroma delete** (retention-expired chunk GC). Per ticket-intake §7 it needs a **Contract Ledger**, and #11641's body lacks one. As the claiming maintainer I will author the ledger into the ticket body before branching — mirroring the #11640 / #11642 resolution. Until then #11641 is `needs-contract-alignment` (no branch / code).
> 
> ### Reality classification
> 
> `valid-as-written` for the core GC-daemon premise + scope, with the documented Tier-2 scope-refinement above. Readiness gate before code: the §7 Contract-Ledger gate (I author it next). Note: #11641's `Blocked-by: Phase 4B` is effectively satisfied — the 4B↔4C seam is "4B tombstones, 4C reclaims," which needs no 4B *code* dependency (4C reclaims soft-deletes regardless of which path created them).
> 
> **Verdict:** Accepted into the Phase-4C lane; self-assigned. Immediate next step — author the Contract Ledger into this ticket body. Implementation follows: the verified 3-part daemon split (`ai/scripts/kb-gc-daemon.mjs` wrapper + an `ai/daemons/KbGarbageCollectionService.mjs` poll-loop class + a pure retention-policy engine helper, mirroring #11640 / #11642) + tests. Peers / operator: flag now if the scope-refinement (dropping config-orphan detection → #11640) warrants a different cut.
> 
> ---
> 
> Origin Session ID: 470c38e7-1ffc-4851-867d-d30c1b6fbdb2

- 2026-05-21T07:42:47Z @neo-opus-ada cross-referenced by #11712
- 2026-05-21T07:42:57Z @neo-opus-ada marked this issue as being blocked by #11712
### @neo-opus-ada - 2026-05-21T07:43:20Z

**Input from Neo Opus 4.7 (Claude Code):**

> ❋ ## Ticket Intake — Update: substrate V-B-A → #11641 is blocked on a prerequisite
> 
> Following my intake comment (https://github.com/neomjs/neo/issues/11641#issuecomment-4505776781), I ran the pre-Contract-Ledger substrate sweep and hit a blocker that re-shapes #11641's readiness.
> 
> ### Substrate V-B-A findings
> 
> 1. **No `VectorService.delete({ids})`.** The ticket's "The Fix" prescribes `VectorService.delete({ids})`; that method does not exist (`VectorService` exposes `deleteCollection`; chunk-level deletes are internal `collection.delete({ids})` calls inside `embed`). Minor — the GC daemon would do `collection.delete({ids})` directly via `ChromaManager.getKnowledgeBaseCollection()`, the #11640 `tombstoneOrphans` pattern. Not a blocker.
> 
> 2. **Chunks carry no ingest timestamp — the blocker.** A `grep` of `parsed-chunk-v1.schema.json` + `VectorService` (`resolveTenantStamp` / `applyTenantStamp`): a stamped chunk's metadata is `{tenantId, repoSlug, visibility, tenantConfigVersion, originAgentIdentity, ...parsed-chunk fields}` — **there is no `ingestedAt` / timestamp field.** The same gap that forced #11640's version-gap design (over a wall-clock grace), now hit again here.
> 
> ### Why this blocks #11641's core
> 
> #11641's AC: "Retention policy structure: per-tenant config (**time-based** / **count-based** / **version-based**)."
> 
> - **Time-based** ("retain last 90 days") — needs a per-chunk ingest timestamp. **No substrate.**
> - **Count-based** ("retain last N chunks") — needs an ingest ordering (timestamp or sequence). **No substrate.**
> - **Version-based** — `tenantConfigVersion` is on every chunk → substrate-ready, **but**: version-based "retention" (expire chunks N+ config-versions old) is **operationally identical to #11640's config-invalidation reconciliation** — the same `tenantConfigVersion`-lag signal, the same delete. A version-based-only #11641 V1 would **duplicate #11640**, not deliver retention.
> 
> So #11641's genuine value — *policy-driven* retention (age / count bounds, independent of config changes) — **requires a per-chunk ingest timestamp.** Version-based-only is not a valid narrowed V1; it is #11640 rebranded.
> 
> ### Reclassification → `needs-relinking` (blocked-by a prerequisite)
> 
> Filed the narrow enabling prerequisite: **#11712** — "Add a server-stamped `ingestedAt` timestamp to tenant KB chunk metadata." #11641 is now linked `blocked-by #11712`. Per ticket-intake §1 the `needs-relinking` verdict **halts implementation**: I am not branching / coding #11641 until #11712 lands.
> 
> This is also a **friction → gold** signal: the no-ingest-timestamp gap has now affected three features — #11640's wall-clock grace, #11711's manifest-grace, and #11641's time/count retention. Three independent features wanting the same one-field stamp = a substrate insufficiency; #11712 closes it once for all three.
> 
> ### What stays valid
> 
> The earlier scope-refinement holds (config-orphan detection → #11640's domain). Once #11712 lands, #11641 V1 = **time / count-based retention-policy expiration + physical defrag-reclaim** — a coherent, non-#11640-overlapping daemon. The Contract Ledger is authored at that point; the readiness gate is now: #11712 merged → ledger → implement.
> 
> **Verdict update:** `needs-relinking` — #11641 remains accepted + assigned to me, but implementation is **held** pending the #11712 prerequisite. Peers / operator: flag if you read the blocker differently.
> 
> ---
> 
> Origin Session ID: 470c38e7-1ffc-4851-867d-d30c1b6fbdb2

- 2026-05-21T07:52:55Z @neo-opus-ada cross-referenced by PR #11713
- 2026-05-21T09:33:38Z @tobiu referenced in commit `d03179a` - "feat(ai): stamp ingestedAt on tenant KB chunks (#11712) (#11713)

* feat(ai): stamp ingestedAt on tenant KB chunks (#11712)

Tenant KB chunks carried no ingest timestamp — a substrate gap that blocked
#11641's time/count-based retention, #11640's wall-clock orphan-grace, and
#11711's manifest-grace. This adds the one missing field.

- ai/services/knowledge-base/VectorService.mjs — resolveTenantStamp now stamps
  ingestedAt (epoch ms, Date.now() at embed time) onto the server-derived
  tenant stamp; it flows via applyTenantStamp onto every embedded chunk's
  Chroma metadata. ingestedAt joins TENANT_GUARDED_FIELDS — purely
  server-derived, so a client-supplied ingestedAt is overwritten (or rejected
  under spoofRejectionMode: 'reject'), consistent with tenantConfigVersion.
  Re-push re-embeds + re-stamps it: last-ingested-at semantics.
- VectorService.tenantStamping.spec.mjs — 3 tests: server-stamp inclusion +
  propagation onto chunk metadata, client-spoof overwrite + diagnostics,
  reject-mode rejection before any Chroma write.

Consumers fail-safe on a missing ingestedAt (a chunk embedded before #11712) —
unknown-age, never expire/action it (mirrors #11640's missing-stamp skip).

10/10 VectorService.tenantStamping.spec.mjs tests pass; node --check clean.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* fix(ai): correct ingestedAt re-push semantics — embed-time, not last-push (#11712)

Addresses @neo-gpt's PR #11713 Cycle-1 Required Action. Commit 659a92700's
JSDoc + the PR/ticket prose claimed a same-content re-push "re-embeds +
re-stamps ingestedAt — last-ingested-at semantics." That is wrong: embed()'s
zero-change fast path dedupes already-known content-hash IDs out of
chunksToProcess, so an unchanged re-push does not upsert and does not refresh
ingestedAt.

- VectorService.mjs — resolveTenantStamp JSDoc corrected: ingestedAt marks when
  the chunk row is actually embedded/upserted; a same-content re-push retains
  its prior ingestedAt (a content change yields a new content-hash chunk row
  with its own fresh ingestedAt).
- VectorService.tenantStamping.spec.mjs — +1 test proving the corrected
  contract: a byte-identical re-push leaves spy.calls.upsert at 1 and the row's
  ingestedAt unchanged.

No production-logic change — resolveTenantStamp still stamps ingestedAt at
embed time; only the documented semantic boundary is corrected. The embed-time
("first-embedded-at for this content") semantics is the correct retention
anchor anyway — re-pushing identical content should not reset its age.

11/11 VectorService.tenantStamping.spec.mjs tests pass; node --check clean.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
### @neo-opus-ada - 2026-05-21T09:38:47Z

**Input from Neo Opus 4.7 (Claude Code):**

> ❋ ## Ticket Intake — Update: #11712 merged → #11641 unblocked; V1 design
> 
> #11712 ("server-stamped `ingestedAt` chunk metadata", PR #11713) **merged to `dev`** 2026-05-21. #11641's blocker is cleared — tenant KB chunks now carry a server-stamped `ingestedAt`, so time- and count-based retention have substrate. Picking up the Phase 4C lane; recording the V1 design before authoring the Contract Ledger.
> 
> ### V1 scope (per the earlier intake scope-refinement)
> 
> V1 = **retention-policy chunk expiration + physical defrag-reclaim.** Config-orphan detection was dropped at intake — it overlaps #11640's config-invalidation reconciliation (see the prior intake comment). The loose ACs above are refined by the design below + the forthcoming Contract Ledger.
> 
> ### Retention schema
> 
> Now that `ingestedAt` (#11712) is on every chunk's metadata:
> - **Time-based** — a chunk is expired when `now − ingestedAt > maxAgeMs`.
> - **Count-based** — retain the `maxCount` most-recent chunks (by `ingestedAt`) per tenant; expire the rest.
> - Version-based is **not** V1 — that is #11640's `tenantConfigVersion`-reconciliation domain (the intake de-dup).
> 
> ### Config — `aiConfig.knowledgeBase` (5 new keys, all default-safe)
> 
> - `gcEnabled` (Boolean, default `false`) — master opt-in; the daemon exits early when false.
> - `gcIntervalMs` (Number, default `86400000` = 24h) — poll-tick interval (ticket AC).
> - `gcRetention` (Object, default `{}`) — `{maxAgeMs?, maxCount?}`. Empty ⇒ nothing expires (conservative default, per the ticket's "default conservative" handoff hint).
> - `gcAutoDelete` (Boolean, default `false`) — opt-in for the destructive Chroma delete. Default-off ⇒ the daemon detects retention-expired chunks + emits Phase 4A telemetry only, issuing no delete (the #11640 `reconciliationAutoTombstone` precedent).
> - `gcDefragThreshold` (Number, default `0`) — when `> 0`, a tick whose cumulative deletion exceeds this fraction of collection size spawns `ai:defrag-kb`.
> 
> ### Decisions / deltas from the ticket
> 
> - **Per-tenant retention → global for V1.** The ticket says "per-tenant retention," but `getTenantConfig` has a fixed 8-field projection (no retention field). V1 applies one **global** `gcRetention` policy *per tenant* (each tenant's chunks evaluated against it); a per-tenant *override* needs a `getTenantConfig`-projection extension (#11637 surface) → V1.x. Mirrors #11640's global-`reconciliationOrphanVersionGap` V1.
> - **Auto-defrag default-off (vs the ticket's "default 10%").** Auto-spawning `ai:defrag-kb` — a whole-collection nuke-and-pave — from a poll-loop daemon needs a defrag-vs-ingest concurrency-coordination story. V1 ships the trigger *capability* behind `gcDefragThreshold` but defaults it to `0` (disabled); an operator opts in. The coordination story + a sane default is a documented V1.x item.
> - **GC delete is opt-in (`gcAutoDelete`), default-off** — destructive-action conservatism.
> 
> ### Shape
> 
> The verified 3-part daemon split (mirrors #11640 / #11642): `ai/services/knowledge-base/helpers/KbGarbageCollectionEngine.mjs` (pure retention-expiry classifier) + `ai/daemons/KbGarbageCollectionService.mjs` (poll loop) + `ai/scripts/kb-gc-daemon.mjs` (wrapper). GC deletes emit `recordIngestionMetric({eventType: 'tombstone', …})`. Tenant-scoped via the `where: {tenantId}` Chroma read (RLS-safe).
> 
> **Next:** author the Contract Ledger (T3) into this body, then branch `agent/11641-kb-gc-daemon` from the updated `dev` + implement. Peers / operator: flag now if the auto-defrag-default-off or global-retention-V1 calls warrant a different cut.
> 
> ---
> 
> Origin Session ID: 470c38e7-1ffc-4851-867d-d30c1b6fbdb2

- 2026-05-21T10:06:14Z @neo-gpt cross-referenced by PR #11714
- 2026-05-21T10:12:17Z @neo-opus-ada cross-referenced by PR #11715
- 2026-05-21T11:05:01Z @neo-opus-ada cross-referenced by #11179
- 2026-05-21T11:19:32Z @tobiu referenced in commit `82ea006` - "feat(ai): KB garbage-collection daemon — Phase 4C (#11641) (#11715)

* feat(ai): add KB garbage-collection retention engine (#11641)

Phase 4C (#11641) substrate — the pure retention-expiry classification core
of the forthcoming KB garbage-collection daemon (KbGarbageCollectionService).

ai/services/knowledge-base/helpers/KbGarbageCollectionEngine.mjs —
dependency-free (no Neo, no I/O, no clock):
- selectExpiredChunks — classifies a tenant's Chroma rows under OR-expiry: a
  chunk is expired if time-expired (now - ingestedAt > maxAgeMs) OR
  count-expired (beyond the maxCount most-recent of its {tenantId, repoSlug}
  bucket, ranked ingestedAt desc then chunk id asc — a deterministic
  tie-breaker for the batch-shared ingestedAt case). Built on the #11712
  ingestedAt chunk stamp.
- resolveRetention — normalizes the {maxAgeMs, maxCount} operator policy
  (each dimension independently degrades to null/disabled when invalid).
- formatGcDetail — pure Phase 4A telemetry detail payload.

Fail-safe: a chunk with a missing/non-numeric ingestedAt (a pre-#11712
ingest) is never flagged — unaged + unrankable, excluded from both expiry
paths (mirrors #11640's missing-tenantConfigVersion skip).

Per the #11641 Contract Ledger (ticket body, T3 — refined per @neo-gpt's
pre-branch peer review): the explicit OR-expiry, the deterministic count
tie-breaker, and {tenantId, repoSlug} count-bucketing are engine contract.
The daemon (poll loop + Chroma I/O + opt-in delete + the defrag-recommended
signal) lands in follow-up commits.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* feat(ai): add KB garbage-collection daemon (#11641)

Phase 4C (#11641) — the KB garbage-collection daemon: periodically evaluates
each tenant's persisted Chroma chunks against an operator retention policy and
(opt-in) deletes the retention-expired ones. Builds on the retention engine
from 03974a30f.

- ai/daemons/KbGarbageCollectionService.mjs — the poll-loop daemon (Neo
  singleton, the KbReconciliationService pattern: start/stop/scheduleNext/
  pulse). Each pulse enumerates tenants (distinct tenantId from
  getTenantIngestionRollup), runs the pure KbGarbageCollectionEngine against
  each tenant's where:{tenantId} Chroma rows, and — when gcAutoDelete is set —
  deletes the expired chunks; a two-pass pulse then stamps the post-loop
  defragRecommended fact onto each tenant's 'tombstone' telemetry and emits a
  defrag-recommended logger.warn when cumulative deletion exceeds
  gcDefragThreshold. All I/O behind test-stubbable seam methods; pulse() is
  try/catch/finally + per-tenant try/catch.
- ai/scripts/kb-gc-daemon.mjs — the thin Neo-bootstrap wrapper.
- ai/config.template.mjs — 5 new aiConfig.knowledgeBase keys (gcEnabled /
  gcIntervalMs / gcRetention / gcAutoDelete / gcDefragThreshold), read
  defensively.
- package.json — the ai:kb-gc script.

Per the #11641 Contract Ledger (ticket body, T3 — refined per @neo-gpt's
pre-branch peer review): V1 is purely additive (touches no merged Phase 2
code); the daemon is opt-in (gcEnabled=false default), the destructive delete
a second opt-in (gcAutoDelete=false default), and V1 emits a defrag-recommended
signal rather than spawning ai:defrag-kb (the spawn + its concurrency story
are V1.x).

node --check clean on both .mjs files; config.template.mjs valid; package.json
valid JSON. The 2 unit specs land in a follow-up commit.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

* test(ai): KB GC engine + daemon specs (#11641)

Phase 4C (#11641) — unit coverage for the KB garbage-collection daemon: 17
pure engine tests + 18 daemon tests, 35 passing.

- test/playwright/unit/ai/services/knowledge-base/KbGarbageCollectionEngine.spec.mjs
  — covers selectExpiredChunks (time-expiry strict-greater-than, count-expiry
  per {tenantId,repoSlug} bucket, the deterministic id-asc tie-break on equal
  ingestedAt, the OR-union, the missing-ingestedAt fail-safe, the empty-policy
  no-op), resolveRetention (per-dimension validation), formatGcDetail. Pure
  module -> no setup() harness.
- test/playwright/unit/ai/daemons/KbGarbageCollectionService.spec.mjs — covers
  start/stop (opt-in gate, idempotency, interval config), pulse (empty-tenant
  early return, per-tenant loop, reschedule-on-failure, null-result isolation,
  the defrag-recommended threshold signal both ways), collectTenant
  (clean-tenant no-delete, auto-delete opt-in gating both ways, never-throws),
  fetchTenants dedup, deleteChunks empty-guard, the recordGcMetric tombstone
  payload. Seam-stubbing on the singleton per the KbReconciliationService.spec
  precedent.

35/35 passing (npm run test-unit). Satisfies the #11641 Contract Ledger
Evidence columns.

Co-Authored-By: Neo Opus 4.7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T11:19:32Z @tobiu closed this issue

