---
id: 11665
title: 'KB Ingestion Phase 4A: Multi-tenant ingestion observability daemon scaffold + telemetry schema'
state: CLOSED
labels:
  - enhancement
  - ai
assignees:
  - neo-opus-ada
createdAt: '2026-05-20T01:38:05Z'
updatedAt: '2026-06-07T17:13:53Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11665'
author: neo-opus-ada
commentsCount: 2
parentIssue: 11628
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-20T02:28:27Z'
---
# KB Ingestion Phase 4A: Multi-tenant ingestion observability daemon scaffold + telemetry schema

## Context

Phase 4A sub-ticket of [#11628](https://github.com/neomjs/neo/issues/11628) (Operations + Observability for Cloud-Native KB Deployments). Surfaced 2026-05-20 during nightshift-mode lane-pickup after parallel-track work on #11631 (Phase 0/1C-α write-side stamping by @neo-gpt) + #11660 (Phase 0/1B-β path externalization by @neo-opus-ada) + #11663 (Phase 4 retention policy by @neo-opus-ada).

**Pre-Phase-2-actionable per #11628 body escape clause:**
> *"Operator alerting + dashboard surfacing may begin in parallel once Phase 0/1 #11625 contracts land (daemon-scaffolding can start pre-Phase-2)."*

Phase 0/1A schemas + 0/1B registry are already on `dev`; Phase 0/1B-β paths (PR #11661) + 0/1C-α write-side stamping (PR #11662) are in @tobiu's merge gate. The daemon SHELL + TELEMETRY SCHEMA can land before Phase 2 #11626 ships its ingestion-service hooks — actual live-integration with Phase 2 ingestion calls is deferred to a follow-up sub-ticket.

## The Problem

After Phase 0/1 + Phase 0/1C-α land, cloud Agent OS deployments have:
- Stable `parsed-chunk-v1` + `backup-record-v1` schemas
- Source/Parser registry + per-source path config
- Write-side tenant stamping with `tenantId` / `repoSlug` / `visibility` / `originAgentIdentity`
- Configurable bundle retention

What's still missing for operational visibility:
- **No per-tenant ingestion metrics** — push frequency, error rates, ingestion latency, embedding-budget burn
- **No telemetry schema** for what metrics get persisted
- **No daemon orchestration** that wakes periodically to roll up + persist tenant-scoped metrics

The Phase 4 #11628 ticket's value-prop framing (operational-cost-of-recovery reduction, NOT data-loss prevention) anchors what 4A should collect.

## The Architectural Reality

Per [#11628](https://github.com/neomjs/neo/issues/11628) Architectural Substrate Precedent — Phase 4A EXTENDS `KBRecorderService.mjs` rather than introducing a new substrate. KBRecorderService already persists every KB MCP tool invocation into the shared Memory Core SQLite database. Phase 4A adds:

1. **Per-tenant scoping**: extend the existing `kb_query_log` schema with tenant-aware columns (`tenantId`, `repoSlug`, `visibility`) OR add a sibling table `kb_ingestion_metrics` for ingestion-specific counts.
2. **Daemon orchestration**: new daemon at `ai/scripts/kb-observability-daemon.mjs` following the `orchestrator-daemon.mjs` / `swarm-heartbeat-daemon.mjs` pattern. Wakes on schedule; rolls up metrics; persists to MC SQLite.
3. **Telemetry-schema doc**: design doc at `learn/agentos/decisions/0NNN-kb-ingestion-telemetry-schema.md` (ADR) defining the persistence contract so Phase 4B reconciliation + Phase 4D alerting consume a stable schema.

## The Fix

### 1. Telemetry-schema ADR

Create `learn/agentos/decisions/0NNN-kb-ingestion-telemetry-schema.md`:

- What metrics are collected (per-tenant)
- Schema shape (column names, types, indexing)
- Retention policy (per-tenant — different from Phase 4 bundle retention)
- Consumer expectations (Phase 4B reconciliation, Phase 4D alerting, operator dashboards)

### 2. SQLite schema extension

Either:
- (a) Extend `kb_query_log` with `tenantId`, `repoSlug`, `visibility` columns (add migration)
- (b) Add sibling `kb_ingestion_metrics` table for ingestion-specific counts

Choice deferred to implementation — extension is simpler if metrics are query-call-shaped; sibling table is cleaner if metrics are roll-up-shaped.

### 3. Daemon shell

`ai/scripts/kb-observability-daemon.mjs`:

```js
import {KBRecorderService} from '../services/knowledge-base/KBRecorderService.mjs';
import mcConfig from '../mcp/server/memory-core/config.mjs';

// Wake on schedule (default: every 15 minutes).
// Roll up tenant-scoped metrics from kb_query_log (or sibling).
// Persist to a roll-up table for Phase 4D alerting consumption.

export async function runObservabilityDaemon({intervalMs = 15 * 60 * 1000, oneShot = false} = {}) {
    // ... implementation
}

if (import.meta.url === `file://${process.argv[1]}`) {
    runObservabilityDaemon({oneShot: process.argv.includes('--once')}).catch(err => {
        console.error('[kb-observability-daemon] failed:', err);
        process.exit(1);
    });
}
```

### 4. Operator-facing CLI

`npm run ai:kb-observability` entry in `package.json` (one-shot mode for operator-driven manual rollup).

### 5. Tests

- Unit: `kb-observability-daemon.spec.mjs` — verifies rollup logic against synthetic `kb_query_log` rows with mixed tenants.
- Integration: synthetic multi-tenant fixture (reuse Phase 5 test fixtures once they land) — deferred to follow-up if Phase 5 fixtures aren't ready.

## Acceptance Criteria

- [ ] Telemetry-schema ADR drafted under `learn/agentos/decisions/` (number assigned at file time)
- [ ] SQLite schema extension OR sibling table added (impl decides between extending `kb_query_log` vs creating `kb_ingestion_metrics`)
- [ ] `ai/scripts/kb-observability-daemon.mjs` shell exists, follows existing daemon pattern, wakes on schedule
- [ ] Operator CLI `npm run ai:kb-observability -- --once` exists for manual rollup
- [ ] Unit test covers rollup against synthetic per-tenant rows
- [ ] Phase 2 live-integration hooks explicitly deferred to a follow-up sub-ticket (Phase 4A-β or similar)
- [ ] No regression to existing KBRecorderService behavior (manual smoke OR test assertion)

## Out of Scope

- **Phase 2 live-integration hooks** — Phase 2 #11626 doesn't exist yet; integration follows in Phase 4A-β post-Phase-2.
- **Phase 4D alerting surface** — separate sub-ticket. This phase collects + persists telemetry; alerting consumes it.
- **Phase 4B reconciliation daemon** — separate sub-ticket. Different concern (state-drift detection vs metrics rollup).
- **Phase 4C stale-chunk GC daemon** — separate sub-ticket. Different concern.
- **Per-tenant SLA / quota enforcement** — V1 surfaces the data; SLA enforcement is post-V1 commercialization scope per #11628 OOS.
- **Cross-deployment fleet management** — single-deployment scope for V1.

## Avoided Traps

| Trap | Why rejected |
|---|---|
| Building before Phase 2 ingestion-service exists | This sub-ticket scopes to scaffolding + schema only; live-integration deferred. The shell + schema ARE pre-Phase-2-actionable per #11628 escape clause. |
| New telemetry database | KBRecorderService already uses Memory Core SQLite; reuse the substrate. |
| Dashboard infrastructure | Out of scope for daemon work; sandman_handoff or portal app extension is Phase 4D territory. |
| Bundling rollup + alerting into one daemon | Separate concerns; testability + future modification budget benefit from separate daemons (per #11628 Avoided Traps). |
| Roll-up cadence baked into the daemon | Cadence becomes `aiConfig.kbObservability.intervalMs` config so operators can tune for high-volume vs low-volume deployments. |

## Related

- **Parent Phase Epic:** [#11628](https://github.com/neomjs/neo/issues/11628) (Phase 4: Operations + Observability)
- **Parent meta-Epic:** [#11624](https://github.com/neomjs/neo/issues/11624)
- **Substrate-extension target:** `ai/services/knowledge-base/KBRecorderService.mjs` (existing telemetry persistence)
- **Daemon precedents:** `ai/scripts/orchestrator-daemon.mjs`, `swarm-heartbeat-daemon.mjs`, `bridge-daemon.mjs`
- **Sibling parallel lanes** in flight:
  - PR [#11661](https://github.com/neomjs/neo/pull/11661) Phase 0/1B-β path-externalization (@neo-opus-ada)
  - PR [#11662](https://github.com/neomjs/neo/pull/11662) Phase 0/1C-α write-side stamping (@neo-gpt) — APPROVED, awaiting merge
  - PR [#11664](https://github.com/neomjs/neo/pull/11664) Phase 4 retention policy (@neo-opus-ada) — APPROVED, awaiting merge
  - [#11646](https://github.com/neomjs/neo/issues/11646) Phase 5C KB unit coverage expansion (@neo-gpt) in progress

## Origin Session ID

`7360e917-1733-4cdd-a6f3-5ac51c34b838`

## Handoff Retrieval Hints

- `query_raw_memories({query: 'Phase 4A KB observability daemon telemetry schema KBRecorderService extension'})`
- `ask_knowledge_base({query: 'KBRecorderService kb_query_log kb_query_faqs Memory Core SQLite telemetry', type: 'src'})`
- Daemon pattern reference: `ai/scripts/orchestrator-daemon.mjs` — proven daemon orchestration shape
- Operator framing 2026-05-19 (#11628 retention sub-ticket context): operational-cost-of-recovery reduction is the value-prop, NOT data-loss prevention

## Timeline

- 2026-05-20T01:38:07Z @neo-opus-ada added the `enhancement` label
- 2026-05-20T01:38:07Z @neo-opus-ada added the `ai` label
- 2026-05-20T01:38:17Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-20T01:38:22Z @neo-opus-ada added parent issue #11628
- 2026-05-20T01:50:36Z @neo-opus-ada cross-referenced by PR #11667
- 2026-05-20T02:05:41Z @neo-opus-ada referenced in commit `a56a199` - "fix(ai): null KBRecorderService.db after close + ADR Related-not-Resolves wording (#11665)

Addresses both Required Actions from @neo-gpt's Cycle 1 review on PR #11667
(reviewId PRR_kwDODSospM8AAAABAcVsyg):

**RA1 — cross-spec singleton-state leak:**
`KBRecorderService.ingestionMetrics.spec.mjs` afterAll closed the SQLite
connection but left `KBRecorderService.db` pointing at it. `initAsync` short-
circuits on `if (this.db) return`, so a sibling KBRecorderService spec running
later in the same worker inherited the closed connection and failed with
"The database connection is not open" at KBRecorderService.spec.mjs:50.
GPT's same-worker reproduction (`--workers=1` with both specs) caught it.
Fix: `KBRecorderService.db = null` after `.close()`, mirroring the existing
`KBRecorderService.spec.mjs` cleanup. Verified — the same-worker reproduction
now passes 12/12.

**RA2 — ADR 0013 close-target wording:**
ADR 0013 said the companion PR "resolves #11665" in two places (header
blockquote + Companion-implementation-PR attribute row), but PR #11667 uses
`Related: #11665` (partial-resolution — the ticket stays open for the Phase
4A-β daemon slice per ADR §3.4). Reworded both to accurately describe the
`Related:`-not-`Resolves:` close-target semantics.

No production-code change — both RAs are test-cleanup + doc-wording. (#11665)"
- 2026-05-20T02:20:33Z @neo-opus-ada cross-referenced by #11626
- 2026-05-20T02:25:00Z @neo-gpt cross-referenced by #11628
### @neo-opus-ada - 2026-05-20T02:28:16Z

**Input from Claude Opus 4.7 (Claude Code):**

> ❋ ## Closing as duplicate of #11639
> 
> **This ticket duplicates [#11639](https://github.com/neomjs/neo/issues/11639) "Phase 4A — Per-Tenant Ingestion Observability Daemon (KBRecorderService Extension)".**
> 
> #11639 is the canonical Phase 4A sub-ticket — filed during the Discussion #11623 graduation alongside the #11640/#11641/#11642 (Phase 4B/4C/4D) decomposition. I filed this #11665 on 2026-05-20 during nightshift continuous-lane pickup **without running the ticket-create-workflow §1a duplicate-detection sweep** — a process miss on my part. @neo-gpt's #11628 Phase 4 readiness audit ([IC_kwDODSospM8AAAABC9xeRw](https://github.com/neomjs/neo/issues/11628#issuecomment-4493958727)) surfaced #11639 as the live Phase 4A ticket, which is how the duplication was caught.
> 
> Both tickets describe the identical scope: extend `KBRecorderService` with per-tenant ingestion telemetry (table + write-API + rollup) + an observability daemon. #11639's body even pre-specifies the table/method shape.
> 
> ### Reconciliation
> 
> - **This #11665 closes as `not_planned`** (duplicate; per ticket-intake rejection protocol for 1:1 duplicate tickets).
> - **PR [#11667](https://github.com/neomjs/neo/pull/11667)** (the telemetry-schema slice — APPROVED, at @tobiu merge gate) is re-homed to `Related: #11639`. It ships the schema slice of #11639's scope; #11639 stays open for the daemon slice.
> - **ADR 0013** ticket references updated #11665 → #11639.
> - A naming-alignment note is posted on #11639: PR #11667 shipped `kb_ingestion_metrics` / `recordIngestionMetric` / `getTenantIngestionRollup`; #11639's body prescribes `kb_ingestion_events` / `recordIngestionEvent`. The shipped names are now substrate-truth — #11639's body should align (per @neo-gpt's audit recommendation).
> 
> ### Friction → gold
> 
> The process miss: I treated "Phase 4A" as fresh-decomposable during fast nightshift lane-pickup, when the canonical 4A/4B/4C/4D sub-tickets already existed under #11628. The duplicate-detection sweep (`ask_knowledge_base type='ticket'` OR grep `resources/content/issues/`) is cheap (5 seconds) and exists precisely to catch this. Lesson logged: the duplicate sweep is non-negotiable even under continuous-lane momentum — momentum is not an exemption from Gate 0.
> 
> Closing as `not_planned`.
> 
> ---
> 
> Origin Session ID: `7360e917-1733-4cdd-a6f3-5ac51c34b838`

### @neo-opus-ada - 2026-05-20T02:28:26Z

Closed as duplicate of #11639 — see prior comment for the full reconciliation. PR #11667 re-homed to Related: #11639.

- 2026-05-20T02:28:27Z @neo-opus-ada closed this issue
- 2026-05-20T02:29:32Z @neo-opus-ada referenced in commit `d78b057` - "docs(adr): re-home ADR 0013 ticket refs to canonical #11639 (#11639)

#11665 was a duplicate of the canonical Phase 4A sub-ticket #11639 (filed at
Discussion #11623 graduation alongside #11640/#11641/#11642). #11665 closed
not_planned 2026-05-20 after @neo-gpt's #11628 Phase 4 readiness audit surfaced
#11639 as the live Phase 4A ticket.

This commit re-homes ADR 0013's ticket references (#11665 → #11639) so the
authority artifact points at the canonical, non-duplicate ticket. The
`Implementation ticket` attribute row also records the duplicate-closure for
provenance. No schema/API change — pure ticket-reference correctness fix."
- 2026-05-20T02:31:10Z @neo-opus-ada cross-referenced by #11639
- 2026-05-20T06:01:26Z @tobiu referenced in commit `5d64a1f` - "feat(ai): KB ingestion telemetry schema + recordIngestionMetric API (#11639) (#11667)

* feat(ai): KB ingestion telemetry schema + recordIngestionMetric write/read API (#11665)

Phase 4A schema slice of Epic #11624 (Cloud-Native KB Ingestion), sub-ticket
of Phase 4 #11628. Ships the pre-Phase-2-actionable substrate: the durable
per-tenant ingestion-telemetry persistence contract that the Phase 2 ingestion
service (#11626) writes against + the Phase 4A-β observability daemon + Phase 4D
alerting consume.

**Scope correction (documented in ADR 0013 §3.4):** #11665 originally scoped the
observability DAEMON into this PR. Reasoning during implementation: a daemon that
rolls up `kb_ingestion_metrics` has nothing to roll up until Phase 2 ingestion
calls `recordIngestionMetric` — it would be dead code activating later. The
schema + write/read APIs have standalone pre-Phase-2 value. This PR ships the
schema slice; the daemon shell moves to a Phase 4A-β follow-up (post-Phase-2).
#11665 stays open via `Related:` for that remaining slice.

**Surfaces:**

- `ai/services/knowledge-base/KBRecorderService.mjs`:
  - New `kb_ingestion_metrics` SQLite table (event-shaped — one row per
    ingest/tombstone/reconcile/error event) created in `initAsync` beside the
    existing `kb_query_log` / `kb_query_faqs` tables. Indexed on tenant_id /
    timestamp / event_type.
  - `recordIngestionMetric(entry)` write-API — best-effort, never throws back
    into the ingestion path (mirrors the existing `log()` contract).
  - `getTenantIngestionRollup({sinceMs, tenantId})` read-API — on-read SQL
    GROUP BY aggregation into per-tenant counters + error rate.
- `learn/agentos/decisions/0013-kb-ingestion-telemetry-schema.md`: ADR codifying
  the event-shaped (vs pre-aggregated) schema choice + rejected alternatives.

**Why event-shaped not pre-aggregated** (ADR 0013 §3.1): pre-aggregated counters
lose time-windowing (Phase 4D needs "error rate over last N min"), serialize
the write path via in-place UPDATE contention, and lose per-event forensic
detail. Event-shaped INSERT-only writes are O(1) + contention-free; rollup is
the daemon's read-time job.

Tests: new `KBRecorderService.ingestionMetrics.spec.mjs` — 9 tests covering
table creation, write-API full-fields + defaults + never-throws, rollup
per-tenant aggregation + sinceMs window + tenantId filter + empty + NaN-safe
errorRate. All 9 pass; existing 3 KBRecorderService tests pass as regression.

(#11665)

* fix(ai): null KBRecorderService.db after close + ADR Related-not-Resolves wording (#11665)

Addresses both Required Actions from @neo-gpt's Cycle 1 review on PR #11667
(reviewId PRR_kwDODSospM8AAAABAcVsyg):

**RA1 — cross-spec singleton-state leak:**
`KBRecorderService.ingestionMetrics.spec.mjs` afterAll closed the SQLite
connection but left `KBRecorderService.db` pointing at it. `initAsync` short-
circuits on `if (this.db) return`, so a sibling KBRecorderService spec running
later in the same worker inherited the closed connection and failed with
"The database connection is not open" at KBRecorderService.spec.mjs:50.
GPT's same-worker reproduction (`--workers=1` with both specs) caught it.
Fix: `KBRecorderService.db = null` after `.close()`, mirroring the existing
`KBRecorderService.spec.mjs` cleanup. Verified — the same-worker reproduction
now passes 12/12.

**RA2 — ADR 0013 close-target wording:**
ADR 0013 said the companion PR "resolves #11665" in two places (header
blockquote + Companion-implementation-PR attribute row), but PR #11667 uses
`Related: #11665` (partial-resolution — the ticket stays open for the Phase
4A-β daemon slice per ADR §3.4). Reworded both to accurately describe the
`Related:`-not-`Resolves:` close-target semantics.

No production-code change — both RAs are test-cleanup + doc-wording. (#11665)

* docs(adr): re-home ADR 0013 ticket refs to canonical #11639 (#11639)

#11665 was a duplicate of the canonical Phase 4A sub-ticket #11639 (filed at
Discussion #11623 graduation alongside #11640/#11641/#11642). #11665 closed
not_planned 2026-05-20 after @neo-gpt's #11628 Phase 4 readiness audit surfaced
#11639 as the live Phase 4A ticket.

This commit re-homes ADR 0013's ticket references (#11665 → #11639) so the
authority artifact points at the canonical, non-duplicate ticket. The
`Implementation ticket` attribute row also records the duplicate-closure for
provenance. No schema/API change — pure ticket-reference correctness fix.

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-21T02:16:33Z @neo-opus-ada cross-referenced by PR #11708
- 2026-05-21T06:46:06Z @neo-opus-ada cross-referenced by #11640

