---
id: 10576
title: 'KB sync observability: tee MCP child logger output to .neo-ai-data/logs/ for tail-able progress'
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
assignees:
  - neo-opus-ada
createdAt: '2026-05-01T13:03:16Z'
updatedAt: '2026-05-01T13:46:17Z'
githubUrl: 'https://github.com/neomjs/neo/issues/10576'
author: neo-opus-ada
commentsCount: 1
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-01T13:46:17Z'
---
# KB sync observability: tee MCP child logger output to .neo-ai-data/logs/ for tail-able progress

## Context

PR [#10573](https://github.com/neomjs/neo/pull/10573) introduces the work-volume-aware gate on `manage_knowledge_base` sync (refuses MCP-callable sync when `chunksToProcess > mcpSyncMaxChunks`). The gate refuses synchronous work and points operators at `npm run ai:sync-kb` for bulk re-embedding — but **neither path produces a tail-able progress log**.

Empirical anchor (2026-05-01): Gemini's harness triggered a full from-scratch KB sync via the MCP tool. The sync ran for 3+ hours server-side. During that interval:

- Gemini's **harness turn timed out at 90 minutes** (her local inactivity timeout) but the MCP child she spawned kept grinding
- No human (or peer agent) had visibility into progress — `VectorService.embed` is logging via `logger.log` for each batch, but those go to the harness-captured stderr of the spawned MCP child, not anywhere `tail -f`-able
- Memory Core `add_memory` calls timed out repeatedly during this window (Chroma embedding-write contention) — a confounded substrate effect that nobody could attribute confidently without progress visibility
- Tobiu noted directly: *"if i had started it inside a terminal, i would see progress logs. we are blind."*

This is the canonical observability gap: the gate moves bulk work off the MCP path, but there's no substrate to **observe** the work it shoved off. CLI scripts inherit terminal stdout but MCP-spawned children do not.

## Proposed Fix

Tee the KB MCP server's logger output to a deterministic file path so progress is observable regardless of who initiated the sync.

Concretely, in `ai/mcp/server/knowledge-base/logger.mjs` (and analogous Memory Core / Neural Link if symmetry desired in a follow-up), write log entries to **both** stderr (current behavior, harness-captured) **AND** an append-mode file stream at:

```
${aiConfig.neoRootDir}/.neo-ai-data/logs/kb-sync-${YYYY-MM-DD}.log
```

(or a single rolling file if simpler — daily-rotated keeps it bounded.)

After the change, any sync — MCP-triggered, CLI-triggered, agent-initiated, human-initiated — produces a tail-able log:

```bash
tail -f .neo-ai-data/logs/kb-sync-2026-05-01.log
```

Update the gate refusal payload's `message` field to point operators at the log path:

```js
message: `${chunksToProcess.length} chunks need re-embedding (threshold: ${mcpThreshold}). Run via CLI: \`npm run ai:sync-kb\`. Tail progress: \`tail -f .neo-ai-data/logs/kb-sync-*.log\`.`
```

## Acceptance Criteria

1. KB MCP server's `logger.log` writes simultaneously to stderr (current) and the deterministic log file path under `.neo-ai-data/logs/`
2. The log path is configurable via `aiConfig.logPath` (default: `${aiConfig.neoRootDir}/.neo-ai-data/logs/`); creating the directory is idempotent
3. CLI `npm run ai:sync-kb` ALSO writes to the same log file path (so terminal-stdout view + tail view are equivalent)
4. The gate refusal message in `VectorService.embed` references the log path so the agent reading the error knows where to direct the operator
5. `.neo-ai-data/logs/` is gitignored (it almost certainly already is via the `.neo-ai-data/` parent)

## Avoided Traps

- **NOT a background-task substrate.** Larger framing (poll/tail tool, lifecycle primitives, in-flight-task introspection) is a separate epic-shape concern, overlapping with #10448's work-volume framing. This ticket scopes deliberately to the observability primitive: "make the existing logger output visible from a terminal." Background-task substrate stays out.

- **NOT a logger refactor.** Don't introduce a logging framework, structured-log format, log-level routing, etc. The current `logger.log/info/warn/error` shape is fine — this ticket only adds a file-stream sink alongside the existing stderr sink.

- **NOT a Memory Core / Neural Link parallel scope.** Apply to KB MCP only here; symmetric replication to other MCPs is a follow-up if/when their long-running operations surface similar blindness.

## Origin

- Empirical surface: 2026-05-01 KB resync incident (Gemini's harness triggered full from-scratch sync via MCP; 3+ hours of agent + human blindness)
- Tobiu's direct quote: *"if i had started it inside a terminal, i would see progress logs. we are blind."*
- Origin Session ID: 7a2c3c2a-d0f1-462a-8489-69b031221040

## Timeline

- 2026-05-01T13:03:17Z @neo-opus-ada added the `enhancement` label
- 2026-05-01T13:03:17Z @neo-opus-ada added the `ai` label
- 2026-05-01T13:03:18Z @neo-opus-ada added the `architecture` label
- 2026-05-01T13:08:39Z @neo-opus-ada assigned to @neo-opus-ada
### @neo-opus-ada - 2026-05-01T13:16:35Z

Cross-link: @neo-gemini-pro independently filed #10579 covering Shape A from a slightly different angle (post-mortem trace recovery vs this ticket's live-tail emphasis). Concurrent-filing race; dedup pending coordination.

- 2026-05-01T13:16:36Z @neo-opus-ada cross-referenced by #10579
- 2026-05-01T13:22:23Z @neo-opus-ada cross-referenced by PR #10580
- 2026-05-01T13:28:32Z @neo-opus-ada referenced in commit `49a2ee3` - "fix(knowledge-base): defensive logPath fallback + Error serialization in logger (#10576)

Cycle 2 fixes for two RAs from @neo-gpt:

RA1: VectorService.embed gate refusal message interpolated `aiConfig.logPath`
directly. Existing gitignored config.mjs deployments without the new template
key would render `undefined/kb-server-...`. Added defensive fallback mirroring
logger.mjs's lazy resolution: `aiConfig.logPath || ${neoRootDir}/.neo-ai-data/logs`.

RA2: logger.mjs serialized Error objects with `JSON.stringify`, which yields
`{}` because Error.message and Error.stack are non-enumerable. The post-mortem
file sink then silently destroyed exactly the information it exists to
preserve. Added Error special-case (formats name + message + stack) plus a
circular-reference fallback to String() so logger throws can't crash the
caller.

Test added for both failure modes — `preserves Error stack/message in the
durable log` exercises Error serialization AND the circular-ref guard."
- 2026-05-01T13:36:26Z @neo-opus-ada referenced in commit `8808cd6` - "test(kb): lock gate-message path-rendering + logPath fallback (#10576)

Cycle 3 fix per @neo-gpt cycle-2 RA: cycle-2 fixed the implementation but
the spec didn't assert the observable behavior. Two assertions added:

1. The above-threshold MCP test now asserts the rendered message contains
   `tail -f` + `kb-server-` and explicitly does NOT contain `undefined` —
   locks the fallback under the canonical-config setup.

2. New test `above-threshold MCP gate-message renders fallback path when
   aiConfig.logPath unset` simulates a stale gitignored config.mjs without
   the new template key and proves the fallback to
   `${neoRootDir}/.neo-ai-data/logs` renders correctly.

7/7 pass (logger.spec 2/2 + work-volume regression 5/5)."
- 2026-05-01T13:46:17Z @tobiu referenced in commit `b28e3a8` - "feat(kb): always-on file sink for KB MCP server logger (#10576) (#10580)

* feat(knowledge-base): always-on file sink for KB MCP server logger (#10576)

Adds a daily-rotated file sink to the KB MCP server's logger that runs
regardless of `aiConfig.debug`, so long-running operations leave a tail-able
diagnostic trail. The stderr sink remains debug-flag-gated (preserves prior
behavior; avoids stdio transport corruption).

Closes the observability gap surfaced 2026-05-01 when a full from-scratch KB
resync ran for 3+ hours server-side with no operator visibility — the gate
introduced in #10573 refuses synchronous MCP-callable bulk work, but until
this change there was no substrate to *observe* the work the gate shoved off.

Changes:
- ai/mcp/server/knowledge-base/logger.mjs: dual-sink (file always-on, stderr
  debug-gated). Daily rotation + log-dir resolution lazy per-write so config
  overrides take effect without re-import.
- ai/mcp/server/knowledge-base/config.template.mjs: adds `logPath` default
  pointing at `${neoRootDir}/.neo-ai-data/logs/`.
- ai/mcp/server/knowledge-base/services/VectorService.mjs: gate refusal
  message now references the log path so agents reading the error know
  where to direct the operator.
- test/playwright/unit/ai/mcp/server/knowledge-base/logger.spec.mjs: locks
  the always-on file behavior with debug=false; covers ISO-timestamp prefix
  and daily-rotated filename pattern.

Resolves #10576

* fix(knowledge-base): defensive logPath fallback + Error serialization in logger (#10576)

Cycle 2 fixes for two RAs from @neo-gpt:

RA1: VectorService.embed gate refusal message interpolated `aiConfig.logPath`
directly. Existing gitignored config.mjs deployments without the new template
key would render `undefined/kb-server-...`. Added defensive fallback mirroring
logger.mjs's lazy resolution: `aiConfig.logPath || ${neoRootDir}/.neo-ai-data/logs`.

RA2: logger.mjs serialized Error objects with `JSON.stringify`, which yields
`{}` because Error.message and Error.stack are non-enumerable. The post-mortem
file sink then silently destroyed exactly the information it exists to
preserve. Added Error special-case (formats name + message + stack) plus a
circular-reference fallback to String() so logger throws can't crash the
caller.

Test added for both failure modes — `preserves Error stack/message in the
durable log` exercises Error serialization AND the circular-ref guard.

* test(kb): lock gate-message path-rendering + logPath fallback (#10576)

Cycle 3 fix per @neo-gpt cycle-2 RA: cycle-2 fixed the implementation but
the spec didn't assert the observable behavior. Two assertions added:

1. The above-threshold MCP test now asserts the rendered message contains
   `tail -f` + `kb-server-` and explicitly does NOT contain `undefined` —
   locks the fallback under the canonical-config setup.

2. New test `above-threshold MCP gate-message renders fallback path when
   aiConfig.logPath unset` simulates a stale gitignored config.mjs without
   the new template key and proves the fallback to
   `${neoRootDir}/.neo-ai-data/logs` renders correctly.

7/7 pass (logger.spec 2/2 + work-volume regression 5/5).

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-01T13:46:18Z @tobiu closed this issue
- 2026-05-01T13:55:23Z @neo-opus-ada cross-referenced by #10582
- 2026-05-01T14:01:49Z @neo-opus-ada cross-referenced by PR #10583
- 2026-05-16T13:44:05Z @neo-gpt cross-referenced by #11459
- 2026-05-16T15:07:22Z @neo-opus-ada cross-referenced by PR #11460

