---
id: 12138
title: Orchestrator recycles Chroma after a max runtime (kill→restart→defrag)
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
assignees:
  - neo-opus-ada
createdAt: '2026-05-28T16:50:15Z'
updatedAt: '2026-06-07T17:16:45Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12138'
author: neo-opus-ada
commentsCount: 1
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy:
  - '[x] 12140 defragChromaDB orphan-cleanup deletes live HNSW segment dirs in the unified store'
  - '[x] 12136 Orchestrator must guarantee a single Chroma daemon (probe + host-pin)'
blocking: []
closedAt: '2026-05-28T20:56:13Z'
---
# Orchestrator recycles Chroma after a max runtime (kill→restart→defrag)

## Context

PR #12137 makes the orchestrator reap duplicate Chroma daemons (singularity). Separately, a long-lived Chroma process accumulates two distinct problems: process-level cruft (memory/fd) and on-disk HNSW index fragmentation (we observed **307 segment dirs** in the unified store this session). The operator does not want a Chroma daemon running unrecycled for long stretches (e.g. a month).

## The Problem

A never-recycled Chroma daemon (a) carries process-level cruft and (b) serves an increasingly fragmented on-disk index. SIGTERM does **not** work on chroma — empirically confirmed across hundreds of attempts — only `kill -9` reliably stops it.

## The Architectural Reality

- The orchestrator already supervises chroma as a continuous task (`ai/daemons/orchestrator/Orchestrator.mjs` `poll()` + `TaskDefinitions.mjs` chroma task). Daemon age is trackable via `taskState('chroma').lastRunAt` (set at spawn; survives orchestrator restarts via persisted state).
- The recycle order MUST be **kill → restart → defrag**: `defragChromaDB.mjs` extracts/rebuilds via the ChromaClient API and needs a LIVE daemon, so it runs against the fresh restarted process. kill+restart first heals a corrupted/hung process; defrag then compacts the index on the healthy daemon.
- `engines.chroma.{host,port}` are the unified-store coordinates; `Config.maintenance.defrag.intervalMs` (7d) already exists as a maintenance-cadence precedent.

## The Fix

1. New config `orchestrator.chroma.maxRuntimeMs` (+ env `NEO_CHROMA_MAX_RUNTIME_MS`; concrete default in the template, e.g. 24h).
2. Orchestrator: when `now - taskState('chroma').lastRunAt > maxRuntimeMs`, run the chain — `kill -9` the tracked daemon → existing supervisor respawn → once the fresh daemon is `chromaConnect`-ready, run the safe defrag.
3. **Prerequisite (dependency):** a unified-store-safe defrag. Today's `defragChromaDB.mjs` is unsafe for the unified store — its step-6 orphan-cleanup (`:471`) deletes MC's segment dirs in the shared persist dir → MC data loss. The safe defrag must rebuild ALL collections (KB + MC) and never delete sibling-collection dirs; ideally shadow-swap (rebuild into a shadow collection, atomic-swap) for zero/minimal downtime — KB already has shadow-swap machinery; MC would need it for the unified rebuild.
4. Gated by `chromaDaemonEnabled` (off in cloud, where chroma is externally managed).

## Contract Ledger

| Target Surface | Source of Authority | Proposed Behavior | Fallback | Docs | Evidence |
|---|---|---|---|---|---|
| `orchestrator.chroma.maxRuntimeMs` (config) | `ai/config.template.mjs` (+ `NEO_CHROMA_MAX_RUNTIME_MS`) | new; concrete default ~24h | n/a (new) | config JSDoc | new |
| `defragChromaDB.mjs` unified-store safety | `ai/scripts/maintenance/defragChromaDB.mjs:471` | step-6 orphan-cleanup must preserve ALL collections' segment dirs in a shared instance | current behavior = MC data loss | JSDoc | read 2026-05-28 |

## Decision Record impact

`aligned-with ADR 0003` (unified Chroma topology).

## Acceptance Criteria

- [ ] `orchestrator.chroma.maxRuntimeMs` config + env binding; concrete default in the template.
- [ ] Orchestrator recycles chroma when daemon age > `maxRuntimeMs`, in kill→restart→defrag order, gated by `chromaDaemonEnabled`.
- [ ] Recycle uses `kill -9` (SIGTERM is not attempted — chroma ignores it).
- [ ] Defrag runs only after the restarted daemon is connection-ready, and uses the unified-store-safe defrag.
- [ ] Unit tests: max-runtime trigger logic; OS-level kill is a no-op under `UNIT_TEST_MODE` (no real process touched).
- [ ] (post-merge) operator-observed: a daemon older than `maxRuntimeMs` is recycled and the store is compacted afterward.

## Out of Scope

- The kill-dups singularity net (PR #12137).
- External-only mode / removing `manage_database` etc. (separate ticket).

## Related

- #12136 / PR #12137 (kill-dups singularity — this builds on it).
- Sibling: "KB + MC external-only Chroma; orchestrator as single source of truth".

Origin Session ID: 51ac9efd-4b5a-488e-b3dd-aa2673acbec9

Filed by @neo-opus-ada.

## Timeline

- 2026-05-28T16:50:16Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-28T16:50:16Z @neo-opus-ada added the `enhancement` label
- 2026-05-28T16:50:17Z @neo-opus-ada added the `ai` label
- 2026-05-28T16:50:17Z @neo-opus-ada added the `architecture` label
- 2026-05-28T16:52:06Z @neo-opus-ada marked this issue as being blocked by #12136
### @neo-opus-ada - 2026-05-28T16:57:37Z

Reaper polish from @neo-gpt's #12137 review (non-blocking; fold in when this ticket reworks the reaper/supervision for recycle):

1. `reapDuplicateListeners` swallows a `listPortListeners` throw as `[]`. Correct for the lsof-absent / nothing-listening case, but an *unexpected* lsof failure then silently no-ops the reap. Add a debug log distinguishing lsof-absent from unexpected-failure.
2. The reaper logs `WARN` per reaped pid; under a pathological flap (a peer repeatedly respawning chroma) that is noisy. Bound it with a dedup window like `shouldLogRunningSkip`.


- 2026-05-28T17:30:29Z @neo-opus-ada marked this issue as being blocked by #12140
- 2026-05-28T17:44:03Z @neo-opus-ada cross-referenced by PR #12141
- 2026-05-28T17:58:25Z @neo-opus-ada cross-referenced by #12142
- 2026-05-28T17:58:34Z @neo-opus-ada cross-referenced by #12143
- 2026-05-28T20:03:49Z @neo-opus-ada cross-referenced by PR #12149
- 2026-05-28T20:20:39Z @neo-opus-ada referenced in commit `d72e080` - "fix(ai): chroma recycle skips uninitialized lastRunAt=0 (#12138)

isChromaRecycleDue computed uptime as now-lastRunAt; a running task with the uninitialized lastRunAt=0 (test-harness default, or any never-stamped state) read as now-0 = huge uptime and spuriously recycled. With a real maxRuntimeMs present this fired inside unrelated orchestrator poll() tests, polluting their outcome/spawn assertions (green locally only because the stale local config.mjs lacks orchestrator.chroma; CI hydrates it from the template). Guard with lastRunAt > 0 — a genuinely running daemon always carries a real markStarted stamp. Adds a regression test for the running + lastRunAt=0 + maxRuntimeMs condition.

Co-Authored-By: neo-opus-4-7 <neo-opus-4-7@neomjs.com>"
- 2026-05-28T20:44:08Z @neo-opus-ada referenced in commit `b197936` - "docs(ai): chroma recycle defrag is best-effort, not restart-durable (#12138)

Per @neo-gpt review of PR #12149: the post-recycle defrag uses a process-local pending flag, so a mid-recycle orchestrator restart (after killTask, before the defrag) skips it for that cycle. Corrected the inaccurate 're-derived from uptime' comment to state the best-effort contract — the next max-runtime recycle compacts the store; persisting the intent would add a pending-forever/endless-retry failure mode for an idempotent compaction step; the kill->restart (primary recycle value) is unaffected. Comment-only; no behavior change.

Co-Authored-By: neo-opus-4-7 <neo-opus-4-7@neomjs.com>"
- 2026-05-28T20:56:14Z @tobiu referenced in commit `f518f50` - "feat(ai): orchestrator recycles chroma after max runtime (#12138) (#12149)

* feat(ai): add orchestrator.chroma.maxRuntimeMs config + getter (#12138)

Config scaffolding for the Chroma max-runtime recycle: orchestrator.chroma.maxRuntimeMs (24h default) + NEO_CHROMA_MAX_RUNTIME_MS env binding + Orchestrator.chromaMaxRuntimeMs getter (reads config verbatim, no hidden fallback). The poll() recycle logic that consumes it follows in this branch.

Co-Authored-By: neo-opus-4-7 <neo-opus-4-7@neomjs.com>

* feat(ai): orchestrator recycles chroma after max runtime (#12138)

Long-lived chroma daemons accumulate process cruft + on-disk HNSW
fragmentation. The orchestrator now recycles the supervised chroma daemon
once its uptime exceeds orchestrator.chroma.maxRuntimeMs (default 24h, env
NEO_CHROMA_MAX_RUNTIME_MS): SIGKILL the over-age process (chroma ignores
SIGTERM), let the existing supervisor branch respawn it, then run the
unified-store-safe ai:defrag-kb against the fresh daemon once it is
connection-ready. Gated by chromaDaemonEnabled (no-op in cloud / externally
managed). MC defrag is intentionally deferred (no rebuild-from-source
fallback) to #12142.

Implementation:
- Orchestrator: isChromaRecycleDue predicate + probeChromaReady TCP readiness
  gate + poll() recycle/defrag branches driven by an in-memory pending-defrag
  flag (re-derived from uptime on orchestrator restart, so unpersisted).
- ProcessSupervisorService.killTask: SIGKILLs the tracked pid (no-op under
  UNIT_TEST_MODE) + markRecycled + records the outcome.
- TaskStateService.markRecycled: clears running/pid so the supervisor loop
  respawns the daemon (robust for adopted processes; deterministic in tests).
- TaskDefinitions: one-shot chromaDefrag task (node defragChromaDB.mjs
  --target knowledge-base).
- config.template.mjs: maintenance.defrag doc carve-out noting the recycle is
  the one orchestrator path that auto-spawns defrag.

Tests: 7 recycle specs (trigger / disabled / cloud-gated / within-ceiling /
ready / not-ready) + 3 killTask specs + the createInitialTaskState keys
update. Orchestrator 38, ProcessSupervisor 14, TaskState 4 — all green.

Co-Authored-By: neo-opus-4-7 <neo-opus-4-7@neomjs.com>

* fix(ai): chroma recycle skips uninitialized lastRunAt=0 (#12138)

isChromaRecycleDue computed uptime as now-lastRunAt; a running task with the uninitialized lastRunAt=0 (test-harness default, or any never-stamped state) read as now-0 = huge uptime and spuriously recycled. With a real maxRuntimeMs present this fired inside unrelated orchestrator poll() tests, polluting their outcome/spawn assertions (green locally only because the stale local config.mjs lacks orchestrator.chroma; CI hydrates it from the template). Guard with lastRunAt > 0 — a genuinely running daemon always carries a real markStarted stamp. Adds a regression test for the running + lastRunAt=0 + maxRuntimeMs condition.

Co-Authored-By: neo-opus-4-7 <neo-opus-4-7@neomjs.com>

* docs(ai): chroma recycle defrag is best-effort, not restart-durable (#12138)

Per @neo-gpt review of PR #12149: the post-recycle defrag uses a process-local pending flag, so a mid-recycle orchestrator restart (after killTask, before the defrag) skips it for that cycle. Corrected the inaccurate 're-derived from uptime' comment to state the best-effort contract — the next max-runtime recycle compacts the store; persisting the intent would add a pending-forever/endless-retry failure mode for an idempotent compaction step; the kill->restart (primary recycle value) is unaffected. Comment-only; no behavior change.

Co-Authored-By: neo-opus-4-7 <neo-opus-4-7@neomjs.com>

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-28T20:56:14Z @tobiu closed this issue
- 2026-05-28T23:41:55Z @neo-gpt cross-referenced by PR #12152
- 2026-06-01T03:30:35Z @neo-gpt cross-referenced by #12180

