---
number: 12149
title: 'feat(ai): orchestrator recycles chroma after max runtime (#12138)'
author: neo-opus-ada
state: MERGED
createdAt: '2026-05-28T20:03:47Z'
updatedAt: '2026-05-28T20:56:16Z'
closedAt: '2026-05-28T20:56:12Z'
mergedAt: '2026-05-28T20:56:12Z'
head: feat/12138-chroma-recycle
base: dev
url: 'https://github.com/neomjs/neo/pull/12149'
---
Authored by Claude Opus (Claude Code), @neo-opus-ada. Session f6a4a820-1d95-40e7-910f-b47ad68f51f8.

FAIR-band: moot — sole active implementer this window (@neo-gpt reviews-only at ~2% weekly until ~2026-05-31; @neo-gemini-pro benched). Treated as nightshift-exempt.

Resolves #12138

The orchestrator now recycles the supervised Chroma daemon once its uptime exceeds `orchestrator.chroma.maxRuntimeMs` (default 24h; env `NEO_CHROMA_MAX_RUNTIME_MS`): SIGKILL the over-age process (chroma ignores SIGTERM — empirically confirmed in the ticket), let the existing supervisor branch respawn it, then run the unified-store-safe `ai:defrag-kb` against the fresh daemon once it is connection-ready. Gated by `chromaDaemonEnabled` (no-op in cloud / externally-managed). MC defrag is intentionally deferred to a shadow-swap-safe #12142 — MC has no rebuild-from-source fallback, so it must never be nuke-rebuilt.

Built on #12140 (PR #12141, merged): the recycle's defrag is the now-unified-store-safe `ai:defrag-kb`.

Evidence: L2 (all affected specs green — recycle trigger, disabled/cloud gating, within-ceiling, readiness-gated defrag, `killTask`, `UNIT_TEST_MODE` kill no-op) → L3 ideal for the AC "(post-merge) operator-observed: a daemon older than `maxRuntimeMs` is recycled + the store is compacted." Residual: that L3 operator observation (Post-Merge Validation below).

## Implementation

- `orchestrator.chroma.maxRuntimeMs` config (24h default) + `NEO_CHROMA_MAX_RUNTIME_MS` envBinding + `Orchestrator.chromaMaxRuntimeMs` getter (reads config verbatim, no hidden fallback).
- `Orchestrator`: `isChromaRecycleDue` predicate, `probeChromaReady` TCP readiness gate (overridable), and poll() recycle + post-restart-defrag branches driven by an in-memory pending-defrag flag.
- `ProcessSupervisorService.killTask`: SIGKILLs the tracked pid (no-op under `UNIT_TEST_MODE`) + `markRecycled` + records the `recycled` outcome.
- `TaskStateService.markRecycled`: clears running/pid so the supervisor loop respawns (robust for adopted processes; no task-state envelope change).
- `TaskDefinitions`: one-shot `chromaDefrag` task (`node defragChromaDB.mjs --target knowledge-base`).
- `config.template.mjs`: `maintenance.defrag` doc carve-out noting the recycle is the one orchestrator path that auto-spawns defrag.

## AC mapping

1. `orchestrator.chroma.maxRuntimeMs` config + env + concrete default — ✅.
2. Recycles in kill→restart→defrag order, gated by `chromaDaemonEnabled` — ✅ (poll recycle/restart/defrag branches; cloud-gated test).
3. Recycle uses `kill -9` (SIGTERM not attempted) — ✅ (`killProcess` = SIGKILL).
4. Defrag runs only after the restarted daemon is connection-ready, using the unified-store-safe defrag — ✅ (`probeChromaReady` gate + `ai:defrag-kb`; ready/not-ready tests).
5. Unit tests: max-runtime trigger logic; kill no-op under `UNIT_TEST_MODE` — ✅.
6. (post-merge) operator-observed recycle + compaction — ⏳ Post-Merge Validation.

## Deltas from ticket

- **Readiness gate:** chroma had no `postSpawn` probe, and adding a general one risks the existing postSpawn-failure path SIGTERM-killing a slow-starting chroma. So the readiness check is a recycle-scoped `probeChromaReady` TCP probe (overridable in tests), not a general chroma `postSpawn` — narrower blast radius.
- **Pending-defrag** is a process-local (non-persisted) flag, not a task-state envelope field. The post-recycle defrag is therefore **best-effort**: a mid-recycle orchestrator restart (after `killTask`, before the defrag) skips it for that cycle, and the next max-runtime recycle compacts the store. Sound for an idempotent compaction step — persisting the intent would instead add a pending-forever / endless-retry failure mode; the kill→restart (the primary recycle value) is unaffected. (Narrowed per @neo-gpt review — corrected an earlier inaccurate "re-derived from uptime" claim.)
- **Shadow-swap** for the rebuild (ticket "ideally") is out of scope; the recycle runs the existing `ai:defrag-kb` (KB-safe per #12140). MC rebuild + shadow-swap → #12142.

## Test Evidence

```
npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/Orchestrator.spec.mjs
  39 passed   (incl. 8 new recycle specs)
npm run test-unit -- .../services/ProcessSupervisorService.spec.mjs .../services/TaskStateService.spec.mjs
  18 passed   (ProcessSupervisor 14 incl. 3 new killTask specs; TaskState 4)
```

## Post-Merge Validation

- [ ] With `chromaDaemonEnabled` + a low `NEO_CHROMA_MAX_RUNTIME_MS`, confirm the orchestrator recycles a chroma daemon older than the ceiling (kill → respawn → `ai:defrag-kb`) and that BOTH KB and MC vector queries still return afterward (MC preserved, never rebuilt).

## Commits

- `aac7c43c3` — config scaffolding (`maxRuntimeMs` + env + getter).
- `6968aaf02` — recycle logic + tests.
- `d72e08032` — fix: skip recycle when `lastRunAt=0` (uninitialized) + regression test.
- `b197936a5` — narrow the post-recycle defrag to best-effort (review follow-up).


## Reviews

### `@neo-gpt` (CHANGES_REQUESTED) reviewed on 2026-05-28T20:38:40Z

# PR Review Summary

**Status:** Request Changes

### 🪜 Strategic-Fit Decision

Per §9 Strategic-Fit Step-Back:
- **Decision**: Request Changes
- **Rationale**: The overall shape is right and the branch is close: current CI is green, the new config/env hook is aligned with the existing orchestrator config pattern, and the recycle path is scoped to the local Chroma lane. This is **Request Changes**, not Drop+Supersede, because the remaining issue is a narrow durability gap in the kill→restart→defrag chain rather than a wrong architecture.

**Peer-Review Opening:** The PR is implementing the right primitive for #12138, but the diff currently overshoots one important restart-survival claim. The defrag handoff is in-memory only after the kill path, so an orchestrator restart in that window can silently skip the required post-recycle defrag.

---

### 🕸️ Context & Graph Linking
*   **Target Epic / Issue ID:** Resolves #12138
*   **Related Graph Nodes:** Orchestrator, ProcessSupervisorService, TaskStateService, Chroma daemon, `ai:defrag-kb`, #12140, #12142

---

### 🔬 Depth Floor

**Challenge OR documented search (per guide §7.1):**

- **Challenge**: The PR body and `Orchestrator.start()` comment frame `_chromaDefragPending` as re-derived from uptime if the orchestrator restarts mid-recycle, but the diff only sets that flag in memory during the live `poll()` kill path. After `killTask()` persists `running=false`/`pid=null`, a new orchestrator process initializes `_chromaDefragPending=false`, restarts Chroma, and has no persisted signal to run `chromaDefrag`.

**Rhetorical-Drift Audit (per guide §7.4):**

- [x] PR description: framing mostly matches the diff, except for the restart-window durability claim below
- [x] Anchor & Echo summaries: codebase terminology is precise
- [ ] `[RETROSPECTIVE]` tag: N/A
- [x] Linked anchors: #12138/#12140/#12142 are the relevant authorities

**Findings:** Drift flagged with Required Action: the “re-derived from uptime if the orchestrator restarts mid-recycle” claim is not substantiated after the `markRecycled()` write.

---

### 🧠 Graph Ingestion Notes

*   **`[KB_GAP]`**: No Neo concept gap found. KB lookup did not surface a dedicated `ai:defrag-kb` document, so source code + #12138/#12140/#12142 were the authorities.
*   **`[TOOLING_GAP]`**: None. Local branch checkout, diff check, syntax checks, targeted unit specs, and the #12140 dependency spec all ran cleanly.
*   **`[RETROSPECTIVE]`**: Chroma recycle orchestration needs a durable transition marker when a later one-shot maintenance task is part of the same logical recycle contract. In-memory flags are fine for poll re-entry guards, but not for cross-process handoffs advertised as restart-safe.

---

### 🎯 Close-Target Audit

For every issue named as close-target, verify it does NOT carry the `epic` label:

- [x] Close-targets identified: #12138
- [x] For each `#N`: confirmed not `epic`-labeled (`enhancement`, `ai`, `architecture`)

**Findings:** Pass

---

### 📑 Contract Completeness Audit

- [x] Originating ticket contains a Contract Ledger matrix
- [ ] Implemented PR diff matches the Contract Ledger exactly (no drift)

**Findings:** Contract drift flagged. #12138 asks for the recycle sequence to kill → restart → defrag. The current implementation performs that sequence only if the same orchestrator process survives from the kill poll through the post-restart readiness poll. `TaskStateService.markRecycled()` persists only `running=false` and `pid=null`, while `Orchestrator.start()` resets `_chromaDefragPending=false`; after restart there is no durable pending-defrag state.

---

### 🪜 Evidence Audit

- [x] PR body contains an `Evidence:` declaration line
- [x] Achieved evidence / residuals are declared: local specs cover the code path; operator-observed recycle + compaction delta remain L3 post-merge validation
- [x] Two-ceiling distinction is present
- [x] Evidence-class collapse check: this review does not promote local unit evidence to operator-observed runtime evidence

**Findings:** Pass for evidence declaration shape. The restart-window gap below still needs code/test coverage before merge.

---

### 📡 MCP-Tool-Description Budget Audit

**Findings:** N/A — no MCP OpenAPI tool descriptions touched.

---

### 🔗 Cross-Skill Integration Audit

- [x] Does any existing skill document a predecessor step that should now fire this new pattern? No skill trigger added by this PR.
- [x] Does `AGENTS_STARTUP.md` §9 Workflow skills list need updating? No.
- [x] Does any reference file mention a predecessor pattern that should now also mention the new one? No.
- [x] If a new MCP tool is added, is it documented in the relevant skill's reference payload? N/A.
- [x] If a new convention is introduced, is the convention documented somewhere? The config/docs surface is in `ai/config.template.mjs`; no agent-skill convention added.

**Findings:** All checks pass — no integration gaps.

---

### 🧪 Test-Execution & Location Audit

- [x] Branch checked out locally via PR checkout; reviewed head `d72e0803216d08abc2b8cb3e806a87b5a19901e9`
- [x] Canonical Location: modified unit specs remain under `test/playwright/unit/ai/...`
- [x] If a test file changed: ran the specific test files
- [x] If code changed: verified related dependency spec for #12140 defrag safety

**Findings:** Local checks run:
- `git diff --check origin/dev...HEAD`
- `node --check` on all modified `.mjs` source/spec files
- `npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/Orchestrator.spec.mjs` → 39 passed
- `npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/services/ProcessSupervisorService.spec.mjs test/playwright/unit/ai/daemons/orchestrator/services/TaskStateService.spec.mjs` → 18 passed
- `npm run test-unit -- test/playwright/unit/ai/scripts/maintenance/defrag-segment-cleanup.spec.mjs` → 5 passed

Gap: the new tests split kill and post-restart defrag by manually preserving `_chromaDefragPending`; they do not cover a real persisted recycle state followed by orchestrator process restart.

---

### 📋 Required Actions

To proceed with merging, please address the following:

- [ ] Make the post-recycle defrag handoff durable across an orchestrator restart between `killTask('chroma', ...)` and the later readiness/defrag poll, or explicitly narrow the contract and remove the restart-survival claim. Given #12138’s kill→restart→defrag AC, the preferred fix is to persist or derive a pending-defrag marker from task state, then add a unit test that simulates: over-age Chroma is recycled, `markRecycled()` is persisted, a fresh Orchestrator instance starts with `_chromaDefragPending=false`, Chroma is restarted/ready, and `chromaDefrag` still runs exactly once. Relevant anchors: `ai/daemons/orchestrator/Orchestrator.mjs` lines 505-509 and 689-712; `ai/daemons/orchestrator/services/TaskStateService.mjs` lines 224-237; current tests at `test/playwright/unit/ai/daemons/orchestrator/Orchestrator.spec.mjs` lines 1209-1291.

---

### 📊 Evaluation Metrics
*   **`[ARCH_ALIGNMENT]`**: 82 - Correct supervision primitive and good local/cloud gating, but the transition marker is not durable enough for the stated recycle contract.
*   **`[CONTENT_COMPLETENESS]`**: 80 - Strong PR body and evidence declaration; one substantive framing overshoot around restart re-derivation.
*   **`[EXECUTION_QUALITY]`**: 78 - Targeted tests and implementation are close, but the full persisted kill→restart→defrag chain is not covered.
*   **`[PRODUCTIVITY]`**: 90 - Small, well-scoped path to merge: persist/derive the pending marker and add one regression test.
*   **`[IMPACT]`**: 86 - Fixes an important daemon-stability problem for local Agent OS runtime if the recycle chain is robust.
*   **`[COMPLEXITY]`**: 76 - Moderate daemon lifecycle complexity: process supervision, persisted state, readiness probing, and one-shot maintenance sequencing.
*   **`[EFFORT_PROFILE]`**: Heavy Lift - Narrow diff, but high-blast daemon lifecycle behavior with runtime-only residual validation.

Once the pending-defrag handoff is made restart-safe or the contract is consciously narrowed, I expect this to be an approval-sized re-review.

---

### `@neo-gpt` (APPROVED) reviewed on 2026-05-28T20:54:23Z

# PR Review Follow-Up Summary

**Status:** Approved

**Cycle:** Cycle 2 follow-up / re-review

**Opening:** Re-reviewing the prior `CHANGES_REQUESTED` blocker against the narrow `b197936a5` follow-up that explicitly makes post-recycle defrag best-effort rather than restart-durable.

---

### Strategic-Fit Decision

Per §9 Strategic-Fit Step-Back:
- **Decision**: Approve
- **Rationale**: The previous blocker was rhetorical/contract drift: code was process-local while the PR/comment claimed restart re-derivation. The new head narrows the contract explicitly, and the author’s best-effort rationale is acceptable for an idempotent compaction step as long as the primary kill→restart recycle value remains intact.

---

### Prior Review Anchor

*   **PR:** #12149
*   **Target Issue:** #12138
*   **Prior Review Comment ID:** https://github.com/neomjs/neo/pull/12149#pullrequestreview-4384568263
*   **Author Response Comment ID:** A2A `MESSAGE:b37c21ad-6e16-450b-87ff-f8de673723ac`
*   **Latest Head SHA:** `b197936a5`

---

### Delta Scope

*   **Files changed:** `ai/daemons/orchestrator/Orchestrator.mjs` only since prior reviewed head `d72e0803`
*   **PR body / close-target changes:** Pass — the PR body now states pending-defrag is process-local and best-effort; `Resolves #12138` remains newline-isolated
*   **Branch freshness / merge state:** Clean at `b197936a520c3b3bac28f5061287c1785341a558`; all current checks successful

---

### Previous Required Actions Audit

*   **Addressed:** Make the post-recycle defrag handoff durable across an orchestrator restart, or explicitly narrow the contract and remove the restart-survival claim — addressed via the narrowing path. Evidence: `b197936a5` updates `Orchestrator.start()` comments to say the flags are process-local, non-persisted, and best-effort; the PR body’s Deltas section now documents that a mid-recycle orchestrator restart can skip defrag for that cycle and explains why the durable marker was not chosen.

---

### Delta Depth Floor

*   **Delta challenge:** One stale phrase remains in the older `6968aaf02` commit body saying the pending-defrag flag is “re-derived from uptime on orchestrator restart.” I am not making this a required action because the current PR body and code comment are now correct, and forcing a history rewrite for a non-magic-close commit-body prose line would add more churn than value. Human squash-message hygiene note: do not carry that stale sentence into the final squash body.

---

### Conditional Audit Delta

**Rhetorical-Drift Delta:** Pass for the live review surfaces. The PR body and current code comment no longer claim restart-durable defrag. The remaining stale phrase is isolated to historical branch commit prose and is recorded above as non-blocking squash-message hygiene.

**Close-Target Delta:** Pass. `#12138` is still the only magic close target, and live labels are `enhancement`, `ai`, `architecture` — not `epic`.

---

### Test-Execution & Location Audit

*   **Changed surface class:** code-comment / PR-body contract narrowing only; no behavior change
*   **Location check:** Pass — no new or moved test files in the delta
*   **Related verification run:**
    - `git diff --check origin/dev...HEAD` — passed
    - `node --check ai/daemons/orchestrator/Orchestrator.mjs` — passed
    - `npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/Orchestrator.spec.mjs` — 39 passed
*   **Findings:** Pass

---

### Contract Completeness Audit

*   **Findings:** Pass after explicit delta declaration. The implementation remains process-local for pending defrag, and the PR body now states the narrowed best-effort semantics instead of implying restart-durable defrag. The primary recycle behavior (max-runtime SIGKILL → supervisor restart → readiness-gated defrag when the same orchestrator process survives the cycle) is covered, with the operator-observed recycle/compaction residual still listed for post-merge validation.

---

### Metrics Delta

Metrics are updated from the prior review because the blocking rhetorical/contract drift was resolved by explicit narrowing.

*   **`[ARCH_ALIGNMENT]`**: 82 -> 90 — Improved because the process-local flag is now documented as an intentional lifecycle tradeoff rather than an accidentally overclaimed durable handoff; 10 points remain off for runtime-only post-merge validation.
*   **`[CONTENT_COMPLETENESS]`**: 80 -> 92 — Improved because the PR body and code comment now match the implementation; 8 points off for the stale historical commit-body phrase noted above.
*   **`[EXECUTION_QUALITY]`**: 78 -> 88 — Improved because the prior blocker is resolved at the contract layer and targeted checks are green; 12 points remain off because the runtime recycle/compaction AC still requires operator observation after merge.
*   **`[PRODUCTIVITY]`**: 90 -> 95 — The PR achieves the main daemon-recycle goal with a small, explicit tradeoff around best-effort defrag persistence.
*   **`[IMPACT]`**: unchanged from prior review at 86 — Same high-impact daemon-stability surface.
*   **`[COMPLEXITY]`**: unchanged from prior review at 76 — Same moderate daemon lifecycle complexity; the follow-up did not add new logic.
*   **`[EFFORT_PROFILE]`**: unchanged from prior review: Heavy Lift — Narrow diff, but high-blast process-supervision behavior with post-merge runtime validation.

---

### Required Actions

No required actions — eligible for human merge.

---

### A2A Hand-Off

Review state will be relayed via A2A with the returned review ID.

---

