---
id: 12019
title: 'Cloud-mode orchestrator: spawn chroma ENOENT loop on first boot'
state: CLOSED
labels:
  - bug
  - ai
  - refactoring
assignees:
  - neo-gpt
createdAt: '2026-05-26T10:46:24Z'
updatedAt: '2026-05-26T11:37:16Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12019'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-26T11:37:16Z'
---
# Cloud-mode orchestrator: spawn chroma ENOENT loop on first boot

## Context

Fifth friction signal from the first-real-world cloud-deployment exercise (siblings: #12014 #12015 #12016 #12017). With the cloud profile booted via `docker compose --profile cloud up`, the `orchestrator` container reaches `unhealthy` because its `ProcessSupervisor` enters an infinite loop trying to launch `chroma` as a host child process — but in the cloud topology, `chroma` is a peer container with its own lifecycle, not an orchestrator-managed subprocess.

The other cloud-safe lane gates (primary-dev-sync, kbSync, bridgeDaemon, golden-path-repo-enrichment, MLX) all have explicit env toggles. The chroma-supervision lane does not, so it cannot be turned off declaratively, and the orchestrator log fills with `spawn chroma ENOENT` every 15 seconds.

## The Problem

Boot a stack mirroring [ai/deploy/docker-compose.yml](ai/deploy/docker-compose.yml) `--profile cloud --profile local-model` (or any cloud-mode deployment); container logs confirm:

```
[Orchestrator] Started. summaryInterval=600000ms kbSyncInterval=1800000ms poll=3000ms.
[ProcessSupervisor] Starting chroma daemon (supervisor-restart).
[ProcessSupervisor] chroma daemon failed to start: spawn chroma ENOENT
[ProcessSupervisor] Starting chroma daemon (supervisor-restart).
[ProcessSupervisor] chroma daemon failed to start: spawn chroma ENOENT
... repeats every ~15 seconds indefinitely ...
```

The deploy-image (`node:24-alpine`-based) has no `chroma` binary — by design, since the cloud topology delegates Chroma to a separate `chroma` container reached over the compose network at `chroma:8000`. The orchestrator process correctly connects to that container for its own data-plane work; only the supervisor restart loop is wrong-shape in cloud mode.

Cloud-mode env vars currently passed (canonical):

```yaml
- NEO_AI_DEPLOYMENT_MODE=cloud
- NEO_ORCHESTRATOR_PRIMARY_DEV_SYNC_ENABLED=false
- NEO_ORCHESTRATOR_KB_SYNC_ENABLED=false
- NEO_ORCHESTRATOR_BRIDGE_DAEMON_ENABLED=false
- NEO_ORCHESTRATOR_GOLDEN_PATH_REPO_ENRICHMENT_ENABLED=false
- NEO_ORCHESTRATOR_MLX_ENABLED=false
```

The chroma-supervisor lane is missing from the gate set; `NEO_AI_DEPLOYMENT_MODE=cloud` is not consulted by the supervisor's lane registry.

Healthcheck consequence: the container is `unhealthy` after boot. `docker compose up --wait` returns non-zero on `--profile cloud`. The deployment otherwise functions (KB, MC, chroma, local-model all healthy; MC `add_memory` round-trips work once embedding models are pulled), but the orchestrator failure is visible to any monitoring/healthcheck gate.

## The Architectural Reality

Affected:
- `ai/daemons/orchestrator/Orchestrator.mjs` — the lane registry; needs awareness of cloud topology.
- `ai/daemons/orchestrator/services/` — `ProcessSupervisorService.mjs` (introduced by #11022) is the spawning surface.
- `ai/deploy/docker-compose.yml:138-144` — orchestrator env block (the gate-toggle list).

Sibling context: ADR 0014 explicitly separates the orchestrator from chroma as distinct cloud-deployable containers; this is the topology the lane registry must match.

## The Fix

Two viable shapes (implementer's call):

1. **Implicit gate via deployment-mode**: in `Orchestrator.mjs` lane registry, when `NEO_AI_DEPLOYMENT_MODE === 'cloud'`, skip the chroma supervisor lane entirely. Consistent with how `primaryDevSync` is treated (local-only by design) — same shape, applied to chroma.

2. **Explicit toggle**: add `NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED` (default `true` for backwards compatibility), set `=false` in `ai/deploy/docker-compose.yml` cloud env block. Operator-visible; matches the existing toggle pattern for the other five lanes.

Option 1 is the smaller change and matches the spirit of the cloud-vs-local lane split. Option 2 is more discoverable but adds another env var to the contract. No strong preference; implementer should pick whichever fits the existing lane-registry shape better.

Either way: the orchestrator container should become `healthy` on first cloud-profile boot without the supervisor loop.

## Contract Ledger

| Target Surface | Source of Authority | Proposed Behavior | Fallback | Docs | Evidence |
|---|---|---|---|---|---|
| Orchestrator chroma-supervisor lane | `NEO_AI_DEPLOYMENT_MODE=cloud` / `ADR 0014` cloud topology | Skip the lane in cloud mode (chroma is a peer container) | None — local-mode behavior unchanged | Inline comment + DeploymentCookbook lane table when it lands | Container logs: `spawn chroma ENOENT` repeated every ~15s on cloud-profile first boot |
| Orchestrator healthcheck | `ai/deploy/docker-compose.yml:178-182` healthcheck contract | `state.json` mtime updates as expected; container reaches `healthy` | None | None | `docker compose ps`: orchestrator `unhealthy` after fix-target boot |

## Decision Record impact

`aligned-with ADR 0014` — completes the cloud topology contract for the orchestrator's own supervision boundary.

## Acceptance Criteria

- [ ] `docker compose -f ai/deploy/docker-compose.yml --profile cloud --profile local-model up -d --wait` returns 0 against dev HEAD (regression guard).
- [ ] Container logs for `orchestrator` show no `spawn chroma ENOENT` lines after the first 30 seconds of boot.
- [ ] Local-mode behavior unchanged — `npm run orchestrator` (or the local launcher) still supervises chroma as a child process when `NEO_AI_DEPLOYMENT_MODE` is unset.
- [ ] If Option 2 is chosen: `NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED` documented alongside the other five orchestrator lane toggles.

## Out of Scope

- Reworking the entire ProcessSupervisor service. Scope is the chroma lane gate.
- Cloud-mode behavior of other supervised lanes (currently only chroma is host-vs-peer-ambiguous in the cloud topology).
- Backup-task `spawn git ENOENT` failure observed in the same logs — separate symptom, separate ticket if filed.

## Avoided Traps

- Filing as `enhancement` — rejected; the cloud-profile deployment is `unhealthy` on first boot, which is a correctness bug. `--profile cloud` is currently broken end-to-end at the orchestrator boundary.
- Removing chroma supervision entirely — rejected; local-mode developers still use the single-process orchestrator-supervises-chroma model.

## Related

- #11022 — introduced `ProcessSupervisorService` (the surface that needs the gate).
- #11844 — orchestrator co-location refactor (where the lane registry now lives).
- #11722 — deployment maintenance config (where the cloud-mode env toggles were established).
- ADR 0014 — cloud topology authority (the boundary that chroma-supervision crosses in cloud mode).
- #12014 — sibling first-real-world deployment friction.
- #12015 / #12016 / #12017 — siblings from the same exercise.

## Handoff Retrieval Hints

- Container log evidence: `docker logs <orchestrator-container> 2>&1 | grep "spawn chroma ENOENT"` on a freshly booted cloud-profile stack.
- Semantic: "orchestrator cloud-mode chroma supervisor lane", "ProcessSupervisor spawn ENOENT loop".
- File anchors: `ai/daemons/orchestrator/Orchestrator.mjs` lane registry, `ai/daemons/orchestrator/services/ProcessSupervisorService.mjs`, `ai/deploy/docker-compose.yml:138-144`.

## Timeline

- 2026-05-26T10:46:25Z @neo-opus-ada added the `bug` label
- 2026-05-26T10:46:25Z @neo-opus-ada added the `ai` label
- 2026-05-26T10:46:26Z @neo-opus-ada added the `refactoring` label
- 2026-05-26T10:49:05Z @neo-gpt assigned to @neo-gpt
- 2026-05-26T11:07:36Z @neo-gpt referenced in commit `ca1a73b` - "fix(orchestrator): gate chroma supervision in cloud mode (#12019)"
- 2026-05-26T11:09:35Z @neo-gpt cross-referenced by PR #12018
- 2026-05-26T11:37:15Z @tobiu referenced in commit `55865f8` - "fix(deploy): restore cloud compose buildability (#12014, #12016, #12017, #12019) (#12018)

* fix(orchestrator): align daemon entrypoint references (#12014)

* fix(deploy): add root dockerignore for compose context (#12017)

* fix(deploy): quote local-model resource interpolation (#12016)

* fix(deploy): colocate dockerignore with Dockerfile (#12017)

* fix(orchestrator): gate chroma supervision in cloud mode (#12019)

* chore(ci): retrigger PR checks (#12014)

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-26T11:37:16Z @tobiu closed this issue
- 2026-05-26T13:09:13Z @neo-opus-ada cross-referenced by #12022
- 2026-05-26T13:34:37Z @neo-opus-ada cross-referenced by #12025
- 2026-05-26T15:40:34Z @neo-opus-ada cross-referenced by #12026
- 2026-05-26T15:46:16Z @neo-opus-ada cross-referenced by PR #12024
- 2026-05-26T16:11:23Z @neo-opus-ada cross-referenced by #12028
- 2026-05-26T16:12:27Z @neo-opus-ada cross-referenced by #12029
- 2026-05-26T17:02:06Z @neo-opus-ada cross-referenced by PR #12020

