---
id: 12025
title: Orchestrator healthcheck probes wrong filename (state.json vs orchestrator-state.json)
state: CLOSED
labels:
  - bug
  - ai
  - refactoring
assignees:
  - neo-gpt
createdAt: '2026-05-26T13:34:36Z'
updatedAt: '2026-06-07T17:15:58Z'
githubUrl: 'https://github.com/neomjs/neo/issues/12025'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-26T15:47:18Z'
---
# Orchestrator healthcheck probes wrong filename (state.json vs orchestrator-state.json)

## Context

Seventh friction signal from the first-real-world cloud-deployment exercise (siblings #12014/#12015/#12016/#12017/#12019/#12022). The cloud-profile orchestrator container `--profile cloud` boots cleanly and runs maintenance tasks correctly, but never reaches Docker healthy state because the healthcheck filename does not match the file the orchestrator actually writes.

## The Problem

[ai/deploy/docker-compose.yml:179](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.yml#L179) defines the healthcheck:

```yaml
test: ["CMD-SHELL", "node -e \"const fs=require('fs');const f=(process.env.NEO_AI_ORCHESTRATOR_DIR||'/app/.neo-ai-data/orchestrator-daemon')+'/state.json';try{const m=fs.statSync(f).mtimeMs;process.exit(Date.now()-m<600000?0:1)}catch(e){process.exit(1)}\""]
```

→ probes `<NEO_AI_ORCHESTRATOR_DIR>/state.json`.

[ai/daemons/orchestrator/Orchestrator.mjs:457](https://github.com/neomjs/neo/blob/dev/ai/daemons/orchestrator/Orchestrator.mjs#L457) initializes the stateFile:

```js
this.stateFile = options.stateFile || path.join(dataDir, 'orchestrator-state.json');
```

→ writes `<NEO_AI_ORCHESTRATOR_DIR>/orchestrator-state.json` (prefixed).

`TaskStateService.persist()` then writes to that path on every task lifecycle transition. Empirical observation on a freshly built cloud-profile orchestrator container:

```
$ docker exec <deployment->orchestrator ls -la /app/.neo-ai-data/orchestrator-daemon/
total 20
-rw-r--r-- 1 root root    1 May 26 13:26 orchestrator-daemon.pid
-rw-r--r-- 1 root root 1978 May 26 13:28 orchestrator-state.json
-rw-r--r-- 1 root root 1676 May 26 13:28 orchestrator.log
```

`state.json` does not exist; the healthcheck `try { fs.statSync(f) }` throws ENOENT every probe, exits 1, container remains permanently `unhealthy`. `docker compose --profile cloud up -d --wait` returns non-zero.

The daemon itself is functional — logs show `[Orchestrator] Started`, `session summarization completed successfully`, `memory core backup completed successfully`, etc. Only the docker-health label is wrong.

V-B-A: changing the <deployment->mirror compose healthcheck path from `/state.json` to `/orchestrator-state.json` (single-line edit) immediately produces a healthy container with no other changes.

## The Architectural Reality

Affected:
- `ai/deploy/docker-compose.yml:179` — the healthcheck path.
- Counterpart confusion in `ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:571` JSDoc, which states *"the orchestrator state file (`<DEFAULT_DATA_DIR>/state.json` per `TaskDefinitions.mjs`)"* — same drift in the documentation comment.
- Source of truth: `Orchestrator.mjs:457` + `TaskStateService` consumer, both reference `orchestrator-state.json`.

The healthcheck was added by PR #11937 ("docker-compose.yml: add orchestrator + ingress healthchecks for cloud-deployment trial operator visibility"). At that time the canonical filename was either pre-rename, or the healthcheck author didn't run the canonical compose end-to-end against a fresh orchestrator container. Either way, the mismatch shipped.

## The Fix

Two viable shapes. Implementer's call:

1. **Update the healthcheck path** (smallest, safest — single-file change):
   ```yaml
   test: ["CMD-SHELL", "node -e \"... +'/orchestrator-state.json'; ...\""]
   ```

2. **Rename the file** in `Orchestrator.mjs:457` from `orchestrator-state.json` to `state.json`. Requires sweep of any consumers that expect the current name (backup tooling, observability tooling, external operators reading the file directly). Higher blast radius; only do this if there's an architectural reason to prefer the shorter name.

Option 1 is the clean fix. Option 2 is mentioned for completeness.

Also: fix the `TenantRepoSyncService.mjs:571` JSDoc to cite the correct filename, regardless of which fix shape lands.

## Contract Ledger

| Target Surface | Source of Authority | Proposed Behavior | Fallback | Docs | Evidence |
|---|---|---|---|---|---|
| `ai/deploy/docker-compose.yml:179` orchestrator healthcheck | `ai/daemons/orchestrator/Orchestrator.mjs:457` actual writer | Probe `/orchestrator-state.json` instead of `/state.json` | None — the probe permanently fails ENOENT otherwise | Inline comment | `docker exec <orch> ls /app/.neo-ai-data/orchestrator-daemon/` shows `orchestrator-state.json`, no `state.json` |
| `ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:571` JSDoc | Same | Cite the actual filename | None — pure docs drift | Inline JSDoc | Grep confirms current cite is `state.json` |

## Decision Record impact

`aligned-with ADR 0014` — implementation alignment under existing cloud-topology authority. No ADR successor risk.

## Acceptance Criteria

- [ ] `ai/deploy/docker-compose.yml` orchestrator healthcheck path matches the filename `Orchestrator.mjs` writes (either by updating the probe path or by renaming the written file).
- [ ] `docker compose -f ai/deploy/docker-compose.yml --profile cloud up -d --wait` returns 0 against dev HEAD (regression guard).
- [ ] `docker inspect <orchestrator-container> --format '{{.State.Health.Status}}'` returns `healthy` within `start_period + interval*retries` after first boot.
- [ ] `TenantRepoSyncService.mjs:571` JSDoc reflects the canonical filename.
- [ ] `grep -rn "/state\.json\b" ai/ | grep -v node_modules` returns zero matches OR each match is explicitly justified (regression guard against re-drift).

## Out of Scope

- Other orchestrator-state-related concerns (e.g., `spawn git ENOENT` from backup task — separate ticket-worthy item).
- Renaming `NEO_AI_ORCHESTRATOR_DIR` or the directory layout (`.neo-ai-data/orchestrator-daemon/`).
- Migrating healthcheck format to invoke `mcpHealthcheck.mjs` against the orchestrator (orchestrator has no MCP endpoint; the mtime check is correct shape).

## Avoided Traps

- Filing as `documentation` only — rejected; the compose healthcheck is functionally broken, not just mis-documented. `bug` + `refactoring` captures both surfaces.
- Skipping the docs-fix piece — rejected; leaving JSDoc cite of `state.json` would re-introduce drift on the next refactor cycle.

## Related

- #11937 — originating PR for the healthcheck (closed); shipped the path mismatch.
- #11833, #11844 — Orchestrator refactors; the `stateFile` default may have evolved during these without the healthcheck path being audited.
- #12014, #12015, #12016, #12017, #12019, #12022 — sibling friction tickets from the same external-deployment exercise.
- ADR 0014 — cloud-deployment topology authority.

## Handoff Retrieval Hints

- Empirical: boot `--profile cloud`, then `docker exec <orch> ls /app/.neo-ai-data/orchestrator-daemon/`. Shows `orchestrator-state.json` only; no `state.json`.
- Source anchor: `grep -n 'state\.json' ai/daemons/orchestrator/Orchestrator.mjs ai/deploy/docker-compose.yml ai/daemons/orchestrator/services/*.mjs`.
- Semantic: "orchestrator healthcheck state.json filename mismatch", "cloud-profile orchestrator permanently unhealthy".

## Timeline

- 2026-05-26T13:34:38Z @neo-opus-ada added the `bug` label
- 2026-05-26T13:34:39Z @neo-opus-ada added the `ai` label
- 2026-05-26T13:34:40Z @neo-opus-ada added the `refactoring` label
- 2026-05-26T13:39:36Z @neo-gpt assigned to @neo-gpt
- 2026-05-26T13:41:56Z @neo-gpt referenced in commit `efc46fb` - "fix(deploy): align orchestrator healthcheck state path (#12025, #12023)"
- 2026-05-26T13:42:56Z @neo-gpt cross-referenced by PR #12024
- 2026-05-26T14:06:49Z @neo-gpt cross-referenced by PR #12020
- 2026-05-26T14:06:49Z @neo-gpt cross-referenced by PR #12021
- 2026-05-26T15:40:34Z @neo-opus-ada cross-referenced by #12026
- 2026-05-26T15:47:16Z @tobiu referenced in commit `05a165a` - "fix(deploy): restore ingress TLS and integration fixtures (#12022, #12023) (#12024)

* fix(deploy): restore ingress TLS and integration fixtures (#12022, #12023)

* fix(deploy): align orchestrator healthcheck state path (#12025, #12023)

---------

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-26T15:47:18Z @tobiu closed this issue
- 2026-05-26T16:11:23Z @neo-opus-ada cross-referenced by #12028
- 2026-05-26T16:12:27Z @neo-opus-ada cross-referenced by #12029

