---
id: 11937
title: 'docker-compose.yml: add orchestrator + ingress healthchecks for cloud-deployment trial operator visibility'
state: CLOSED
labels:
  - enhancement
  - ai
  - architecture
assignees:
  - neo-opus-ada
createdAt: '2026-05-24T22:55:13Z'
updatedAt: '2026-06-07T17:15:26Z'
githubUrl: 'https://github.com/neomjs/neo/issues/11937'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-25T00:06:00Z'
---
# docker-compose.yml: add orchestrator + ingress healthchecks for cloud-deployment trial operator visibility

## Context

40h cloud-deployment trial pending (~2026-05-26). Production `ai/deploy/docker-compose.yml` has healthchecks for `chroma`, `kb-server`, `mc-server` but **not** for `orchestrator` (cloud profile) or `ingress` (caddy reverse proxy). For a trial where the deployment operator needs real-time visibility into Agent OS state, missing healthchecks on 2 of 5 services creates monitoring gaps.

Distinct from [#11791](https://github.com/neomjs/neo/issues/11791) (tenant-repo ingestion docs + telemetry — focused on the pull-based ingestion lane shipping under #11731). This ticket is about general docker-substrate health visibility across ALL deployed services.

## The Problem

`orchestrator` service in `docker-compose.yml` lines 132-177:
- Maintenance daemon — no HTTP endpoint to ping
- TaskStateService writes a JSON state file at `${NEO_AI_ORCHESTRATOR_DIR:-.neo-ai-data/orchestrator-daemon}/state.json` on every task lifecycle event
- A fresh state file (mtime within last N minutes) = orchestrator is alive + polling

`ingress` service (caddy) in lines 184-203:
- Has TCP port 443 exposed
- No healthcheck declared — operator can't distinguish "caddy crashed" from "no requests arriving"
- Standard caddy admin endpoint at `:2019/config/` (admin API) OR simpler TCP check on 443

## The Fix

**For orchestrator** — add healthcheck verifying state file freshness:

```yaml
orchestrator:
  # ... existing config ...
  healthcheck:
    test: ["CMD-SHELL", "node -e \"const fs=require('fs');const f=process.env.NEO_AI_ORCHESTRATOR_DIR+'/state.json';const m=fs.statSync(f).mtimeMs;process.exit(Date.now()-m<600000?0:1)\""]
    interval: 60s
    timeout: 5s
    retries: 3
    start_period: 90s
```

Polls every 60s; state file must be mutated within the last 10 minutes (Orchestrator's `swarmHeartbeatIntervalMs` + safety margin). 90s start period accommodates first-poll latency.

**For ingress** — add TCP-port healthcheck on 443:

```yaml
ingress:
  # ... existing config ...
  healthcheck:
    test: ["CMD-SHELL", "wget --no-check-certificate --quiet --tries=1 --spider https://127.0.0.1/ 2>&1 | grep -E '^(HTTP|Connecting)' || exit 1"]
    interval: 30s
    timeout: 5s
    retries: 3
    start_period: 30s
```

(Alternative: bind-mount a tiny caddy-admin probe; rejected as scope-creep for trial-window.)

## Acceptance Criteria

- [x] `orchestrator` service in `docker-compose.yml` has a `healthcheck` block verifying state file mtime within tolerance window — PR #11939
- [x] `ingress` service has a `healthcheck` block verifying caddy port 443 responsiveness — PR #11939 (final shape: `nc -z 127.0.0.1 443`, TCP-level, per cycle-1 RA on silent-success-empty-output failure mode)
- [ ] `docker compose config` validates clean (no schema errors) — **deferred to operator pre-trial validation** (PR review host has no local docker; YAML-library structure parse confirms all 6 services have valid healthcheck blocks as preflight evidence)
- [ ] Local manual smoke test: `docker compose --profile cloud --profile ingress up`; observe `docker ps` shows both services in `(healthy)` state after start_period — **deferred to operator pre-trial validation** (same reason)

## Deferred Evidence (PR #11939)

PR #11939 satisfied AC1 + AC2 mechanically (compose-file edits + YAML structure validation via `yaml` library). AC3 + AC4 require live docker not available on the review host; both are formally deferred to operator pre-trial smoke test per the [#11939](https://github.com/neomjs/neo/pull/11939) FAIR-band/Evidence framing. Residual operator actions:
1. Run `docker compose config` (parses + validates compose-file against docker schema)
2. Run `docker compose --profile cloud --profile ingress up` and observe `docker ps` reports `(healthy)` for both `orchestrator` (90s start_period) and `ingress` (30s start_period)
3. If either fails, healthcheck command tuning is small follow-up PR shape; no service-config change needed

This deferred-evidence annotation matches PR #11939's AC3-partial / AC4-deferred state so the close-target issue and the PR substrate stay aligned (per `pull-request §10 Evidence Audit`).

## Out of Scope

- Tenant-repo ingestion health/telemetry → owned by [#11791](https://github.com/neomjs/neo/issues/11791)
- Active push-style alerting / external observability integration (e.g., Prometheus/Grafana export) → post-MVP
- chroma / kb-server / mc-server healthcheck tuning (already declared)

## Avoided Traps

- **Trap**: HTTP healthcheck on the orchestrator (it doesn't expose one). **Rejection**: state-file-mtime is the canonical liveness signal already present in the substrate; introducing an HTTP server would balloon scope.
- **Trap**: caddy admin API healthcheck on `:2019`. **Rejection**: admin port is internal-only by default; activating it for healthcheck would expose extra surface area for marginal value beyond TCP-443 check.
- **Trap**: tight `interval: 10s` matching the other services. **Rejection**: orchestrator polling cadence is minutes, not seconds; 60s interval matches the actual lane semantics + reduces healthcheck noise.

## Empirical Anchor

Cloud-deployment trial readiness review 2026-05-24. Operator's framing: "in theory we are there → many new deployment guides, orchestrator enhancements, docker, integration tests." This ticket addresses the docker substrate slice — orchestrator + ingress healthcheck gap was visible on inspection of `docker-compose.yml`.

— @neo-opus-ada

## Timeline

- 2026-05-24T22:55:14Z @neo-opus-ada assigned to @neo-opus-ada
- 2026-05-24T22:55:14Z @neo-opus-ada added the `enhancement` label
- 2026-05-24T22:55:14Z @neo-opus-ada added the `ai` label
- 2026-05-24T22:55:14Z @neo-opus-ada added the `architecture` label
- 2026-05-24T22:57:40Z @neo-opus-ada cross-referenced by PR #11939
- 2026-05-24T23:18:53Z @neo-opus-ada referenced in commit `1e578d0` - "fix(deploy): ingress healthcheck via nc -z (no silent-success-empty-output) (#11937)

Per @neo-gpt #11939 cycle-1 review:
https://github.com/neomjs/neo/pull/11939#pullrequestreview-4353556762

Previous `wget --quiet --spider | grep` could fail-closed when wget succeeded quietly with no stderr output (grep on empty input exits 1). Replaced with direct `nc -z 127.0.0.1 443` — unambiguous TCP-level liveness. caddy:2-alpine includes BusyBox nc; exit 0 = port accepts connection, non-zero = caddy crashed."
- 2026-05-25T00:06:00Z @tobiu closed this issue
- 2026-05-26T13:34:37Z @neo-opus-ada cross-referenced by #12025

