---
id: 10908
title: 'Chroma healthcheck fails in CI: chromadb/chroma image lacks curl'
state: CLOSED
labels:
  - bug
  - ai
  - architecture
  - build
assignees:
  - neo-opus-ada
createdAt: '2026-05-07T15:51:44Z'
updatedAt: '2026-05-09T21:15:56Z'
githubUrl: 'https://github.com/neomjs/neo/issues/10908'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-07T15:59:12Z'
---
# Chroma healthcheck fails in CI: chromadb/chroma image lacks curl

## Context

Surfaced 2026-05-07 by Lane C [#10899](https://github.com/neomjs/neo/pull/10899) integration-row CI re-fire after [#10904](https://github.com/neomjs/neo/pull/10904) (Dockerfile fix) merged. The Dockerfile fix unblocked the build path; the next failure mode is now visible: Chroma container's healthcheck fails because `curl` is not installed in the `chromadb/chroma:1.5.9` base image, leading to `dependency failed to start: container neo-integration-test-chroma-1 is unhealthy` after 60s and a 240s `webServer` timeout in Playwright.

## The Problem

[`ai/deploy/docker-compose.test.yml:7`](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.test.yml#L7) declares:

```yaml
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8000/api/v2/heartbeat"]
  interval: 5s
  timeout: 5s
  retries: 12
```

The `chromadb/chroma:1.5.9` image is debian-slim-based and does **not** include `curl` by default. Docker reports the healthcheck as failing every 5s; after 12 retries (60s) the container is marked unhealthy. `kb-server` and `mc-server` both have `depends_on: chroma: condition: service_healthy`, so neither starts → `composeWebServer.mjs` waits → 240s webServer-timeout → integration suite fails.

**Empirical anchor:** [Lane C CI run 25506320077](https://github.com/neomjs/neo/actions/runs/25506320077) integration job, log lines:
- `15:45:28` — Chroma starts, banner output observed (process is healthy, server listening).
- `15:46:28` — `dependency failed to start: container neo-integration-test-chroma-1 is unhealthy` (exactly 60s, matching `5s × 12 retries`).
- `15:48:39` — `Error: Timed out waiting 240000ms from config.webServer.`

The Chroma server itself **is** running and listening on port 8000 (verified via banner). The healthcheck just can't probe it because curl is absent.

## Why this wasn't caught earlier

Same root pattern as [#10902](https://github.com/neomjs/neo/issues/10902): local Docker layer caching. On a developer machine that previously ran `docker compose up` against an older Chroma image (which may have included curl) OR where `--no-cache` was never invoked, the cache masks the issue. Lane C ([#10899](https://github.com/neomjs/neo/pull/10899)) is the first PR to run a truly clean ubuntu-latest Docker build → bug surfaces on first `--no-cache` build of `chromadb/chroma:1.5.9`.

## The Architectural Reality

- [`ai/deploy/docker-compose.test.yml:7`](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.test.yml#L7) — current healthcheck.
- The `chromadb/chroma:1.5.9` image is a **Python application**; Python is guaranteed to be present (it's the runtime). curl/wget are not.
- The Chroma `/api/v2/heartbeat` endpoint exists and is the canonical health probe (verified via image documentation + previous PR-#10880 design intent).
- We need a healthcheck that uses something **guaranteed** to exist in the image, not curl.

## The Fix

Replace the `curl`-based healthcheck with a Python `urllib`-based check (uses the runtime that's already in the image):

```diff
   healthcheck:
-    test: ["CMD", "curl", "-f", "http://localhost:8000/api/v2/heartbeat"]
+    test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/api/v2/heartbeat',timeout=2).read() else 1)"]
     interval: 5s
     timeout: 5s
     retries: 12
```

The Python check:
- Uses `urllib.request.urlopen` (stdlib, no dependencies).
- 2s socket timeout per probe (well under the 5s `timeout` envelope).
- Returns exit 0 on any 2xx + non-empty body, exit 1 on any failure (HTTP error, connection refused, timeout, malformed response).
- Compatible with all chromadb image variants (Python is the runtime).

## Contract Ledger (T3)

| Target Surface | Source of Authority | Proposed Behavior | Fallback / Edge Case | Docs | Evidence |
|---|---|---|---|---|---|
| `ai/deploy/docker-compose.test.yml:7` healthcheck | This ticket; surfaced by [#10897](https://github.com/neomjs/neo/issues/10897) Lane C CI re-fire post-#10904 | Healthcheck uses Python `urllib.request` (stdlib) instead of `curl` (absent from image). Same probe target (`http://localhost:8000/api/v2/heartbeat`), same interval/timeout/retries. | If chromadb ever ships an image variant without Python (unlikely — Chroma is a Python app), fallback to TCP-port check via `nc`/`netcat` or `bash`-builtin `/dev/tcp`. Document in inline comment. | Inline `docker-compose.test.yml` comment cross-linking this ticket. | L3 — Lane C CI ([#10899](https://github.com/neomjs/neo/pull/10899)) `integration` job runs green on the rebased head after this fix lands. |

## Acceptance Criteria

- [ ] [`ai/deploy/docker-compose.test.yml:7`](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.test.yml#L7) healthcheck uses the Python-stdlib probe per Ledger row 1.
- [ ] Inline comment cross-links this ticket as the rationale.
- [ ] Lane C CI ([#10899](https://github.com/neomjs/neo/pull/10899)) `Tests / integration` matrix row passes on next rebase + run.
- [ ] No regression to local Docker workflow (still works for `docker compose -f ai/deploy/docker-compose.test.yml up` on dev machines).

## Out of Scope

- Audit of OTHER healthchecks elsewhere in the repo for similar curl-dependency assumptions (only `ai/deploy/docker-compose.test.yml` has compose-level healthchecks today; cross-substrate audit is future ticket).
- Switching to a different chromadb base image variant.
- Custom Dockerfile FROM the chromadb image with curl pre-installed (heavier intervention; Python-stdlib path is leaner).

## Avoided Traps / Gold Standards Rejected

- **Rejected: install curl in a custom Chroma Dockerfile.** Adds image-build complexity for a healthcheck-only utility. Python-stdlib path is leaner.
- **Rejected: drop the healthcheck + use `service_started` instead of `service_healthy`.** Race condition: kb-server / mc-server may start before Chroma is actually ready and fail with connection-refused on first MCP request. Healthcheck is the right substrate.
- **Rejected: switch to `wget`.** Same problem as curl — not guaranteed in chromadb image. Python is the safe-by-construction choice (it's the runtime).
- **Rejected: bash `/dev/tcp` probe.** Requires bash (not just sh); chromadb image basis isn't guaranteed to have bash. Less portable than Python.

## Related

- **Surfacing context:** [Lane C CI run 25506320077](https://github.com/neomjs/neo/actions/runs/25506320077) integration job, post-#10904 rebase.
- **Sibling substrate fix (predecessor):** [#10902](https://github.com/neomjs/neo/issues/10902) → PR [#10904](https://github.com/neomjs/neo/pull/10904) — Dockerfile prepare-lifecycle fix. Same pattern: substrate config bug masked by local Docker layer caching, surfaced by clean-CI.
- **Original substrate:** [#10801](https://github.com/neomjs/neo/issues/10801) → PR [#10880](https://github.com/neomjs/neo/pull/10880) (Docker artifacts). The current healthcheck shape was introduced there.
- **Downstream impact:** [Lane C #10899](https://github.com/neomjs/neo/pull/10899) (CI workflow gating depends on this), [Lane B #10898](https://github.com/neomjs/neo/pull/10898) (rebased; integration evidence path live but blocked by this), and all future integration-test PRs.

Origin Session ID: `7e897a0b-33ce-4d6c-b1a9-a1ff93e4e571`

Retrieval Hint: `query_raw_memories(query="chroma healthcheck curl docker-compose Lane C CI integration substrate")`

## Timeline

- 2026-05-07T15:51:45Z @neo-opus-ada added the `bug` label
- 2026-05-07T15:51:45Z @neo-opus-ada added the `ai` label
- 2026-05-07T15:51:46Z @neo-opus-ada added the `architecture` label
- 2026-05-07T15:51:46Z @neo-opus-ada added the `build` label
- 2026-05-07T15:53:38Z @neo-opus-ada cross-referenced by PR #10909
- 2026-05-07T15:59:12Z @tobiu referenced in commit `3f94362` - "fix(deploy): use python urllib for Chroma healthcheck (curl missing) (#10908) (#10909)

The chromadb/chroma:1.5.9 image is debian-slim-based and does NOT
include curl. The healthcheck at ai/deploy/docker-compose.test.yml:7
'test: [CMD, curl, -f, ...]' fails every 5s for 60s, marks the chroma
container unhealthy, and blocks kb-server + mc-server from starting
(both have depends_on: chroma: service_healthy). Result: integration
tests time out at the 240s webServer envelope.

Same root pattern as #10902/#10904 (Dockerfile prepare-lifecycle):
substrate config bug masked by local Docker layer caching, surfaces
on first clean ubuntu-latest CI build via Lane C #10899.

Fix: use Python urllib (stdlib, guaranteed in the chromadb image
since Chroma is itself a Python app). Same /api/v2/heartbeat
endpoint, same interval/timeout/retries, no external binary
dependency.

Empirical anchor: Lane C CI run 25506320077 integration job —
'dependency failed to start: container neo-integration-test-chroma-1
is unhealthy' at 60s, then 240s webServer timeout.

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-07T15:59:12Z @tobiu closed this issue
- 2026-05-07T16:10:29Z @neo-opus-ada cross-referenced by #10911
- 2026-05-07T16:11:39Z @neo-opus-ada cross-referenced by PR #10912
- 2026-05-07T16:17:18Z @tobiu referenced in commit `2a897c9` - "fix(deploy): use python3 binary in Chroma healthcheck (#10911) (#10912)

The chromadb/chroma:1.5.9 image is built on python:3.11-slim-bookworm.
Slim variants ship python3 but the 'python' symlink may be absent.
The healthcheck after #10909 used 'python -c' which silently failed
with exit 127 (command not found), so docker reported chroma as
unhealthy after 60s and integration tests timed out at 240s.

Two surgical changes:
1. python → python3 (binary path portability)
2. Drop sys.exit(0 if ... else 1) wrapper — redundant. urlopen
   raises on any HTTP/connection error, exception propagates, Python
   exits non-zero naturally. Easier to debug, same semantic.

Empirical anchor: Lane C run 25507191916 integration job — same
Chroma banner output as previous runs but healthcheck still failing
at exactly 60s (5s × 12 retries) on the post-#10909 image.

Iteration on #10908/#10909 in the Lane-C-CI substrate-fragility
discovery cascade.

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-07T16:30:29Z @neo-opus-ada cross-referenced by #10913
- 2026-05-07T17:00:22Z @neo-opus-ada cross-referenced by #10915
- 2026-05-07T18:08:10Z @neo-opus-ada cross-referenced by PR #10899
- 2026-05-07T18:24:02Z @neo-opus-ada cross-referenced by PR #10922
- 2026-05-09T21:15:56Z @neo-gemini-pro assigned to @neo-opus-ada

