---
id: 10911
title: 'Chroma healthcheck still fails: `python` binary not in PATH (chromadb slim image uses `python3`)'
state: CLOSED
labels:
  - bug
  - ai
  - regression
  - build
assignees:
  - neo-opus-ada
createdAt: '2026-05-07T16:10:28Z'
updatedAt: '2026-05-09T21:15:54Z'
githubUrl: 'https://github.com/neomjs/neo/issues/10911'
author: neo-opus-ada
commentsCount: 0
parentIssue: null
subIssues: []
subIssuesCompleted: 0
subIssuesTotal: 0
blockedBy: []
blocking: []
closedAt: '2026-05-07T16:17:17Z'
---
# Chroma healthcheck still fails: `python` binary not in PATH (chromadb slim image uses `python3`)

## Context

Iteration on [#10908](https://github.com/neomjs/neo/issues/10908) (Chroma healthcheck curl-missing). The PR [#10909](https://github.com/neomjs/neo/pull/10909) merged the `curl` → `python urllib` switch, but Lane C [#10899](https://github.com/neomjs/neo/pull/10899) integration row STILL fails with the same 60s healthcheck timeout. Empirical anchor: [Lane C run 25507191916](https://github.com/neomjs/neo/actions/runs/25507191916) — same Chroma banner shown, same `dependency failed to start: container neo-integration-test-chroma-1 is unhealthy` after exactly 60s.

## The Problem

The current healthcheck in [`ai/deploy/docker-compose.test.yml:10`](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.test.yml#L10):

```yaml
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/api/v2/heartbeat',timeout=2).read() else 1)"]
```

The `chromadb/chroma:1.5.9` image is built on `python:3.11-slim-bookworm`. In Debian slim Python images, the **`python` symlink may not be present** — only `python3` (and `python3.11`) is guaranteed in PATH. When Docker runs the healthcheck command, `python -c '...'` fails with exit 127 (command not found) silently — Docker reports the container as unhealthy without surfacing the underlying error in the compose log.

The Chroma server itself runs fine (banner output observed at the canonical timestamp on every run) — chromadb's entrypoint uses `python3` internally, so the lack of `python` symlink doesn't break the server, just our healthcheck.

## The Architectural Reality

- [`ai/deploy/docker-compose.test.yml:10`](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.test.yml#L10) — current healthcheck (after #10909).
- `chromadb/chroma:1.5.9` base: `python:3.11-slim-bookworm`. Slim variants typically lack the `python` shim.
- The probe target `/api/v2/heartbeat` is the canonical Chroma 1.5.x endpoint; URL is correct.
- The Python expression itself is sound — it's the binary path that doesn't resolve.

## The Fix

Two surgical changes:

```diff
-    test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/api/v2/heartbeat',timeout=2).read() else 1)"]
+    test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/v2/heartbeat', timeout=2)"]
```

1. **`python` → `python3`** — addresses the binary path issue.
2. **Drop the `sys.exit(0 if ... else 1)` wrapper** — redundant. If `urlopen` raises (any HTTP error, connection refused, timeout), the exception propagates and Python exits non-zero naturally. If it succeeds, Python exits 0 implicitly. Same semantic, less code, easier to debug.

## Acceptance Criteria

- [ ] [`ai/deploy/docker-compose.test.yml:10`](https://github.com/neomjs/neo/blob/dev/ai/deploy/docker-compose.test.yml#L10) uses `python3` and the simplified expression.
- [ ] Inline comment updated to clarify `python3` rationale.
- [ ] Lane C [#10899](https://github.com/neomjs/neo/pull/10899) `Tests / integration` matrix row passes on next rebase + CI run.

## Out of Scope

- Switching to a different probe entirely (e.g., chroma CLI, TCP probe). `python3 + urllib` is the cleanest HTTP-readiness probe for this image; only the binary path needed correction.
- Increasing `start_period` for cold-start tolerance — the 60s × 12 retries window with a working probe should be ample on ubuntu-latest. Re-evaluate only if this fix doesn't pass.

## Avoided Traps / Gold Standards Rejected

- **Rejected: drop the healthcheck and use `service_started`.** Race condition: kb/mc may start before Chroma's HTTP server is ready, causing connection-refused on first MCP request. Healthcheck is the right substrate.
- **Rejected: switch to bash `/dev/tcp` TCP probe.** chromadb slim image isn't guaranteed to have bash — sh-only.
- **Rejected: install `python` symlink in a custom Dockerfile.** Adds image-build complexity for a 1-character fix.

## Related

- **Predecessor:** [#10908](https://github.com/neomjs/neo/issues/10908) → PR [#10909](https://github.com/neomjs/neo/pull/10909) (curl-missing fix; established the Python urllib pattern but used wrong binary name).
- **Surfacing context:** [Lane C CI run 25507191916](https://github.com/neomjs/neo/actions/runs/25507191916) integration job — 3rd Lane-C-exposed substrate-config bug in this lineage (after [#10902](https://github.com/neomjs/neo/issues/10902) Dockerfile prepare-lifecycle and [#10908](https://github.com/neomjs/neo/issues/10908) curl-missing).
- **Downstream impact:** [Lane C #10899](https://github.com/neomjs/neo/pull/10899) integration row blocked until this lands.

Origin Session ID: `7e897a0b-33ce-4d6c-b1a9-a1ff93e4e571`

Retrieval Hint: `query_raw_memories(query="chroma healthcheck python3 binary path Lane C CI substrate fix iteration")`

## Timeline

- 2026-05-07T16:10:29Z @neo-opus-ada added the `bug` label
- 2026-05-07T16:10:29Z @neo-opus-ada added the `ai` label
- 2026-05-07T16:10:30Z @neo-opus-ada added the `regression` label
- 2026-05-07T16:10:30Z @neo-opus-ada added the `build` label
- 2026-05-07T16:11:39Z @neo-opus-ada cross-referenced by PR #10912
- 2026-05-07T16:17:17Z @tobiu referenced in commit `2a897c9` - "fix(deploy): use python3 binary in Chroma healthcheck (#10911) (#10912)

The chromadb/chroma:1.5.9 image is built on python:3.11-slim-bookworm.
Slim variants ship python3 but the 'python' symlink may be absent.
The healthcheck after #10909 used 'python -c' which silently failed
with exit 127 (command not found), so docker reported chroma as
unhealthy after 60s and integration tests timed out at 240s.

Two surgical changes:
1. python → python3 (binary path portability)
2. Drop sys.exit(0 if ... else 1) wrapper — redundant. urlopen
   raises on any HTTP/connection error, exception propagates, Python
   exits non-zero naturally. Easier to debug, same semantic.

Empirical anchor: Lane C run 25507191916 integration job — same
Chroma banner output as previous runs but healthcheck still failing
at exactly 60s (5s × 12 retries) on the post-#10909 image.

Iteration on #10908/#10909 in the Lane-C-CI substrate-fragility
discovery cascade.

Co-authored-by: tobiu <tobiasuhlig78@gmail.com>"
- 2026-05-07T16:17:18Z @tobiu closed this issue
- 2026-05-07T16:30:29Z @neo-opus-ada cross-referenced by #10913
- 2026-05-07T17:00:22Z @neo-opus-ada cross-referenced by #10915
- 2026-05-07T18:08:10Z @neo-opus-ada cross-referenced by PR #10899
- 2026-05-07T18:24:02Z @neo-opus-ada cross-referenced by PR #10922
- 2026-05-09T21:15:54Z @neo-gemini-pro assigned to @neo-opus-ada

