Testing

Zaxy follows test-first development. Public behavior should have a test before implementation. The full suite has a broad 92 percent pytest coverage gate plus a coverage ratchet that currently requires at least 92.00% total line coverage from coverage.xml. Unit tests mock external dependencies such as Neo4j and Pathlight. Integration tests use Docker services and are marked with integration.

Common commands:

pytest
pytest -m integration --no-cov
scripts/integration-check.sh --start
ruff check src tests
mypy src
pytest tests/test_packet_memory_e2e.py --no-cov -q
zaxy doctor --beta-readiness
scripts/beta-uat.sh
scripts/release-check.sh --root .
zaxy benchmark-freeze --json

The default pytest command includes coverage reporting and --cov-fail-under=92 from pyproject.toml. CI and scripts/release-check.sh also run scripts/check-coverage.py against the generated XML report. The ratchet floor lives in [tool.zaxy.coverage], is based on the canonical CI Python 3.13 measurement, and can be intentionally raised after coverage improvements. Integration-only runs use --no-cov because the project-level coverage gate is intended for the full suite. Before running integration tests, start the Neo4j services:

./scripts/generate-certs.sh .certs
docker compose --profile integration up -d neo4j-test neo4j-tls

For local full-suite checks, prefer the integration helper so the Neo4j dependency is explicit:

scripts/integration-check.sh --start
scripts/integration-check.sh --require
scripts/integration-check.sh --skip-if-unavailable

Use --start when Docker is available and the helper should generate TLS certs, boot neo4j-test and neo4j-tls, then run pytest. Use --require when services should already be running and absence should fail fast. Use --skip-if-unavailable for development loops where graph integration tests should be omitted only after the helper verifies the Neo4j test ports are not reachable.

Tests are organized by module: event log integrity, extraction, graph behavior, query routing, MCP tools, tracing, configuration, embeddings, operations scripts, packaging, and site/docs validation. New modules should get focused tests rather than relying only on high-level workflows.

The contract-validation path (introduced with the v0.9 freeze) includes fuzz-style parametrized checks for Eventloom payload validation, hash-chain replay, and bounded MCP inputs. These tests intentionally exercise malformed payload shapes, oversized JSON objects, sequence-tampered but hash-valid Eventloom records, and invalid direct memory_append handler inputs so contract validation does not depend only on MCP client schema enforcement.

The packet-memory product path has an explicit smoke check:

pytest tests/test_packet_memory_e2e.py --no-cov -q

Keep packet-memory smoke coverage active so the analyzer-to-projection-to-context workflow remains tested. Public benchmark gates are documented separately in benchmarks.md; do not add release checks that depend on archived benchmark reports unless the benchmark hub is updated first.

The beta hardening path has two additional checks. zaxy doctor --beta-readiness is a fast local inventory of release metadata, release gate coverage, clean-repo UAT coverage, documentation, and deterministic capture posture. scripts/beta-uat.sh performs a clean first-run exercise in a throwaway workspace: install, zaxy init, deterministic capture startup, zaxy memory bootstrap, zaxy memory checkout, doctor, hook status, capture status, capture soak, and memory status. It also runs the bare embedded init path and verifies the generated profile includes PROJECTION_BACKEND=embedded, NEO4J_AUTO_START=false, and the repo-local embedded projection path, so the zero-friction default cannot silently drift back to a sidecar requirement. The bare embedded branch also runs zaxy memory status --eventloom-path .eventloom --graph, zaxy memory inferred-status, and zaxy reproject so clean UAT proves the repo-local embedded projection can be inspected, audited, rebuilt, and rechecked without backend flags. The UAT path runs zaxy hook-status --min-activation-rate 1.0, so it fails if its clean first-run captured sessions did not all start substantive work after fresh checkout. zaxy capture-soak is the beta evidence command for deterministic capture: it checks transcript, tool-call, command, and file-edit observation coverage, freshness, latest seq/hash, and remediation steps. zaxy doctor --beta-readiness also reads docs/examples/first-run-timing-report.json; keep time_to_successful_doctor_seconds and time_to_first_successful_example_seconds at or below 300 seconds, and keep requires_sidecar false for the default local path.

For activation hardening, zaxy hook-status --json reports activation efficiency under memory_activation.activation_efficiency. The metric counts high-context sessions that have command, file-edit, tool-call, or transcript activity, then reports what percentage had a fresh memory.checkout.completed event before the first substantive captured event. Keep this as a product KPI: capture without fresh checkout means Zaxy observed the work but did not become the model's working context. Treat activation efficiency as a release-readiness signal for launcher, hook, and dashboard work. When hooks emit memory.reminder.suggested, zaxy hook-status --json also exposes memory_activation.latest_reminder so the warning is tied back to an auditable Eventloom event. Checkout activity markers now preserve numeric token_efficiency diagnostics, including prompt-token estimates and current facts per 1k prompt tokens, so hook-status and dashboard status can show whether activation is both fresh and token-disciplined.

Use the same command as a guardrail when the evidence fixture should prove that models are actually starting work with memory loaded:

zaxy hook-status --eventloom-path .eventloom --json --min-activation-rate 0.8 \
  --max-checkout-prompt-tokens 5000 \
  --min-checkout-facts-per-1k-tokens 0.1

The release gate names every public smoke path it expects. A command may be set to SKIP:<reason> only when the skip is intentional and auditable; otherwise zaxy doctor --beta-readiness reports the missing surface through release_gate_surface_coverage.

The historical v0.9 gate evidence is archived in v09-gate-audit.md. That audit intentionally leaves the external-user feedback gate pending until feedback from outside the current implementation session exists.

The command exits non-zero when fewer than 80% of high-context sessions had fresh checkout before substantive captured work, when the latest checkout exceeds the prompt-token ceiling, or when checkout facts per 1k prompt tokens falls below the required floor.

For graph changes, write both mock tests for Cypher behavior and integration tests against Neo4j when the real database semantics matter. For security changes, test both accepted and rejected inputs. For scripts, use temporary fixtures and injectable command stubs so tests can assert ordering and fail-fast behavior without running destructive commands.

Benchmark tests cover extraction latency, append latency, graph upsert latency, query latency, and retrieval harness behavior. Benchmarks are useful for detecting large regressions, but correctness tests decide release readiness.

The current public benchmark evidence is intentionally narrow and is documented in benchmarks.md. Active public claims should point only to:

Older benchmark suites, backend shootouts, StateRecoveryBench, PurposeBench, LongMemBench adapter artifacts, debug runs, and partial LongMemEval iterations are archived development history. Use those artifacts for engineering investigation only, not as current public benchmark claims.

The 2.0.0-rc.1 StateRecoveryBench, CoordinationBench, and PurposeBench artifacts under reports/benchmarks/ are active RC release guardrails, but remain project-defined internal evidence unless a future release publishes a separate external validation boundary for them.

When staging the next full 500-question run, write it to a new report directory under reports/benchmarks/, update benchmarks.md to select one headline report, and move the superseded headline into reports/archive/.

For the 2.0 RC.1 release candidate, run:

zaxy benchmark-freeze --json

This validates the frozen headline LongMemEval-compatible 500 report, Harvey LAB external-anchor artifacts, StateRecoveryBench, CoordinationBench, PurposeBench, and project-defined 2.0 internal lanes. It is a release evidence and claim-boundary gate; it must not be used to tune answers or mix internal guardrail lanes into public benchmark claims.

For consolidation safety checks, use the identity-collapse workload. It creates near-duplicate source records with distinct durable identifiers and adds an identity-recall metric to the report. The centroid baseline intentionally models semantic consolidation that keeps one representative text, so it can look topically relevant while losing exact source identities:

scripts/live-benchmark.sh --workload consolidation --documents 100 --runs 1 --reset-graph

Use this lane to detect whether a compaction strategy preserves exact event, document, transcript, or entity identity under retrieval, not just broad topic coverage.

Interpret the frozen temporal results narrowly. The suite workload is broader, but still synthetic; use it to measure Zaxy's target problem before making broad market claims: current versus historical facts, stale-context avoidance, graph connections, cited document recall, transcript recall, mixed context assembly, latency, and returned context size on the same paired workload.

CI runs lint, mypy, the full test matrix, package artifact validation, and integration tests. The local release gate mirrors the important pieces. See operations.md, deployment.md, and README.md. The public docs entry is site/index.html.