A deterministic, idempotent 0→60 build of the ai-memory v0.7.0 federated test fleet on DigitalOcean. Terraform stands the infrastructure up; a push-based SSH toolkit brings every node to a verified federated state; a verification harness proves it. Everything a reviewer needs to reproduce both the environment and the results ships inside release/v0.7.0 under deploy/hive-1461/. Seven nodes across two regions, all traffic TLS+mTLS, every step re-runnable.
One entrypoint reproduces the whole thing: make seed up provision validate test builds the fleet, brings every node to a verified federated state, runs the verification harness, then runs the full-spectrum P3 suite. make down tears it all back down. Every probe in both harnesses goes over the real TLS+mTLS path; the canonical green reports are committed at deploy/hive-1461/results/.
Hostnames encode each node's function: hive-<function>-<region>-<NN>. The 3 peers form a W=2-of-N=3 write-quorum mesh; agents and ctrl are pure mTLS clients of that mesh — they hold a client cert only, never a server cert, and run no inbound HTTPS daemon.
| Host | Role | Region | Size | Runs |
|---|---|---|---|---|
hive-peer-nyc3-01 | peer | nyc3 | s-4vcpu-8gb | federated serve + PG16/AGE + Ollama embedder |
hive-peer-nyc3-02 | peer | nyc3 | s-4vcpu-8gb | federated serve + PG16/AGE + Ollama embedder |
hive-peer-sfo2-01 | peer | sfo2 | s-4vcpu-8gb | federated serve + PG16/AGE + Ollama embedder |
hive-agent-nyc3-01 | agent | nyc3 | s-1vcpu-2gb | xAI grok-4.3 NHI client (mTLS API client) |
hive-agent-nyc3-02 | agent | nyc3 | s-1vcpu-2gb | xAI grok-4.3 NHI client |
hive-agent-nyc3-03 | agent | nyc3 | s-1vcpu-2gb | xAI grok-4.3 NHI client |
hive-ctrl-nyc3-01 | ctrl | nyc3 | s-8vcpu-16gb | loadgen / chaos / orchestration (mTLS client) |
Account stays well within the 15-droplet cap (7 in use). All fleet traffic is TLS + mTLS: the peer HTTPS port enforces client_auth_mandatory — a connection is accepted only if the SHA-256 of the client cert's DER bytes is on the mtls-allowlist.txt (fingerprint pinning, the SSH known_hosts model). Outbound quorum/API clients verify peer server certs against the campaign CA, whose SAN pins each peer's public IP.
Every target is idempotent and re-runnable. Provisioning is push-based over SSH; nothing is installed on the operator host beyond the CLI prereqs (terraform jq openssl ssh scp curl). The provision steps run in this exact deterministic order:
| Step | Script | Effect |
|---|---|---|
00 | 00_render_inventory.sh | project terraform output fleet → inventory.json |
05 | 05_wait_ssh.sh | block until every node accepts SSH |
10 | 10_binary.sh | fan out the golden binary; assert version + sha256 |
20 | 20_pg_age.sh | per-peer PG16/AGE/pgvector container + schema-init (v55) |
25 | 25_ollama_embed.sh | per-peer CPU Ollama sidecar serving nomic-embed-text (768-dim) |
30 | 30_config.sh | render + push per-role config.toml + secret EnvironmentFile |
40 | 40_tls.sh | campaign CA + per-node leaf certs + mTLS allowlist fan-out |
50 | 50_federation.sh | per-peer systemd unit; start the quorum mesh; health-gate |
The verification harness exercises the live fleet over the real TLS+mTLS path and emits a JSON + tabular report. Exit status is 0 iff every check is green. Per node it asserts binary sha256 + --version; per peer it asserts /api/v1/health, storage_backend == postgres, db_schema_version == 55, single-instance, and systemd-active; and a fleet federation-convergence probe writes a collective-scope marker to one peer and reads it back by id on another over the encrypted path.
Canonical green baseline committed at results/verify-baseline.{json,tsv}: TOTAL=40 PASS=40 FAIL=0.
The P3 suite runs against the live fleet, gated behind a green make validate. Every probe goes over the real TLS+mTLS path and authenticates with x-api-key; throwaway markers land in the _test / _verify namespaces and are best-effort deleted, so the baseline corpus is never mutated. Canonical green report committed at results/test-full-spectrum.{json,tsv}: TOTAL=22 PASS=22 FAIL=0.
000); non-allowlisted client cert refused (000); wrong server CA refused (000); privileged endpoint without x-api-key → 401; with key → 200; /health exempt → 200; admin endpoint as non-admin → 403.
7/7 PASS
agent-alpha (an mTLS client node) writes a collective memory to a peer over the network; agent-beta (a different client identity on a different node) reads it back on the write peer and on a federated peer.
3/3 PASS
agent-alpha drives a live xAI/grok expand_query decision over the mesh, commits the LLM-derived term as a collective memory, and the decision converges on a federated peer — a full NHI decision → commit → federate loop.
3/3 PASS
The live AI-NHI loop is the capstone: in the canonical run, grok returned the term paxos algorithm, the harness committed it as a collective memory on hive-peer-nyc3-01, and it converged on hive-peer-nyc3-02 — a real LLM decision crossing the encrypted mesh and federating cross-node.
provision/lib.sh): binary sha256, version 0.7.0, schema v55, ollama/ollama:0.6.8, apache/age:release_PG16_1.6.0, embedder/LLM model ids — all single-source constants, overridable by env for forks.inventory.json is a pure projection of Terraform state; the whole toolkit drives off it.make validate + make test exercise the live fleet over the real TLS+mTLS path and emit JSON + tabular reports; exit 0 iff every check is green.OPENROUTER_API_KEY (peer chat LLM) + XAI_API_KEY (agent grok NHI driver) are written only into the gitignored run dir (mode 0600) and pushed to mode-0400 EnvironmentFiles. Never committed, never echoed, never placed on an SSH command line.make seed up provision validate test (teardown: make down)→
deploy/hive-1461/provision/Push-based 0→60 toolkit: steps 00–50 + lib.sh pinned-constant source + pg-age/→
results/verify-baseline.tsvCanonical green verification report — TOTAL=40 PASS=40 FAIL=0→
results/test-full-spectrum.tsvCanonical green full-spectrum report — TOTAL=22 PASS=22 FAIL=0→
ai-memory v0.7.0, features sal,sal-postgres,sqlite-bundled5e86de3ab0be6a19e02f760390651d9c425ba7ca73b9b8c7db0ce3b6f25a0aa7nomic-embed-text (768-dim) via CPU Ollama sidecar per peergrok-4.3 (agents); peer chat LLM via OpenRouterclient_auth_mandatory fingerprint-pinned allowlistrelease/v0.7.0219a0e39a (P3 harness) · 10a9d97eb (docs + canonical results)