---
title: "fak hardware bench plan: what to benchmark next per machine"
description: "Auto-generated hardware benchmark plan: the next highest-value test per bench-node (macbook / datacenter-A100 / cloud-L4 / RTX-laptop) across coverage, regression, and new-data intents, ranked from the benchmark catalog."
---

# Hardware bench plan — 20260630T022248Z

_Auto-generated by `tools/bench_plan.py --md`. Do not hand-edit; re-run the tool (or the `FleetBenchPlanDoc` task) to refresh._

**PLAN ONLY -- no benchmark was run; every figure is a literal catalog fact and all ages are computed against --now.**

Generated from `catalog.json` (last_updated 2026-06-29T13:35:42.135429+00:00): 7 bench-nodes, 61 feasible cells (48 empty coverage holes, 2 with a recorded baseline), 2 infeasible cells, 58 runs on record.

## Coverage matrix (bench-node × workload-kind)

`.` = empty feasible cell (a coverage hole) · `-` = infeasible on this hardware (e.g. a CUDA kind on the Apple mac) · a number = runs on record.

| machine | model | gpu | qwen36 | radix | session | fan | agent-live | turn-tax | parity |
|---|---|---|---|---|---|---|---|---|---|
| a100 | . | . | . | . | . | . | . | . | . |
| cpu-server-a | . | - | . | . | . | . | . | . | . |
| gcp-a3-high-h100-1g | . | . | . | . | . | . | . | . | . |
| gcp-g2-l4 | . | . | . | . | . | . | . | . | . |
| gcp-g2-l4-32 | . | . | . | . | . | . | . | . | . |
| node-macos-a | 2 | - | 5 | 1 | 1 | . | 1 | . | . |
| workstation-a | 4 | 1 | 9 | 2 | . | 2 | 5 | 2 | 4 |

## Next test per machine

| machine | next workload | model / precision | intent | score | why |
|---|---|---|---|---|---|
| a100 | gpu-benchmark | qwen2.5-3b / Q8_0 **NEW** | learn-collect | 0.84 | a100 has 0 runs (last_run=None); the gpu-benchmark cell has never run -- a first-ever measurement on this node. |
| cpu-server-a | model-benchmark | SmolLM2-135M-Instruct / q8_0 **NEW** | coverage | 0.74 | cpu-server-a has run other workloads but the model-benchmark cell is empty -- a coverage gap to fill (SmolLM2-135M-Instruct). |
| gcp-a3-high-h100-1g | gpu-benchmark | qwen2.5-3b / Q8_0 **NEW** | coverage | 0.74 | gcp-a3-high-h100-1g has run other workloads but the gpu-benchmark cell is empty -- a coverage gap to fill (qwen2.5-3b). |
| gcp-g2-l4 | gpu-benchmark | qwen2.5-3b / Q8_0 **NEW** | coverage | 0.69 | gcp-g2-l4 has run other workloads but the gpu-benchmark cell is empty -- a coverage gap to fill (qwen2.5-3b). |
| gcp-g2-l4-32 | gpu-benchmark | qwen2.5-3b / Q8_0 **NEW** | coverage | 0.74 | gcp-g2-l4-32 has run other workloads but the gpu-benchmark cell is empty -- a coverage gap to fill (qwen2.5-3b). |
| node-macos-a | fan-benchmark | None / n/a | coverage | 0.5882 | node-macos-a has run other workloads but the fan-benchmark cell is empty -- a coverage gap to fill (agent workload). |
| workstation-a | session-benchmark | None / n/a | coverage | 0.5747 | workstation-a has run other workloads but the session-benchmark cell is empty -- a coverage gap to fill (agent workload). |

## Do next — global ranked plan

| # | machine | workload | intent | score | suggested command (hint) |
|---|---|---|---|---|---|
| 1 | a100 | gpu-benchmark | learn-collect | 0.84 | `go run -tags cuda ./cmd/gpucheck` |
| 2 | a100 | model-benchmark | learn-collect | 0.84 | `go run ./cmd/modelbench -quant` |
| 3 | a100 | qwen36 | learn-collect | 0.84 | `fak serve + fak agent (qwen3.6-27b via gateway)` |
| 4 | a100 | radix-benchmark | learn-collect | 0.84 | `go run ./cmd/radixbench` |
| 5 | a100 | agent-live | learn-collect | 0.77 | `go run ./cmd/fak agent --task <task>` |
| 6 | a100 | fan-benchmark | learn-collect | 0.77 | `go run ./cmd/fanbench` |
| 7 | a100 | parity | learn-collect | 0.77 | `go run ./cmd/paritybench` |
| 8 | a100 | session-benchmark | learn-collect | 0.77 | `go run ./cmd/sessionbench` |
| 9 | a100 | turn-tax | learn-collect | 0.77 | `go run ./cmd/fak turntax --suite turntax-airline` |
| 10 | cpu-server-a | model-benchmark | coverage | 0.74 | `go run ./cmd/modelbench -quant` |

## By intent

_The planner guarantees all four operator intents are surfaced._

### Benchmark perf
- **workstation-a → gpu-benchmark** (score 0.3154): workstation-a/gpu-benchmark ran 1x, last 11.09d ago (re-check 7d) but recorded no number -- re-run to capture one.
- **node-macos-a → session-benchmark** (score 0.2624): node-macos-a/session-benchmark ran 1x, last 10.54d ago (re-check 14d) but recorded no number -- re-run to capture one.
- **node-macos-a → radix-benchmark** (score 0.2404): node-macos-a/radix-benchmark ran 1x, last 6.69d ago (re-check 14d) but recorded no number -- re-run to capture one.
- **node-macos-a → agent-live** (score 0.2152): node-macos-a/agent-live ran 1x, last 4.89d ago (re-check 30d) but recorded no number -- re-run to capture one.
- **workstation-a → radix-benchmark** (score 0.1992): workstation-a/radix-benchmark ran 2x, last 11.77d ago (re-check 14d) but recorded no number -- re-run to capture one.

### Learn / collect new data
- **a100 → gpu-benchmark** (score 0.84): a100 has 0 runs (last_run=None); the gpu-benchmark cell has never run -- a first-ever measurement on this node.
- **a100 → model-benchmark** (score 0.84): a100 has 0 runs (last_run=None); the model-benchmark cell has never run -- a first-ever measurement on this node.
- **a100 → qwen36** (score 0.84): a100 has 0 runs (last_run=None); the qwen36 cell has never run -- a first-ever measurement on this node.
- **a100 → radix-benchmark** (score 0.84): a100 has 0 runs (last_run=None); the radix-benchmark cell has never run -- a first-ever measurement on this node.
- **a100 → agent-live** (score 0.77): a100 has 0 runs (last_run=None); the agent-live cell has never run -- a first-ever measurement on this node.

### Prevent regression
- **workstation-a → model-benchmark** (score 0.3778): workstation-a/model-benchmark has a recorded 31.02 tok/s baseline aged 11.48d (re-check every 7d) -- re-measure to catch drift.
- **node-macos-a → model-benchmark** (score 0.3749): node-macos-a/model-benchmark has a recorded 38.07 tok/s baseline aged 6.69d (re-check every 7d) -- re-measure to catch drift.

### Fill coverage gaps
- **cpu-server-a → model-benchmark** (score 0.74): cpu-server-a has run other workloads but the model-benchmark cell is empty -- a coverage gap to fill (SmolLM2-135M-Instruct).
- **cpu-server-a → qwen36** (score 0.74): cpu-server-a has run other workloads but the qwen36 cell is empty -- a coverage gap to fill (qwen3.6-27b).
- **cpu-server-a → radix-benchmark** (score 0.74): cpu-server-a has run other workloads but the radix-benchmark cell is empty -- a coverage gap to fill (SmolLM2-135M).
- **gcp-a3-high-h100-1g → gpu-benchmark** (score 0.74): gcp-a3-high-h100-1g has run other workloads but the gpu-benchmark cell is empty -- a coverage gap to fill (qwen2.5-3b).
- **gcp-a3-high-h100-1g → model-benchmark** (score 0.74): gcp-a3-high-h100-1g has run other workloads but the model-benchmark cell is empty -- a coverage gap to fill (SmolLM2-135M-Instruct).

## Frontier & honesty call-outs

- `a100` is a total coverage frontier — every feasible workload-kind (9) has never run here.
- `cpu-server-a` is a total coverage frontier — every feasible workload-kind (8) has never run here.
- `gcp-a3-high-h100-1g` is a total coverage frontier — every feasible workload-kind (9) has never run here.
- `gcp-g2-l4` is a total coverage frontier — every feasible workload-kind (9) has never run here.
- `gcp-g2-l4-32` is a total coverage frontier — every feasible workload-kind (9) has never run here.

## Excluded (not a bench target)

- `desktop` — role=agent-host (run-on-bench-nodes-by-default).

## Methodology & honesty

Each feasible `(machine × workload-kind)` cell is scored on five dimensions, each in `[0,1]`:

| dimension | weight | intent | reads |
|---|---|---|---|
| coverage_gap | 0.34 | coverage | run count in the cell (empty = 1.0) |
| machine_novelty | 0.20 | learn-collect | machine total runs / last_run |
| staleness_overdue | 0.16 | regression | newest cell timestamp vs --now, per-kind interval |
| baseline_drift | 0.16 | regression | a recorded peak_tok_per_sec aging past its interval |
| model_diversity | 0.14 | coverage | anti-monoculture: how rare the proposed model is |

Cells sort empty-tier-first (coverage dominance), then by weighted score. Feasibility (no CUDA on the mac) is applied before scoring. `--now` is injected, never read from the wall clock, so a fixed stamp yields identical output. marginal-information-gain (novelty / diversity) is a heuristic proxy, not a measured quantity — the ranking is a guide, not a verdict. This tool only PLANS; a run is a later action on the remote bench-node.

