LLM编码基准

Rebuild The Report Only

If the runs are already on disk and you just want to rebuild the Markdown summary:

python scripts/run_benchmark.py --report-only

If your warmup file lives somewhere else:

python scripts/run_benchmark.py \
  --report-only \
  --ollama-warmup-results path/to/ollama_warmup.json

Local Opencode Benchmark Config

Benchmark runs do not rely on mutating your home opencode config.

Before execution, scripts/run_benchmark.py writes a local config file:

config/opencode.benchmark.json

That file is built from your installed ~/.config/opencode/opencode.json, but trimmed to the providers and models needed for the selected benchmark run. Benchmark subprocesses are launched with:

OPENCODE_CONFIG=config/opencode.benchmark.json

This keeps the benchmark reproducible and lets the harness apply safe per-model context values without rewriting your global setup.

Generate The Local Benchmark Config Only

If you want to refresh the local benchmark config without starting model runs:

python scripts/run_benchmark.py --sync-ollama-contexts-only

That will regenerate config/opencode.benchmark.json from the current model config and warmup results.

2. Add the model to `config/models.json`

Append a new entry to the models array:

{
  "slug": "vendor_name_version",
  "id": "openrouter/vendor/model-id",
  "label": "Vendor Model Name",
  "provider": "openrouter",
  "selection_reason": "One-line context (pricing, why it was added, known caveats)."
}

For local llama-swap models, also add "llama_swap_model": "vendor:tag" matching the name configured on the llama-swap server. For models you don't want to run by default, add "skip_by_default": true.

Benchmark Workflow

Recommended order:

1. Run the Ollama warmup first if you intend to benchmark local models.
2. Inspect or override any fragile local model context settings.
3. Generate the local benchmark opencode config.
4. Run the full benchmark or a subset.
5. For OpenRouter runs, let the harness continue into the second validation prompt before treating the run as final.
6. Rebuild the report from saved artifacts whenever needed.

Two Local Backend Profiles: AMD Server vs NVIDIA Workstation

This repo supports running the local-llama-swap subset of the benchmark against two different machines, with separate config files and result directories so the runs don't overwrite each other:

The NVIDIA profile is a strict subset of the AMD profile: only the local llama-swap models that fit in 32 GB of VRAM are included, with smaller benchmark_context_override values to keep KV cache within budget. The OpenRouter and Z.ai cloud models are not duplicated — those go in config/models.json alone since they don't depend on local hardware.

The Docker setup for the NVIDIA workstation lives at ~/Projects/llama-swap-docker (separate repo). It builds llama.cpp from source against CUDA 12.8 with CMAKE_CUDA_ARCHITECTURES=120 so the kernels target Blackwell directly.

Ollama vs llama-swap

This project supports two local model backends. Ollama was the original backend; llama-swap was added after Ollama proved unreliable for unattended benchmark runs.