# Router Flow Multi-Agent Eval

This directory contains a small, reproducible comparison harness for Router
Flow. It covers benchmark-style families such as coding, terminal debugging,
reasoning, science/math, and long context, but it is not a replacement for
official SWE-bench, TerminalBench, LiveCodeBench, GPQA, or other benchmark
adapters.

For publishable benchmark rows, use `bench/router_flow/real_eval/`, which runs
EvalScope benchmark adapters and collects EvalScope report JSON into the
Fugu-aligned table/charts. The lightweight `flow_eval.py` runner below is only
for development smoke tests and qualitative regression triage.

For the signal/decision design contract and staged eval order, see
[`SIGNAL_DECISION_EVAL_PLAN.md`](SIGNAL_DECISION_EVAL_PLAN.md). Current
publishable evals use one benchmark-specific `vllm-sr/auto` recipe per row.
Closed-model recipes are shown as `VSR 1.0 Pro`, and single-open-model recipes
are shown as `VSR 1.0`. Current recipes include
`configs/amd_auto_gpqa_omni.yaml`, `configs/amd_auto_livecode_omni.yaml`,
`configs/amd_auto_livecode_glm52_omni.yaml`, and
`configs/amd_auto_livecode_kimi_k27_code_omni.yaml`.

Use the lightweight runner to compare the same prompt set across arms such as:

- `vsr=vllm-sr/auto`
- `single=<worker-model>`

## Quick Run

```bash
python bench/router_flow/flow_eval.py \
  --dataset bench/router_flow/frontier_proxy_prompts.jsonl \
  --base-url http://localhost:8899/v1 \
  --arm vsr=vllm-sr/auto \
  --arm single=claude-worker \
  --output-dir bench/router_flow/results/local
```

With an OpenAI-compatible judge:

```bash
export OPENROUTER_API_KEY=...
python bench/router_flow/flow_eval.py \
  --dataset bench/router_flow/frontier_proxy_prompts.jsonl \
  --base-url http://localhost:8899/v1 \
  --arm vsr=vllm-sr/auto \
  --arm single=claude-worker \
  --judge-base-url https://openrouter.ai/api/v1 \
  --judge-model openai/gpt-4.1-mini \
  --output-dir bench/router_flow/results/amd-flow
```

The runner writes:

- `samples.jsonl`: per-item answers, score, latency, token usage, looper headers;
- `summary.json`: pass rate, mean score, mean latency, token totals by arm and
  category.

For local vLLM smoke runs with Qwen chat templates, add
`--request-extra-json '{"chat_template_kwargs":{"enable_thinking":false}}'` so
single-model baselines return visible answer text instead of reasoning-only
content. Use `--judge-request-extra-json` the same way when the judge is also a
local Qwen/vLLM endpoint.

The `results/` directory is git-ignored. Do not commit API keys, private host
details, or raw private validation logs.

## Report Rendering

After running arms into separate result directories, render a table and SVG
charts:

```bash
python bench/router_flow/render_report.py \
  --results bench/router_flow/results/amd-auto \
  --results bench/router_flow/results/amd-fusion \
  --results bench/router_flow/results/amd-flow \
  --reference-json bench/router_flow/public_reference_scores.json \
  --output-dir bench/router_flow/results/amd-report
```

The report script writes:

- `benchmark_table.md`
- `benchmark_table.csv`
- `overall_bars.svg`
- `benchmark_bars.svg`
- `metadata.json`

Rows from `public_reference_scores.json` are contextual public reference
columns. They are not generated by this proxy harness and should be labeled as
such in any external writeup.
