# Accuracy Smoke (Harbor + Terminal-Bench-2)

This directory hosts a benchmark-backed accuracy smoke layer for `pythinker-code`.

It is intentionally separate from `tests_ai/` root so we can:

- keep existing fast policy checks unchanged;
- evolve benchmark-based checks independently;
- run smoke/nightly tracks with different budgets.

## Selected tasks (local CPU-focused)

Task list:

- `terminal_bench_2_tasks_default.txt`: default discriminative set (15 tasks)

Selection rules:

- no external API keys required by task logic;
- no GPU requirement in `task.toml`;
- moderate runtime suitable for development-tool CI smoke checks.

## Scripts

- `scripts/prepare_env.sh`: install pinned Harbor and clone/update pinned Terminal-Bench-2.
- `scripts/run_smoke.sh`: run selected tasks one-by-one with Harbor.

## Quick start

```bash
bash tests_ai/accuracy_smoke/scripts/prepare_env.sh
bash tests_ai/accuracy_smoke/scripts/run_smoke.sh
```

## Version pinning

Both Harbor and Terminal-Bench-2 are pinned by default for stable regression signals:

- Harbor: `0.5.0` (in `scripts/prepare_env.sh`)
- Terminal-Bench-2 ref: `53ff2b87d621bdb97b455671f2bd9728b7d86c11` (in `scripts/prepare_env.sh`)

Override when needed:

```bash
HARBOR_VERSION=0.5.0 \
TERMINAL_BENCH_2_REF=53ff2b87d621bdb97b455671f2bd9728b7d86c11 \
  bash tests_ai/accuracy_smoke/scripts/prepare_env.sh
```

Network-restricted environments can optionally set a GitHub mirror prefix:

```bash
GH_MIRROR_PREFIX=http://ghfast.top/ \
  bash tests_ai/accuracy_smoke/scripts/prepare_env.sh
```

## API key configuration

Harbor's custom local `pythinker-code` agent reads credentials from this environment variable
before running smoke tasks:

- `PYTHINKER_API_KEY`

Example:

```bash
export PYTHINKER_API_KEY="your_api_key"
bash tests_ai/accuracy_smoke/scripts/run_smoke.sh
```

For CI, store the key in secret variables and inject it as an environment
variable at runtime. Do not commit API keys into this repository.

Model selection:

- `HARBOR_MODEL` (default: `pythinker/pythinker-for-coding`)

## Use current pythinker-code source (not release build)

`run_smoke.sh` defaults to evaluating the current repository commit by using a
custom Harbor agent import path:

- `tests_ai.accuracy_smoke.local_pythinker_code_agent:LocalPythinkerCli`

By default, it builds a local wheel from your current workspace and installs
that wheel inside the benchmark container. This means local unpushed changes
are included automatically.

Wheel controls:

- `PYTHINKER_CLI_WHEEL_DIR` (default: `dist/accuracy_smoke`)
- `HARBOR_JOBS_DIR` (default: `jobs`)
- `UV_PYTHON` (default: local `python3`)
- `UV_PYTHON_INSTALL_MIRROR` (optional mirror for uv Python downloads)

Example override:

```bash
PYTHINKER_CLI_WHEEL_DIR=/tmp/pythinker-wheel-cache \
  bash tests_ai/accuracy_smoke/scripts/run_smoke.sh
```

If `GH_MIRROR_PREFIX` is set and `UV_PYTHON_INSTALL_MIRROR` is unset,
`run_smoke.sh` automatically prefixes uv Python-download URLs with
`GH_MIRROR_PREFIX`.

After each task run, `run_smoke.sh` collects reward stats from the latest
`jobs/<timestamp>/result.json` and writes a summary TSV file under
`tests_ai/accuracy_smoke/` (filename starts with `accuracy_smoke_rewards_`).

## Notes

- Default list is intentionally larger for better discrimination.
