# `ugphysics_judge` resource server

Verifies undergraduate-physics free-form answers using a TRUE/FALSE
LLM-judge cascade with `math_verify` symbolic-equivalence as a
fast-path.  Ports NeMo Skills' `UGPhysicsMetrics` + the
`judge/ugphysics.yaml` four-shot physics judge prompt char-for-char.

## When to use

Use this server when grading physics problems where:

- Answers are free-form mathematical expressions or numerical values
  with optional units.
- `math_verify`'s symbolic equivalence is sufficient for the
  numerically-correct cases but breaks down on physical equivalence
  (e.g. equivalent units, factored vs. expanded expressions, alternate
  variable choices).
- A physics-aware LLM judge can resolve the residual cases, given the
  problem statement, the canonical reference solution, and the
  reference answer.

The verdict cascade is:

1. `math_verify` symbolic equivalence on the boxed student answer
   versus the boxed expected answer.  Match → `TRUE`.
2. On a symbolic miss (or when `should_use_judge=False`), the judge is
   asked once whether the student's answer is equivalent to the
   reference.  The judge prompt embeds the problem, reference solution,
   reference answer, and student solution, and is parsed for
   `## Equivalence Judgement\nTRUE|FALSE`.

## Per-row schema

`verify` requires:

| Field             | Type   | Notes                                                                                                         |
| ----------------- | ------ | ------------------------------------------------------------------------------------------------------------- |
| `question`        | str    | Problem statement.                                                                                            |
| `expected_answer` | str    | Canonical answer (may be wrapped in `\(…\)`/`$…$` — stripped before symbolic comparison).                     |
| `solution`        | str    | Reference solution embedded in the judge prompt under `**Reference Solution**:`. Empty string is acceptable.  |
| `subject`         | str    | UGPhysics subject (e.g. `ClassicalMechanics`); used by `compute_metrics` for the per-subject Tier-2 breakdown. |

## Reasoning models

When the policy model emits `<think>…</think>` chains, start vLLM with
`--reasoning-parser <name>` (e.g. `deepseek_r1` for Nemotron-3-Nano) so
the chain-of-thought is routed to a separate reasoning output item and
the `\boxed{…}` extractor / judge prompt only see the final assistant
message.

## Example usage

The server config references a `judge_model` that callers must
provide.  The bundled `benchmarks/ugphysics/judge_gptoss20b.yaml`
(`openai/gpt-oss-20b` via NVIDIA's public NIM) is a runnable default;
swap it for any other `responses_api_models/*` config that exposes a
`judge_model` server name to use a different judge.

```bash
# Running servers
gym env start \
    --model-type vllm_model \
    --resources-server ugphysics_judge \
    --config benchmarks/ugphysics/judge_gptoss20b.yaml

# Collecting rollouts (5-example smoke test)
gym eval run --no-serve \
    --agent ugphysics_judge_simple_agent \
    --input resources_servers/ugphysics_judge/data/example.jsonl \
    --output results/ugphysics_judge_rollouts.jsonl \
    --num-repeats 1
```

## Metrics

- **Tier 1** — pass@k / pass@1[avg-of-k] / majority@k for both
  `judge_accuracy` (final reward) and `symbolic_accuracy`
  (math-verify-only reward).  Auto-generated by
  `compute_pass_majority_metrics`.
- **Tier 2** — per-subject pass@k via `compute_subset_metrics` keyed on
  the `subject` field (matches Skills' `subset_for_metrics`
  stratification across the 13 UGPhysics subjects).
