---
title: LLM Graders
description: Customizable LLM-based evaluation
sidebar:
  order: 2
---

LLM graders use a language model to evaluate agent responses against custom criteria defined in a prompt file.

## Default Grader

When a test defines `criteria` but has **no `assertions` field**, a default `llm-grader` runs automatically. The built-in prompt evaluates the response against your `criteria` and `expected_output`:

```yaml
tests:
  - id: simple-eval
    criteria: Correctly explains the bug and proposes a fix
    input: "Debug this function..."
    # No assertions needed — default llm-grader evaluates against criteria
```

When `assertions` **is** present, no default grader is added. To use an LLM grader alongside other graders, declare it explicitly. See [How criteria and assertions interact](/docs/evaluation/eval-cases/#how-criteria-and-assertions-interact).

## Configuration

Reference an LLM grader in your eval file:

```yaml
assertions:
  - name: semantic_check
    type: llm-grader
    prompt: ./graders/correctness.md
    target: grader_gpt_5_mini   # optional: route this grader to a named LLM target
```

Use `target:` when you want different `llm-grader` entries in the same eval to run on different grader models. This is useful for grader panels, majority-vote ensembles, and grader A/B benchmarks.

## Prompt Files

The prompt file defines evaluation criteria and scoring guidelines. It can be a markdown text template or a TypeScript/JavaScript dynamic template.

### Markdown Template

Write evaluation instructions as markdown. Template variables are interpolated:

```markdown
# Evaluation Criteria

Evaluate the candidate's response to the following question:

**Question:** {{input_text}}
**Criteria:** {{criteria}}
**Reference Answer:** {{expected_output_text}}
**Candidate Answer:** {{output_text}}

## Scoring

Score the response from 0.0 to 1.0 based on:
1. Correctness — does the output match the expected outcome?
2. Completeness — does it address all parts of the question?
3. Clarity — is the response clear and well-structured?
```

### Available Template Variables

| Variable | Source |
|----------|--------|
| `input_text` | First user message content |
| `output_text` | Last candidate response content |
| `expected_output_text` | Last expected message content |
| `criteria` | Test `criteria` field |
| `input` | Full resolved input array, JSON-serialized |
| `expected_output` | Full resolved expected array, JSON-serialized |
| `output` | Full provider output array, JSON-serialized |
| `file_changes` | Unified diff of workspace file changes (populated when `workspace` is configured) |
| `tool_calls` | Formatted summary of tool calls from agent execution (tool name + key inputs per call) |

## Per-Grader Target

By default, an `llm-grader` uses the suite target's `grader_target`. Override it per grader when you need multiple grader models in one run:

```yaml
assertions:
  - name: grader-gpt
    type: llm-grader
    target: grader_gpt_5_mini
    prompt: ./prompts/pass-fail.md
  - name: grader-haiku
    type: llm-grader
    target: grader_claude_haiku
    prompt: ./prompts/pass-fail.md
```

Each `target:` value must match a named LLM target in `.agentv/targets.yaml`.

### TypeScript Template

For dynamic prompt generation, use the `definePromptTemplate` function from `@agentv/eval`:

```typescript
#!/usr/bin/env bun
import { definePromptTemplate } from '@agentv/eval';

export default definePromptTemplate((ctx) => {
  const rubric = ctx.config?.rubric as string | undefined;

  return `You are evaluating an AI assistant's response.

## Question
${ctx.inputText}

## Candidate Answer
${ctx.outputText}

${ctx.expectedOutputText ? `## Reference Answer\n${ctx.expectedOutputText}` : ''}

${rubric ? `## Evaluation Criteria\n${rubric}` : ''}

Evaluate and provide a score from 0 to 1.`;
});
```

## How It Works

1. AgentV renders the prompt template with variables from the test
2. The rendered prompt is sent to the grader target (configured in targets.yaml)
3. The LLM returns a structured evaluation with score, assertions array, and reasoning
4. Results are recorded in the output JSONL

## Command Configuration

When using TypeScript templates, configure them in YAML with optional `config` data passed to the command:

```yaml
assertions:
  - name: custom-eval
    type: llm-grader
    prompt:
      command: [bun, run, ../prompts/custom-grader.ts]
      config:
        rubric: "Your rubric here"
        strictMode: true
```

The `config` object is available as `ctx.config` inside the template function.

## Preprocessing File Outputs

If an agent returns a `ContentFile` block instead of plain text, you can preprocess that file into text before `llm-grader` builds the candidate prompt.

AgentV always tries a default UTF-8 text read first. That is enough for text-based formats such as CSV, JSON, SQL, Markdown, YAML, HTML, XML, and plain text. For binary formats such as `.xlsx`, `.pdf`, or `.docx`, add a preprocessor command:

```yaml
preprocessors:
  - type: xlsx
    command: ["bun", "run", "scripts/preprocessors/xlsx-to-csv.ts"]

tests:
  - id: spreadsheet-output
    criteria: Output includes the revenue rows
    input: Generate the spreadsheet report
    assertions:
      - name: spreadsheet-check
        type: llm-grader
        prompt: |
          Check whether the transformed spreadsheet text contains the revenue rows:

          {{ output }}
```

`type` accepts either a short alias such as `xlsx` or a full MIME type such as `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`.

Resolution order:

- per-grader `preprocessors` override suite-level entries
- if no preprocessor matches, AgentV falls back to a UTF-8 text read
- if the fallback read looks binary or invalid, the grader receives a warning note instead of failing the test run

The implicit default `llm-grader` also inherits suite-level `preprocessors`, so you can omit `assertions` and still preprocess file outputs before grading.

See [`examples/features/preprocessors/`](../../../../examples/features/preprocessors/) for a runnable example with a file-producing target and a custom preprocessor script.

## Available Context Fields

TypeScript templates receive a context object with these fields:

| Field | Type | Description |
|-------|------|-------------|
| `inputText` | `string` | First user message content |
| `outputText` | `string` | Last entry in `output` |
| `expectedOutputText` | `string` | Last entry in `expected_output` |
| `criteria` | `string` | Test `criteria` field |
| `expectedOutput` | `Message[]` | Full resolved expected output |
| `output` | `Message[]` | Full provider output messages |
| `trace` | `TraceSummary` | Execution metrics summary |
| `config` | `object` | Custom config from YAML |

## Template Variable Derivation

Template variables are derived internally through three layers:

### 1. Authoring Layer

What users write in YAML or JSONL:

- `input` or `input` -- two syntaxes for the same data. `input: "What is 2+2?"` expands to `[{ role: "user", content: "What is 2+2?" }]`. If both are present, `input` takes precedence.
- `expected_output` or `expected_output` -- two syntaxes for the same data. `expected_output: "4"` expands to `[{ role: "assistant", content: "4" }]`. If both are present, `expected_output` takes precedence.

### 2. Resolved Layer

After parsing, canonical message arrays replace the shorthand fields:

- `input: TestMessage[]` -- canonical resolved input
- `expected_output: TestMessage[]` -- canonical resolved expected output

At this layer, `input` and `expected_output` no longer exist as separate fields.

### 3. Template Variable Layer

Derived strings injected into grader prompts:

| Variable | Derivation |
|----------|------------|
| `input_text` | Content of the first `user` role entry in `input` |
| `criteria` | Passed through from the test field |
| `expected_output_text` | Content of the last entry in `expected_output` |
| `output_text` | Content of the last entry in `output` |
| `input` | Full resolved input array, JSON-serialized |
| `expected_output` | Full resolved expected array, JSON-serialized |
| `output` | Full provider output array, JSON-serialized |
| `file_changes` | Unified diff of workspace file changes (populated when `workspace` is configured) |
| `tool_calls` | Formatted summary of tool calls from agent execution (tool name + key inputs per call) |

**Example flow:**

```yaml
# User writes:
input: "What is 2+2?"
expected_output: "The answer is 4"
```

```
# Resolved:
input:    [{ role: "user", content: "What is 2+2?" }]
expected_output: [{ role: "assistant", content: "The answer is 4" }]

# Derived template variables:
input_text:           "What is 2+2?"
expected_output_text: "The answer is 4"
output_text: (extracted from provider output at runtime)
```
