---
title: Eval Files
description: YAML and JSONL evaluation file formats
sidebar:
  order: 1
---

Evaluation files define the test cases, targets, and graders for an evaluation run. AgentV supports two formats: YAML and JSONL.

## Suites

An eval file is a **suite**: it binds test cases to execution context (workspace, hooks, targets, trials). Test cases can be inline or loaded from an external file via `tests: ./cases.yaml` for reuse across suites.

## YAML Format

The primary format. A single file contains metadata, execution config, and tests:

```yaml
description: Math problem solving evaluation
execution:
  target: default

assertions:
  - name: correctness
    type: llm-grader
    prompt: ./graders/correctness.md

tests:
  - id: addition
    criteria: Correctly calculates 15 + 27 = 42
    input: What is 15 + 27?
    expected_output: "42"
```

### Top-level Fields

| Field | Description |
|-------|-------------|
| `description` | Human-readable description of the evaluation |
| `suite` | Optional suite identifier |
| `execution` | Default execution config (`target`, `fail_on_error`, `threshold`, etc.) |
| `workspace` | Suite-level workspace config — inline object or string path to an [external workspace file](/docs/guides/workspace-pool/#external-workspace-config) |
| `tests` | Array of individual tests, or a string path to an external file or directory |
| `assertions` | Suite-level graders appended to each test unless `execution.skip_defaults: true` is set on the test |
| `input` | Suite-level input messages prepended to each test's input unless `execution.skip_defaults: true` is set on the test |

### Metadata Fields

You can add structured metadata to your eval file using these optional top-level fields. Metadata is parsed when the `name` field is present:

| Field | Description |
|-------|-------------|
| `name` | Machine-readable identifier (lowercase, hyphens, max 64 chars). Triggers metadata parsing. |
| `description` | Human-readable description (max 1024 chars) |
| `version` | Eval version string (e.g., `"1.0"`) |
| `author` | Author or team identifier |
| `tags` | Array of string tags for categorization |
| `license` | License identifier (e.g., `"MIT"`, `"Apache-2.0"`) |
| `requires` | Dependency constraints (e.g., `agentv: ">=0.30.0"`) |

```yaml
name: export-screening
description: Evaluates export control screening accuracy
version: "1.0"
author: acme-compliance
tags: [compliance, agents]
license: Apache-2.0
requires:
  agentv: ">=0.30.0"

tests:
  - id: denied-party
    criteria: Identifies denied parties correctly
    input: Screen "Acme Corp" against denied parties list
```

### Suite-level Assertions

The `assertions` field is the canonical way to define suite-level graders. Suite-level assertions are appended to every test's graders unless a test sets `execution.skip_defaults: true`.

```yaml
description: API response validation
assertions:
  - type: is-json
    required: true
  - type: contains
    value: "status"

tests:
  - id: health-check
    criteria: Returns health status
    input: Check API health
```

`assertions` supports all grader types, including deterministic assertion types (`contains`, `regex`, `is_json`, `equals`) and `rubrics`. See [Tests](/docs/evaluation/eval-cases/#per-test-assertions) for per-test assertions usage.

### Assertion Includes

Reusable assertion sets can be factored into template files and referenced from any `assertions` array:

```yaml
assertions:
  - include: safe-response
  - include: ./shared/format.yaml
```

Resolution rules:
- `include: name` resolves to `.agentv/templates/{name}.yaml` with the closest matching directory winning
- Relative paths resolve from the eval file location, so `include: ./shared/format.yaml` works as expected
- Nested includes are allowed up to depth 3 to keep cycles and runaway recursion bounded
- Suite-level includes follow the same merge behavior as other suite-level assertions and still respect `execution.skip_defaults: true`

### Suite-level Input

The `input` field defines messages that are **prepended** to every test's input. This avoids repeating the same prompt or system context in each test case — following the same pattern as suite-level `assertions`.

```yaml
description: Travel assistant evaluation
input:
  - role: user
    content:
      - type: file
        value: ./system-prompt.md

tests: ./cases.yaml
```

Each test in `cases.yaml` only needs its own query:

```yaml
- id: japan-spring
  criteria: Recommends spring for cherry blossoms
  input: When is the best time to visit Japan?
```

The effective input at runtime becomes `[...suite input, ...test input]`.

Suite-level `input` accepts the same formats as test-level `input`:
- **String** — wrapped as `[{ role: "user", content: "..." }]`
- **Message array** — used as-is, including file references

To opt out for a specific test, set `execution.skip_defaults: true` (same flag that skips suite-level `assertions`).

### Suite-level Input Files

The `input_files` field provides a shorthand for attaching shared file references to every test. When a test has a string `input`, the suite-level files are prepended as `type: file` content blocks in a single user message — the same shape produced by per-test `input_files`.

```yaml
description: Schema review evaluation
input_files:
  - ./shared-context.md
  - ./schema.json

tests:
  - id: summarize
    criteria: Summarizes the important constraints
    input: Summarize the important constraints.
  - id: validate
    criteria: Identifies validation gaps
    input: What validation is missing?
```

Each test's effective input becomes a single user message with `[file blocks..., text block]`.

Per-test `input_files` overrides the suite-level value (it does not merge). To opt out, set `execution.skip_defaults: true` on the test.

### Tests as String Path

Instead of inlining tests in the same file, you can point `tests` to an external YAML or JSONL file. This is the inverse of the sidecar pattern — the metadata file references the test data:

```yaml
name: my-eval
description: My evaluation suite
execution:
  target: default
tests: ./cases.yaml
```

The path is resolved relative to the eval file's directory. The external file should contain a YAML array of test objects or a JSONL file with one test per line.

### Tests as Directory Path

When `tests` points to a directory, AgentV auto-discovers test cases from subdirectories. Each subdirectory containing a `case.yaml` (or `case.yml`) becomes a test case:

```
my-eval/
  EVAL.yaml
  cases/
    fix-null-check/
      case.yaml
    add-greeting/
      case.yaml
      workspace/        # optional per-case workspace template
        setup-files...
```

```yaml
# EVAL.yaml
name: my-benchmark
tests: ./cases/
```

Each `case.yaml` is a single YAML object (not an array) with the same fields as an inline test:

```yaml
# cases/fix-null-check/case.yaml
criteria: Fixes the null reference bug in the parser module
input: Fix the null check bug in parser.ts
```

**Behavior:**

- **Directory name as `id`:** If `case.yaml` doesn't specify an `id`, the directory name is used (e.g., `fix-null-check`)
- **Alphabetical ordering:** Subdirectories are sorted alphabetically for deterministic order
- **Per-case workspace:** A `workspace/` subdirectory inside the case directory automatically sets `workspace.template` to that path, unless the case already defines a `workspace` field
- **Skipped directories:** Subdirectories without `case.yaml` are skipped with a warning
- **Suite-level config applies:** Suite-level `assertions`, `input`, `workspace`, and `execution` still apply to directory-discovered cases

This pattern is useful for benchmarks with many cases, where each case benefits from its own directory for workspace templates, supporting files, or documentation.

## Environment Variable Interpolation

All string fields in eval files support `${{ VAR }}` syntax for environment variable interpolation. This enables portable eval configs that work across machines and CI environments without hardcoded paths.

```yaml
workspace:
  repos:
    - path: ./RepoA
      source:
        type: local
        path: "${{ REPO_A_PATH }}"

tests:
  - id: test-1
    input: "Evaluate the code in ${{ PROJECT_NAME }}"
    criteria: "${{ EVAL_CRITERIA }}"
```

### Behavior

- **Syntax:** `${{ VARIABLE_NAME }}` with optional whitespace around the name
- **Missing variables** resolve to an empty string
- **Partial interpolation** is supported: `${{ HOME }}/repos/${{ PROJECT }}` becomes `/home/user/repos/myproject`
- **Non-string values** (numbers, booleans) are not affected
- Interpolation is applied recursively to all nested objects and arrays
- Works in YAML eval files, external YAML/JSONL case files, and external workspace config files
- `.env` files in the directory hierarchy are loaded automatically before interpolation

### Example: Portable Workspace Config

```yaml
# workspace.yaml — works on any machine
repos:
  - path: ./my-repo
    source:
      type: local
      path: "${{ MY_REPO_LOCAL_PATH }}"
```

For local sources, omit `checkout.resolve`. If you need to pin the local clone to a specific revision, use `checkout.ref` and optionally `checkout.ancestor`.

```bash
# .env
MY_REPO_LOCAL_PATH=/home/dev/repos/my-repo
```

## Per-Test Template Variables

Eval YAML also supports per-test `vars` for data-driven prompt templates. Use `{{name}}` placeholders in test-facing text fields, and AgentV resolves them when the suite loads.

```yaml
input: "Answer clearly: {{question}}"

tests:
  - id: capital
    vars:
      question: What is the capital of France?
      expected_answer: Paris
    criteria: "Answers {{question}} correctly"
    input:
      - role: user
        content: "Question: {{question}}"
    expected_output: "{{expected_answer}}"
```

### Behavior

- `vars` is defined per test as an object
- `{{name}}` and dotted paths like `{{ user.name }}` are supported
- Substitution applies to suite-level `input`, test `input`, `input_files`, `criteria`, `expected_output`, and conversation turn `input` / `expected_output`
- When the whole string is a single placeholder, the original JSON value is preserved
- Missing variables are left unchanged, so unrelated template syntax is not silently blanked out
- `vars` interpolation is separate from environment interpolation: `{{question}}` uses test data, `${{ PROJECT_NAME }}` uses environment variables

## JSONL Format

For large-scale evaluations, AgentV supports JSONL (JSON Lines) format. Each line is a single test:

```jsonl
{"id": "test-1", "criteria": "Calculates correctly", "input": "What is 2+2?"}
{"id": "test-2", "criteria": "Provides explanation", "input": "Explain variables"}
```

### Sidecar Metadata

An optional YAML sidecar file provides metadata and execution config. Place it alongside the JSONL file with the same base name:

`dataset.jsonl` + `dataset.eval.yaml`:

```yaml
description: Math evaluation dataset
suite: math-tests
execution:
  target: azure-base
assertions:
  - name: correctness
    type: llm-grader
    prompt: ./graders/correctness.md
```

### Benefits of JSONL

- **Streaming-friendly** — process line by line
- **Git-friendly** — diffs show individual case changes
- **Programmatic generation** — easy to create from scripts
- **Industry standard** — compatible with DeepEval, LangWatch, Hugging Face datasets

## Converting Between Formats

Use the `convert` command to switch between YAML and JSONL:

```bash
agentv convert evals/dataset.eval.yaml --format jsonl
agentv convert evals/dataset.jsonl --format yaml
```
