---
title: Running Evaluations
description: CLI commands for running and managing evaluations
sidebar:
  order: 4
---

## Run an Evaluation

```bash
agentv eval evals/my-eval.yaml
```

Results are written to `.agentv/results/runs/<timestamp>/index.jsonl`. Each line is a JSON object with one result per test case, and the run workspace also stores the manifest and related artifacts.

Each `scores[]` entry includes per-grader timing:

```json
{
  "scores": [
    {
      "name": "format_structure",
      "type": "llm-grader",
      "score": 0.9,
      "verdict": "pass",
      "assertions": [
        { "text": "clear structure", "passed": true }
      ],
      "duration_ms": 9103,
      "started_at": "2026-03-09T00:05:10.123Z",
      "ended_at": "2026-03-09T00:05:19.226Z",
      "token_usage": { "input": 2711, "output": 2535 }
    }
  ]
}
```

The `duration_ms`, `started_at`, and `ended_at` fields are present on every grader result (including `code-grader`), enabling per-grader bottleneck analysis.

## Common Options

### Override Target

Run against a different target than specified in the eval file:

```bash
agentv eval --target my-target evals/**/*.yaml
```

### Experiment Label

Tag a pipeline run with an experiment name to track different conditions (e.g. with vs without skills):

```bash
agentv pipeline run evals/my-eval.yaml --experiment with_skills
agentv pipeline run evals/my-eval.yaml --experiment without_skills
```

The experiment label is written to `manifest.json` and propagated to each entry in `index.jsonl` by `pipeline bench`. The eval file stays the same across experiments — what changes is the environment. Dashboards can filter and compare results by experiment.

### Run Specific Test

Run a single test by ID:

```bash
agentv eval --test-id case-123 evals/my-eval.yaml
```

### Dry Run

Test the harness flow with mock responses (does not call real providers):

```bash
agentv eval --dry-run evals/my-eval.yaml
```

:::note
Dry-run returns mock responses that don't match grader output schemas. Use it only for testing harness flow, not grader logic.
:::

### Custom Output Directory

Write all artifacts (index.jsonl, benchmark.json, per-test grading/timing) to a specific directory:

```bash
agentv eval evals/my-eval.yaml --output ./my-results
```

### Export Additional Formats

Write additional output files alongside the artifact directory. Format is inferred from the file extension (`.jsonl`, `.json`, `.xml`, `.yaml`, `.html`):

```bash
# Export JUnit XML for CI test reporters
agentv eval evals/my-eval.yaml --export results.xml

# Export multiple formats
agentv eval evals/my-eval.yaml --output ./my-results --export results.xml --export results.html
```

### Trace Persistence

Export execution traces (tool calls, timing, spans) to files for debugging and analysis:

By default, AgentV writes a per-run workspace with `index.jsonl` as the canonical manifest for
result-oriented workflows. For full-fidelity span inspection, export OTLP JSON explicitly.

```bash
# Summary-level inspection from the run manifest
agentv inspect stats .agentv/results/runs/<timestamp>/index.jsonl

# Full-fidelity OTLP JSON trace (importable by OTel backends like Jaeger, Grafana)
agentv eval evals/my-eval.yaml --otel-file traces/eval.otlp.json

# Inspect the OTLP export
agentv inspect show traces/eval.otlp.json --tree
```

`index.jsonl` contains aggregate metrics such as score, latency, cost, token usage, and summary
trace counters. `--otel-file` writes standard OTLP JSON that can be imported into any
OpenTelemetry-compatible backend.

### Live OTel Export

Stream traces directly to an observability backend during evaluation using `--export-otel`:

```bash
# Use a backend preset (braintrust, langfuse, confident)
agentv eval evals/my-eval.yaml --export-otel --otel-backend braintrust

# Include message content and tool I/O in spans (disabled by default for privacy)
agentv eval evals/my-eval.yaml --export-otel --otel-backend braintrust --otel-capture-content

# Group messages into turn spans for multi-turn evaluations
agentv eval evals/my-eval.yaml --export-otel --otel-backend braintrust --otel-group-turns
```

#### Braintrust

Set up your environment:

```bash
export BRAINTRUST_API_KEY=sk-...
export BRAINTRUST_PROJECT=my-project    # associates traces with a Braintrust project
```

Run an eval with traces sent to Braintrust:

```bash
agentv eval evals/my-eval.yaml --export-otel --otel-backend braintrust --otel-capture-content
```

The following environment variables control project association (at least one is required):

| Variable | Format | Example |
|----------|--------|---------|
| `BRAINTRUST_PROJECT` | Project name | `my-evals` |
| `BRAINTRUST_PROJECT_ID` | Project UUID | `proj_abc123` |
| `BRAINTRUST_PARENT` | Raw `x-bt-parent` header | `project_name:my-evals` |

Each eval test case produces a trace with:
- **Root span** (`agentv.eval`) — test ID, target, score, duration
- **LLM call spans** (`chat <model>`) — model name, token usage (input/output/cached)
- **Tool call spans** (`execute_tool <name>`) — tool name, arguments, results (with `--otel-capture-content`)
- **Turn spans** (`agentv.turn.N`) — groups messages by conversation turn (with `--otel-group-turns`)
- **Grader events** — per-grader scores attached to the root span

:::tip[Claude provider + trace-claude-code plugin]
When using the Claude provider, AgentV injects `CC_PARENT_SPAN_ID` and `CC_ROOT_SPAN_ID` into the Claude subprocess. If the [trace-claude-code](https://github.com/braintrustdata/braintrust-claude-plugin) plugin is installed, it attaches Claude Code CLI-level tool spans (Read, Write, Bash, etc.) as children of the AgentV eval trace, giving you full visibility into both the eval framework and the agent's internal actions.
:::

#### Langfuse

```bash
export LANGFUSE_PUBLIC_KEY=pk-...
export LANGFUSE_SECRET_KEY=sk-...
# Optional: export LANGFUSE_HOST=https://cloud.langfuse.com

agentv eval evals/my-eval.yaml --export-otel --otel-backend langfuse --otel-capture-content
```

#### Custom OTLP Endpoint

For backends not covered by presets, configure via environment variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-backend/v1/traces
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer token"

agentv eval evals/my-eval.yaml --export-otel
```

### Parallelism

The `--workers N` flag controls how many **test cases run in parallel within each eval file** (default: 3). Eval files always run sequentially — one file completes before the next starts.

```bash
agentv eval evals/my-eval.yaml --workers 4
# Up to 4 test cases from the file run concurrently

agentv eval evals/file1.yaml evals/file2.yaml evals/file3.yaml --workers 3
# Files run one at a time; within each file, up to 3 test cases run in parallel
```

This matches the standard model used by eval frameworks (promptfoo, deepeval, OpenAI Evals) and avoids cross-file workspace races without any special configuration.

### Workspace Modes and Finish Policy

Use workspace mode and finish policies instead of multiple conflicting booleans:

```bash
# Mode: pooled | temp | static
agentv eval evals/my-eval.yaml --workspace-mode pooled

# Static mode path
agentv eval evals/my-eval.yaml --workspace-mode static --workspace-path /path/to/workspace

# Pooled reset policy override: standard | full (CLI override)
agentv eval evals/my-eval.yaml --workspace-clean full

# Finish policy overrides: keep | cleanup (CLI)
agentv eval evals/my-eval.yaml --retain-on-success cleanup --retain-on-failure keep
```

Equivalent eval YAML:

```yaml
workspace:
  mode: pooled           # pooled | temp | static
  path: null             # workspace path for mode=static; auto-materialised when empty/missing
  hooks:
    enabled: true        # set false to skip all hooks
    after_each:
      reset: fast        # none | fast | strict
```

Notes:
- Pooling is default for shared workspaces with repos when mode is not specified.
- `mode: static` (or `--workspace-mode static`) uses `path` / `--workspace-path`. When the path is empty or missing, the workspace is auto-materialised (template copied + repos cloned). Populated directories are reused as-is.
- Static mode is incompatible with `isolation: per_test`.
- `hooks.enabled: false` skips all lifecycle hooks (setup, teardown, reset).
- Pool slots are managed separately (`agentv workspace list|clean`).

### Resume an Interrupted Run

AgentV ships three flags for picking up a partial run. They differ only in **which prior results are skipped**; in all three modes the new results are merged with the prior run.

| Flag | What it skips | What it re-runs | Use when |
|------|---------------|-----------------|----------|
| `--resume` | Anything that finished without an `execution_error` (passes, fails, threshold misses) | Errors and missing cases | The run was interrupted (Ctrl-C, crash, OOM) and you just want it to finish |
| `--rerun-failed` | Only cases with `executionStatus === 'ok'` | Errors **and** test failures (assertion misses, threshold misses) | A grader change or model swap means you want to re-grade everything that wasn't already passing |
| `--retry-errors <path>` | Anything that completed without an `execution_error` (same set as `--resume`) | Errors and missing cases | You want to point at an arbitrary prior run/manifest by path, instead of resuming the run dir you're currently writing to |

`--resume` and `--rerun-failed` both append to the existing `index.jsonl`. When `--output <dir>` is given they target that directory; when omitted they default to the **last run dir for the current cwd**, recorded in `.agentv/cache.json` and updated after every eval. This matches promptfoo's `--resume [evalId]` and OpenCompass's `-r [timestamp]` "latest by default" convention. `--retry-errors` takes the prior run's path directly (a directory or an `index.jsonl`).

```bash
# Resume the last run — no args needed; AgentV finds it from .agentv/cache.json
agentv eval evals/my-eval.yaml --resume

# Or target a specific run dir explicitly
agentv eval evals/my-eval.yaml --output .agentv/results/runs/<timestamp> --resume

# Re-run errors AND failed cases against the last run dir
agentv eval evals/my-eval.yaml --rerun-failed

# Re-run only execution errors from any prior run by path
agentv eval evals/my-eval.yaml --retry-errors .agentv/results/runs/<timestamp>/index.jsonl
```

After any failing run, the CLI prints the exact `--rerun-failed` command for the run dir that just completed — copy/paste it.

The interactive wizard (`agentv eval` with no arguments) remembers the last run's artifact directory and surfaces a **"Resume last run"** entry in the main menu when one exists.

### Execution Error Tolerance

Control whether the eval run halts on execution errors using `execution.fail_on_error` in the eval YAML:

```yaml
execution:
  fail_on_error: false    # never halt on errors (default)
  # fail_on_error: true   # halt on first execution error
```

| Value | Behavior |
|-------|----------|
| `true` | Halt immediately on first execution error |
| `false` | Continue despite errors (default) |

When halted, remaining tests are recorded with `failureReasonCode: 'error_threshold_exceeded'`. With concurrency > 1, a few additional tests may complete before halting takes effect.

### Suite-Level Quality Threshold

Set a per-test score threshold for the eval suite. Each test case must score at or above this value to pass. If any test scores below the threshold, the CLI exits with code 1 — useful for CI/CD quality gates.

**CLI flag:**

```bash
agentv eval evals/ --threshold 0.8
```

**YAML config:**

```yaml
execution:
  threshold: 0.8
```

The CLI `--threshold` flag overrides the YAML value. The threshold is a number between 0 and 1 (default: 0.8). Execution errors are excluded from the count.

When active, the summary line shows how many tests met the threshold:

```
RESULT: PASS  (28/31 scored >= 0.8, mean: 0.927)
```

The threshold also controls JUnit XML pass/fail: tests with scores below the threshold are marked as `<failure>` in JUnit output. When no threshold is set, JUnit defaults to 0.5.

## Validate Before Running

Check eval files for schema errors without executing:

```bash
agentv validate evals/my-eval.yaml
```

## Run a Single Assertion

Run a code-grader assertion in isolation without executing a full eval suite:

```bash
agentv eval assert <name> --agent-output <text> --agent-input <text>
```

The command discovers the assertion script by walking up directories looking for `.agentv/graders/<name>.{ts,js,mts,mjs}`, then passes the input via stdin and prints the result JSON to stdout.

```bash
# Run an assertion with inline arguments
agentv eval assert rouge-score \
  --agent-output "The fox jumps over the lazy dog" \
  --agent-input "Summarise the article"

# Or pass a JSON payload file
agentv eval assert rouge-score --file result.json
```

The `--file` option reads a JSON file with `{ "output": "...", "input": "..." }` fields.

**Exit codes:** 0 if score >= 0.5 (pass), 1 if score < 0.5 (fail).

This is the same interface that agent-orchestrated evals use — the EVAL.yaml transpiler emits `assertions` instructions for code graders so external grading agents can execute them directly.

## Offline Grading

Grade existing agent sessions without re-running them. Import a transcript, then run deterministic graders:

```bash
# List sessions and import one
agentv import claude --list
agentv import claude --session-id <uuid>

# Run graders against the imported transcript
agentv eval evals/my-eval.yaml --transcript .agentv/transcripts/claude-<id>.jsonl
```

See the [Import tool docs](/docs/tools/import/) for all providers and options.

## Version Requirements

Declare the minimum AgentV version needed by your eval project in `.agentv/config.yaml`:

```yaml
required_version: ">=2.12.0"
```

The value is a **semver range** using standard npm syntax (e.g., `>=2.12.0`, `^2.12.0`, `~2.12`, `>=2.12.0 <3.0.0`).

| Condition | Interactive (TTY) | Non-interactive (CI) |
|-----------|-------------------|---------------------|
| Version satisfies range | Runs silently | Runs silently |
| Version below range | Warns + prompts to continue | Warns to stderr, continues |
| `--strict` flag + mismatch | Warns + exits 1 | Warns + exits 1 |
| No `required_version` set | Runs silently | Runs silently |
| Malformed semver range | Error + exits 1 | Error + exits 1 |

Use `--strict` in CI pipelines to enforce version requirements:

```bash
agentv eval --strict evals/my-eval.yaml
```

## Config File Defaults

Set default execution options so you don't have to pass them on every CLI invocation. Both `.agentv/config.yaml` and `agentv.config.ts` are supported.

### YAML config (`.agentv/config.yaml`)

```yaml
execution:
  verbose: true
  keep_workspaces: false
  otel_file: .agentv/results/otel-{timestamp}.json
```

| Field | CLI equivalent | Type | Default | Description |
|-------|---------------|------|---------|-------------|
| `verbose` | `--verbose` | boolean | `false` | Enable verbose logging |
| `keep_workspaces` | `--keep-workspaces` | boolean | `false` | Always keep temp workspaces after eval |
| `otel_file` | `--otel-file` | string | none | Write OTLP JSON trace to file |

### TypeScript config (`agentv.config.ts`)

```typescript
import { defineConfig } from '@agentv/core';

export default defineConfig({
  execution: {
    verbose: true,
    keepWorkspaces: false,
    otelFile: '.agentv/results/otel-{timestamp}.json',
  },
});
```

The `{timestamp}` placeholder is replaced with an ISO-like timestamp (e.g., `2026-03-05T14-30-00-000Z`) at execution time.

**Precedence:** CLI flags > `.agentv/config.yaml` > `agentv.config.ts` > built-in defaults.

## Environment Variables

### AGENTV_HOME

Override the data directory for heavy runtime artifacts — workspaces, workspace pool, subagents, trace state, git cache, and downloaded dependencies. Lightweight config and cache files (`version-check.json`, `last-config.json`, `projects.yaml`) always stay in `~/.agentv` regardless of this setting.

```bash
# Linux/macOS
export AGENTV_HOME=/data/agentv

# Windows (PowerShell)
$env:AGENTV_HOME = "D:\agentv"

# Windows (CMD)
set AGENTV_HOME=D:\agentv
```

When set, AgentV logs `Using AGENTV_HOME: <path>` on startup to confirm the override is active.

:::tip[Windows long paths]
If you use a custom `AGENTV_HOME` on Windows for large monorepo workspaces, enable long path support:
```powershell
git config --system core.longpaths true
```
Or set the registry key: `HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled = 1`
:::

## All Options

Run `agentv eval --help` for the full list of options including workers, timeouts, output formats, and trace dumping.
