---
title: Batch CLI Evaluation
description: Evaluate external tools that process all tests in a single invocation
sidebar:
  order: 5
---

Batch CLI evaluation handles tools that process multiple inputs at once — bulk classifiers, screening engines, or any runner that reads all tests and outputs results in one pass.

## Overview

Use batch CLI evaluation when:

- An external tool processes multiple inputs in a single invocation (e.g., AML screening, bulk classification)
- The runner reads the eval YAML directly to extract all tests
- Output is JSONL with records keyed by test `id`
- Each test has its own grader to validate its corresponding output record

## Execution Flow

1. **AgentV** invokes the batch runner once, passing `--eval <yaml-path>` and `--output <jsonl-path>`
2. **Batch runner** reads the eval YAML, extracts all tests, processes them, and writes JSONL output keyed by `id`
3. **AgentV** parses the JSONL and routes each record to its matching test by `id`
4. **Per-test graders** validate the output for each test independently

## Eval File Structure

```yaml
description: Batch CLI demo using structured input
execution:
  target: batch_cli

tests:
  - id: case-001
    criteria: |-
      Batch runner returns JSON with decision=CLEAR.

    expected_output:
      - role: assistant
        content:
          decision: CLEAR

    input:
      - role: system
        content: You are a batch processor.
      - role: user
        content:
          request:
            type: screening_check
            jurisdiction: AU
          row:
            id: case-001
            name: Example A
            amount: 5000

    assertions:
      - name: decision-check
        type: code-grader
        command: [bun, run, ./scripts/check-output.ts]
        cwd: .

  - id: case-002
    criteria: |-
      Batch runner returns JSON with decision=REVIEW.

    expected_output:
      - role: assistant
        content:
          decision: REVIEW

    input:
      - role: system
        content: You are a batch processor.
      - role: user
        content:
          request:
            type: screening_check
            jurisdiction: AU
          row:
            id: case-002
            name: Example B
            amount: 25000

    assertions:
      - name: decision-check
        type: code-grader
        command: [bun, run, ./scripts/check-output.ts]
        cwd: .
```

## Batch Runner Contract

The batch runner reads the eval YAML directly and processes all tests in one invocation.

### Input

The runner receives the eval file path via `--eval` and an output path via `--output`:

```bash
bun run batch-runner.ts --eval ./my-eval.yaml --output ./output.jsonl
```

### Output

JSONL where each line is a JSON object with an `id` matching a test:

```json
{"id": "case-001", "text": "{\"decision\": \"CLEAR\", ...}"}
{"id": "case-002", "text": "{\"decision\": \"REVIEW\", ...}"}
```

The `id` field must match the test `id` for AgentV to route output to the correct grader.

### Output with Tool Trajectory

To enable `tool_trajectory` evaluation, include `output` with `tool_calls`:

```json
{
  "id": "case-001",
  "text": "{\"decision\": \"CLEAR\", ...}",
  "output": [
    {
      "role": "assistant",
      "tool_calls": [
        {
          "tool": "screening_check",
          "input": { "origin_country": "NZ", "amount": 5000 },
          "output": { "decision": "CLEAR", "reasons": [] }
        }
      ]
    },
    {
      "role": "assistant",
      "content": { "decision": "CLEAR" }
    }
  ]
}
```

AgentV extracts tool calls directly from `output[].tool_calls[]` for `tool_trajectory` graders.

## Grader Implementation

Each test has its own grader that validates the batch runner output. The grader receives the standard `code_grader` input via stdin.

**Input (stdin):**
```json
{
  "output_text": "{\"id\":\"case-001\",\"decision\":\"CLEAR\",...}",
  "expected_output": [{"role": "assistant", "content": {"decision": "CLEAR"}}],
  "input": [...]
}
```

**Output (stdout):**
```json
{
  "score": 1.0,
  "assertions": [
    { "text": "decision matches: CLEAR", "passed": true }
  ],
  "reasoning": "Batch runner decision matches expected."
}
```

### Example Grader

```typescript
import fs from 'node:fs';

type EvalInput = {
  output_text?: string;
  expected_output?: Array<{ role: string; content: unknown }>;
};

function main() {
  const stdin = fs.readFileSync(0, 'utf8');
  const input = JSON.parse(stdin) as EvalInput;

  const expectedDecision = findExpectedDecision(input.expected_output);

  let candidateDecision: string | undefined;
  try {
    const parsed = JSON.parse(input.output_text ?? '');
    candidateDecision = parsed.decision;
  } catch {
    candidateDecision = undefined;
  }

  const assertions: Array<{ text: string; passed: boolean }> = [];

  if (expectedDecision === candidateDecision) {
    assertions.push({ text: `decision matches: ${expectedDecision}`, passed: true });
  } else {
    assertions.push({ text: `mismatch: expected=${expectedDecision} actual=${candidateDecision}`, passed: false });
  }

  const passed = assertions.every(a => a.passed);

  process.stdout.write(JSON.stringify({
    score: passed ? 1 : 0,
    assertions,
    reasoning: passed
      ? 'Batch runner output matches expected.'
      : 'Batch runner output did not match expected.',
  }));
}

function findExpectedDecision(messages?: Array<{ role: string; content: unknown }>) {
  if (!messages) return undefined;
  for (const msg of messages) {
    if (typeof msg.content === 'object' && msg.content !== null) {
      return (msg.content as Record<string, unknown>).decision as string;
    }
  }
  return undefined;
}

main();
```

## Structured Content

Use structured objects in `expected_output` to define expected output fields for easy validation:

```yaml
expected_output:
  - role: assistant
    content:
      decision: CLEAR
      confidence: high
      reasons: []
```

The grader extracts these fields and compares them against the parsed candidate output.

## Target Configuration

Configure the batch CLI provider in your targets file or eval file:

```yaml
# In agentv-targets.yaml or eval file
targets:
  batch_cli:
    provider: cli
    command: bun run ./scripts/batch-runner.ts --eval {EVAL_FILE} --output {OUTPUT_FILE}
    provider_batching: true
```

Key settings:

| Setting | Description |
|---------|-------------|
| `provider: cli` | Use the CLI provider |
| `provider_batching: true` | Run once for all tests instead of per-test |
| `{EVAL_FILE}` | Placeholder replaced with the eval file path |
| `{OUTPUT_FILE}` | Placeholder replaced with the JSONL output path |

## Best Practices

1. **Use unique test IDs** -- the batch runner and AgentV use `id` to route outputs to the correct grader
2. **Structured input** -- put structured data in `user.content` for the runner to extract
3. **Structured expected_output** -- define expected output as objects for easy comparison
4. **Deterministic runners** -- batch runners should produce consistent output for reliable testing
5. **Healthcheck support** -- add a `--healthcheck` flag for runner validation:
   ```typescript
   if (args.includes('--healthcheck')) {
     console.log('batch-runner: healthy');
     return;
   }
   ```
