---
title: Coding Agents
description: Evaluate coding agent targets
sidebar:
  order: 3
---

Coding agent targets evaluate AI coding assistants and CLI-based agents. These targets require a `grader_target` (also accepts `judge_target` for backward compatibility) to run LLM-based graders.

## Prompt format

Agent providers receive a structured prompt document with two sections: a **preread block** listing files the agent must read, and the **user query** containing the eval input.

### File handling

When an eval test includes `type: file` inputs, agent providers do **not** receive the file content inline. Instead, they receive:

1. A preread block with `file://` URIs pointing to absolute paths on disk
2. The user query with `<file: path="...">` reference tags

The agent is expected to read the files itself using its filesystem tools.

This differs from [LLM providers](/docs/targets/llm-providers), which receive file content embedded directly in the prompt as XML:

```xml
<file path="src/example.ts">
// file content is inlined here
</file>
```

### Example prompt

Given an eval with file inputs:

```yaml
input:
  - role: user
    content:
      - type: file
        value: ./src/example.ts
      - type: text
        value: Review this code
```

The agent receives a prompt like:

```
Read all input files:
* [example.ts](file:///abs/path/src/example.ts).

If any file is missing, fail with ERROR: missing-file <filename> and stop.
Then apply system_instructions on the user query below.

[[ ## user_query ## ]]
<file: path="./src/example.ts">
Review this code
```

The preread block instructs the agent to read input files before processing the query. If a `system_prompt` is configured on the target, it is passed separately via the provider SDK (not in the prompt document).

## Claude

```yaml
targets:
  - name: claude_agent
    provider: claude
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `executable` | No | CLI binary name or path (default: `claude`). Accepts a bare name looked up on PATH (e.g. `claude-zai`) or an absolute/relative file path. |
| `cwd` | No | Working directory |
| `grader_target` | Yes | LLM target for evaluation |

## cc-mirror

[cc-mirror](https://github.com/numman-ali/cc-mirror) creates isolated Claude Code variants that route through alternative providers (Z.ai, Kimi, MiniMax, OpenRouter, etc.). The `cc-mirror` provider alias resolves to `claude-cli` and auto-discovers the binary path from `~/.cc-mirror/<variant>/variant.json`.

```yaml
targets:
  # Explicit variant with known executable
  - name: claude-zai
    provider: cc-mirror
    executable: claude-zai
    grader_target: azure-base

  # Auto-discover binary from variant.json
  - name: my-kimi
    provider: cc-mirror
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `executable` | No | CLI binary name or path. When set, used directly (skips variant.json lookup). |
| `variant` | No | Variant name (directory under `~/.cc-mirror/`). Defaults to target `name`. Used to locate `variant.json` when `executable` is not set. |
| `cwd` | No | Working directory |
| `grader_target` | Yes | LLM target for evaluation |

Setup a variant first, then reference it by name:

```bash
npx cc-mirror quick --provider zai --name claude-zai --api-key "$Z_AI_API_KEY"
```

Since `cc-mirror` resolves to `claude-cli`, all Claude target fields (model, system_prompt, timeout_seconds, etc.) are also supported.

## Codex CLI

```yaml
targets:
  - name: codex_target
    provider: codex
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `cwd` | No | Working directory |
| `grader_target` | Yes | LLM target for evaluation |

## Copilot CLI

```yaml
targets:
  - name: copilot
    provider: copilot
    model: gpt-5-mini
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `model` | No | Model to use (defaults to copilot's default) |
| `cwd` | No | Working directory |
| `grader_target` | Yes | LLM target for evaluation |

## Pi Coding Agent

```yaml
targets:
  - name: pi_target
    provider: pi-coding-agent
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `cwd` | No | Working directory |
| `grader_target` | Yes | LLM target for evaluation |

## VS Code

```yaml
targets:
  - name: vscode_dev
    provider: vscode
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `executable` | No | Path to VS Code binary. Supports `${{ ENV_VAR }}` syntax or literal paths. Defaults to `code` (or `code-insiders` for the insiders provider). |
| `grader_target` | Yes | LLM target for evaluation |

Using a custom executable path:

```yaml
targets:
  - name: vscode_dev
    provider: vscode
    executable: ${{ VSCODE_CMD }}
    grader_target: azure-base
```

## VS Code Insiders

```yaml
targets:
  - name: vscode_insiders
    provider: vscode-insiders
    grader_target: azure-base
```

Same configuration as VS Code.

## Custom CLI Agent

Evaluate any command-line agent:

```yaml
targets:
  - name: local_agent
    provider: cli
    command: 'python agent.py --prompt-file {PROMPT_FILE} --output {OUTPUT_FILE}'
    grader_target: azure-base
```

| Field | Required | Description |
|-------|----------|-------------|
| `command` | Yes | Command to run. `{PROMPT}` is inline prompt text and `{PROMPT_FILE}` is a temp file path containing the prompt. |
| `cwd` | No | Working directory |
| `grader_target` | Yes | LLM target for evaluation |

## Mock Provider

For testing the evaluation harness without calling real providers:

```yaml
targets:
  - name: mock_target
    provider: mock
```

## Known limitations

### VS Code

The VS Code provider uses a **subagent file-messaging architecture**. AgentV provisions pre-configured VS Code workspace directories (subagents), dispatches requests by writing prompt files, and the AI agent writes its response to a file. Lock files control concurrency.

- **Per-target worker limit**: VS Code evals run with 1 worker per target because the provider requires window focus to dispatch requests. When multiple targets are configured (e.g., `vscode` + `copilot`), they run concurrently — the single-worker limit only applies within each VS Code target. Subagents are provisioned automatically if needed.
- **Windows only**: VS Code is not available on Linux CI. E2E testing must be done on a Windows machine.
- **`.code-workspace` support**: When your eval uses `workspace.template` with a `.code-workspace` file, the template folders are opened in the VS Code window alongside the subagent directory.

### Copilot CLI

- **MCP OAuth token expiration**: If your copilot CLI has MCP servers configured that use OAuth authentication, **expired tokens will block eval execution**. The copilot CLI attempts to re-authenticate via a browser OAuth flow, which cannot complete in non-interactive mode and causes the eval to hang indefinitely. Before running evals, either re-authenticate your MCP servers manually (`copilot` → `/mcp`) or remove MCP servers with expired tokens. See [copilot-cli#1797](https://github.com/github/copilot-cli/issues/1797) and [copilot-cli#1491](https://github.com/github/copilot-cli/issues/1491) for upstream tracking.
- **Windows shell shim vs process spawn**: On Windows, `copilot -h` may work in PowerShell while AgentV still fails with `spawn copilot ENOENT`. Shell commands can execute `copilot.ps1`/`copilot.bat`, but AgentV launches a subprocess that expects a directly spawnable executable path. If this occurs, set an explicit target executable (for example via env var):

```yaml
targets:
  - name: copilot
    provider: copilot
    executable: ${{ COPILOT_EXE }}
    grader_target: azure-base
```

Use a native binary path for `COPILOT_EXE` (for example `copilot.exe` from `@github/copilot-win32-x64`).

### Claude Code

- **Run evals externally**: Run agentv evals from **outside** Claude Code. Running `agentv eval` with the `claude` target from within a Claude Code session can cause unintended behavior — the spawned Claude agent may interfere with the parent session.
- **`ANTHROPIC_API_KEY` overrides subscription auth**: Claude Code loads `.env` from the working directory on startup. If your `.env` contains `ANTHROPIC_API_KEY`, the spawned Claude Code process will use that API key instead of your Claude subscription (Max/Pro). If the API key has insufficient credits, evals will fail with "Credit balance is too low". To use subscription auth, remove `ANTHROPIC_API_KEY` from your `.env` file.
