---
title: Studio
description: Visual dashboard for reviewing evaluation results
sidebar:
  order: 6
---

import { Image } from 'astro:assets';
import studioRuns from '../../../../assets/screenshots/studio-runs.png';
import studioRunDetail from '../../../../assets/screenshots/studio-run-detail.png';
import studioExperiments from '../../../../assets/screenshots/studio-experiments.png';
import studioProjects from '../../../../assets/screenshots/studio-projects.png';
import studioProjectsMulti from '../../../../assets/screenshots/studio-projects-multi.png';
import studioCompareAggregated from '../../../../assets/screenshots/studio-compare-aggregated.png';
import studioComparePerRun from '../../../../assets/screenshots/studio-compare-per-run.png';
import studioCompareSideBySide from '../../../../assets/screenshots/studio-compare-side-by-side.png';
import studioRunsBench from '../../../../assets/screenshots/studio-runs-bench.png';
import studioAnalyticsAggregated from '../../../../assets/screenshots/studio-analytics-aggregated.png';
import studioAnalyticsCharts from '../../../../assets/screenshots/studio-analytics-charts.png';
import studioAnalyticsTrend from '../../../../assets/screenshots/studio-analytics-trend.png';

The `studio` command launches a web-based dashboard for browsing evaluation runs, inspecting individual test results, and reviewing scores. It shows both local runs and runs synced from a remote results repository.

<Image src={studioRunsBench} alt="AgentV Studio showing evaluation runs with pass rates, targets, and experiment names" />

## Usage

```bash
agentv studio
```

Studio auto-discovers run workspaces from `.agentv/results/runs/` in the current directory and opens at `http://localhost:3117`.

You can also point it at a specific run workspace or `index.jsonl` manifest:

```bash
agentv studio .agentv/results/runs/2026-03-30T11-45-56-989Z/index.jsonl
# or
agentv studio .agentv/results/runs/2026-03-30T11-45-56-989Z
```

## Options

| Option | Description |
|--------|-------------|
| `--port`, `-p` | Port to listen on (flag > `PORT` env var > 3117) |
| `--dir`, `-d` | Working directory (default: current directory) |
| `--multi` | Launch in multi-project dashboard mode (deprecated; use auto-detect or `--single`) |
| `--single` | Force single-project dashboard mode |
| `--add <path>` | Register a project by path |
| `--remove <id>` | Unregister a project by ID |

## Features

- **Recent Runs** — table of all evaluation runs with source badge (`local` / `remote`), target, experiment, timestamp, test count, pass rate, and mean score
- **Experiments** — group and compare runs by experiment name
- **Targets** — group runs by target (model/agent)
- **Run Detail** — drill into a run to see per-test results, scores, and grader output
- **Human Review** — add feedback annotations to individual test results
- **Analytics** — two modes: an aggregated experiment × target matrix, and a per-run view for selecting individual runs to compare side-by-side with optional retroactive tags. Includes a collapsible charts section with baseline comparison analytics
- **Remote Results** — sync and browse runs pushed from other machines or CI (see [Remote Results](#remote-results))

## Run Detail

Click any run to see a breakdown by suite, per-test scores, target, duration, and cost. The source label (`local` or `remote`) tells you where the run came from.

<Image src={studioRunDetail} alt="AgentV Studio run detail showing 100% pass rate across 5 tests with scores and duration" />

## Experiments

The Experiments tab groups runs by experiment name so you can compare the impact of changes — for example, `with_skills` vs `without_skills`.

<Image src={studioExperiments} alt="AgentV Studio experiments tab comparing with_skills (100%) vs without_skills (60%) pass rates" />

## Analytics

The **Analytics** tab has two modes: **Aggregated** for the classic experiment × target matrix, and **Per run** for selecting individual runs and pitting them side-by-side. Toggle between them from the mode switch on the right of the masthead.

<Image src={studioCompareSideBySide} alt="AgentV Studio side-by-side comparison of two runs tagged improved-prompt and baseline, with per-test pass rates" />

### Aggregated matrix

The default view shows a cross-experiment, cross-target performance matrix. Numbers are colour-coded by pass rate — green (80%+), amber (50–80%), red (below 50%) — and each cell shows `passed/total` and the mean score. Click any cell to expand the per-test-case breakdown.

<Image src={studioAnalyticsAggregated} alt="AgentV Studio Analytics tab showing aggregated experiment × target matrix with pass rates for baseline, optimized-prompt, and with-rag across claude-sonnet, gemini-pro, and gpt-4o" />

Run the same eval against multiple providers or experiment variants, then open the Analytics tab:

```bash
agentv eval my.EVAL.yaml --target azure --experiment baseline
agentv eval my.EVAL.yaml --target azure --experiment with-caching
agentv eval my.EVAL.yaml --target gemini --experiment baseline
agentv eval my.EVAL.yaml --target gemini --experiment with-caching
agentv studio  # Analytics tab shows 2x2 matrix
```

### Per-run comparison

Running the same `(experiment, target)` twice no longer collapses into a single cell. Switch to **Per run** mode to see every run as its own row, select two or more, and compare them head-to-head.

<Image src={studioComparePerRun} alt="AgentV Studio per-run compare mode with a filter-by-tag chip row and individual runs listing timestamp, tags, experiment, target, and pass rate; experiment-prefixed runs surface the experiment name under the timestamp" />

Use per-run mode when you want to:

- Compare back-to-back runs of the same agent + eval after a prompt or parameter tweak
- Pit a fresh run against a tagged baseline without touching the eval YAML
- Debug flakiness by inspecting two identical-configuration runs side-by-side

Select 2+ rows with the checkboxes and click the sticky **Compare N** action to open the side-by-side view. Column headers show the run's timestamp, with any assigned tags as chips below it. The per-test breakdown reuses the same scoring and colour tones as the aggregated matrix.

### Retroactive tags

Click any row's **Tags** cell to tag a run after the fact. Each run can carry multiple free-form tags (max 20, up to 60 characters each); tags are stored in a `tags.json` sidecar next to `index.jsonl` in the run workspace, so they're mutable, non-destructive, and won't touch your eval YAML or run manifest. The chip editor supports Enter/comma to commit a new tag, Backspace to remove the last chip, and **Clear all** to remove every tag (deletes the sidecar). Remote runs are read-only.

Use tags to annotate ad-hoc variants, experiment cross-cuts, or status flags you didn't plan for up front — `baseline`, `v2-prompt`, `slow`, `after-retry-fix`, `regression`, etc. Unlike `experiment` — which groups runs and is baked into the JSONL at eval-run time — tags are mutable, multi-valued, and never touch the original run data.

### Filtering by tag

Once runs are tagged, a chip row appears above the compare view listing every distinct tag with a usage count. Click a chip to narrow both the aggregated matrix and the per-run table to runs carrying at least one of the selected tags (OR semantics — clicking a second chip widens the set). A **Clear** link resets the filter, and filter selections persist as you switch between Aggregated and Per-run modes.

The same filter is available to API consumers via `GET /api/compare?tags=baseline,v2-prompt`, which returns only the cells and runs whose tags intersect the query.

### Analytics charts

Below the aggregated matrix, a collapsible **Analytics** section provides visual charts for deeper comparison. Select a **baseline target** from the dropdown to compute deltas and normalized gain metrics against that target.

<Image src={studioAnalyticsCharts} alt="AgentV Studio analytics charts showing normalized gain bar chart with baseline selector and score distribution histogram" />

The section includes the following visualizations:

- **Normalized Gain (g)** — horizontal bar chart showing how much of the remaining headroom each experiment × target captured relative to the baseline. Bars are colour-coded green (positive gain), red (regression), or grey (null / no headroom). See [Normalized Gain](/docs/tools/compare#normalized-gain-g) for the formula.
- **Tag × Target Heatmap** — pass-rate grid across tags and targets, colour-coded by performance (emerald for high, amber for medium, red for low).
- **Negative Delta Table** — filtered list of experiment × target pairs that scored worse than the baseline, sorted by largest regression.
- **Score Distribution** — histogram showing the variance of scores across all test cases, binned by 10% intervals.
- **Score Trend Over Time** — line chart plotting mean score per target across runs over time, with a colour-coded legend for each target.

<Image src={studioAnalyticsTrend} alt="AgentV Studio analytics showing score distribution histogram and score trend over time line chart with multi-target legend" />

The baseline comparison is also available via the API: `GET /api/compare?baseline=<target>` adds `delta` and `normalized_gain` fields to each non-baseline cell in the response.

## Projects Dashboard

By default, Studio shows results for the current directory. Register multiple project repos to view them from a single dashboard.

### Registering Projects

Register project repos one at a time:

```bash
agentv studio --add /path/to/my-evals
agentv studio --add /path/to/other-evals
```

Each path must contain a `.agentv/` directory. Registered projects are stored in `~/.agentv/projects.yaml`.

To register a remote repo and keep it synced automatically, add a `source` block to the entry in `~/.agentv/projects.yaml`:

```yaml
projects:
  - id: my-evals
    name: My Evals
    path: /srv/agentv/my-evals
    source:
      url: https://github.com/example/my-evals
      ref: main
```

On each Studio startup, AgentV clones the repo if the path is empty (`git clone --depth 1`) or pulls the latest if a clone already exists (`git pull --ff-only`). You can also trigger a sync manually from the Studio UI's **Sync** button.

### Runtime behavior: no restart needed

`projects.yaml` is the single source of truth. Studio re-reads it on every `/api/projects` request (which the UI polls every ~10 s), so any of these changes appear live without restarting `agentv serve`:

- Adding via the UI's **Add Project** form or `POST /api/projects`.
- Removing via the UI's **Remove** button or `DELETE /api/projects/:id`.
- Editing `~/.agentv/projects.yaml` directly.
- Mounting the file via a Kubernetes ConfigMap — GitOps the ConfigMap and Studio reflects it within the next poll.

This satisfies the 24/7-Studio use case: the server stays up; projects come and go through config edits or API calls.

### Launching the Dashboard

Studio auto-detects the mode based on how many projects are registered:

- `0` or `1` registered: single-project view
- `2+` registered: Projects dashboard

```bash
agentv studio          # auto-detects
agentv studio --single # force single-project view
```

The landing page shows a card for each project with run count, pass rate, and last run time.

<Image src={studioProjectsMulti} alt="AgentV Studio projects dashboard showing project cards with pass rates" />

### Removing a Project

Unregister by its ID:

```bash
agentv studio --remove my-evals
```

IDs are derived from the directory name (e.g., `/home/user/repos/my-evals` becomes `my-evals`).

## Remote Results

Studio can display runs pushed to a remote git repository by other machines or CI — alongside your local runs. Each run in the list carries a source badge: **local** (green) or **remote** (amber).

### Configuration

Add a `results` block to `.agentv/config.yaml`:

```yaml
results:
  repo: EntityProcess/agentv-evals   # GitHub repo (owner/repo or full URL)
  path: runs                         # Directory within the repo
  auto_push: true                    # Push directly to base branch after every eval run
```

With `auto_push: true`, every `agentv eval` or `agentv pipeline bench` pushes results directly to the configured repo's base branch (e.g., `main`). Results appear immediately in Studio without requiring PR merges.

Each run writes to a unique timestamped directory, so concurrent pushes from multiple machines are safe — non-fast-forward conflicts are resolved automatically via rebase retry.

### Authentication

Uses `gh` CLI and `git` credentials already configured on the machine. If authentication is missing, AgentV warns and skips the export — the eval run itself is never blocked.

### Syncing in Studio

Once configured, Studio fetches remote runs on load. Use the **Sync Remote Results** button in the source toolbar to pull the latest. The toolbar also shows when results were last synced and the configured repo.

Use the **All Sources / Local Only / Remote Only** filter to narrow the run list by origin.
