NASDE AI编程智能体评测工具

Prerequisites

• Python 3.12+
• Docker (default) or a cloud sandbox provider — Harbor runs agents in isolated environments
• uv — Package manager
• npm — Required for Gemini CLI (@google/gemini-cli is installed automatically by Harbor)
• Agent credentials (at least one):
• Claude Code: ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN
• OpenAI Codex: CODEX_API_KEY (API key) or codex login (ChatGPT subscription OAuth)
• Gemini CLI: GEMINI_API_KEY (API key), GOOGLE_API_KEY (Vertex AI), or gemini login (Google account OAuth)
• Evaluator CLI — the assessment evaluator spawns the claude CLI by default (or codex if [evaluation] backend = "codex"). That CLI must be installed and authenticated (OAuth subscription or API key — whichever you already use interactively)

1. Install the CLI

uv tool install nasde-toolkit --python 3.13
nasde --version

This installs the latest stable release from PyPI.

Python version: We recommend --python 3.13 (latest stable, broadest wheel availability). --python 3.12 is also supported and tested if your environment standardizes on it. Python 3.14 is not currently supported — a transitive dependency (pyiceberg via supabase) hasn't yet released wheels for cp314. The cap will be lifted once upstream wheels land.

2. Install the authoring skills for Claude Code

nasde install-skills

This copies the bundled nasde-benchmark-* skills into ~/.claude/skills/ so they're available in every Claude Code session. Use --scope project to install into the current project's .claude/skills/ instead, or --force to overwrite after a nasde upgrade.

Note: the authoring helpers are Claude Code skills. Codex and Gemini users can still run NASDE from the CLI — the skills just speed up creating benchmarks; they are not required to run them.

3. From inside your own repo, ask the agent to build a benchmark from git history

Open your own project in Claude Code and say something like:

"Create a NASDE benchmark with a single task, based on a recent piece of work from this repo — a commit, a range of commits, or a merged PR."

Start with one task. Point the skill at whatever unit of work feels self-contained in your workflow — a single commit, a range, a merged MR/PR, or an issue that was closed by a set of commits. The nasde-benchmark-from-history skill proposes a good candidate, and generates one task directory with instruction.md, a Dockerfile, test.sh, and a starter assessment_criteria.md. You review each file before it's written.

Then run it:

nasde run --all-variants -C path/to/generated-benchmark

--all-variants runs every variant the skill scaffolded, so you don't need to know their names yet. If you'd rather burn fewer tokens on the first run, pick just one with --variant NAME — you can run the others later.

Installation reference

The Quick start above uses uv tool install — recommended because it isolates nasde in its own environment and puts only the nasde command on PATH. Alternatives:

```bash

Latest unreleased changes from main (for testing PRs and dev builds)

uv tool install git+https://github.com/NoesisVision/nasde-toolkit.git --python 3.13

harbor_env = "daytona" # Optional: cloud sandbox provider (default: docker)

[docker] base_image = "ubuntu:22.04" build_commands = []

[evaluation] backend = "claude" # "claude" (default) | "codex" model = "claude-opus-4-7" dimensions_file = "assessment_dimensions.json"

Quick start (three steps)

The fastest path from zero to a working benchmark built from your own git history:

Results — four agent configurations scored against the same criteria

The insight: the same "DDD guidance" skill helps Claude a little (+3.5) and badly hurts Codex (-22). The per-dimension breakdown pinpoints where Codex regresses — domain modeling, encapsulation, extensibility — which would be invisible without this assessment. Skill optimization is agent-specific.

Inside an existing virtual environment (3.12 or 3.13)

pip install nasde-toolkit

Configuring the reviewer agent

The reviewer agent (assessment evaluator) is configurable via the [evaluation] section in nasde.toml. By default it uses claude-opus-4-7 with read-only tools (Read, Glob, Grep).

All options

When include_trajectory is enabled, the evaluator can read the agent's full execution trajectory (agent/trajectory.json) — tool calls, timestamps, token usage, errors. This enables assessment dimensions that evaluate the agent's process (efficiency, verification discipline, decision-making) alongside the final output. See examples/nasde-dev-skill for a working example with trajectory-aware dimensions.

`nasde run` options

mcp_config = "./evaluator_mcp.json" # MCP server config for evaluator

CLI cheatsheet

Most users only need nasde run — everything else is occasional. See Commands below for the full reference.

```bash

Gemini CLI variant

nasde run --variant gemini-baseline --model google/gemini-3-flash-preview -C my-benchmark

or: export OPENAI_API_KEY=sk-...

```

API key always takes priority over OAuth when both are present.

The evaluation pipeline, end to end

Stage 1 (the agent does the work in a sandbox) comes from Harbor. The optional experiment-tracking stage at the end uses Opik. NASDE is the glue that connects them and adds the reviewer stage in between — plus the CLI, the benchmark project layout, and the authoring skills (see below).

Benchmarking a Claude Code plugin (`[nasde.plugin]`)

If your task exercises a local Claude Code plugin (a directory with .claude-plugin/plugin.json, skills/, and an MCP server in .mcp.json), declare it once in task.toml — no vendored snapshot, no hand-wired Dockerfile COPY, no hand-written [environment.mcp_servers], no copying the plugin's skills into a variant:

[nasde.plugin]
path = "../../../src/plugins/my-plugin"   # dir containing .claude-plugin/plugin.json
ref = "abc1234"                           # optional git ref, same semantics as [nasde.source]
install_root = "/opt/my-plugin"           # optional, default /opt/<plugin-name>
build = "bun install --frozen-lockfile"   # optional, run at image-build time

[nasde.plugin.env]                        # optional, exported in the MCP server wrapper
CLAUDE_PLUGIN_DATA = "/opt/my-plugin-data"

One declaration ships the whole plugin into the sandbox image (at ref, via a temporary git worktree, for reproducibility), registers the plugin's own skills for the agent (whole skill dir, including references/), and wires its MCP server into the task automatically. Works with or without [nasde.source] and with or without a hand-written environment/Dockerfile. This removes the frozen-snapshot workaround entirely. See ADR-009.