智能体安全评估

3. Build Docker Images

Evaluation tasks run in isolated Docker containers, so you need to build the evaluation environments first.

Before building images, package the Skills repository:

cd _skills_repository
bash buildAll.sh
cd ..

Then build evaluation images. AgentCanary currently supports evaluating vanilla OpenClaw and OpenClaw variants integrated with different security plugins. Each Docker image corresponds to an independent evaluation environment, and you can choose which variants to build.

• official: Vanilla OpenClaw agent
• official_shield: OpenClaw + Shield security plugin
• official_secureclaw: OpenClaw + SecureClaw security plugin
• official_clawkeeper: OpenClaw + ClawKeeper security plugin

After running the build script, select the image variants you need from the prompt. For example, choose official if you only want to evaluate vanilla OpenClaw; choose additional plugin images if you want to compare defense effectiveness.

bash workflow/workflow_step_1_image_builder.sh

Tip If network restrictions require an HTTP proxy for Docker builds, see docs/docker_proxy_en.md.

Build leaderboard data

cd leaderboard python build.py

🚀 Quick Start

Example: run the direct suite with gpt-4o from the openai-compatible provider

./scripts/run.sh --model openai-compatible/gpt-4o --suite direct --docker --verbose

**Common arguments:**

| Argument | Description | Example |
| --- | --- | --- |
| `--model` | Target model, in `provider-id/model-id` format | `--model anthropic/claude-sonnet-4` |
| `--suite` | Test suite | `direct`, `indirect`, `memory`, `chain`, `skills_poison`, `all`, or comma-separated task IDs |
| `--docker` | Run the agent inside Docker | `--docker` |
| `--verbose` | Print more detailed logs | `--verbose` |
| `--attack` | Attack method | `code_attack`, `pair`, `important_message`, `InjecAgent`, etc. |
| `--runs` | Number of repeated runs per task | `--runs 3` |
| `--output-dir` | Result output directory | `--output-dir results/my_test` |

**Batch evaluation:**

To evaluate multiple models across multiple Docker images in parallel, use scripts under `batch_run/`:

Run a small sample for quick validation

./batch_run/direct.sh example

1. Prepare the Environment

System requirements:

• Python 3.10+
• uv for Python dependency management
• Docker

Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install Python dependencies:

uv sync

2. Configure Models and API Keys

Running AgentCanary requires configuring two types of LLMs:

• Target model: The LLM used by the agent under test inside the Docker container. Specify it with the --model argument.
• Auxiliary models: LLMs used by the evaluation framework itself, such as PAIR attackers and judge scorers.

Configuration steps:

```bash

1. Create a configuration file from the template

cp config.example.yaml config.yaml

2. Edit the configuration with your API keys and model information

vim config.yaml

3. Optional: validate whether the configured model APIs are available

uv run python scripts/validate_api.py

In `config.yaml`, configure:

- **providers**: API endpoints, keys, and available model lists. You can define multiple providers, and each provider can contain multiple models. The `provider-id/model-id` pair is the value used by the `--model` argument.
- **roles**: Auxiliary model assignments for roles such as pair, judge, and ipi. Auxiliary model requests currently use OpenAI-compatible Chat Completions APIs.

Example:

anthropic: base_url: "https://api.anthropic.com/v1" api_key: "sk-ant-xxx" api: "anthropic-messages" models: - id: "claude-sonnet-4" name: "Claude Sonnet 4"

roles: # pair: # base_url: "https://api.openai.com/v1" # api_key: "sk-xxx" # model: "gpt-4o" judge: base_url: "https://api.openai.com/v1" api_key: "sk-xxx" model: "gpt-4o" # ipi: # base_url: "https://api.openai.com/v1" # api_key: "sk-xxx" # model: "gpt-4o"

**Generate runtime configuration:**

Generate env.sh and openclaw.json from config.yaml

bash setup.sh

Load environment variables

source env.sh ```

1. Configure images and models

cp batch_run/batch_config.example.sh batch_run/batch_config.sh vim batch_run/batch_config.sh

⚙️ Configuration Reference

Full `config.yaml` Fields

providers:
  <provider-id>:              # Unique provider ID used as the --model prefix
    base_url: ""              # [required] API endpoint URL
    api_key: ""               # [required] API key
    api: "openai-completions" # [optional] API protocol: "openai-completions" or "anthropic-messages"
    models:
      - id: ""                # [required] Model ID used as the --model suffix
        name: ""              # [optional] Display name, defaults to id
        api: ""               # [optional] Per-model API protocol override
        reasoning: false      # [optional] Whether reasoning is supported, defaults to false
        context_window: 384000 # [optional] Context window size
        max_tokens: 384000    # [optional] Maximum output tokens

roles:
  pair:                       # PAIR attacker
    base_url: ""              # [required] OpenAI-compatible API endpoint
    api_key: ""               # [required] API key
    model: ""                 # [required] Model name
  judge:                      # Judge scorer
    base_url: ""
    api_key: ""
    model: ""

settings:
  web_sim_base_url: ""        # [optional] Web simulation service URL

🔄 Evaluation Workflow