古安事数权模式

Option B — Manual Setup (5 minutes)

1. Install

git clone https://github.com/Corbell-AI/evalmonkey
cd evalmonkey
pip install -e .

2. Configure your LLM key (used only as the evaluation judge — never for your agent)

cp .env.example .env

First time setup:

cp .env.example .env # fill in EVAL_MODEL + your LLM provider key pip install -e .

⚡️ Quick Start

1. CSV Example (`evals.csv`)

If using a CSV, just make sure you have the columns id and expected_behavior_rubric. Any other column you add (like question, topic, image_url) will be automatically gathered and sent in the JSON payload directly to your agent!

evalmonkey run-benchmark --scenario get_benefits --eval-file evals.csv

2. JSON / YAML Example (`evals.json`)

If you use JSON or YAML, you must nest the agent payload keys explicitly under an input_payload dict object:

[
  {
    "id": "onboarding_query",
    "description": "Test HR agent's ability to return the onboarding link.",
    "expected_behavior_rubric": "Must contain exactly the URL https://hr.example.com/benefits",
    "input_payload": {
      "question": "Where do I sign up for benefits?"
    }
  }
]

evalmonkey run-benchmark --scenario onboarding_query --eval-file evals.json

---

Experience 1: Local Sample Agents (Single Command Start)

Easiest Experience: Test our built-in sample agents with a single command! EvalMonkey will spawn the sample agent in the background automatically and run the benchmark. ```bash

⚠️ 3 sample(s) scored below threshold — eval assets saved.

Experience 6: One-Command End-to-End Demo (RAG App)

Run the full benchmark + chaos + eval-generation pipeline against the built-in rag_app sample agent:

```bash

Option A — Let Claude Code or Cursor set it up for you (30 seconds)

Open Claude Code, Cursor, or any AI coding assistant and paste this prompt:

Set up EvalMonkey in my project so I can benchmark my AI agent.

1. Clone https://github.com/Corbell-AI/evalmonkey into a sibling folder
2. Run: pip install -e . inside that folder
3. Copy .env.example to .env and ask me which LLM provider I want to use as the benchmark judge (OpenAI, Anthropic, Bedrock, or Ollama) — then fill in the correct key
4. Run: evalmonkey init --framework <my_framework> --name "My Agent" --port <my_port>
   Use the framework my agent is built with (crewai / langchain / openai / bedrock / autogen / ollama / strands / custom)
5. Show me the generated evalmonkey.yaml and ask me to confirm the agent URL and response path are correct
6. Run a quick smoke test: evalmonkey run-benchmark --scenario gsm8k --sample-agent rag_app --limit 2
   to confirm everything is wired up correctly
7. Then run the real benchmark against my agent: evalmonkey run-benchmark --scenario mmlu --limit 5
8. Show me the score and explain what it means

The agent will handle cloning, installing, configuring your .env, and running the first benchmark — all without you typing a single command.

---

Setting Up in Claude Desktop / Cursor

Add the following to your MCP configuration file (e.g. claude_desktop_config.json):

{
  "mcpServers": {
    "evalmonkey": {
      "command": "evalmonkey",
      "args": ["serve-mcp"]
    }
  }
}

Once connected, your AI assistant will gain the ability to list benchmarks, trigger full evaluation runs, inject chaos payload mutators, pull historical trends, and generate improvement eval assets — entirely autonomously while helping you build your agent!

OpenAI-compatible endpoint returning {"choices":[{"message":{"content":""}}]}

evalmonkey run-benchmark --scenario arc \ --target-url http://localhost:8000/v1/chat/completions \ --request-key content \ --response-path choices.0.message.content ```

🤖 MCP Server (Cursor & Claude Integration)

EvalMonkey natively ships with a Model Context Protocol (MCP) server! This allows AI IDEs (like Cursor) or external agents (like Claude Desktop) to invoke EvalMonkey tools automatically while they build your agent.

1. run_full_pipeline(scenario="gsm8k", target_url="...", chaos_profiles="client_prompt_injection,client_payload_bloat")