PestPHP AI评估插件

Agent instance (with constructor dependencies)

it('evaluates a pre-configured agent', function () {
    $agent = new RefundAgent($user);

    expectAgent($agent, 'Can I return a damaged laptop?')
        ->toContain('refund')
        ->toPassJudge('Response explains the refund policy clearly');
});

You can also use Laravel's ::make() method:

it('evaluates agent created with make()', function () {
    expectAgent(RefundAgent::make(user: $user), 'Can I return a damaged laptop?')
        ->toContain('refund');
});

Installation

composer require shipfastlabs/pest-plugin-evals --dev

Publish the config (optional):

php artisan vendor:publish --tag=eval-config

Quick Start

use function ShipFastLabs\PestEval\expectAgent;

it('answers refund questions accurately', function () {
    expectAgent(RefundAgent::class, 'Can I return a damaged laptop?')
        ->toContain('refund')
        ->toContain('return')
        ->toPassJudge('Response explains the refund policy clearly')
        ->toBeRelevant(0.8);
});

Run your evals:

pest --eval

Eval tests are excluded from normal test runs automatically. Place your eval tests in tests/Evals/ — when you run pest without --eval, the plugin excludes that directory so evals never pollute your regular test suite.

pest --eval targets the tests/Evals directory. If it does not exist, it falls back to --group=eval.

Usage Examples

Configuration

// config/eval.php
return [
    'ai' => [
        'scoring' => [
            'provider' => env('EVAL_SCORING_PROVIDER', 'openai'),
            'model' => env('EVAL_SCORING_MODEL', 'gpt-4.1-mini'),
        ],
        'embedding' => [
            'provider' => env('EVAL_EMBEDDING_PROVIDER', 'openai'),
            'model' => env('EVAL_EMBEDDING_MODEL', 'text-embedding-3-small'),
        ],
    ],
];

Faked mode (fast iteration, no agent API calls)

it('eval pipeline works with faked responses', function () {
    expectAgent(
        RefundAgent::class,
        'What is your return policy?',
        fake: ['Our return policy allows returns within 30 days.'],
    )->toContain('30 days')
        ->toMatch('/\d+ days/');
});

Factuality check against reference

it('answers factually', function () {
    expectAgent(CapitalCityAgent::class, 'What is the capital of Japan?')
        ->toBeFactual(expected: 'Tokyo');
});

Custom Expectations Reference

All thresholds default to 0.7 and represent the minimum score (0.0-1.0) required to pass.

`expectAgent()` API

expectAgent(
    string|Closure|Agent $agent, // Agent class name, closure, or instance
    string $prompt,              // The input prompt
    array $fake = [],            // Fake responses (bypasses agent execution)
    array $attachments = [],     // Files to pass to the agent (Document, Image)
): Expectation

// Chain ->repeat(N) for multiple runs:
->repeat(5)                  // Run agent 5 times, all assertions checked on every output

Pest Plugin Eval

A PestPHP plugin for evaluating Laravel AI SDK agents. Build evals with LLM-as-judge, semantic similarity, and deterministic matchers — all with a native Pest expect() API.