# Multi-SDK Benchmark Report

**Date:** 2026-03-16T07:51:37.965Z
**Duration:** 300.0s
**Runs:** 16 (10 success, 6 failed)
**Success Rate:** 62.5%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | balanced | n/a       | 1.8m     | $0.1989  | 63%     |

## Variant Rankings (by avg score)

| Rank | Variant                        | Score | Time  | Cost    | Success |
| ---- | ------------------------------ | ----- | ----- | ------- | ------- |
| 1    | claude-raw-single-pass-c1-0    | n/a   | 2.0m  | $0.3135 | 100%    |
| 2    | claude-raw-iterative-c1-1      | n/a   | 25.5s | $0.0722 | 50%     |
| 3    | claude-sdk-single-pass-c1-2    | n/a   | 2.1m  | $0.1783 | 100%    |
| 4    | claude-sdk-iterative-c1-3      | n/a   | 45.8s | $0.0723 | 50%     |
| 5    | claude-raw-single-pass-c4835-4 | n/a   | 1.9m  | $0.1326 | 50%     |
| 6    | claude-raw-iterative-c4835-5   | n/a   | 2.1m  | $0.1412 | 50%     |
| 7    | claude-sdk-single-pass-c4835-6 | n/a   | 2.6m  | $0.3524 | 50%     |
| 8    | claude-sdk-iterative-c4835-7   | n/a   | 2.3m  | $0.2347 | 50%     |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-raw-single-pass-c1-0
- **Worst:** claude-sdk-iterative-c4835-7

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-single-pass-c1-0
- **Worst:** claude-sdk-single-pass-c1-2

## Failures

- **claude-raw-iterative-c1-1** × periodic-table: Claude raw iterative: no compiled code produced
- **claude-sdk-iterative-c1-3** × periodic-table: Timeout after 300000ms
- **claude-raw-single-pass-c4835-4** × periodic-table: Timeout after 300000ms
- **claude-raw-iterative-c4835-5** × periodic-table: Timeout after 300000ms
- **claude-sdk-single-pass-c4835-6** × periodic-table: Timeout after 300000ms
- **claude-sdk-iterative-c4835-7** × periodic-table: Timeout after 300000ms
