# Multi-SDK Benchmark Report

**Date:** 2026-03-15T16:21:55.600Z
**Duration:** 600.0s
**Runs:** 36 (24 success, 12 failed)
**Success Rate:** 66.7%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | fast     | n/a       | 2.0m     | $0.3099  | 100%    |
| Anthropic Claude | balanced | n/a       | 6.8m     | $1.1844  | 83%     |
| OpenAI           | premium  | n/a       | 0ms      | $0.0000  | 0%      |
| OpenAI           | balanced | n/a       | 48.2s    | $0.0000  | 17%     |
| Google Gemini    | fast     | n/a       | 45.5s    | $0.0511  | 100%    |

## Variant Rankings (by avg score)

| Rank | Variant                  | Score | Time  | Cost    | Success |
| ---- | ------------------------ | ----- | ----- | ------- | ------- |
| 1    | openai-raw-gpt54-sp      | 0.0   | 0ms   | $0.0000 | 0%      |
| 2    | claude-raw-haiku-sp      | n/a   | 2.0m  | $0.3099 | 100%    |
| 3    | claude-raw-sonnet-sp     | n/a   | 6.8m  | $1.1844 | 83%     |
| 4    | openai-sdk-codex-sp      | n/a   | 48.2s | $0.0000 | 17%     |
| 5    | google-raw-flash-sp      | n/a   | 1.2m  | $0.0799 | 100%    |
| 6    | google-raw-flash-lite-sp | n/a   | 17.2s | $0.0223 | 100%    |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Multi-Step Survey Form (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Kanban Task Board (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### E-Commerce Product Page (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Real-Time Chat Interface (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

## Failures

- **claude-raw-sonnet-sp** × chat-interface: Timeout after 600000ms
- **openai-raw-gpt54-sp** × weather-card: 400 Duplicate item found with id fc_0be63870901dba470069b6da4cd39c8196bf81743597c92963. Remove duplicate items from your input and try again.
- **openai-raw-gpt54-sp** × survey-form: 400 Duplicate item found with id fc_09ffff1b327dd42c0069b6da4e4e4481a29af6dfb14ff05d35. Remove duplicate items from your input and try again.
- **openai-raw-gpt54-sp** × kanban-board: 400 Duplicate item found with id fc_034671ba1cb8efaa0069b6da4d19348192a0ed49aed59dc472. Remove duplicate items from your input and try again.
- **openai-raw-gpt54-sp** × periodic-table: 400 Duplicate item found with id fc_06b89c2b8a0663410069b6da4e016481968777b7ff4e7b7f4f. Remove duplicate items from your input and try again.
- **openai-raw-gpt54-sp** × product-page: 400 Duplicate item found with id fc_0d71bd1dac0762cb0069b6da4c9cb881a28e73bf400b2af600. Remove duplicate items from your input and try again.
- **openai-raw-gpt54-sp** × chat-interface: 400 Duplicate item found with id fc_0d6bee1c71d5f4900069b6da4cbccc819f869421f83fc709a3. Remove duplicate items from your input and try again.
- **openai-sdk-codex-sp** × survey-form: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × kanban-board: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × periodic-table: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × product-page: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × chat-interface: OpenAI SDK single-pass: no compiled code produced
