# Multi-SDK Benchmark Report

**Date:** 2026-03-15T16:05:41.721Z
**Duration:** 737.3s
**Runs:** 36 (21 success, 15 failed)
**Success Rate:** 58.3%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | fast     | n/a       | 1.9m     | $0.2700  | 100%    |
| Anthropic Claude | balanced | n/a       | 3.6m     | $0.6578  | 50%     |
| OpenAI           | premium  | n/a       | 0ms      | $0.0000  | 0%      |
| OpenAI           | balanced | n/a       | 24.3s    | $0.0000  | 33%     |
| Google Gemini    | fast     | n/a       | 48.7s    | $0.0521  | 83%     |

## Variant Rankings (by avg score)

| Rank | Variant                  | Score | Time  | Cost    | Success |
| ---- | ------------------------ | ----- | ----- | ------- | ------- |
| 1    | openai-raw-gpt54-sp      | 0.0   | 0ms   | $0.0000 | 0%      |
| 2    | claude-raw-haiku-sp      | n/a   | 1.9m  | $0.2700 | 100%    |
| 3    | claude-raw-sonnet-sp     | n/a   | 3.6m  | $0.6578 | 50%     |
| 4    | openai-sdk-codex-sp      | n/a   | 24.3s | $0.0000 | 33%     |
| 5    | google-raw-flash-sp      | n/a   | 1.3m  | $0.0791 | 83%     |
| 6    | google-raw-flash-lite-sp | n/a   | 21.4s | $0.0250 | 83%     |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-sp

### Multi-Step Survey Form (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Kanban Task Board (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### E-Commerce Product Page (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

### Real-Time Chat Interface (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku-sp
- **Worst:** google-raw-flash-lite-sp

## Failures

- **claude-raw-sonnet-sp** × survey-form: Timeout after 300000ms
- **claude-raw-sonnet-sp** × kanban-board: Timeout after 300000ms
- **claude-raw-sonnet-sp** × periodic-table: Timeout after 300000ms
- **openai-raw-gpt54-sp** × weather-card: 400 Previous response with id 'resp_091598e4a137f4d20169b6d727d0cc819292d959fde8eba842' not found.
- **openai-raw-gpt54-sp** × survey-form: 400 Previous response with id 'resp_0627318a10767f630169b6d72a792c81a2b26b5f839e3c4fa8' not found.
- **openai-raw-gpt54-sp** × kanban-board: 400 Previous response with id 'resp_07f0a962f9e60c100169b6d72d6700819d88d7a270fd0c9edf' not found.
- **openai-raw-gpt54-sp** × periodic-table: 400 Previous response with id 'resp_04fcd771e88d833f0169b6d72eb23c8197abe0314d7910982b' not found.
- **openai-raw-gpt54-sp** × product-page: 400 Previous response with id 'resp_0750b924c2be617e0169b6d72fb79881a38a04fcbd106718c0' not found.
- **openai-raw-gpt54-sp** × chat-interface: 400 Previous response with id 'resp_0b99eccc78c65fb10169b6d73088148196b9617b1b97ab2b2d' not found.
- **openai-sdk-codex-sp** × survey-form: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × kanban-board: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × product-page: OpenAI SDK single-pass: no compiled code produced
- **openai-sdk-codex-sp** × chat-interface: OpenAI SDK single-pass: no compiled code produced
- **google-raw-flash-sp** × chat-interface: Timeout after 300000ms
- **google-raw-flash-lite-sp** × weather-card: Google raw single-pass: no compiled code produced
