# Multi-SDK Benchmark Report

**Date:** 2026-03-15T11:19:59.214Z
**Duration:** 663.9s
**Runs:** 36 (25 success, 11 failed)
**Success Rate:** 69.4%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| Anthropic Claude | fast     | n/a       | 1.9m     | $0.2514  | 100%    |
| Anthropic Claude | balanced | n/a       | 4.3m     | $0.7228  | 100%    |
| OpenAI           | balanced | n/a       | 1.3m     | $0.0000  | 8%      |
| Google Gemini    | fast     | n/a       | 49.7s    | $0.0551  | 100%    |

## Variant Rankings (by avg score)

| Rank | Variant               | Score | Time  | Cost    | Success |
| ---- | --------------------- | ----- | ----- | ------- | ------- |
| 1    | openai-raw-codex      | 0.0   | 0ms   | $0.0000 | 0%      |
| 2    | claude-raw-haiku      | n/a   | 1.9m  | $0.2514 | 100%    |
| 3    | claude-raw-sonnet     | n/a   | 4.3m  | $0.7228 | 100%    |
| 4    | openai-sdk-codex      | n/a   | 1.3m  | $0.0000 | 17%     |
| 5    | google-raw-flash      | n/a   | 1.3m  | $0.0848 | 100%    |
| 6    | google-raw-flash-lite | n/a   | 20.7s | $0.0254 | 100%    |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku
- **Worst:** google-raw-flash-lite

### Multi-Step Survey Form (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku
- **Worst:** google-raw-flash-lite

### Kanban Task Board (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku
- **Worst:** google-raw-flash-lite

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku
- **Worst:** google-raw-flash-lite

### E-Commerce Product Page (complex)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku
- **Worst:** google-raw-flash-lite

### Real-Time Chat Interface (medium)

- **Avg Score:** -1.0
- **Best:** claude-raw-haiku
- **Worst:** google-raw-flash-lite

## Failures

- **openai-raw-codex** × weather-card: 404 This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
- **openai-raw-codex** × survey-form: 404 This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
- **openai-raw-codex** × kanban-board: 404 This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
- **openai-raw-codex** × periodic-table: 404 This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
- **openai-raw-codex** × product-page: 404 This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
- **openai-raw-codex** × chat-interface: 404 This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?
- **openai-sdk-codex** × survey-form: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-sdk-codex** × kanban-board: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-sdk-codex** × periodic-table: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-sdk-codex** × product-page: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-sdk-codex** × chat-interface: OpenAI SDK adapter: no compiled code produced after all turns
