# Multi-SDK Benchmark Report

**Date:** 2026-03-15T10:49:39.758Z
**Duration:** 285.1s
**Runs:** 18 (13 success, 5 failed)
**Success Rate:** 72.2%

## SDK Comparison

| SDK              | Tier | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | ---- | --------- | -------- | -------- | ------- |
| Anthropic Claude | fast | n/a       | 1.6m     | $0.2283  | 100%    |
| OpenAI           | fast | n/a       | 1.4m     | $0.0000  | 17%     |
| Google Gemini    | fast | n/a       | 1.2m     | $0.0776  | 100%    |

## Variant Rankings (by avg score)

| Rank | Variant      | Score | Time | Cost    | Success |
| ---- | ------------ | ----- | ---- | ------- | ------- |
| 1    | claude-haiku | n/a   | 1.6m | $0.2283 | 100%    |
| 2    | openai-mini  | n/a   | 1.4m | $0.0000 | 17%     |
| 3    | google-flash | n/a   | 1.2m | $0.0776 | 100%    |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** google-flash

### Multi-Step Survey Form (medium)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** google-flash

### Kanban Task Board (medium)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** google-flash

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** google-flash

### E-Commerce Product Page (complex)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** google-flash

### Real-Time Chat Interface (medium)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** google-flash

## Failures

- **openai-mini** × survey-form: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-mini** × kanban-board: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-mini** × periodic-table: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-mini** × product-page: OpenAI SDK adapter: no compiled code produced after all turns
- **openai-mini** × chat-interface: OpenAI SDK adapter: no compiled code produced after all turns
