# Multi-SDK Benchmark Report

**Date:** 2026-03-15T10:34:08.414Z
**Duration:** 157.7s
**Runs:** 18 (6 success, 12 failed)
**Success Rate:** 33.3%

## SDK Comparison

| SDK              | Tier | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | ---- | --------- | -------- | -------- | ------- |
| Anthropic Claude | fast | n/a       | 1.8m     | $0.2403  | 100%    |
| OpenAI           | fast | n/a       | 0ms      | $0.0000  | 0%      |
| Google Gemini    | fast | n/a       | 0ms      | $0.0000  | 0%      |

## Variant Rankings (by avg score)

| Rank | Variant      | Score | Time | Cost    | Success |
| ---- | ------------ | ----- | ---- | ------- | ------- |
| 1    | openai-mini  | 0.0   | 0ms  | $0.0000 | 0%      |
| 2    | google-flash | 0.0   | 0ms  | $0.0000 | 0%      |
| 3    | claude-haiku | n/a   | 1.8m | $0.2403 | 100%    |

## Per-Prompt Results

### Weather Card (simple)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** claude-haiku

### Multi-Step Survey Form (medium)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** claude-haiku

### Kanban Task Board (medium)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** claude-haiku

### Interactive Periodic Table (complex)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** claude-haiku

### E-Commerce Product Page (complex)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** claude-haiku

### Real-Time Chat Interface (medium)

- **Avg Score:** -1.0
- **Best:** claude-haiku
- **Worst:** claude-haiku

## Failures

- **openai-mini** × weather-card: 400 The requested model 'codex-5.3-spark' does not exist.
- **openai-mini** × survey-form: 400 The requested model 'codex-5.3-spark' does not exist.
- **openai-mini** × kanban-board: 400 The requested model 'codex-5.3-spark' does not exist.
- **openai-mini** × periodic-table: 400 The requested model 'codex-5.3-spark' does not exist.
- **openai-mini** × product-page: 400 The requested model 'codex-5.3-spark' does not exist.
- **openai-mini** × chat-interface: 400 The requested model 'codex-5.3-spark' does not exist.
- **google-flash** × weather-card: {"error":{"code":404,"message":"models/gemini-3-flash is not found for API version v1beta, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.","status":"NOT_FOUND"}}
- **google-flash** × survey-form: {"error":{"code":404,"message":"models/gemini-3-flash is not found for API version v1beta, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.","status":"NOT_FOUND"}}
- **google-flash** × kanban-board: {"error":{"code":404,"message":"models/gemini-3-flash is not found for API version v1beta, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.","status":"NOT_FOUND"}}
- **google-flash** × periodic-table: {"error":{"code":404,"message":"models/gemini-3-flash is not found for API version v1beta, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.","status":"NOT_FOUND"}}
- **google-flash** × product-page: {"error":{"code":404,"message":"models/gemini-3-flash is not found for API version v1beta, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.","status":"NOT_FOUND"}}
- **google-flash** × chat-interface: {"error":{"code":404,"message":"models/gemini-3-flash is not found for API version v1beta, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.","status":"NOT_FOUND"}}
