# Multi-SDK Benchmark Report

**Date:** 2026-03-18T04:56:53.166Z
**Duration:** 900.1s
**Runs:** 72 (62 success, 10 failed)
**Success Rate:** 86.1%

## SDK Comparison

| SDK              | Tier     | Avg Score | Avg Time | Avg Cost | Success |
| ---------------- | -------- | --------- | -------- | -------- | ------- |
| OpenAI           | balanced | 84.0      | 23.1s    | $0.0704  | 100%    |
| Google Gemini    | balanced | 78.4      | 2.7m     | $0.4483  | 100%    |
| Anthropic Claude | balanced | 80.0      | 4.2m     | $0.8440  | 58%     |

## Variant Rankings (by avg score)

| Rank | Variant             | Score | Time  | Cost    | Success |
| ---- | ------------------- | ----- | ----- | ------- | ------- |
| 1    | openai-sdk-c0437-3  | 86.4  | 20.4s | $0.0000 | 100%    |
| 2    | openai-raw-c0437-2  | 85.1  | 22.2s | $0.1860 | 100%    |
| 3    | openai-raw-c1-0     | 84.1  | 22.5s | $0.0958 | 100%    |
| 4    | google-sdk-c1-5     | 82.9  | 1.8m  | $0.2446 | 100%    |
| 5    | claude-raw-c4835-10 | 80.7  | 8.1m  | $1.2960 | 50%     |
| 6    | openai-sdk-c1-1     | 80.5  | 27.3s | $0.0000 | 100%    |
| 7    | claude-raw-c1-8     | 79.5  | 2.3m  | $0.7681 | 83%     |
| 8    | google-sdk-c3975-7  | 79.4  | 2.3m  | $0.3530 | 100%    |
| 9    | google-raw-c3975-6  | 77.1  | 4.0m  | $0.7082 | 100%    |
| 10   | google-raw-c1-4     | 74.3  | 2.7m  | $0.4876 | 100%    |
| 11   | claude-sdk-c1-9     | n/a   | 40.9s | $0.1191 | 17%     |
| 12   | claude-sdk-c4835-11 | n/a   | 4.6m  | $0.7936 | 83%     |

## Per-Commit Results

### Weather Card (simple)

- **Avg Score:** 83.5
- **Best:** openai-raw-c0437-2
- **Worst:** google-sdk-c3975-7

### Multi-Step Survey Form (medium)

- **Avg Score:** 80.1
- **Best:** openai-sdk-c1-1
- **Worst:** google-raw-c3975-6

### Interactive Periodic Table (complex)

- **Avg Score:** 79.9
- **Best:** google-sdk-c1-5
- **Worst:** google-raw-c1-4

### Chat Interface (medium)

- **Avg Score:** 80.5
- **Best:** openai-raw-c1-0
- **Worst:** google-sdk-c1-5

### Stock Ticker Dashboard (simple)

- **Avg Score:** 85.0
- **Best:** openai-raw-c1-0
- **Worst:** google-raw-c3975-6

### Onboarding Wizard (medium)

- **Avg Score:** 77.7
- **Best:** openai-sdk-c0437-3
- **Worst:** google-raw-c1-4

## Failures

- **claude-raw-c1-8** × survey-form: Claude raw iterative: no compiled code produced (9 turns, 587560 tokens)
  Tool calls (8): self_check(ERR) → compile_component(ERR) → validate_component(OK) → compile_component(ERR) → compile_component(ERR) → compile_component(ERR) → validate_component(OK) → compile_component(ERR)
  Last error: {
  "success": false,
  "phase": "self_check",
  "error": "2 issue(s) found — fix these before recompiling:\n\n[accessib
- **claude-sdk-c1-9** × survey-form: Claude SDK iterative: no compiled code produced (153.5s elapsed, 20 turns)
  sourceCode captured: true, meta captured: false
  sourceCode snippet: import React, { useState } from 'react';
  import {
  Container,
  Card,
  Stack,
  Text,
  Heading,
  Button,
  Input,
  TextArea,
  RadioGroup,
  Chec...
- **claude-sdk-c1-9** × periodic-table: Claude SDK iterative: no compiled code produced (159.9s elapsed, 23 turns)
  sourceCode captured: false, meta captured: false
- **claude-sdk-c1-9** × chat-interface: Claude SDK iterative: no compiled code produced (324.7s elapsed, 102 turns)
  sourceCode captured: true, meta captured: false
  sourceCode snippet: import React, { useEffect, useRef, useState } from 'react';
  import {
  Card,
  Stack,
  Text,
  Input,
  Button,
  Badge,
  Heading,
  Box,
  } from '@g...
- **claude-sdk-c1-9** × stock-ticker: Claude SDK iterative: no compiled code produced (200.7s elapsed, 34 turns)
  sourceCode captured: true, meta captured: false
  sourceCode snippet: import React, { useState, useEffect } from 'react';
  import { Container, Stack, Box, Text, Heading, Badge, Divider } from '@ggui-ai/design/primitives';...
- **claude-sdk-c1-9** × onboarding-wizard: Claude SDK iterative: no compiled code produced (219.0s elapsed, 26 turns)
  sourceCode captured: false, meta captured: false
- **claude-raw-c4835-10** × survey-form: Timeout after 900000ms
- **claude-raw-c4835-10** × stock-ticker: Timeout after 900000ms
- **claude-raw-c4835-10** × onboarding-wizard: Timeout after 900000ms
- **claude-sdk-c4835-11** × onboarding-wizard: Timeout after 900000ms
