Model · harness19 tasks × 5 attemptseasiest → hardest across all runsScoreRun cost
OpenAIOpenAI GPT-66 of our runs
OpenAIGPT-6.1 Sol · maxOur runfast-agent 0.10.44atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass64.2%61/95 · ±10.2recorded 64.2%subset only$95.01OpenAIGPT-6 Astra · highOur runfast-agent 0.10.43atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass62.1%59/95 · ±10.7recorded 62.1%subset only≥$223OpenAIGPT-6.1 Sol · mediumOur runfast-agent 0.10.41atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass60.0%57/95 · ±9.4recorded 60.0%subset only≥$22.42OpenAIGPT-6.1 Sol · highOur runfast-agent 0.10.41atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass58.9%56/95 · ±10.5recorded 58.9%subset only≥$32.88OpenAIGPT-6 Astra · mediumOur runfast-agent 0.10.43atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass55.7%53/95 · ±11.2recorded 55.7%subset only≥$159OpenAIGPT-6.1 Sol · lowOur runfast-agent 0.10.41atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass46.3%44/95 · ±8.9recorded 46.3%subset only$13.41OpenAIGPT-6.1 Sol · maxLeaderboardCodex 0.159.0atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass60.0%57/95 · ±10.7full run 58.1%$94.91OpenAIGPT-6 Astra · highLeaderboardCodex 0.151.0atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 0% of attempts pass58.9%56/95 · ±11.4full run 57.8%$342OpenAIGPT-6 Astra · maxLeaderboardCodex 0.151.0atif-scan basic + imagestelecom-entity-resolution 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass56.8%54/95 · ±10.4full run 58.1%$483OpenAIGPT-6 Astra · lowLeaderboardCodex 0.151.0atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass54.7%52/95 · ±10.9full run 50.6%$221OpenAIGPT-6 Astra · mediumLeaderboardCodex 0.151.0atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass54.7%52/95 · ±11.3full run 54.2%$296OpenAIGPT-6 Astra · xhighLeaderboardCodex 0.151.0atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass53.6%51/95 · ±11.2full run 57.8%$373OpenAIGPT-6 Sol · maxLeaderboardCodex 0.159.0atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass50.5%48/95 · ±9.2recorded 53.6%full run 49.3%$187OpenAIGPT-6 Luna · maxLeaderboardCodex 0.159.0atif-scan basic + imagesreviewedtelecom-entity-resolution 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 77% of attempts passsound-change-cascade 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (1 passed) All runs: 64% of attempts passmvcc-lsm-compaction 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass14.7%14/95 · ±6.1recorded 16.8%full run 16.3%$12.70
OpenAIOpenAI GPT-5.6comparators only
OpenAIGPT-5.6 Sol · maxLeaderboardCodex 0.149.1atif-scan basic + imagestelecom-entity-resolution 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 4 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 20% of attempts passvllm-deepseek-streaming 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass41.0%39/95 · ±7.6full run 37.2%$314OpenAIGPT-5.6 Luna · maxLeaderboardCodex 0.149.1atif-scan basic + imagestelecom-entity-resolution 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass27.3%26/95 · ±7.4full run 17.2%$54.88OpenAIGPT-5.6 Terra · maxLeaderboardCodex 0.149.1atif-scan basic + imagestelecom-entity-resolution 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed error atif-scan: no result for 1 attempt All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass17.8%17/95 · ±6.1full run 21.5%~$244
xAIxAI Grok2 of our runs
xAIGrok 4.7 · xhighOur runfast-agent 0.10.46atif-scan basic + imagesreviewedtelecom-entity-resolution 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 68% of attempts passhof-topology-interpenetration 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed error atif-scan: 2 attempts with a high or critical finding (0 passed) atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 53% of attempts passcad-model 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 22% of attempts passsession-window-debug 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 0% of attempts pass42.1%40/95 · ±9.3recorded 42.1%subset only$658xAIGrok 4.7 · mediumOur runfast-agent 0.10.42atif-scan basic + imagesreviewedtelecom-entity-resolution 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed error, error atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 4 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 0% of attempts pass31.5%30/95 · ±8.1recorded 31.5%subset only$519xAIGrok 4.7 · xhighLeaderboardGrok Build 1.0.34atif-scan basic + imagestelecom-entity-resolution 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 77% of attempts passsound-change-cascade 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed error, error, error atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 4 of 5 attempts passed agent timeout atif-scan: 5 attempts with a medium finding All runs: 53% of attempts passcad-model 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 49% of attempts passreact-lead-form 4 of 5 attempts passed error atif-scan: 5 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 2 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passsession-window-debug 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 0% of attempts pass36.8%35/95 · ±8.4full run 37.5%$633xAIGrok 4.6 · highLeaderboardGrok Build 1.0.5atif-scan basic + imagestelecom-entity-resolution 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 77% of attempts passsound-change-cascade 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (0 passed) 5 attempts ran on another model (fallback) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 53% of attempts passcad-model 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 49% of attempts passreact-lead-form 4 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 2 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (0 passed) 5 attempts ran on another model (fallback) atif-scan: 2 attempts with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 0% of attempts pass24.2%23/95 · ±7.7full run 20.3%~$375xAIGrok 4.5 · highLeaderboardGrok Build 1.0.5atif-scan basic + imagestelecom-entity-resolution 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 77% of attempts passsound-change-cascade 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 5 attempts with a high or critical finding (0 passed) 5 attempts ran on another model (fallback) All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed agent timeout 4 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding atif-scan: no result for 1 attempt All runs: 53% of attempts passcad-model 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 4 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 5 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 0% of attempts pass14.7%14/95 · ±6.8full run 12.4%~$201
Z.aiZ.ai GLM-5.3comparators only
Z.aiGLM-5.3 · maxLeaderboardClaude Code 2.1.207atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 2 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 2 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 0% of attempts pass43.1%41/95 · ±8.7full run 41.8%$464Z.aiGLM-5.3-Flash · noneLeaderboardClaude Code 2.1.285atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 2 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error atif-scan: 1 attempt with a medium finding All runs: 53% of attempts passcad-model 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass40.0%38/95 · ±8.4full run 35.7%$510
DeepSeekDeepSeek V41 of our runs
DeepSeekDeepSeek V4.1 Flash · maxOur runfast-agent 0.10.46atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 77% of attempts passsound-change-cascade 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 60% of attempts passvpp-loss-divergence 3 of 5 attempts passed atif-scan: 5 attempts with a high or critical finding (3 passed) All runs: 56% of attempts passphotonic-waveguide-routing 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 3 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 5 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (0 passed) atif-scan: 3 attempts with a medium finding All runs: 0% of attempts pass45.2%43/95 · ±9.5recorded 45.2%subset only≥$25.48
AnthropicAnthropic Claude4 of our runs
AnthropicOpus 5.5 · highOur runfast-agent 0.10.42atif-scan basic + imagesreviewedtelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 3 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 3 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass56.8%54/95 · ±9.3recorded 56.8%subset only$182AnthropicSonnet 5.5 · mediumOur runfast-agent 0.10.42atif-scan basic + imagesreviewedtelecom-entity-resolution 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass28.4%27/95 · ±8.0recorded 28.4%subset only≥$50.05AnthropicHaiku 5.5 · highOur runfast-agent 0.10.43atif-scan basic + imagesreviewedtelecom-entity-resolution 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass21.0%20/95 · ±6.9recorded 21.0%subset only≥$85.75AnthropicHaiku 5.5 · mediumOur runfast-agent 0.10.43atif-scan basic + imagesreviewedtelecom-entity-resolution 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass9.4%9/95 · ±4.4recorded 9.4%subset only$36.01AnthropicOpus 5.5 · maxLeaderboardClaude Code 2.1.284atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 3 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passproduction-planning 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 0% of attempts pass63.1%60/95 · ±9.7full run 64.8%$949AnthropicSonnet 5.5 · maxLeaderboardClaude Code 2.1.284atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 5 of 5 attempts passed 1 attempt ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed 1 attempt ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 5 of 5 attempts passed 1 attempt ran on another model (fallback) atif-scan: 4 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 25% of attempts passproduction-planning 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed error atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 2 attempts with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass60.0%57/95 · ±10.2full run 61.8%$1,740AnthropicFable 5.1 · xhighLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 2 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed 2 attempts ran on another model (fallback) All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass58.9%56/95 · ±9.0full run 57.8%$820AnthropicOpus 5 · maxLeaderboardClaude Code 2.1.231atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 5 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 0% of attempts pass57.8%55/95 · ±9.3full run 51.8%$1,139AnthropicOpus 5 · xhighLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass56.8%54/95 · ±10.2full run 53.9%$951AnthropicFable 5.1 · highLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 4 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass53.6%51/95 · ±9.7full run 54.5%$528AnthropicFable 5.1 · maxLeaderboardClaude Code 2.1.257atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed 3 attempts ran on another model (fallback) All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 4 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 2 of 5 attempts passed error 2 attempts ran on another model (fallback) atif-scan: 2 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass53.6%51/95 · ±9.2full run 57.8%$1,072AnthropicFable 5.1 · mediumLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 4 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 2 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (0 passed) atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass53.6%51/95 · ±9.9full run 53.9%$368AnthropicOpus 5 · highLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass49.4%47/95 · ±8.7full run 50.3%$752AnthropicFable 5 · maxLeaderboardClaude Code 2.1.231atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed 1 attempt ran on another model (fallback) All runs: 78% of attempts passfin-saccr-rwa 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 2 of 5 attempts passed 1 attempt ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 4 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 2 of 5 attempts passed 3 attempts ran on another model (fallback) All runs: 27% of attempts passvba-userform-port 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass47.3%45/95 · ±9.4full run 44.5%$1,137AnthropicOpus 5 · mediumLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 3 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass40.0%38/95 · ±7.2full run 44.8%$516AnthropicFable 5.1 · lowLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass33.6%32/95 · ±8.5full run 43.3%$303AnthropicOpus 5 · lowLeaderboardClaude Code 2.1.273atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass33.6%32/95 · ±7.7full run 34.8%$328AnthropicOpus 4.8 · maxLeaderboardClaude Code 2.1.231atif-scan basic + imagestelecom-entity-resolution 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass18.9%18/95 · ±6.4full run 23.6%$1,210AnthropicSonnet 5 · maxLeaderboardClaude Code 2.1.231atif-scan basic + imagestelecom-entity-resolution 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 78% of attempts passfin-saccr-rwa 2 of 5 attempts passed error atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 1 of 5 attempts passed error atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 1 of 5 attempts passed error, error atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 1 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed error atif-scan: 1 attempt with a medium finding atif-scan: no result for 1 attempt All runs: 0% of attempts pass13.6%13/95 · ±4.6full run 12.4%~$1,447
GoogleGoogle Geminicomparators only
GoogleGemini 3.8 Flash · highLeaderboardmini-SWE-agent 2.4.6atif-scan basic + imagestelecom-entity-resolution 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 78% of attempts passfin-saccr-rwa 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 77% of attempts passsound-change-cascade 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 3 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 22% of attempts passsession-window-debug 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 3 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 2 attempts with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 0% of attempts pass26.3%25/95 · ±6.8full run 19.0%$292GoogleGemini 3.7 Flash · highLeaderboardmini-SWE-agent 2.4.6atif-scan basic + imagestelecom-entity-resolution 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 42% of attempts passatrx-vep-crispr 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 22% of attempts passgsea-proteomics 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 2 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 0% of attempts pass16.8%16/95 · ±5.4full run 11.2%$149
AlibabaAlibaba Qwencomparators only
AlibabaQwen3.8-Max-0902 · maxLeaderboardClaude Code 2.1.285atif-scan basic + imagestelecom-entity-resolution 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 78% of attempts passfin-saccr-rwa 3 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 77% of attempts passsound-change-cascade 2 of 5 attempts passed agent timeout atif-scan: 5 attempts with a medium finding All runs: 68% of attempts passhof-topology-interpenetration 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed error, error, error, error, error atif-scan: 2 attempts with a medium finding All runs: 53% of attempts passcad-model 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 49% of attempts passreact-lead-form 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passatrx-vep-crispr 3 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 36% of attempts passhtml-js-filter 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 27% of attempts passvba-userform-port 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 25% of attempts passproduction-planning 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 20% of attempts passvllm-deepseek-streaming 0 of 5 attempts passed error, error atif-scan: 5 attempts with a high or critical finding (0 passed) All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass30.5%29/95 · ±6.9full run 26.9%~$769
MetaMeta Musecomparators only
MetaMuse Spark 1.3 · xhighLeaderboardMuse Code 1.4.1atif-scan basic + imagestelecom-entity-resolution 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 78% of attempts passfin-saccr-rwa 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 77% of attempts passsound-change-cascade 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passhof-topology-interpenetration 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmvcc-lsm-compaction 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 60% of attempts passvpp-loss-divergence 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passphotonic-waveguide-routing 0 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 53% of attempts passcad-model 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passsatb-audio-transcription 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passreact-lead-form 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passwal-recovery-ordering 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passatrx-vep-crispr 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 36% of attempts passhtml-js-filter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 27% of attempts passvba-userform-port 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 25% of attempts passproduction-planning 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passsession-window-debug 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 22% of attempts passgsea-proteomics 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 20% of attempts passvllm-deepseek-streaming 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 5% of attempts passcargo-flight-dispatch 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 0% of attempts pass16.8%16/95 · ±6.2full run 14.5%no cost data
our passcomparator passfailtimeouterrorjudge DQatif-scan: high or critical findingtask has a flagged attemptran on another model