Model · harness89 tasks × 5 attemptseasiest → hardest across all runsScoreRun cost
OpenAIOpenAI GPT-68 of our runs
OpenAIGPT-6 Astra · highOur runfast-agent 0.10.436h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 1 attempt with a medium finding All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass88.5%394/445 · ±3.0recorded 88.5%≥$280OpenAIGPT-6.1 Sol · highOur runfast-agent 0.10.436h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 1 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 2 of 5 attempts passed error, error, error atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass85.6%381/445 · ±3.2recorded 85.6%≥$50.04OpenAIGPT-6.1 Sol · mediumOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 76% of attempts passvulnerable-secret 1 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 1 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 1 of 5 attempts passed error, error, error atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass81.8%364/445 · ±3.6recorded 81.8%≥$31.20OpenAIGPT-6 Luna · maxOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 79% of attempts passmteb-leaderboard 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtrain-fasttext 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass80.0%356/445 · ±3.5recorded 80.6%≥$7.20OpenAIGPT-6 Luna · xhighOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 79% of attempts passmteb-leaderboard 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 2 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 2 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass77.0%343/445 · ±3.7recorded 77.9%≥$4.77OpenAIGPT-6 Luna · mediumOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 79% of attempts passmteb-leaderboard 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass74.8%333/445 · ±3.8recorded 75.0%≥$3.30OpenAIGPT-6 Luna · highOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 85% of attempts passfinancial-document-processor 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 79% of attempts passmteb-leaderboard 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (2 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 66% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass73.9%329/445 · ±3.9recorded 75.5%≥$4.48OpenAIGPT-6 Luna · lowOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passcancel-async-tasks 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 66% of attempts passinstall-windows-3.11 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass44.9%200/445 · ±4.5recorded 45.6%≥$0.57
OpenAIOpenAI GPT-5.65 of our runs
OpenAIGPT-5.6 Sol · highOur runfast-agent 0.9.24Provisionalatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: no result for 1 attempt All runs: 59% of attempts passsanitize-git-repo 4 of 5 attempts passed agent timeout atif-scan: 2 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 1 of 5 attempts passed agent timeout, agent timeout atif-scan: no result for 1 attempt All runs: 6% of attempts pass87.1%388/445 · ±2.6recorded 88.3%~$216OpenAIGPT-5.6 Sol · mediumOur runfast-agent 0.9.21Provisionalatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 67% of attempts passpytorch-model-cli 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 66% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 3 of 5 attempts passed agent timeout, agent timeout atif-scan: no result for 1 attempt All runs: 59% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass81.5%363/445 · ±3.2recorded 82.0%~$169OpenAIGPT-5.6 Luna · maxOur runfast-agent 0.10.126h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: no result for 1 attempt All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 4 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 91% of attempts passcount-dataset-tokens 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: no result for 1 attempt All runs: 85% of attempts passfinancial-document-processor 2 of 5 attempts passed atif-scan: no result for 1 attempt All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 79% of attempts passmteb-leaderboard 4 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 74% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 68% of attempts passdb-wal-recovery 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 66% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 59% of attempts passsanitize-git-repo 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtrain-fasttext 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 6% of attempts pass80.9%360/445 · ±3.2recorded 82.2%~$29.54OpenAIGPT-5.6 Terra · highOur runfast-agent 0.9.15Provisionalatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 4 of 5 attempts passed agent timeout atif-scan: no result for 1 attempt All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passpytorch-model-recovery 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 4 of 5 attempts passed agent timeout atif-scan: no result for 1 attempt All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 74% of attempts passtorch-tensor-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 67% of attempts passpytorch-model-cli 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: no result for 3 attempts All runs: 44% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 4 of 5 attempts passed agent timeout atif-scan: no result for 1 attempt All runs: 6% of attempts pass76.4%340/445 · ±3.7recorded 77.7%~$134OpenAIGPT-5.6 Luna · highOur runfast-agent 0.10.466h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 79% of attempts passmteb-leaderboard 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass74.1%330/445 · ±3.6recorded 76.1%≥$8.89
OpenAIGPT-5.6 Sol · highVendor claimOpenAI · GPT-5.6 launch postNo trial data published · score only84.7%reported~$388
OpenAIGPT-5.6 Sol · mediumVendor claimOpenAI · GPT-5.6 launch postNo trial data published · score only81.8%reported~$317
OpenAIGPT-5.6 Terra · maxLeaderboardCodex 0.144.xnot scanneddistribution-search 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passfix-code-vulnerability 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: this run hasn't been scanned All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passgit-leak-recovery 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passhf-model-inference 2 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passportfolio-optimization 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passpypi-server 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passfeal-linear-cryptanalysis 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed agent timeout atif-scan: this run hasn't been scanned All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 93% of attempts passheadless-terminal 2 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 93% of attempts passqemu-alpine-ssh 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 93% of attempts passwrite-compressor 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 93% of attempts passcustom-memory-heap-crash 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 91% of attempts passlarge-scale-text-editing 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 91% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 91% of attempts passcount-dataset-tokens 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 89% of attempts passkv-store-grpc 0 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 85% of attempts passfinancial-document-processor 3 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 84% of attempts passregex-chess 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 84% of attempts passcrack-7z-hash 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 75% of attempts passfix-ocaml-gc 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 71% of attempts passquery-optimize 2 of 5 attempts passed error, agent timeout atif-scan: this run hasn't been scanned All runs: 69% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 68% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 67% of attempts passpytorch-model-cli 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 61% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: this run hasn't been scanned All runs: 59% of attempts passsanitize-git-repo 0 of 5 attempts passed agent timeout atif-scan: this run hasn't been scanned All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed error, error atif-scan: this run hasn't been scanned All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 44% of attempts passtrain-fasttext 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: this run hasn't been scanned All runs: 44% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 43% of attempts passextract-elf 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 38% of attempts passvideo-processing 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 26% of attempts passraman-fitting 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 8% of attempts passmake-doom-for-mips 1 of 5 attempts passed agent timeout atif-scan: this run hasn't been scanned All runs: 6% of attempts pass78.4%349/445 · ±3.3$421
OpenAIGPT-5.6 Terra · highVendor claimOpenAI · GPT-5.6 launch postNo trial data published · score only76.6%reported$280
xAIxAI Grok3 of our runs
xAIGrok 4.7 · mediumOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 91% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 2 attempts with a medium finding All runs: 67% of attempts passpytorch-model-cli 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 46% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 44% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 44% of attempts passprotein-assembly 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 43% of attempts passextract-elf 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 2 attempts with a medium finding All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 32% of attempts passdna-insert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass88.5%394/445 · ±2.8recorded 90.3%≥$523xAIGrok 4.6 · mediumOur runfast-agent 0.10.10Provisionalatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passsam-cell-seg 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), agent timeout atif-scan: 2 attempts with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 2 of 5 attempts passed error, error, agent timeout atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 46% of attempts passdna-assembly 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtrain-fasttext 4 of 5 attempts passed agent timeout atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed error atif-scan: 3 attempts with a medium finding All runs: 32% of attempts passdna-insert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass86.9%387/445 · ±2.9recorded 87.6%$193xAIGrok 4.6 · highOur runfast-agent 0.10.9Provisionalatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 68% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 1 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 49% of attempts passgcode-to-text 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 46% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed error, error atif-scan: 3 attempts with a medium finding All runs: 32% of attempts passdna-insert 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass85.3%380/445 · ±3.1recorded 87.1%$238
Z.aiZ.ai GLM-5.31 of our runs
Z.aiGLM-5.3-Flash · maxOur runfast-agent 0.10.116h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 99% of attempts passreshard-c4-data 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 3 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 91% of attempts passcount-dataset-tokens 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 1 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 81% of attempts passbuild-pov-ray 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 64% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 43% of attempts passextract-elf 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 38% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 26% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 6% of attempts pass82.4%367/445 · ±3.0recorded 84.4%~$77.53
DeepSeekDeepSeek V43 of our runs
DeepSeekDeepSeek V4.1 Flash · maxOur runfast-agent 0.10.246h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 88% of attempts passmcmc-sampling-stan 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 74% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 2 attempts with a medium finding All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 64% of attempts passmteb-retrieve 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 61% of attempts passextract-moves-from-video 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 2 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 46% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 44% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passprotein-assembly 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 43% of attempts passextract-elf 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 38% of attempts passvideo-processing 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 32% of attempts passdna-insert 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 6% of attempts pass84.2%375/445 · ±3.3recorded 95.0%≥$31.20DeepSeekDeepSeek V4 Flash Vision · maxOur runfast-agent 0.10.136h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 4 of 5 attempts passed error atif-scan: 3 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 3 of 5 attempts passed agent timeout atif-scan: 1 attempt with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 74% of attempts passtorch-tensor-parallelism 2 of 5 attempts passed error atif-scan: 1 attempt with a medium finding All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 67% of attempts passpytorch-model-cli 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 3 of 5 attempts passed error atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 2 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 2 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout, error, agent timeout, error atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 4 attempts with a medium finding All runs: 38% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 26% of attempts passraman-fitting 1 of 5 attempts passed error atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 1 attempt with a medium finding All runs: 25% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 6% of attempts pass81.3%362/445 · ±3.0recorded 83.1%~$53.68DeepSeekDeepSeek V4 Flash · maxOur runfast-agent 0.9.30Provisionalatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 1 of 5 attempts passed agent timeout, agent timeout, error, error atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 3 of 5 attempts passed agent timeout, agent timeout atif-scan: 3 attempts with a medium finding All runs: 92% of attempts passlargest-eigenval 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 3 attempts with a medium finding All runs: 85% of attempts passfinancial-document-processor 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passwinning-avg-corewars 4 of 5 attempts passed agent timeout atif-scan: 1 attempt with a medium finding All runs: 81% of attempts passbuild-pov-ray 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 74% of attempts passtorch-tensor-parallelism 2 of 5 attempts passed agent timeout, pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 5 attempts with a medium finding All runs: 68% of attempts passdb-wal-recovery 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), agent timeout, agent timeout atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 3 attempts with a medium finding All runs: 67% of attempts passpytorch-model-cli 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed agent timeout, agent timeout atif-scan: 3 attempts with a medium finding All runs: 66% of attempts passgpt2-codegolf 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 3 attempts with a medium finding atif-scan: no result for 1 attempt All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 2 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, error, error atif-scan: no result for 4 attempts All runs: 59% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 3 attempts with a medium finding All runs: 46% of attempts passdna-assembly 1 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: 2 attempts with a medium finding atif-scan: no result for 1 attempt All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: no result for 4 attempts All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 43% of attempts passextract-elf 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 2 attempts with a medium finding All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 32% of attempts passdna-insert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass73.0%325/445 · ±3.7recorded 75.5%$8.74
AnthropicAnthropic Claude7 of our runs
AnthropicOpus 5.5 · highOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 64% of attempts passmteb-retrieve 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 2 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass88.5%394/445 · ±3.3recorded 88.5%$160AnthropicOpus 5.5 · mediumOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed error atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass86.7%386/445 · ±3.3recorded 86.7%≥$104AnthropicSonnet 5.5 · highOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 64% of attempts passmteb-retrieve 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass84.9%378/445 · ±3.2recorded 84.9%≥$58.62AnthropicHaiku 5.5 · highOur runfast-agent 0.10.436h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 2 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed error atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass83.1%370/445 · ±3.2recorded 83.1%≥$35.95AnthropicHaiku 5.5 · mediumOur runfast-agent 0.10.436h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 89% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass82.7%368/445 · ±3.2recorded 82.7%≥$18.27AnthropicSonnet 5.5 · mediumOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 66% of attempts passinstall-windows-3.11 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass81.1%361/445 · ±3.7recorded 81.8%≥$39.63AnthropicHaiku 5.5 · lowOur runfast-agent 0.10.436h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 6% of attempts pass79.7%355/445 · ±3.3recorded 80.0%$9.80AnthropicFable 5 · xhighLeaderboardClaude Code 2.1.167atif-scan basic + imagesdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed 4 attempts ran on another model (fallback) All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed 3 attempts ran on another model (fallback) All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 81% of attempts passwinning-avg-corewars 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 2 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 2 of 5 attempts passed agent timeout 5 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 4 of 5 attempts passed 1 attempt ran on another model (fallback) All runs: 66% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 64% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 51% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed 1 attempt ran on another model (fallback) All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 4 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtrain-fasttext 2 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 43% of attempts passextract-elf 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 38% of attempts passvideo-processing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 26% of attempts passraman-fitting 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed 2 attempts ran on another model (fallback) All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout 2 attempts ran on another model (fallback) All runs: 6% of attempts pass83.8%373/445 · ±3.2$553AnthropicFable 5 · highLeaderboardTerminus 2 2.0.0atif-scan basic + imagesdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed 4 attempts ran on another model (fallback) All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 3 of 5 attempts passed agent timeout, agent timeout 5 attempts ran on another model (fallback) All runs: 93% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 89% of attempts passpytorch-model-recovery 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout 5 attempts ran on another model (fallback) All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 81% of attempts passsam-cell-seg 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 81% of attempts passwinning-avg-corewars 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 61% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 59% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 2 of 5 attempts passed agent timeout, agent timeout 5 attempts ran on another model (fallback) All runs: 44% of attempts passtrain-fasttext 1 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 43% of attempts passextract-elf 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 26% of attempts passraman-fitting 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass80.4%358/445 · ±3.5$439AnthropicOpus 4.8 · highLeaderboardClaude Code 2.1.205atif-scan basic + imagesdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passllm-inference-batching-scheduler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passqemu-alpine-ssh 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 74% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 67% of attempts passpytorch-model-cli 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding All runs: 59% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 51% of attempts passconfigure-git-webserver 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 49% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass78.8%351/445 · ±3.5$287AnthropicSonnet 5 · highLeaderboardClaude Code 2.1.205atif-scan basic + imagesdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmulti-source-data-merger 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passopenssl-selfsigned-cert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passportfolio-optimization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 97% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpypi-server 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passbuild-cython-ext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passfeal-linear-cryptanalysis 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 96% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passcaffe-cifar-10 4 of 5 attempts passed agent timeout atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passllm-inference-batching-scheduler 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passpolyglot-rust-c 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passsparql-university 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passtune-mjcf 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcompile-compcert 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 93% of attempts passqemu-alpine-ssh 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 93% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 93% of attempts passcustom-memory-heap-crash 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpath-tracing 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passschemelike-metacircular-eval 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 91% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passpassword-recovery 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 89% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 84% of attempts passcancel-async-tasks 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 81% of attempts passwinning-avg-corewars 2 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 81% of attempts passbuild-pov-ray 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 79% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passvulnerable-secret 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passfix-ocaml-gc 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) All runs: 74% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passquery-optimize 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 69% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: nothing above low priority All runs: 68% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 68% of attempts passdb-wal-recovery 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 67% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passgpt2-codegolf 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passmteb-retrieve 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passmodel-extraction-relu-logits 1 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 61% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 61% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding atif-scan: no result for 1 attempt All runs: 59% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 51% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 49% of attempts passgcode-to-text 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 44% of attempts passtrain-fasttext 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passprotein-assembly 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 43% of attempts passextract-elf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 38% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 32% of attempts passdna-insert 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 26% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 25% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 8% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 6% of attempts pass74.6%332/445 · ±3.3$288
our passcomparator passfailtimeouterrorjudge DQatif-scan: high or critical findingtask has a flagged attemptran on another modelnot scannedvendor claim, no trials