Use case provisional data
Best AI models for code: benchmark comparison
Fixing real bugs and writing code that passes tests. SWE-bench Verified is the most used yardstick, but the agent setup changes results a lot.
Direct answer: On SWE-bench Verified, the top score among the models on this page is Claude Opus 4.7, with 83.5%, per Epoch AI.
How to read this table
- Each row is a model. ClickTap the name to open the full page, with sources and version history. ClickTap a header to sort.
- Benchmark scores are percentages, and higher is better. I always compare the same metric, from the same source, in the latest version we checked. In this section, in the 90% to 100% range, I treat a gap of 1 or 2 points as possible measurement noise.
- The "?" next to each column explains what it measures and when it matters. "No data" means there is no measurement in the open sources this section uses. "Provisional data" marks a field derived by a rule, such as use case.
- Input and output prices are the OpenRouter list price, in US dollars per 1 million tokens, and the two can differ a lot. The "to check" badge marks an odd value in the source, which is left out of recommendations. Usage is the token share in OpenRouter's public ranking, for the latest day published by that source. Downloads are the official Hugging Face repository's 30-day window. Epoch's GPQA Diamond and OpenRouter's GPQA Diamond measure the same test, but they can differ.
| ?what it is | ? | ? | ? | ? | ? | ? | ? | ? | ? | |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7Anthropic | closed | US$ 5.00 | US$ 25.00 | 1,000,000 | 90.2% | 83.5% | 77.8% | 51.7% | no data | no data |
| GPT-5.5OpenAI | closed | US$ 5.00 | US$ 30.00 | 1,050,000 | 94% | 80.6% | 74.7% | 63% | no data | no data |
| Gemini 3.5 FlashGoogle | closed | US$ 1.50 | US$ 9.00 | 1,048,576 | 92.8% | 79.3% | 73% | 66.2% | no data | no data |
| GLM-5.2Zhipu AI (Z.ai) | open | US$ 0.007 to check | US$ 6.80 | 1,048,576 | 91.9% | 78.7% | 76.7% | 34.2% | 0.43% | no data |
| Claude Opus 4.6Anthropic | closed | US$ 5.00 | US$ 25.00 | 1,000,000 | 90.5% | 78.7% | 75.8% | 47% | no data | no data |
| DeepSeek V4 ProDeepSeek | open | US$ 0.96 | US$ 1.91 | 1,048,576 | 91.7% | 77.6% | 78.6% | 52.9% | 0.66% | no data |
| Qwen3.7 MaxAlibaba | closed | US$ 1.48 | US$ 4.42 | 1,000,000 | 90.9% | 77.3% | 72% | 55.8% | no data | no data |
| GPT-5.4OpenAI | closed | US$ 2.50 | US$ 15.00 | 1,050,000 | 93.3% | 76.9% | 75% | 45.1% | no data | no data |
| Kimi K2.6Moonshot AI | open | US$ 0.65 | US$ 3.41 | 262,144 | 90.8% | 76.7% | 74.9% | 34.9% | no data | no data |
| Claude Opus 4.5Anthropic | closed | US$ 5.00 | US$ 25.00 | 200,000 | 86% | 76.7% | 76.6% | 45.7% | no data | no data |
| Qwen 3.6 Max (Preview)Alibaba | undisclosed | no data | no data | no data | 87.4% | 76.7% | no data | 52% | no data | no data |
| Gemini 3.1 ProGoogle | closed | US$ 2.00 | US$ 12.00 | 1,048,576 | 94.4% | 75.6% | 77% | 73.5% | no data | no data |
| Gemini 3 FlashGoogle | closed | no data | no data | no data | 89.4% | 75.4% | no data | 66.8% | no data | no data |
| Claude Sonnet 4.6Anthropic | closed | US$ 3.00 | US$ 15.00 | 1,000,000 | 87.4% | 75.2% | 74.8% | 35.5% | no data | no data |
| GPT-5.3-CodexOpenAI | closed | US$ 1.75 | US$ 14.00 | 400,000 | no data | 74.8% | 75.3% | no data | no data | no data |
| GLM 5.1Zhipu AI (Z.ai) | open | US$ 1.40 | US$ 4.40 | 204,800 | 89.9% | 74.2% | 75.8% | 34% | no data | 867,145 |
| GPT-5.2OpenAI | closed | US$ 1.75 | US$ 14.00 | 400,000 | 91.4% | 73.8% | no data | 37.1% | no data | no data |
| Kimi K2.5Moonshot AI | open | US$ 0.53 | US$ 3.33 | 262,144 | 87.6% | 73.8% | 73.8% | 34.3% | no data | 393,277 |
| GPT-5OpenAI | closed | US$ 1.25 | US$ 10.00 | 400,000 | 86.2% | 73.6% | 70.7% | 50.1% | no data | no data |
| Claude Opus 4.1Anthropic | closed | US$ 15.00 | US$ 75.00 | 200,000 | 77.3% | 73.3% | no data | no data | no data | no data |
| Gemini 3 ProGoogle | closed | no data | no data | no data | 92.6% | 72.9% | no data | no data | no data | no data |
| GLM 5Zhipu AI (Z.ai) | open | US$ 1.00 | US$ 3.20 | 204,800 | 87.8% | 72.1% | 76.9% | no data | no data | 1,370,207 |
| Claude Sonnet 4.5Anthropic | closed | US$ 3.00 | US$ 15.00 | 1,000,000 | 82.3% | 71.3% | 74.9% | 30.7% | no data | no data |
| Claude Opus 4Anthropic | closed | no data | no data | no data | 76.3% | 70.7% | no data | no data | no data | no data |
| GPT-5.1OpenAI | closed | US$ 1.25 | US$ 10.00 | 400,000 | 87.6% | 68% | 80% | 48% | no data | no data |
| GPT-5 MiniOpenAI | closed | US$ 0.25 | US$ 2.00 | 400,000 | 75% | 64.7% | 72.4% | 21.6% | no data | no data |
| o3OpenAI | closed | US$ 2.00 | US$ 8.00 | 200,000 | 81.8% | 62.3% | no data | 49.4% | no data | no data |
| Claude 3.7 SonnetAnthropic | closed | no data | no data | no data | 79.7% | 61% | no data | no data | no data | no data |
| Qwen3.6 PlusAlibaba | closed | US$ 0.33 | US$ 1.95 | 1,000,000 | 88.4% | 57.9% | no data | 44.1% | no data | no data |
| Gemini 2.5 ProGoogle | closed | US$ 1.25 | US$ 10.00 | 1,048,576 | 85.3% | 57.6% | 60% | no data | no data | no data |
| GPT-4.1OpenAI | closed | US$ 2.00 | US$ 8.00 | 1,047,576 | 66.9% | 48.5% | 52.3% | 31.1% | no data | no data |
| GPT-4oOpenAI | closed | US$ 2.50 | US$ 10.00 | 128,000 | 49.2% | 31% | 43.2% | 26% | no data | no data |
| Qwen3-CoderAlibaba | open | US$ 0.30 | US$ 1.00 | 262,144 | no data | no data | 48% | no data | no data | 58,791 |
| Seed-2.0-CodeByteDance | undisclosed | no data | no data | no data | no data | no data | no data | no data | no data | no data |
| Codestral 2508Mistral AI | undisclosed | US$ 0.30 | US$ 0.90 | 256,000 | no data | no data | no data | no data | no data | no data |
| Devstral 2 2512Mistral AI | open | US$ 0.40 | US$ 2.00 | 262,144 | no data | no data | no data | no data | no data | 6,656 |
| Kimi K2.7 CodeMoonshot AI | open | US$ 0.60 | US$ 3.50 | 262,144 | 87.9% | no data | 77.3% | 36.5% | no data | 91,687 |
| GPT-5.1-CodexOpenAI | undisclosed | US$ 1.25 | US$ 10.00 | 400,000 | no data | no data | no data | no data | no data | no data |
| GPT-5.1-Codex-MaxOpenAI | undisclosed | US$ 1.25 | US$ 10.00 | 400,000 | no data | no data | no data | no data | no data | no data |
| GPT-5.1-Codex-MiniOpenAI | undisclosed | US$ 0.25 | US$ 2.00 | 400,000 | no data | no data | no data | no data | no data | no data |
| GPT-5.2-CodexOpenAI | undisclosed | US$ 1.75 | US$ 14.00 | 400,000 | no data | no data | no data | no data | no data | no data |
| Qwen2.5 Coder 32B InstructAlibaba | open | US$ 0.66 | US$ 1.00 | 32,768 | no data | no data | no data | no data | no data | 547,627 |
| Qwen3 Coder 30B A3B InstructAlibaba | open | US$ 0.15 | US$ 0.60 | 262,144 | no data | no data | 46% | no data | no data | 1,679,625 |
| Qwen3 Coder FlashAlibaba | undisclosed | US$ 0.20 | US$ 0.97 | 1,000,000 | no data | no data | no data | no data | no data | no data |
| Qwen3 Coder NextAlibaba | open | US$ 0.12 | US$ 0.80 | 262,144 | no data | no data | 60.7% | no data | no data | 425,899 |
Which model to use for what
This pick follows the data and changes when the data does.
Code
- Rule: Highest score on SWE-bench VerifiedClaude Opus 4.7 83.5%
- Rule: Highest open weights score on SWE-bench VerifiedGLM-5.2 78.7%
- Rule: Cheapest per 1M input tokens among the top 5 on SWE-bench VerifiedGemini 3.5 Flash US$ 1.50
SWE-bench Verified ?
What does this mean?
What it measures: SWE-bench Verified, on the Epoch AI leaderboard (CC BY 4.0, latest version we checked), is 500 real issues from Python repositories on GitHub. The model has to produce the fix and pass the automated tests. The score is the share of tasks solved.
When it matters: This is the coding yardstick this table uses. I think of it as fixing a real bug in the company repo, not writing a function in isolation. Always compare under the same agent setup, because that setup changes the score.
Compare with
Sources
- Epoch AI, CC BY 4.0, latest version we checked.
- OpenRouter Benchmarks, CC BY 4.0, latest version we checked.
- OpenRouter, dados públicos da API; exports de benchmarks e rankings em CC BY 4.0, latest version we checked.
- Hugging Face Hub, metadados públicos do Hub; licença de cada modelo no repositório, latest version we checked.
Dates come from the sources: OpenRouter listing or Hugging Face repo creation, which may differ from the official launch. Family and version were grouped by rule from the name.