Glossary
SWE-bench Verified
SWE-bench Verified, on the Epoch AI leaderboard (CC BY 4.0, latest version we checked), is 500 real issues from Python repositories on GitHub. The model has to produce the fix and pass the automated tests. The score is the share of tasks solved.
What it measures
SWE-bench Verified, on the Epoch AI leaderboard (CC BY 4.0, latest version we checked), is 500 real issues from Python repositories on GitHub. The model has to produce the fix and pass the automated tests. The score is the share of tasks solved.
When it matters
This is the coding yardstick this table uses. I think of it as fixing a real bug in the company repo, not writing a function in isolation. Always compare under the same agent setup, because that setup changes the score.
How to read it
The score is the share of issues solved, with tests passing, on the Epoch AI leaderboard, in the latest version we checked. This test has 500 issues, so 1 percentage point is 5 issues. Always compare under the same agent setup.
Who leads
- Claude Opus 4.7 Anthropic83.5%
- GPT-5.5 OpenAI80.6%
- Gemini 3.5 Flash Google79.3%
- GLM-5.2 Zhipu AI (Z.ai)78.7%
- Claude Opus 4.6 Anthropic78.7%
- DeepSeek V4 Pro DeepSeek77.6%
- Qwen3.7 Max Alibaba77.3%
- GPT-5.4 OpenAI76.9%
- Kimi K2.6 Moonshot AI76.7%
- Claude Opus 4.5 Anthropic76.7%
Source: Epoch AI, latest version we checked.