How to read AI benchmarks for each use case

A new table arrives every week. The lab publishes a cut, the number climbs onto a slide and, soon enough, becomes an argument for a choice. I have already spent too much time treating that cut as if it closed the conversation.

A benchmark measures a model's performance on a specific task, under a specific condition, on a specific date. Change the program that drives the model, change the index version or the time limit, and the number moves with them. The reading that serves me starts with the use case: what that metric captures, what it leaves out, and whether the figure came from the lab itself or from an independent board.

What to look at in any comparison

Before I open the table, I run through a short checklist.

Who measured it. A lab's own table, the so-called self reported kind, is a cut with a method, useful for watching one family move from release to release. An independent ranking, such as Arena or τ-bench, carries more weight among rivals, and still comes with a bias of method, language and framing.

Under what condition. The same benchmark with another harness (the program that drives the model), another time limit or another reasoning effort produces another result. Further down, Qwen and MiniMax publish Terminal-Bench 2.1 with different harnesses and time limits, and those figures do not compare.

Which version. "Qwen3.8 Max" shows up as a preview, as 0902 and in different effort settings. Indexes change version too, and the old number does not fit the new scale.

Cost per task, not only the price per token. A model that looks cheap on the list and then thinks for a long time can still cost more on each finished job.

Latency, license and what the board does not see. Time to the first token matters in live support. The license matters before the model becomes a product: open weights do not equal unrestricted use, and Qwen3.8 Max has its own license with a revenue cap for MaaS and AI Work Assistant. Brazilian Portuguese, data residency, Brazil's LGPD and SLA almost never enter the ranking.

What the table shows

Every "reference today" cell carries a source. Where the official leaderboard was not successfully read in this pass, that is stated plainly. Artificial Analysis does not enter this table; an isolated point in the prose goes with credit and, if it is the Index, the version.

This table lives only on this hub and should be read with the date in view. The change log sits at the end of the page.

Text and reasoning

For writing, analysis and reasoning, I look at the Arena text board with the rank range, at HLE when I want a knowledge ceiling, and at a test of my own in Portuguese. MMLU, MMLU-Pro and GPQA stay out of a choice among frontier models: they have already saturated at the top, and a gap of 1 to 2 points is usually noise. In Qwen's table, Qwen3.8-Max scores 92.6 on GPQA Diamond, in the same cut where the rivals on that board sit between 92.0 and 94.1.

On the Arena text board with style control, in the latest version we checked, Gemini 4 Carbon (High) led with an Elo of 1526. Qwen3.8-Max sat 23rd, at 1484, with a range from 13 to 37, in the same statistical pack as dozens of models. On the same board, GLM-5.3 (Max) came in at 1483 under an MIT license, and Qwen3.8-27B, Apache 2.0, at 1433.

Two caveats complete that reading. None of the major rankings measures Brazilian Portuguese in a central way; the test that matters most, for my use, is with text and tasks from here. And the list price per million tokens does not close the bill. Qwen3.8-max, on Model Studio in Singapore, lists at US$2 for input and US$6 for output per million. In a one-off citation from Artificial Analysis, on Index v5, Qwen3.8 Max (0902) comes out at US$10.02 per task and takes 56.8 minutes, with 114 seconds to the first token in the 0902 setup. Source: Artificial Analysis (artificialanalysis.ai). That cost depends on the provider and the configuration they used; a low price per token, with a lot of reasoning, can still make the work expensive.

There is also the episode of an index that changes scale. In the Apsara release, Alibaba said Qwen3.8-Max had moved from 40 to 45 on Artificial Analysis. Qwen3.8 Max (0902) sat at 34 on the current Index (v5). Source: Artificial Analysis (artificialanalysis.ai). The change of Index version is enough to refuse any "Index" figure cited without the version next to it.

Code

For code, I look at SWE-bench Pro, at Terminal-Bench 2.x always with the harness and time limit in view, and at LiveCodeBench when I want logic on problems published after the model's cutoff date. Standalone SWE-bench Verified and HumanEval stay out: the first is already saturated and heavily dependent on the harness, the second is far from day-to-day engineering.

In this pass, I could not successfully read the official SWE-bench Pro leaderboard or the Terminal-Bench board, which on the official site was already on version 4.1, while Qwen and MiniMax still cite 2.1. Without that reading, I do not publish an independent position as if it were settled.

What exists with a source is the labs' own cuts, under different conditions. In Qwen's table, with the Claude Code harness, Fable 5 stands at 80.0 on SWE-bench Pro and Qwen3.8-Max at 67.7, plus 86.6 on Terminal Bench 2.1. MiniMax reports 59.0 on SWE-bench Pro (Claude Code) and 66.0 on Terminal-Bench 2.1, with Terminus 2, an 8C16G sandbox, a 2 hour limit and 128K output. Qwen's 86.6, with Claude Code and 5 hours, does not compare with MiniMax's 66.0. Even inside the same family the bill does not close on its own: Qwen3.8-27B reports 73.0 on Terminal Bench 2.1 with the Terminus harness, while the Max run uses Claude Code. MiniMax also reports 80.5 on SWE-bench Verified, with a swapped system prompt and an average of 4 runs: partial transparency, and one more reason not to decide on that cut alone.

Qwen Code CLI is Apache 2.0, had 29,157 stars and was on release v0.27.1, according to the GitHub API. The free OAuth plan ended, in PR #3291. Sensitive code still calls for a conversation about where the data runs, including the option to run locally.

Agents and tools

For agents, I look at τ-bench with pass^k (succeeding several times in a row, not only once), at BFCL when the question is function calling, and at newer cuts such as Toolathlon and MCP Atlas, still with little independent measurement. A launch demo and an Automation-Bench without pass^k stay out of the decision.

On Sierra's official τ-bench, in the latest version we checked, Qwen3.5-397B-A17B led τ² (retail, airline, telecom) at 87.9%, ahead of Gemini 3.0 Pro (85.4%) and Claude Opus 4.5 (85.3%). On the τ³ Banking cut, an agent that consults about 700 documents and applies policy, Qwen 3.8 Max led with 55.2% pass^1, ahead of Claude Opus 5 (48.7%) and Grok 4.5 (47.9%). The board does not include every model from the period, the simulated user is itself a model, and 55% still means missing almost half of the tasks in that cut.

An independent replay helps me calibrate the noise. ZenML ran 40 τ² Telecom tasks: GLM-5.3 with 39 out of 40, Qwen3.8 27B with 36, DeepSeek V4.1 Flash with 35, GPT-5.6 Sol with 33 and GPT-5.2 with 27. GPT-5.2, run twice, matched the same total and still won 6 and lost 6 different tasks. Every model swap has to fit inside variation of that kind.

The official BFCL board was not read in this pass. What exists, self reported, is Qwen3.8-Max at 72.5 on Toolathlon, in Qwen's table, and MiniMax M3 at 74.2 on MCP Atlas: a signal, not a verdict. The test that decides remains yours, with your tools, your audit trail and your sandbox.

Image

For image, I look at the Arena text-to-image board, with Elo and a range, plus price, license and what the launch photo does not show: Portuguese text inside the image, editing and a transparent background. Hand-picked galleries stay out.

On the Arena image board, GPT-Image-2.5 Sunburst led with an Elo of 1429. Qwen-Image-3.0-Pro sat 14th, at 1254, and the best open Qwen option on that board was Qwen-Image (2512), Apache 2.0, at 1124. On Model Studio in Singapore, wan2.6-t2i lists at US$0.03 per image.

Video

For video, I look at Arena, the price per second and the maximum duration, and I keep the board with audio separate from the board without audio when both exist. A single ranking is not enough: two serious boards can disagree.

On the Arena video board, Gemini Omni 1.2 Flash led with an Elo of 1540, still marked preliminary. Seedance 2.0 came in at 1483 and Wan 3.0 sat 8th, at 1476. MiniMax H3, with open weights and a community license, was at 1456.

In a one-off citation from Artificial Analysis, in the latest version we checked, on the text-to-video cut without audio, Wan 3.0 was in front, at an Elo of 1175. Source: Artificial Analysis (artificialanalysis.ai). The same model was eighth on Arena. Saying "Wan 3.0 is number one" only holds if the board and the presence or absence of audio travel with the sentence.

The list price for Wan 3.0 on Model Studio in Singapore is US$0.10 per second at 720p and US$0.20 at 1080p, with a 30% promotion in this pass.

Voice and cloning

For voice, I look at human preference, price per million characters, streaming latency and whether pt-BR voices exist. Demo samples and a MOS declared by the vendor stay out. Cloning requires consent and a reading of the use policy; a Brazilian accent and rights enter that conversation before the rank.

I did not find, in this pass, an independent arena devoted only to voice cloning.

What I cite is an isolated point from Artificial Analysis, in the latest version we checked: Eleven v4 led the Speech Arena with an Elo of 1338, at US$80 per million characters, and Qwen-Audio-3.1-TTS-Plus sat 3rd, at 1299. Source: Artificial Analysis (artificialanalysis.ai). There is a price divergence I have not reconciled: the same reading lists Qwen at US$19.31 per million, while Model Studio in Singapore lists qwen-audio-3.1-tts-plus at US$0.3 per 10,000 characters, which equals US$30 per million. It may be a different region or a different cut.

Transcription

For transcription, what decides in my use is the word error on audio in Brazilian Portuguese, together with speed, price per thousand minutes and diarization. LibriSpeech WER, already saturated and in read English, does not serve that choice.

The Hugging Face Open ASR Leaderboard exists, but the figures from that board were not collected in this pass.

In a one-off citation from Artificial Analysis, in the latest version we checked, Alibaba's Fun-Realtime-ASR-preview shows an AA-WER of 1.7%, at US$2.82 per thousand minutes. Source: Artificial Analysis (artificialanalysis.ai). The metric weights AgentTalk at 50%, VoxPopuli at 25% and Earnings22 at 25%, a mostly English mix, and the "preview" label itself can change. For a meeting, an interview or support in pt-BR, a test on your own audio still comes ahead of that cut.

Multimodal work, documents and embeddings

For vision and documents, I look at MMMU-Pro with the variant declared (4 or 10 choices change the number), at Video-MME, at OCR tests and at a trial with your own papers. The original MMMU, already saturated, stays out. MiniMax reports 78.1 on MMMU-Pro and 85.4 on Video-MME in the M3 blog, self reported, and the evaluation material itself records doubt about the MMMU-Pro variant. In this pass, I did not collect a current official table for Qwen-VL or Qwen-Omni.

For search and RAG, the path is the retrieval column of a multilingual board such as MMTEB, the vector size, the license, the price and a test with real questions about your documents. The English MTEB average on its own does not decide, and I do not republish scores that came from a third-party mirror. On Model Studio in Singapore, text-embedding-v4 lists at US$0.07 per million and qwen3-rerank at US$0.10. Qwen3-Embedding-8B is Apache 2.0; a ranking score only enters this hub when it comes from the official page, with a date.

Traps that keep coming back

The first is comparing what was not measured under the same condition. Harness, time limit, number of runs and a swapped system prompt change the result under the same benchmark label.

The second is treating a launch table as a permanent photograph. Qwen's launch comparison uses Opus 4.8, Fable 5 and GPT 5.6. Later there was already Fable 5.2, Opus 5.5 and the GPT-6 line. A table of that kind ages in weeks.

The third is the in-house benchmark, created by the party being measured. Cuts such as QwenSWEBench, QwenQoderBench and QwenReactBench deserve extra caution.

The fourth is a claim without a table. The SenseNova 6.8 Flash releasesays the model outperformed some international leaders in agent collaboration, data analysis and presentation generation, without publishing benchmark names or figures. It works as a positioning signal, not as evidence.

The fifth is mixing usage with quality. On the OpenRouter usage rankings, in the latest version we checked, DeepSeek accounted for 31.8% of requests. Source: OpenRouter (openrouter.ai/rankings). Token volume measures what people are calling, not what gets your task right.

The sixth is forgetting the range: a 23rd place with a band from 13 to 37 is a wide tie. The seventh is repeating an index without the version, as in the case of 45 and 34 for Qwen3.8 Max.

How I have been using these boards

I do not pick a model from an isolated cell, and I do not throw the boards away either. They help me leave saturated tests aside, see when the test condition does not compare, and get faster to a trial of my own, in the language I work in, with the cost and latency of real use. What the work actually needs almost always lives outside the table, and that is where I take the conversation.

The full comparison table, by use case, lives on this hub.

The episodes below go deeper into each case. Paths; in these drafts, the slug is the same in all three languages:

• Text
• Code
• Agents
• Image
• Video
• Voice
• Transcription

Portuguese and Spanish versions: /ia/como-ler-benchmarks/ and /es/ia/como-ler-benchmarks/.

Frequently asked questions

What is an AI model benchmark? It is a standardized test that measures a model's performance on a task, under a condition, on a date. Outside that cut, a comparison still needs the model version, the harness, the source and what the test does not cover, such as Brazilian Portuguese or data residency.

How do I read an AI model ranking without getting lost in the number? Start with who measured it, the date and the version. Check whether the board is independent or self reported, whether the condition is the same among rivals, and whether the metric has anything to do with your use case. Only then look at the position, preferably with the range.

Why do two rankings name different winners for the same model? Because they measure different things, under different conditions. Arena captures human preference; a SWE-bench captures repository issues; a video cut with audio is not the same cut without audio. In video, Wan 3.0 sat 8th on Arena and, in a one-off citation from Artificial Analysis, in the latest version we checked, in front on the board without audio. Source: Artificial Analysis (artificialanalysis.ai). An honest sentence carries the name of the board with it.

Can I compare a number from one lab with a number from another? Only when the condition is the same, or when the text makes clear that it is not. Qwen's Terminal-Bench 2.1, with Claude Code and 5 hours, does not compare with MiniMax's run with Terminus 2 and 2 hours, even though the benchmark label is the same.

What is the best AI model today? It depends on the task, the cost per finished job, the latency, the license and the language. In the latest version we checked, Gemini 4 Carbon (High) led the Arena text board. GPT-Image-2.5 Sunburst led the image board and Gemini Omni 1.2 Flash, still preliminary, led the video board. On τ-bench, the reference changes from τ² to τ³. There is no single winner for every case.

Why does the price per token fail to close the bill? Because a model that looks cheap on the list can spend many reasoning tokens on each task, or take a long time to produce the first answer. The useful bill is the finished job, on the provider and in the region where you will run it, including cache, batch and context tiers. On Model Studio, batch carries a 50% discount and the price of Qwen3.8-max changes across Singapore, Beijing, Frankfurt and Hong Kong.

Do international rankings measure Brazilian Portuguese? Not in a central way. The major boards in this pass are mostly in English. For text, voice, transcription and search, a test with Brazilian material remains the most valuable one.

Update

First version of this hub.

Table change log

• Draft with Arena (text, image and video), τ-bench, Qwen tables and MiniMax tables, plus one-off citations from Artificial Analysis. No figures, in this pass, from the official SWE-bench Pro, Terminal-Bench 4.1, BFCL, Open TTS, Open ASR or MMTEB leaderboards.