How to read benchmarks · Guide

Agents and tools

For agents, I look at τ-bench with pass^k (succeeding several times in a row, not only once), at BFCL when the question is function calling, and at newer cuts such as Toolathlon and MCP Atlas, still with little independent measurement. A launch demo and an Automation-Bench without pass^k stay out of the decision.

On Sierra's official τ-bench, in the latest version we checked, Qwen3.5-397B-A17B led τ² (retail, airline, telecom) at 87.9%, ahead of Gemini 3.0 Pro (85.4%) and Claude Opus 4.5 (85.3%). On the τ³ Banking cut, an agent that consults about 700 documents and applies policy, Qwen 3.8 Max led with 55.2% pass^1, ahead of Claude Opus 5 (48.7%) and Grok 4.5 (47.9%). The board does not include every model from the period, the simulated user is itself a model, and 55% still means missing almost half of the tasks in that cut.

An independent replay helps me calibrate the noise. ZenML ran 40 τ² Telecom tasks: GLM-5.3 with 39 out of 40, Qwen3.8 27B with 36, DeepSeek V4.1 Flash with 35, GPT-5.6 Sol with 33 and GPT-5.2 with 27. GPT-5.2, run twice, matched the same total and still won 6 and lost 6 different tasks. Every model swap has to fit inside variation of that kind.

The official BFCL board was not read in this pass. What exists, self reported, is Qwen3.8-Max at 72.5 on Toolathlon, in Qwen's table, and MiniMax M3 at 74.2 on MCP Atlas: a signal, not a verdict. The test that decides remains yours, with your tools, your audit trail and your sandbox.

Open source board

This case has no number board. The versioned file does not yet hold an open source measurement here.