Market · AI in practice
With the model fixed, the harness shows up in cost and speed
In an open evaluation dated November 4, 2026, Garry Tan held the model and tasks fixed and swapped only the harness; pass rates were statistically indistinguishable, while cost per solved task diverged.
What the sources report
Garry Tan open-sourced g-harness-evals and, on November 8, 2026, pointed to the matrix on X. The design holds the model and the tasks fixed and swaps system prompt, tools and memory. On 20 public tasks, pass rates sat in a band of about 71–79%, with no measurable separation across harnesses. Cost and speed diverged.
On Opus 5.5, cost per solved task ran about $0.19–$0.24 for QM, Claude Code and Hermes, and $0.44 for OpenClaw, also the slowest at 99.4 s per trial. On GPT-6.1 Sol, Codex came in at $0.17 and 30.1 s. About 81% of OpenClaw’s extra cost over Claude Code on Opus came from cache writes: a larger system prompt and larger tool results resent every turn. The README notes conflicts of interest and that results are development runs on public tasks, not held-out confirmation.
Adriano's reading
The thesis: at scale, the harness becomes an operating-cost line. I would put cost per solved task on the same dashboard as accuracy — and treat cache and prompt/tool size as an engineering decision with a budget owner.