AI models

Methodology of AI models

This page describes the rules I use on /ia/modelos/ to compare API models: where the data comes from, when a comparison enters search engines,

This page describes the rules I use on /ia/modelos/ to compare API models: where the data comes from, when a comparison enters search engines, how I decide a statistical tie, how I treat price and license, and how to cite what is on the site. This is version 2, November 21, 2026.

The guide for reading the scores is at /en/ia/como-ler-benchmarks/.

Sources and weekly collection

The weekly collection runs every Sunday at 7 p.m. São Paulo time. The scores that enter the index come only from Epoch AI, OpenRouter, and official lab sites that allow this use. The collection reads the OpenRouter catalog, the public Epoch AI benchmarks CSV, and OpenRouter's own evaluations. LMArena, through its dataset on Hugging Face, and weight metadata on Hugging Face appear here only cited as collection sources and do not enter the index scores. The lab's official price, when it exists, I record separately from the official pricing page, and it prevails over the OpenRouter price. Each row of /en/ia/modelos/mudancas/ shows the source and the date.

Minimum of 4 tests for the overall index

A model only gets a rank in the overall index, the list that puts all tests into one order, when it has a score on at least 4 distinct tests. GPQA Diamond scores from Epoch AI and from OpenRouter count as one test. With fewer than 4, the model stays out of the overall index calculation and its page shows the label «insufficient data» in place of that rank. In the current collection, 85 of the 258 models have a rank in the overall index, and 173 show «insufficient data».

For those models, everything else stays the same. The model page shows every score, each test has its own ranking, with range and badges, and «Good for» still comes from the scores on each test. A comparison between two models still requires at least 3 distinct tests in common, and in the current collection that applies to 5,718 pairs. In the use categories, a model enters when it has a score on both tests of the category, even with fewer than 4 in total. Saturated and low discrimination tests also count toward both minimums.

I only index the comparison between two models when they have a score in at least 3 distinct benchmarks in common; in this collection, that applies to 5,718 pairs, among 258 models with some score. It was 3,145 pairs before November 20, 2026, when OTIS Mock AIME started counting toward this minimum.

OTIS Mock AIME is saturated: the score stays in the table with the badge, without a rank and outside «Good for», and it counts toward the minimum of 3 tests in common. GPQA Diamond scores from Epoch AI and from OpenRouter count as one test. τ-bench Verified (airline), a low discrimination test, also counts toward this minimum.

## How I calculate the range in the overall index

In the overall index, each model's rank comes from three ranking methods, which look at the same data in different ways. BT, the Bradley-Terry model, treats each pair of models, on each test both have, as a match: the one above outside the margin wins, and a statistical tie is worth half a point to each side; from all the matches, the method calculates each model's strength. IRT, item response theory, estimates each model's ability while accounting for how hard each test is and how well it separates models; on both GPQA Diamond sources, the calculation accounts for a 25% chance of answering correctly by guessing. The percentile shows where the model stands on each test, among the models ranked in the overall index, and then averages the tests it has.

To measure how much a rank can move by chance alone, I use the bootstrap: I redo the whole calculation 2,000 times, and in each of these replicates I draw again how many questions each model gets right on each test, based on the published score and the number of questions in the test. The tests themselves stay fixed, because each model has scores on 4 to 6 of them, and drawing whole tests would erase part of the data for some models. The draw uses the seed 20261121, the number that fixes the random sequence, so anyone who runs the same calculation reaches the same results.

In each method, the range goes from the 2.5th to the 97.5th percentile of the 2,000 ranks, which drops the most extreme results on both sides. The published range combines the three methods: it starts at the best of the three starting points and ends at the worst of the three end points. Combining the three avoids a gap, because for 7 of the 85 models the ranges of the three methods share no position at all.

The rank appears as a single number, in the format «Xth of N», when the range has a width of 1, meaning it starts and ends at the same position. Otherwise it appears as a range, in the format «between Xth and Yth of N». If two models end up with the same exact rank, both show the same number, for example «3rd of 85», with no tiebreak. The list follows the start of the range and then the end. In the current collection, none of the 85 models has an exact rank in the overall index, and the median range covers 23 positions.

Statistical tie and 95% margin

I treat two models as a statistical tie when the difference between scores, without rounding, falls within the 95% margin: |d| < max(1.96·√(SE_A²+SE_B²); 1 p.p.). The p for the difference still comes from the source, without rounding and without k/n.

To size the margin of each score, I start from the score published by the source and the number of questions in the test: from these I calculate how many questions the model got right and the standard error, which measures how much the score could vary if the test had other questions of the same kind. When the model gets every question right or none, the simple calculation would give zero error, and in that case I use the Wilson interval, an adjustment that keeps a margin at the ceiling and at zero. This calculation applies to every test and replaces the standard error published by Epoch AI. The tests have 198 questions on both GPQA Diamond sources, 285 on FrontierMath (tiers 1 to 3), 1,000 on SimpleQA Verified, 484 on SWE-bench Verified, 45 on OTIS Mock AIME and 50 on τ-bench Verified (airline). The «±» that OpenRouter publishes is the standard deviation across runs, and I do not use it as a standard error.

Low resolution at the top

The S index measures whether the top of a test still separates models. It compares the score of the model in 1st place with the one in 5th and accounts for the size of the test: the closer to 1, the more crowded the top. The calculation follows the paper that proposed the index (Akhtar, Reuel and coauthors, 2026), with α = 0.5, the paper's default value, which sets how much the number of questions in the test weighs in the result.

When the S index reaches 0.7 or more, the test gets the caveat «low resolution at the top». The score, the range, the badges and «Good for» stay as they are; the caveat warns that, at the top of that test's list, the gap between scores says little. In the current collection, five tests carry the caveat: GPQA Diamond from Epoch AI (0.970), GPQA Diamond from OpenRouter (0.952), FrontierMath, tiers 1 to 3 (0.820), SWE-bench Verified (0.849) and τ-bench Verified, airline (0.991). On SimpleQA Verified, the S index is 0.012, and the test has no caveat.

GPQA Diamond keeps its rank, badges and «Good for», and carries only this caveat. In the current collection, the top score on GPQA Diamond from Epoch AI is 96.8, held by one model, and 28 of the 195 models get one of the two badges; on the OpenRouter source, it is 21 of 134.

Saturated test

A test is saturated when the S index reaches 0.7 or more and at least two models get every question right. The score stays in the table and the badge still applies, but the test leaves the rank order and «Good for». It still counts toward the minimum of 3 tests for comparisons and the minimum of 4 for the overall index, and it enters the overall index calculation. In the current collection, only OTIS Mock AIME is saturated: its S index is 1.000, 16 of the 182 models are at 100%, and 65 of the 182 get one of the two badges.

Low discrimination test

When more than half of the models on a test would get a badge, the test is marked «low discrimination»: the score is shown, without a rank and without a badge, and it leaves «Good for». It still counts toward the minimums of 3 and 4 tests and enters the overall index calculation. In the current collection, this happens only on τ-bench Verified (airline), where 96 of the 130 models would get a badge. Its S index is also above 0.7, and the caveat «low resolution at the top» appears with it.

## Use categories

Two use categories combine two tests each, with equal weight: Reasoning and math, with GPQA Diamond from Epoch AI and FrontierMath (tiers 1 to 3), across 80 models, and Knowledge and factual accuracy, with GPQA Diamond from Epoch AI and SimpleQA Verified, across 84 models. A model enters when it has a score on both tests of the category, and I do not estimate missing scores. GPQA Diamond from OpenRouter stays out of the categories.

Each model's score is its win share. On each test, I count how many models in the category it beats outside the margin, with half a point for each statistical tie, and divide by the number of opponents. Then I average the two tests.

The range uses the same 2,000 replicates as the overall index. In each replicate, besides drawing the correct answers again, I also draw the weights of the two tests from a Dirichlet(5, 5) distribution, which keeps each weight close to half and lets it vary a little. That way the range also shows how much the rank depends on the chosen weight. The range goes from the 2.5th to the 97.5th percentile, and the label follows the same format as the overall index. In the current collection, 1 of the 80 models has an exact rank in Reasoning and math, and none of the 84 in Knowledge and factual accuracy.

Coding has a single test, SWE-bench Verified, so it appears as that test's ranking, with the caveat «one test only».

Price and license

Price review: A price change on OpenRouter only enters the change log on its own if it is greater than 10% and the new price repeats in the next collection. A change in the lab's official price goes through review in the data PR, and does not enter under this rule. Retired models leave cost ranking and the recommender.

Scheduled price: When the lab announces a price change with a date, the site switches on its own on that date, at midnight São Paulo time. If the lab also gives a time with time zone, the site uses that instant.

Gemini 4 Argon and Gemini 4 Carbon go from US$ 2.00 to US$ 4.00 per 1M input tokens and from US$ 10.00 to US$ 20.00 per 1M output tokens on December 1, 2026, according to the Google official pricing page consulted November 17, 2026.

**License:** This site's own data is released under the CC BY 4.0 license, with attribution «Adriano Meirinho, meirinho.com.br». Third-party data follows each source's license; the license on own data does not relicense third-party data. Prices, official or from OpenRouter, appear only as a link to the source, without the number.

CSV and JSON

The open table is at /ia/modelos/dados.csv and at /ia/modelos/dados.json. It includes identifiers and catalog curation, under CC BY 4.0 with attribution «Adriano Meirinho, meirinho.com.br», and benchmark scores whose source licenses redistribution: Epoch AI internal evaluations and OpenRouter's own evaluations, each with the source license and credit. Version 1 of the CSV and JSON ships without the LMArena rating and without the OpenRouter usage slice. Price values, test questions and answer keys, Hugging Face downloads and likes, and any data from a source without clear redistribution license are excluded or appear only as a link.

How to cite

If you use this data, cite the page and the date you accessed it.

Adriano Meirinho. "Methodology of AI models". meirinho.com.br. https://meirinho.com.br/en/ia/modelos/metodologia/. Accessed [date of your access].

Third-party data follows each source's license. The full table is available as CSV and as JSON.

Version history

Version 1, November 20, 2026: first version of the page, with the rules already in effect.

Version 2, November 21, 2026: adds the minimum of 4 tests for the overall index, the range from three methods with bootstrap, the S index with the low resolution at the top caveat, and the use categories with a range.

Markdown