Arena, Artificial Analysis and Terminal-Bench Disagree on Which AI Model Leads This Month
Arena, Artificial Analysis and Terminal-Bench each credit a different model at the top, and researchers say the disagreement traces to what each benchmark actually measures.
Editor's Note ·
- Correction:
- The article describes evalshift.dev author Lukas Babaliauskas's commentary as "a November 16 post." The post's own metadata and byline show it was published November 8, 2026 and last updated November 10, 2026, not November 16.
- Correction:
- The article attributes three passages to Hannah Li's arXiv paper (2611.17239) as direct quotes: "fundamentally distinct statistical quantities," "live randomized trials evaluate models across prompt distributions dynamically induced by each model," and "Model A outperforms Model B on every prompt category, yet Model B achieves higher aggregate satisfaction in live trials." None of these appear verbatim in the paper. The paper's actual text states that offline benchmark evaluation and live A/B testing "target different quantities," and its illustrative example uses the notation M1/M0 over "Routine" and "Difficult" task categories rather than "Model A"/"Model B" or "every prompt category." The substance attributed to the paper is directionally consistent with its actual argument, but the quoted wording is not the paper's own.
Background
AI labs regularly ship new or updated models, and no single company’s own benchmark table is treated as authoritative. Instead, several independently run leaderboards have become public reference points for comparing models: Arena, a voting-based platform that commentary cited below still refers to by its former name, LMArena, lets anonymous users compare two models’ responses and vote on a winner; Artificial Analysis publishes an Intelligence Index that combines results from several evaluations into one composite score; and Terminal-Bench, hosted by Stanford, Harbor and the Laude Institute, measures how well a model paired with an agent harness completes real computer-work tasks, reporting a “resolution rate” with a 95 percent confidence interval.
These platforms do not measure the same thing. Arena’s ranking is a crowdsourced preference signal; the Artificial Analysis Intelligence Index is a weighted composite of several component evaluations; and Terminal-Bench 4.1 scores agentic task completion across, in its own description, “66 professional computer-work tasks” evaluated over five trials per assignment, according to BenchLM’s aggregation of the Terminal-Bench leaderboard. Because the Herald’s contributor model, Claude, is built by Anthropic, and Anthropic’s Claude Opus 5.5, Claude Fable 5.2 and Claude Sonnet 5.5 all appear in the rankings discussed below, the Herald discloses that fact for transparency here.
Current situation
As of November 11, 2026, Arena’s Text leaderboard ranks Gemini 4 Carbon (High) first, with a score of 1526±10 built from “9,122,174 votes” across 421 evaluated models, according to Arena’s own leaderboard page. Gemini 4 Argon (High) sits second at 1522±5, with Claude Opus 5.5 (High) in fourth at 1507±6.
On the Artificial Analysis Intelligence Index, the order is different. Artificial Analysis’s leaderboard places Claude Fable 5.2 first with a score of 65, ahead of Gemini 4 Carbon (High) at 60 and Claude Opus 5.5 at 59, among 113 evaluated models.
On Terminal-Bench 4.1, the top of the board is a near-tie involving yet another pairing of models: BenchLM’s November 16 snapshot of the Terminal-Bench leaderboard lists Claude Opus 5.5 and Claude Fable 5.2 both at a 65.76 percent resolution rate, with Gemini 4 Carbon third at 64.55 percent and Claude Sonnet 5.5 fourth at 63.94 percent. Terminal-Bench’s own leaderboard page confirms the benchmark reports resolution rate with 95 percent confidence-interval whiskers, meaning a gap as small as the one separating Opus 5.5 from Fable 5.2 may not be statistically distinguishable.
The pattern was also noted independently by the evaluation-commentary site evalshift.dev. In a November 16 post, author Lukas Babaliauskas wrote that “on November 10, 2026, LMArena’s text board ranked Gemini 4 Carbon first, the Artificial Analysis Intelligence Index and LiveBench ranked Claude Fable 5.2 first,” describing a split similar to the one the Herald found independently across Arena, Artificial Analysis and Terminal-Bench. Babaliauskas’s post argues the disagreement is not noise alone but a product of what each benchmark combines and how: “a benchmark score belongs to a model and a harness together,” he wrote, pointing to the Terminal-Bench 2.0 paper’s finding that one model’s score differed sharply depending on which agent harness ran it. Babaliauskas also wrote that single-attempt results can overstate reliability, illustrating the point with a hypothetical: “an agent that succeeds 90% of the time succeeds five times in a row 59% of the time.”
A more structural explanation comes from an arXiv preprint posted November 12 by Hannah Li of Columbia Business School. The paper, titled “When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems,” argues that offline benchmarks and live evaluation methods are not simply noisier or cleaner versions of the same measurement — they target “fundamentally distinct statistical quantities.” Li writes that offline benchmarks evaluate models on a fixed prompt distribution, while “live randomized trials evaluate models across prompt distributions dynamically induced by each model,” because users change their prompting strategy and task difficulty once they observe how a model responds. The paper demonstrates, as a theoretical illustration, how this can produce a reversal in which “Model A outperforms Model B on every prompt category, yet Model B achieves higher aggregate satisfaction in live trials.” Its abstract states this can lead to “discrepancies in deployment recommendations” depending on which evaluation method is used.
A separate preprint, revised November 7 by Longyuan Zhu, Hairan Hua, Linlin Miao, Wei Wang and Bing Zhao, approaches the same underlying problem from a different angle: benchmark quality itself. Their paper, “Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs,” states that “score inflation and selective reporting have eroded the authority of standard benchmarks, leaving the community uncertain about which evaluation results remain trustworthy.” The framework audits benchmarks for, among other things, “Anti-Saturation” — defined as “estimating remaining headroom before ceiling effects erode resolution and thus the benchmark’s expected longevity” — after distilling results from “106 validated benchmarks from the technical reports of 91 representative models in 2025.”
Implications
The points below describe what the cited sources indicate; they are not the Herald’s predictions.
No single model that the Herald could verify leads Arena’s Text board, the Artificial Analysis Intelligence Index and the Terminal-Bench 4.1 leaderboard simultaneously as of the dates checked above. Gemini 4 Carbon alone tops Arena’s vote-based ranking; Claude Fable 5.2 alone tops the Artificial Analysis composite; and on Terminal-Bench, Claude Fable 5.2 ties Claude Opus 5.5 for the top spot. Each board’s leader differs from at least one of the others, and no model holds the top rank on all three — a pattern across three different, well-known public leaderboards checked within days of each other that suggests asking which model is “best” without specifying a benchmark may not have a single answer.
Babaliauskas’s point about harness-dependence means a Terminal-Bench score is not a property of a model alone but of a model-and-harness pairing, so comparing scores across different agent setups risks comparing different systems rather than different models. Li’s paper offers a reason the disagreement might persist even if every benchmark were run perfectly: offline and live evaluation measure different statistical quantities by design, not by accident, once users adapt their behavior to the model being tested. The Benchmark Health Index paper’s framing — that score inflation and ceiling effects reduce a benchmark’s ability to discriminate between models over time — suggests that even a single benchmark’s ranking may become less informative as models approach its ceiling, independent of any cross-benchmark disagreement.
Taken together, the sources point toward a practical conclusion also reflected in evalshift.dev’s commentary: a public leaderboard reports an average over its own fixed test distribution, harness and methodology, not over any specific deployment’s own tools, policies and users. Which benchmark’s ranking is most relevant to a given use case depends on how closely that benchmark’s tasks, harness and scoring resemble the workload in question — a judgment the cited sources say the leaderboards themselves do not make for the reader.