How Popular AI Benchmarks Are Built
Nine case studies: how tasks are created and scored, why researchers use them, and what could improve.
A useful benchmark turns a question about AI into a test that other people can run. But every part of that test involves a choice: where tasks come from, what the model is allowed to do, how success is judged, and which failures count.
This article examines nine benchmark families used in recent LLM and agent evaluations. For each, I look at how it was built, what its design makes possible, why other teams have adopted it, and what could improve. The emphasis is on recent releases and revisions, mostly from 2025–2026. Earlier papers are included where they explain the design of a benchmark still in use.
The evidence has three levels: documented methods and implementation details; explanations of adoption based on public records; and proposed improvements that still need testing. Adoption is observable. Its causes are harder to establish. This is a study of published work and selected evaluation code, not a full reproduction of the benchmarks.
1. Terminal-Bench: ship the task, environment, and checks together
What problem does it address?
Writing a piece of code and completing a task in a computer environment demand different things. The latter may require inspecting files, installing dependencies, running programs, diagnosing errors, and changing approach. Terminal-Bench makes these terminal-based tasks an evaluation target.
The construction process below mainly concerns Terminal-Bench 2.0, released in November 2025. Later revisions show how the project evolved. Results from 2.0 and the August 2026 release of 4.0 need to be treated separately. Project timeline
How was it built?
Collect tasks, then review them. Mike A. Merrill, Alexander G. Shaw, and colleagues report that 93 contributors submitted 229 tasks, from which 89 were selected for 2.0. Selection considered difficulty and quality, with experienced reviewers checking the submissions. Paper, §2.2
Package each task so it can run. A task includes instructions, a container with the necessary software and files, a human-written reference solution, and tests. The reference solution helps establish that the task is solvable. The tests check the result of the agent’s work. Harbor provides tools for running and recording these evaluations. Task repository, Harbor
Review execution as well as text. The team ran reference solutions, checked that doing nothing did not earn a passing score, inspected failures from multiple models, and used an agent instructed to look for shortcuts. These checks help distinguish a model failure from a broken task or test. Paper, §2.3 and Appendix B
How does evaluation work?
The agent receives instructions and an initialized environment. It operates on that environment, and the evaluator checks the final state. The intended target is whether the task was completed, rather than whether the agent followed the reference solution’s command sequence. The paper also distinguishes model–agent combinations: a different agent framework can change the performance of the same model. Paper, §2.1 and §3
A useful result therefore identifies the task version, model, agent framework, resources, number of attempts, and success rate. Harbor helps organize runs and their records. Framework
What works well, and why has it been adopted?
Task completion becomes something executable tests can check. Different approaches can be accepted when they reach an acceptable result. This is a better fit for many work tasks than matching a reference sequence of actions.
Other teams can reuse the environment. Tasks, tools, and records reduce the amount of evaluation infrastructure each team must rebuild. Claude, Gemini, and Kimi reports use Terminal-Bench 2.0; Artificial Analysis has incorporated 4.0 into its index. Claude, Gemini, Kimi, Artificial Analysis
My interpretation is that terminal agents needed a shared test, and the project supplied both tasks and the means to run them. The work behind continued use is also visible: Kelly Buchanan and others organized the 2.1 revision, while Ryan Marten and colleagues developed approaches to continuous maintenance. These are documented contributions, although their individual effects on adoption have not been isolated. 2.1 revision, Continuous Benchmarks
What could improve?
| Problem and current status | Possible improvement | How to check whether it helps |
|---|---|---|
| Instructions, tests, and environments can disagree. Version 2.1 revised 28 tasks despite earlier reviews. Revision record | Add discovered mistakes, alternative valid solutions, and shortcuts to a regression collection that runs as tasks change. | Measure both valid solutions rejected and invalid solutions accepted. A higher model score alone does not establish better evaluation. |
| Resources affect results. Anthropic observed score differences when changing resources under controlled conditions; 4.0 already includes resource calibration. Study, 4.0 | Build on that calibration with fixed-time or fixed-cost comparisons, showing how much additional success more resources buy. | Hold the model and framework fixed while varying resources. Check ranking stability and resource-related failures. |
| Public tasks and solutions can leak. Tasks with public solutions were among those removed in 4.0. Release | Keep a public development set and add independently written, periodically refreshed evaluation tasks under comparable conditions. | Check whether improvements carry over to the new tasks. The cost is additional task creation and review. |
What to borrow: publish the tools for running, inspecting, and reporting failures alongside the tasks. Quality needs attention after release.
2. BrowseComp: make searching difficult and answers easier to check
What problem does it address?
Doing research includes finding sources, evaluating evidence, analyzing it, and writing. BrowseComp selects a narrower target: finding difficult-to-locate facts on the web and returning a correct short answer. Jason Wei, Zhiqing Sun, Spencer Papay, and colleagues released it in April 2025. Authors’ introduction
How was it built?
- Start with a fact supported by evidence. The answer should be short and, as far as possible, stable over time.
- Write an inverted question around it. For example, begin with a known event and describe it through several constraints, asking the solver to identify it. This illustrates the method; it is not a benchmark item.
- Check that it is difficult. Contributors tried models available at the time and simple searches. Some questions were also attempted by another contributor.
- Review the answer and wording. When another valid answer was found, the question could be revised. The paper explicitly acknowledges that this process cannot guarantee a unique valid answer for every question. Paper, §2
This is deliberate selection for difficulty. The resulting distribution is not a natural sample of everyday browsing requests, and the models used during construction help determine which questions survive.
How does evaluation work?
An agent searches and submits a response. The intended scoring procedure extracts its final answer and uses a model to judge whether it means the same thing as the reference answer, then aggregates correctness. A short answer simplifies that comparison. It does not establish that the reference covers every valid answer or that the response’s evidence is sound. A version-specific implementation issue is discussed in the code sidebar. Reference implementation
The design makes it easier to answer “Was the final answer right?” than “Why did the agent succeed or fail?” If systems use different search services, the result also includes differences between those services.
What works well, and why has it been adopted?
Browsing gets a clear comparison target. Short factual answers are easier to compare than open-ended reports, while the search itself can still require substantial effort. Authors’ introduction
OpenAI, Anthropic, Google, and Moonshot report BrowseComp results. My interpretation is that this clear, reusable test met a need as teams started comparing browsing agents. The publisher’s reach may have helped visibility, but the available records do not quantify how much each factor contributed. OpenAI, Anthropic, Google, Moonshot
What could improve?
| Problem and evidence | Possible improvement | How to check whether it helps |
|---|---|---|
| The reference answer may not be the only valid answer. The authors acknowledge this risk. Paper, §2.1 | Review different answers that include supporting evidence. After expert confirmation, expand the accepted answers or revise the question. | Audit responses marked wrong and count how many satisfy the question. Also check that accepting alternatives does not make the test too permissive. |
| The live web and search services change. BrowseComp-Plus already addresses part of this problem with a fixed corpus. Follow-up paper | Evaluate a fixed corpus and the live web separately. Use controlled retrieval to compare components, and live browsing to compare complete systems. | Hold one component fixed and vary another to locate the source of a difference. A fixed corpus, however, cannot capture every change on the real web. |
| A correct answer need not have valid supporting citations. The original scoring focus is the final answer. Code | Report citation support, search calls, cost, and latency separately instead of compressing everything into one score. | Audit whether cited sources support the answers, and compare the rate of correct, supported answers at matched budgets. |
BrowseComp-Plus is a useful example of a follow-up design: a fixed corpus and evidence labels make it easier to distinguish failure to retrieve evidence from failure to use it. This is an existing improvement, not a new proposal made here. Paper, Project
What to borrow: narrow the problem enough to score it reliably, then examine what that simplification leaves out.
3. τ²-bench: build a world whose state can be checked
What problem does it address?
Support tasks often require cooperation. An agent can change backend settings, while a user can operate their own device; each side has different information. τ²-bench evaluates this kind of multi-turn interaction. The June 2025 paper is by Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. Paper
How was it built?
The new Telecom domain illustrates the process:
- Build the backend and user device. Models help generate databases, tools, and tests, which people refine. The agent operates a backend; a simulated user operates a simulated device.
- Define small faults. Each has an initial state, a feasible solution, and conditions for checking that it was resolved.
- Combine faults. Avoid conflicting combinations and execute reference actions to verify solvability.
- Select evaluation tasks. Sample combinations covering different intents and levels of complexity.
- Add policies and user instructions, then review them together. These materials determine what each side knows and can do. Paper, §3.2
How does evaluation work?
The tested agent talks to a model acting as the user, with tools available according to each participant’s role. Criteria are specified by task. The inspected environment evaluator supports comparison against a target database state and checks of particular state conditions; other evaluation modules handle other criteria. Environment evaluator
In the paper, Telecom mainly uses state assertions. The experiments also compare normal cooperation, an agent controlling all tools without a user, and cooperation with a reference action plan provided. These comparisons help locate problems in planning and collaboration. Since information and control also change, the score differences should not be treated as perfectly isolated measurements of a single ability. Paper, §3.3 and §4.2
Repeated success matters. pass^k concerns succeeding on the same task in all k runs. It differs from pass@k, where at least one success among k attempts is sufficient. The former becomes stricter as k increases and is useful for studying service reliability. Metric implementation
What works well, and why has it been adopted?
A conversation produces state changes that can be checked. Reference actions help verify that tasks are solvable, and composing smaller faults makes task construction more systematic.
The experiments can help explain failures. Cooperation and reference-plan comparisons provide more information than an aggregate success rate alone. GPT-5.2 and Gemini 3.1 Pro report results on τ² domains. GPT-5.2, Gemini
My interpretation is that Sierra’s familiarity with service workflows helped turn gaps in single-tool-call tests into concrete tasks. The later τ³ release adds company-document retrieval and voice conditions, explicitly connecting these extensions to deployment experience. τ³ authors’ account
What could improve?
| Problem and evidence | Possible improvement | How to check whether it helps |
|---|---|---|
| The simulated user affects the result. A model-based user does not necessarily make the same mistakes or communicate like a person. Paper | Vary the user model, wording, and expertise, then calibrate with a smaller set of human interactions. | Check ranking stability and manually attribute failures to the agent, simulator, or task description. Human validation costs more. |
| A successful final state does not establish that every process requirement was met. Telecom’s state checks have a defined scope. Paper, §3.3 | For tasks with genuine process requirements, add checks for authorization, confirmation, or forbidden actions. Report task completion and process compliance separately. | Test trajectories that reach the correct state through an impermissible action, while still accepting different legitimate action sequences. |
| More combinations may not mean more new capabilities. Reusing the same building blocks in development and evaluation can limit what is learned. This is a risk to test. | Hold out combinations, tools, or business rules separately, and state what each split evaluates. | Check whether improvements survive these held-out conditions, and verify that the new tasks remain solvable. |
The repository’s July 2026 v1.0.1 update fixes some banking_knowledge grading issues and states that results for that domain before and after the fix are not directly comparable. Domain, simulator, and grading version all belong in a result description. Repository notice
What to borrow: define a checkable task world before studying interaction, and validate both the simulator and the evaluator.
4. OSWorld-Verified: real software creates value and maintenance work
How was it designed and built?
The original OSWorld framework appeared in 2024; the focus here is the Verified revision released in 2025. Tasks draw on actual software-use scenarios, with files, windows, and application states prepared for each one. Evaluation restores a virtual machine, initializes the task, lets the agent act, and then collects files or application state for checking. Original paper, §2–3
Evaluation is more than screenshot similarity. The framework can inspect application settings and document or spreadsheet contents. Experimental configurations may allow different observations and actions, including screenshots, accessibility information, keyboard and mouse actions, or commands. These conditions need to be reported. Original paper
Verified addresses web changes, initialization failures, ambiguous tasks, and grading issues. XLANG reports processing more than 300 pieces of feedback, with roughly ten people working for two months. This is a count of feedback items, not a count of broken tasks. The repair and re-evaluation contributions of Mengqi Yuan, Zilong Zhou, and others are documented. Revision account
Strengths and adoption
Real applications make the target understandable and allow cross-application tasks. A shared environment saves teams from rebuilding a desktop setup. GPT-5.4 and Sonnet 4.6 use Verified, demonstrating recent adoption for computer-use evaluation. GPT-5.4, Sonnet
My interpretation is that the task definition, reusable environment, and continued maintenance support adoption together. OSWorld 2.0 adds longer workflows in 2026; it is a later design whose results should be reported separately. 2.0 paper
Problems and possible improvements
| Problem and current status | Possible improvement | How to check whether it helps |
|---|---|---|
| Web changes, loading failures, and incorrect initialization have occurred. Verified account | Check the starting state before each run, preserve environment information, and record unavailable environments separately from normal task failures. | Repeat reference actions on different dates and measure failures without a model involved. Report exclusions rather than silently dropping tasks. |
| Valid solutions may fall outside a grader’s expectations, and instructions may be ambiguous. Verified account | Build examples of different correct outcomes. Check values, content, and appearance separately, with human review where needed. | Audit automatically failed results and measure false rejections before and after revisions. |
| Post-processing may affect what is measured. The framework supports actions such as saving files before checking; whether this hides an omitted agent action requires a task-by-task audit. Original paper | Preserve the state before and after evaluator post-processing. If saving is part of the task, check that the agent performed it. | Test trajectories that edit a file without saving. This is an audit proposal, not a claim that every task has this defect. |
What to borrow: initialization and result-collection code are part of the evaluation method. Real environments need continuing checks.
5. WebArena: make websites reproducible, then check whether the task is done
What problem does it address?
WebArena asks whether an agent can carry out a request inside a website, including changing stored data. Shuyan Zhou, Frank F. Xu, and colleagues at Carnegie Mellon built working sites for shopping, forums, software collaboration, and content management, with supporting tools and reference sites. Project
The original release dates to 2023. It belongs here because it continued to serve as a comparison point: OpenAI used WebArena for its Computer-Using Agent in January 2025, and WebArena-Verified revisited its evaluation in late 2025. These are different evaluation versions; a score should identify which one was used. Original repository, CUA report, Verified workshop publication
How was it built?
- Make the sites runnable by other teams. The repository supplies website environments, setup instructions, task configurations, and agent-running code. It distinguishes demonstration sites from the self-hosted environments required for evaluation, and documents resetting the sites. Repository
- Write goals, then vary their details. Authors created 241 task templates and expanded them into 812 requests. Templates specify a goal with replaceable details; instances can still require different actions. Paper, §3.1
- Check the answers and the checks. Information-seeking answers were annotated twice, with disagreements resolved by a third annotator. Authors writing evaluation programs executed tasks and inspected states. Paper, §3.2
As an illustration, imagine asking an agent to find a project, add a note with a given title, and return its link. This is an invented example, not a dataset item. A useful test must distinguish finding the project, opening the editor, and actually saving the right note.
How does evaluation work?
The original implementation combines checks selected for each task: compare a returned answer, inspect a URL, or locate and inspect page content. Answer checks include exact matching, required text, and a model-based semantic comparison. Thus “programmatic evaluation” does not mean that every original check is a deterministic test of the database. Pinned evaluator
The design aims to accept different action paths that achieve the goal. A particular checker can still miss part of that goal. For the invented note example, checking only that the editor opened would be too weak; requiring one exact click sequence would be too restrictive.
What did WebArena-Verified change?
Amine El hattami, Megh Thakkar, Nicolas Chapados, and Christopher Pal report auditing all 812 tasks, including ambiguous instructions and misaligned evaluators. This is a concrete maintenance contribution to an existing benchmark. Workshop publication
The revision makes three changes especially useful to study:
- Make the expected response explicit. Agents return structured fields describing the operation, outcome, and retrieved data. This helps distinguish reaching a page from completing a retrieval request. Response format
- Normalize data before comparing it. Type-aware rules replace loose substring checks and LLM judgments, so equivalent representations can receive the same result. A repeatable rule still needs checking against human judgments. Scoring design
- Save evidence that can be scored again. Captured browser–server traffic supports evaluation after the run, without keeping the websites running for that later scoring step. Running the agent still requires an environment. Network evaluation
These changes also alter what is measured. The documentation describes rewriting some open-ended requests into factual retrieval tasks so they can be checked deterministically. That reduces grading ambiguity, but also reduces the test of synthesis. Requirements about rendered content or client-side state need evidence beyond network-level checks. Task reformulation, Trace limitations
The cited repository version lists 812 tasks and a 258-task Hard subset; the earlier workshop abstract lists 137 Hard tasks. Even the subset name needs a version. A full-set score, a Hard-subset score, and an original-WebArena score should be reported separately. Pinned repository, Earlier abstract
What works well, and why has it been adopted?
It gives browser-agent research a reusable experimental setting. Researchers can study planning, recovery, and task completion in working applications. The original repository now recommends AgentLab and BrowserGym for experiments, showing how surrounding tools can carry a benchmark forward. Repository
Its use extends beyond the originating team. OpenAI’s CUA report and Microsoft Research’s Magentic-One study both include WebArena, although their agents and evaluation protocols differ. That is evidence of adoption, not a controlled comparison between the systems. CUA, Magentic-One
My interpretation is that the timing helped: as researchers moved toward agents that take actions, WebArena already offered applications, tasks, and a way to score completion. The original team made the test usable; later framework and evaluator work reduced the burden of continuing to use it. These contributions are visible, but their individual effects on popularity have not been measured.
What could improve?
| Problem and current status | Possible improvement | How to check whether it helps |
|---|---|---|
| An automated check can miss a requirement. Verified already repairs documented evaluation problems. Audit | Build a collection of valid alternative solutions and convincing failures. Include cases where a request was sent but the intended change did not persist. | Compare evaluator decisions with human inspection and saved state; measure both false passes and false failures. These are proposed audit cases, not claims that Verified currently fails them. |
| Related tasks can appear on both sides of a split. Magentic-One used a split by template to separate development from testing. Study, §5.1 | Keep entire templates together; add held-out workflows and sites when testing broader generalization. | Compare performance on new details within familiar templates, new templates, and new sites. These test different kinds of transfer. |
| Stable sites cover only part of web use. WebArena-Pro, reported in July 2026, already broadens coverage to 300 tasks across 20 applications, including audio and video. Pro publication | Pair fixed-version tests with controlled changes to layouts, data, and permissions. Broader coverage and robustness to change deserve separate measurements. | Hold task goals constant and measure the drop after each change. Check new tasks remain solvable; do not assume a newer benchmark has inherited the original’s adoption. |
| Resetting and running sites adds experimental overhead. The original repository requires reset procedures; Verified also provides environment tooling. Original setup, Verified tools | Record initial-state checks and isolate runs that modify the same data. Report environment failures separately. | Repeat selected tasks in different orders and compare sequential with parallel runs. Investigate changes caused by shared state before attributing them to the model. |
What to borrow: build the task, the environment, and the evidence for success together. WebArena-Verified adds a second lesson: keeping a useful benchmark trustworthy requires revisiting how it assigns scores.
6. HLE: difficult questions require expert coordination and review
How was it designed and built?
Humanity’s Last Exam was built by CAIS, Scale, and subject experts, with contributors including Long Phan, Dan Hendrycks, and Summer Yue. It targets difficult academic questions with relatively definite answers. Paper
Contributors submit questions, answers, and explanations. Questions are screened for difficulty using several contemporary models, reviewed and revised by people with relevant expertise, and approved by organizers or reviewers. Prizes and authorship opportunities help recruit expert contributions. Paper, §3
Feedback after release also matters. In April 2025, the finalized set was reduced to 2,500 questions after confirmed errors were addressed and readily searchable questions were removed. Final-version account
How is it scored?
Accuracy is the main outcome, while reported confidence helps assess whether models remain confident when wrong. The official method uses a model to extract and judge answers. Even with reference answers, wording, precision, and judge choice can affect edge cases. Tool access and the inclusion of images need separate reporting. Evaluation method
Strengths and adoption
Expert recruitment supplies difficult questions across subjects, while a common format makes integration easier. HLE offers new comparison material when older tests struggle to distinguish strong models. Claude, Gemini, Qwen, and Kimi reports include HLE results. Claude, Gemini, Qwen, Kimi
My interpretation is that demand, expert coordination, common scoring, and distribution all contributed. Expert review can improve quality, but expert authorship does not make errors impossible.
Problems and possible improvements
| Problem and evidence | Possible improvement | How to check whether it helps |
|---|---|---|
| Errors and directly searchable questions required revisions. Final-version account | Continue accepting evidence-backed challenges, use independent subject review, and retain change records with affected results. | Sample questions for reviewer disagreement, errors, and score changes caused by corrections. |
| Filtering on model failure may favor the screening models’ weaknesses. The size of this effect needs testing. Construction method | Keep a smaller comparison set of expert questions not selected by model performance, and evaluate models that were not used during screening. | Check whether discrimination remains and whether a few subjects or unusual question types dominate the result. |
| Closed-answer success does not demonstrate autonomous research. The project states this boundary. Scope | Audit reasoning and evidence on a subset. If the target is research, add separate research or experimental tasks and report them separately. | Measure correct answers with faulty reasoning, and test whether knowledge scores predict performance on the independent tasks. Review adds cost. |
What to borrow: check difficulty, correctness, and intended use separately. Making questions harder does not automatically broaden what the score means.
7. GDPval / GDPval-AA: compare work products without treating scores as productivity
How was it designed and built?
Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, and colleagues built GDPval around knowledge work in the U.S. economy. They selected sectors and occupations and asked experienced professionals to create requests, reference materials, and deliverables based on their work. The full set has 1,320 tasks; the public gold subset has 220. Paper, §2
Tasks pass through model-assisted screening and multiple rounds of human review. The original study uses blinded comparisons by occupational experts, who explain their judgments, and also introduces an experimental automated grader. Paper, §2.4–2.5
How is it run and scored?
The model receives a request and materials, then produces documents, spreadsheets, or other files. Reviewers compare those products with reference work. The deliverables resemble real work, but the experiment still supplies particular materials and conditions. Release account
GDPval-AA has a different execution and scoring procedure. Artificial Analysis uses the 220 public tasks, tools provided through Stirrup, and Elo ratings derived from blind pairwise comparisons. This must be distinguished from the original GDPval expert comparisons and win or tie rates. AA methodology
Strengths and adoption
Readers can connect the result to a recognizable work product. Experts help align tasks and quality judgments with occupational expectations. The AA platform adds a cross-model evaluation service; Claude and Gemini reports use GDPval-AA. Claude, Gemini
My interpretation is that, as agents began producing useful files, this evaluation addressed a practical question: is the output usable? The personnel requirements are concrete—professional experience, task creation, review, and grading time.
Problems and possible improvements
| Problem and evidence | Possible improvement | How to check whether it helps |
|---|---|---|
| The initial tasks generally supply the context upfront instead of testing repeated clarification. Paper, §5 | Preserve the original evaluation and add an interactive version with questions, additional materials, and revision requests. | Compare initial quality, revised quality, and user time. Publish separate scores when the task scope changes. |
| Human judgments differ, and automated grading does not fully replace experts. Paper, §5 | Review factual accuracy, calculations, requirement coverage, and presentation separately. Use multiple blinded reviewers and resolve important disagreements. | Include polished but incorrect deliverables and accurate but plain ones, checking that the evaluator distinguishes the relevant qualities. |
| The covered tasks are not an entire occupation. The study has explicit limits on work types and conditions. Release account | Report human review and correction time, broken down by occupation and task type. | On independent work tasks, test whether total human effort falls while quality stays acceptable. Model generation time alone is not a productivity measure. |
What to borrow: work products can make an evaluation more useful, but economic implications need their own validation.
8. ARC-AGI-2: keep the format simple and invest in task design and human testing
How is it designed and scored?
François Chollet, Mike Knoop, Gregory Kamradt, and colleagues retain the input–output grid format: show examples, provide a new input, and ask the system to construct the corresponding output. Discrete colors and bounded grids make the output directly checkable. Paper, §3
The project tests tasks with human participants, using completion rates and times to examine difficulty and calibrate subsets. Public tasks, held-out evaluation tasks, competition rules, and resource conditions form the broader project. Attempt limits and budgets should be stated for the particular competition or leaderboard being cited. Paper, §4, Release and competition
Strengths and adoption
The interface is simple and the question is clear: can a system use a few examples to handle a new rule? Retaining the format also supports reuse of existing tools. Competitions and public resources provide ways to participate; Gemini and Sonnet 4.6 report ARC-AGI-2 results. Gemini, Sonnet
My interpretation is that a clear research target and sustained organization helped build a shared test. Grid performance and competition activity do not, on their own, establish general intelligence.
Problems and possible improvements
| Problem | Possible improvement | How to check whether it helps |
|---|---|---|
| Extensive search can raise scores under different costs. The authors explicitly discuss brute-force search and efficiency. Paper | Compare matched cost, time, and attempt limits alongside best-achieved scores. | Check whether a method remains better across budgets, rather than only in its most expensive configuration. |
| Human-solvable grids remain a limited task family. | Test independently designed rule families and other interaction tasks, stating which abilities are expected to transfer. | Check whether improvement extends beyond familiar grid patterns and interfaces. |
| Repeated feedback on a held-out set can encourage adaptation to that set. The paper raises this concern about the earlier benchmark. Paper | Limit repeated probing and periodically introduce independently created, difficulty-calibrated tasks. | Compare old and new sets with difficulty references. Maintenance and historical comparison become more expensive. |
ARC-AGI-3 already moves toward interactive environments. That is an existing extension, but its different tasks need separate analysis; a new release does not itself prove transfer. ARC-AGI-3 release
9. LiveCodeBench: make task dates part of the experiment
How is it designed and scored?
Naman Jain and colleagues collect contest problems from platforms including LeetCode, AtCoder, and Codeforces and record their publication dates. In code generation, models receive a problem statement and examples, and unseen tests check their programs. The project also includes repair, execution, and test-output prediction scenarios. A single result in a model report should not be assumed to cover all of them. Paper
The v6 release used by Qwen3.5 and Kimi K2.5 is documented as containing 1,055 problems from May 2023 through April 2025. The repository also distinguishes a faster, reduced-test configuration from the full tests. Repository, Qwen, Kimi
LiveCodeBench Pro is a separate work. Zihan Zheng, Zerui Cheng, and colleagues involve programming-contest experts in analyzing tasks and failed submissions. Gemini 3.1 Pro reports Pro results. Authors, tasks, and scores should not be merged with v6. Pro paper, Gemini
Strengths and adoption
New contests provide a continuing source of tasks, and timestamps make contamination questions easier to examine. Executable tests also make comparison practical. My interpretation is that this addresses concern about models seeing old coding questions while remaining easy to integrate into established evaluation workflows.
Problems and possible improvements
| Problem | Possible improvement | How to check whether it helps |
|---|---|---|
| A recently released model may have seen older problems. The v6 window ends in April 2025; “Live” is not a guarantee against contamination. Repository | Report task dates and, where possible, evaluate on problems known to postdate training data or a frozen model. Mark unknown training coverage explicitly. | Compare difficulty-matched periods and check duplicates or reformulations. A gap between old and new tasks can also reflect difficulty, so it is not proof of contamination by itself. |
| Passing depends on test coverage, and reduced and full test sets differ. Repository | Add boundary tests to a subset and use deliberately altered, incorrect programs to probe the tests. | Compare disagreements between full and reduced testing, while checking that added tests do not reject valid solutions. |
| Contest performance does not directly measure maintenance, requirements discussion, or deployment. | Report repository-level work separately rather than treating all coding tasks as one ability. | Check whether gains survive on independent software-engineering tasks. More contest problems alone do not expand the scope. |
What to borrow: dates, sources, and test settings belong in the score description. “New” and “live” cannot substitute for validation.
What these designs have in common
These benchmarks did not succeed through one identical design. BrowseComp makes answers easier to judge; τ² builds a controlled interaction world; Terminal-Bench, OSWorld, and WebArena supply task environments; HLE and GDPval rely on expert coordination. Each turns an important question into a test other teams can use.
Each useful choice also creates a cost:
| Design choice | What it enables | What still needs checking |
|---|---|---|
| Short answers or automated tests | Lower-cost comparison | Incomplete references, mistaken grading, and missed process requirements |
| Real websites and applications | Tasks closer to actual use | Environment changes, initialization, and reproducibility |
| Selecting questions that defeat current models | Better separation of contemporary models | Model-specific weaknesses and task representativeness |
| Public data, code, and reference solutions | Research, reproduction, and adoption | Exposure during training or browsing |
| Frequent task and grader revisions | More reliable results and longer useful life | Versions, rerun costs, and comparability with old results |
The useful question is specific: which design choice solved which problem, what did it cost, and how can that cost be measured? Labels such as “realistic,” “hard,” and “comprehensive” need this explanation behind them.
Three directions worth testing
These are research suggestions drawn from the cases, not claims of new methods or demonstrated improvements.
- Audit evaluators systematically. Assemble alternative valid solutions, superficially convincing failures, and trajectories that finish through invalid actions. Have experts label them, then measure false rejections, false acceptances, review cost, and performance across task types.
- Separate models, tools, and execution conditions. Keep tasks fixed while changing the agent framework for one model, or the model inside one framework. Control budgets and environments to explain where differences come from.
- Evaluate controlled conditions and real change together. For example, test a fixed corpus and the live web separately, then examine whether the results agree. Report both stability and the range of real situations the evaluation covers.
A small check of the grading code
Examining a pinned BrowseComp reference implementation reveals a concrete mismatch. At commit 652c89d, the parser returns correct: yes, while its caller compares the result with yes. Along that path, even a positive mock judgment is not counted as correct. Pinned source, lines 92–108
The function was extracted from that source and checked with mock grader outputs. This involved no model calls, benchmark questions, or full benchmark run. The essential string comparison is:
import re
reply = "correct: yes" # Synthetic grader output
parsed = re.search(r"correct: (yes|no)", reply).group(0)
print(parsed == "yes") # False: parsed still includes "correct: "
This finding concerns that reference-code version. It does not establish that any paper or model provider used the same path, or that its published scores are wrong. The practical lesson is to test evaluation code with known positive, negative, and unparseable outputs alongside reviewing the questions themselves.
Sources and scope
Methods are described for the cited versions, with later revisions identified separately. Explanations of adoption are based on public records rather than author interviews or causal comparisons. Proposed improvements include ways to test them, but have not been validated here. Apart from the local parser check, no full benchmark was reproduced, and no benchmark questions or answers are reproduced in this article.
The main sources are grouped below; links within each case point to the evidence for individual claims.
- Terminal-Bench: paper, timeline, tasks, Harbor, 2.1 revisions, 4.0 release, continuous maintenance, infrastructure experiment.
- BrowseComp: paper, authors’ introduction, pinned evaluation code, BrowseComp-Plus paper, follow-up repository.
- τ-bench family: τ² paper, environment evaluator, metrics, repository and version notes, τ³ release.
- OSWorld: original framework, Verified revision, 2.0 paper.
- WebArena: project, original paper, repository, pinned evaluator, CUA adoption, Magentic-One and its split.
- WebArena extensions: Verified workshop publication, pinned Verified repository, response format, normalization, network evaluation, Pro publication.
- HLE: original paper, final-version and evaluation details.
- GDPval: paper, release, GDPval-AA methodology.
- ARC: ARC-AGI-2 paper, release and competition, ARC-AGI-3 release.
- LiveCodeBench: original paper, repository and versions, Pro paper.
- Adoption records: GPT-5.2, GPT-5.4, GPT-5.6, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro, Qwen3.5, Kimi K2.5, Artificial Analysis index.