Shiyi Du · Blog

How Popular AI Benchmarks Are Built

2026-09-09T23:45:00-07:00

A useful benchmark turns a question about AI into a test that other people can run. But every part of that test involves a choice: where tasks come from, what the model is allowed to do, how success is judged, and which failures count.

This article examines nine benchmark families used in recent LLM and agent evaluations. For each, I look at how it was built, what its design makes possible, why other teams have adopted it, and what could improve. The emphasis is on recent releases and revisions, mostly from 2025–2026. Earlier papers are included where they explain the design of a benchmark still in use.

The evidence has three levels: documented methods and implementation details; explanations of adoption based on public records; and proposed improvements that still need testing. Adoption is observable. Its causes are harder to establish. This is a study of published work and selected evaluation code, not a full reproduction of the benchmarks.

1. Terminal-Bench: ship the task, environment, and checks together

What problem does it address?

Writing a piece of code and completing a task in a computer environment demand different things. The latter may require inspecting files, installing dependencies, running programs, diagnosing errors, and changing approach. Terminal-Bench makes these terminal-based tasks an evaluation target.

The construction process below mainly concerns Terminal-Bench 2.0, released in November 2025. Later revisions show how the project evolved. Results from 2.0 and the August 2026 release of 4.0 need to be treated separately. Project timeline

How was it built?

Collect tasks, then review them. Mike A. Merrill, Alexander G. Shaw, and colleagues report that 93 contributors submitted 229 tasks, from which 89 were selected for 2.0. Selection considered difficulty and quality, with experienced reviewers checking the submissions. Paper, §2.2

Package each task so it can run. A task includes instructions, a container with the necessary software and files, a human-written reference solution, and tests. The reference solution helps establish that the task is solvable. The tests check the result of the agent’s work. Harbor provides tools for running and recording these evaluations. Task repository, Harbor

Review execution as well as text. The team ran reference solutions, checked that doing nothing did not earn a passing score, inspected failures from multiple models, and used an agent instructed to look for shortcuts. These checks help distinguish a model failure from a broken task or test. Paper, §2.3 and Appendix B

How does evaluation work?

The agent receives instructions and an initialized environment. It operates on that environment, and the evaluator checks the final state. The intended target is whether the task was completed, rather than whether the agent followed the reference solution’s command sequence. The paper also distinguishes model–agent combinations: a different agent framework can change the performance of the same model. Paper, §2.1 and §3

A useful result therefore identifies the task version, model, agent framework, resources, number of attempts, and success rate. Harbor helps organize runs and their records. Framework

What works well, and why has it been adopted?

Task completion becomes something executable tests can check. Different approaches can be accepted when they reach an acceptable result. This is a better fit for many work tasks than matching a reference sequence of actions.

Other teams can reuse the environment. Tasks, tools, and records reduce the amount of evaluation infrastructure each team must rebuild. Claude, Gemini, and Kimi reports use Terminal-Bench 2.0; Artificial Analysis has incorporated 4.0 into its index. Claude, Gemini, Kimi, Artificial Analysis

My interpretation is that terminal agents needed a shared test, and the project supplied both tasks and the means to run them. The work behind continued use is also visible: Kelly Buchanan and others organized the 2.1 revision, while Ryan Marten and colleagues developed approaches to continuous maintenance. These are documented contributions, although their individual effects on adoption have not been isolated. 2.1 revision, Continuous Benchmarks

What could improve?

Problem and current status	Possible improvement	How to check whether it helps
Instructions, tests, and environments can disagree. Version 2.1 revised 28 tasks despite earlier reviews. Revision record	Add discovered mistakes, alternative valid solutions, and shortcuts to a regression collection that runs as tasks change.	Measure both valid solutions rejected and invalid solutions accepted. A higher model score alone does not establish better evaluation.
Resources affect results. Anthropic observed score differences when changing resources under controlled conditions; 4.0 already includes resource calibration. Study, 4.0	Build on that calibration with fixed-time or fixed-cost comparisons, showing how much additional success more resources buy.	Hold the model and framework fixed while varying resources. Check ranking stability and resource-related failures.
Public tasks and solutions can leak. Tasks with public solutions were among those removed in 4.0. Release	Keep a public development set and add independently written, periodically refreshed evaluation tasks under comparable conditions.	Check whether improvements carry over to the new tasks. The cost is additional task creation and review.

What to borrow: publish the tools for running, inspecting, and reporting failures alongside the tasks. Quality needs attention after release.

2. BrowseComp: make searching difficult and answers easier to check

What problem does it address?

Doing research includes finding sources, evaluating evidence, analyzing it, and writing. BrowseComp selects a narrower target: finding difficult-to-locate facts on the web and returning a correct short answer. Jason Wei, Zhiqing Sun, Spencer Papay, and colleagues released it in April 2025. Authors’ introduction

How was it built?

Start with a fact supported by evidence. The answer should be short and, as far as possible, stable over time.
Write an inverted question around it. For example, begin with a known event and describe it through several constraints, asking the solver to identify it. This illustrates the method; it is not a benchmark item.
Check that it is difficult. Contributors tried models available at the time and simple searches. Some questions were also attempted by another contributor.
Review the answer and wording. When another valid answer was found, the question could be revised. The paper explicitly acknowledges that this process cannot guarantee a unique valid answer for every question. Paper, §2

This is deliberate selection for difficulty. The resulting distribution is not a natural sample of everyday browsing requests, and the models used during construction help determine which questions survive.

How does evaluation work?

An agent searches and submits a response. The intended scoring procedure extracts its final answer and uses a model to judge whether it means the same thing as the reference answer, then aggregates correctness. A short answer simplifies that comparison. It does not establish that the reference covers every valid answer or that the response’s evidence is sound. A version-specific implementation issue is discussed in the code sidebar. Reference implementation

The design makes it easier to answer “Was the final answer right?” than “Why did the agent succeed or fail?” If systems use different search services, the result also includes differences between those services.

What works well, and why has it been adopted?

Browsing gets a clear comparison target. Short factual answers are easier to compare than open-ended reports, while the search itself can still require substantial effort. Authors’ introduction

OpenAI, Anthropic, Google, and Moonshot report BrowseComp results. My interpretation is that this clear, reusable test met a need as teams started comparing browsing agents. The publisher’s reach may have helped visibility, but the available records do not quantify how much each factor contributed. OpenAI, Anthropic, Google, Moonshot

What could improve?

Problem and evidence	Possible improvement	How to check whether it helps
The reference answer may not be the only valid answer. The authors acknowledge this risk. Paper, §2.1	Review different answers that include supporting evidence. After expert confirmation, expand the accepted answers or revise the question.	Audit responses marked wrong and count how many satisfy the question. Also check that accepting alternatives does not make the test too permissive.
The live web and search services change. BrowseComp-Plus already addresses part of this problem with a fixed corpus. Follow-up paper	Evaluate a fixed corpus and the live web separately. Use controlled retrieval to compare components, and live browsing to compare complete systems.	Hold one component fixed and vary another to locate the source of a difference. A fixed corpus, however, cannot capture every change on the real web.
A correct answer need not have valid supporting citations. The original scoring focus is the final answer. Code	Report citation support, search calls, cost, and latency separately instead of compressing everything into one score.	Audit whether cited sources support the answers, and compare the rate of correct, supported answers at matched budgets.

BrowseComp-Plus is a useful example of a follow-up design: a fixed corpus and evidence labels make it easier to distinguish failure to retrieve evidence from failure to use it. This is an existing improvement, not a new proposal made here. Paper, Project

What to borrow: narrow the problem enough to score it reliably, then examine what that simplification leaves out.

3. τ²-bench: build a world whose state can be checked

What problem does it address?

Support tasks often require cooperation. An agent can change backend settings, while a user can operate their own device; each side has different information. τ²-bench evaluates this kind of multi-turn interaction. The June 2025 paper is by Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. Paper

How was it built?

The new Telecom domain illustrates the process:

Build the backend and user device. Models help generate databases, tools, and tests, which people refine. The agent operates a backend; a simulated user operates a simulated device.
Define small faults. Each has an initial state, a feasible solution, and conditions for checking that it was resolved.
Combine faults. Avoid conflicting combinations and execute reference actions to verify solvability.
Select evaluation tasks. Sample combinations covering different intents and levels of complexity.
Add policies and user instructions, then review them together. These materials determine what each side knows and can do. Paper, §3.2

How does evaluation work?

The tested agent talks to a model acting as the user, with tools available according to each participant’s role. Criteria are specified by task. The inspected environment evaluator supports comparison against a target database state and checks of particular state conditions; other evaluation modules handle other criteria. Environment evaluator

In the paper, Telecom mainly uses state assertions. The experiments also compare normal cooperation, an agent controlling all tools without a user, and cooperation with a reference action plan provided. These comparisons help locate problems in planning and collaboration. Since information and control also change, the score differences should not be treated as perfectly isolated measurements of a single ability. Paper, §3.3 and §4.2

Repeated success matters. pass^k concerns succeeding on the same task in all k runs. It differs from pass@k, where at least one success among k attempts is sufficient. The former becomes stricter as k increases and is useful for studying service reliability. Metric implementation

What works well, and why has it been adopted?

A conversation produces state changes that can be checked. Reference actions help verify that tasks are solvable, and composing smaller faults makes task construction more systematic.

The experiments can help explain failures. Cooperation and reference-plan comparisons provide more information than an aggregate success rate alone. GPT-5.2 and Gemini 3.1 Pro report results on τ² domains. GPT-5.2, Gemini

My interpretation is that Sierra’s familiarity with service workflows helped turn gaps in single-tool-call tests into concrete tasks. The later τ³ release adds company-document retrieval and voice conditions, explicitly connecting these extensions to deployment experience. τ³ authors’ account

What could improve?

Problem and evidence	Possible improvement	How to check whether it helps
The simulated user affects the result. A model-based user does not necessarily make the same mistakes or communicate like a person. Paper	Vary the user model, wording, and expertise, then calibrate with a smaller set of human interactions.	Check ranking stability and manually attribute failures to the agent, simulator, or task description. Human validation costs more.
A successful final state does not establish that every process requirement was met. Telecom’s state checks have a defined scope. Paper, §3.3	For tasks with genuine process requirements, add checks for authorization, confirmation, or forbidden actions. Report task completion and process compliance separately.	Test trajectories that reach the correct state through an impermissible action, while still accepting different legitimate action sequences.
More combinations may not mean more new capabilities. Reusing the same building blocks in development and evaluation can limit what is learned. This is a risk to test.	Hold out combinations, tools, or business rules separately, and state what each split evaluates.	Check whether improvements survive these held-out conditions, and verify that the new tasks remain solvable.

The repository’s July 2026 v1.0.1 update fixes some banking_knowledge grading issues and states that results for that domain before and after the fix are not directly comparable. Domain, simulator, and grading version all belong in a result description. Repository notice

What to borrow: define a checkable task world before studying interaction, and validate both the simulator and the evaluator.

4. OSWorld-Verified: real software creates value and maintenance work

How was it designed and built?

The original OSWorld framework appeared in 2024; the focus here is the Verified revision released in 2025. Tasks draw on actual software-use scenarios, with files, windows, and application states prepared for each one. Evaluation restores a virtual machine, initializes the task, lets the agent act, and then collects files or application state for checking. Original paper, §2–3

Evaluation is more than screenshot similarity. The framework can inspect application settings and document or spreadsheet contents. Experimental configurations may allow different observations and actions, including screenshots, accessibility information, keyboard and mouse actions, or commands. These conditions need to be reported. Original paper

Verified addresses web changes, initialization failures, ambiguous tasks, and grading issues. XLANG reports processing more than 300 pieces of feedback, with roughly ten people working for two months. This is a count of feedback items, not a count of broken tasks. The repair and re-evaluation contributions of Mengqi Yuan, Zilong Zhou, and others are documented. Revision account

Strengths and adoption

Real applications make the target understandable and allow cross-application tasks. A shared environment saves teams from rebuilding a desktop setup. GPT-5.4 and Sonnet 4.6 use Verified, demonstrating recent adoption for computer-use evaluation. GPT-5.4, Sonnet

My interpretation is that the task definition, reusable environment, and continued maintenance support adoption together. OSWorld 2.0 adds longer workflows in 2026; it is a later design whose results should be reported separately. 2.0 paper

Problems and possible improvements

Problem and current status	Possible improvement	How to check whether it helps
Web changes, loading failures, and incorrect initialization have occurred. Verified account	Check the starting state before each run, preserve environment information, and record unavailable environments separately from normal task failures.	Repeat reference actions on different dates and measure failures without a model involved. Report exclusions rather than silently dropping tasks.
Valid solutions may fall outside a grader’s expectations, and instructions may be ambiguous. Verified account	Build examples of different correct outcomes. Check values, content, and appearance separately, with human review where needed.	Audit automatically failed results and measure false rejections before and after revisions.
Post-processing may affect what is measured. The framework supports actions such as saving files before checking; whether this hides an omitted agent action requires a task-by-task audit. Original paper	Preserve the state before and after evaluator post-processing. If saving is part of the task, check that the agent performed it.	Test trajectories that edit a file without saving. This is an audit proposal, not a claim that every task has this defect.

What to borrow: initialization and result-collection code are part of the evaluation method. Real environments need continuing checks.

5. WebArena: make websites reproducible, then check whether the task is done

What problem does it address?

WebArena asks whether an agent can carry out a request inside a website, including changing stored data. Shuyan Zhou, Frank F. Xu, and colleagues at Carnegie Mellon built working sites for shopping, forums, software collaboration, and content management, with supporting tools and reference sites. Project

The original release dates to 2023. It belongs here because it continued to serve as a comparison point: OpenAI used WebArena for its Computer-Using Agent in January 2025, and WebArena-Verified revisited its evaluation in late 2025. These are different evaluation versions; a score should identify which one was used. Original repository, CUA report, Verified workshop publication

How was it built?

Make the sites runnable by other teams. The repository supplies website environments, setup instructions, task configurations, and agent-running code. It distinguishes demonstration sites from the self-hosted environments required for evaluation, and documents resetting the sites. Repository
Write goals, then vary their details. Authors created 241 task templates and expanded them into 812 requests. Templates specify a goal with replaceable details; instances can still require different actions. Paper, §3.1
Check the answers and the checks. Information-seeking answers were annotated twice, with disagreements resolved by a third annotator. Authors writing evaluation programs executed tasks and inspected states. Paper, §3.2

As an illustration, imagine asking an agent to find a project, add a note with a given title, and return its link. This is an invented example, not a dataset item. A useful test must distinguish finding the project, opening the editor, and actually saving the right note.

How does evaluation work?

The original implementation combines checks selected for each task: compare a returned answer, inspect a URL, or locate and inspect page content. Answer checks include exact matching, required text, and a model-based semantic comparison. Thus “programmatic evaluation” does not mean that every original check is a deterministic test of the database. Pinned evaluator

The design aims to accept different action paths that achieve the goal. A particular checker can still miss part of that goal. For the invented note example, checking only that the editor opened would be too weak; requiring one exact click sequence would be too restrictive.

What did WebArena-Verified change?

Amine El hattami, Megh Thakkar, Nicolas Chapados, and Christopher Pal report auditing all 812 tasks, including ambiguous instructions and misaligned evaluators. This is a concrete maintenance contribution to an existing benchmark. Workshop publication

The revision makes three changes especially useful to study:

Make the expected response explicit. Agents return structured fields describing the operation, outcome, and retrieved data. This helps distinguish reaching a page from completing a retrieval request. Response format
Normalize data before comparing it. Type-aware rules replace loose substring checks and LLM judgments, so equivalent representations can receive the same result. A repeatable rule still needs checking against human judgments. Scoring design
Save evidence that can be scored again. Captured browser–server traffic supports evaluation after the run, without keeping the websites running for that later scoring step. Running the agent still requires an environment. Network evaluation

These changes also alter what is measured. The documentation describes rewriting some open-ended requests into factual retrieval tasks so they can be checked deterministically. That reduces grading ambiguity, but also reduces the test of synthesis. Requirements about rendered content or client-side state need evidence beyond network-level checks. Task reformulation, Trace limitations

The cited repository version lists 812 tasks and a 258-task Hard subset; the earlier workshop abstract lists 137 Hard tasks. Even the subset name needs a version. A full-set score, a Hard-subset score, and an original-WebArena score should be reported separately. Pinned repository, Earlier abstract

What works well, and why has it been adopted?

It gives browser-agent research a reusable experimental setting. Researchers can study planning, recovery, and task completion in working applications. The original repository now recommends AgentLab and BrowserGym for experiments, showing how surrounding tools can carry a benchmark forward. Repository

Its use extends beyond the originating team. OpenAI’s CUA report and Microsoft Research’s Magentic-One study both include WebArena, although their agents and evaluation protocols differ. That is evidence of adoption, not a controlled comparison between the systems. CUA, Magentic-One

My interpretation is that the timing helped: as researchers moved toward agents that take actions, WebArena already offered applications, tasks, and a way to score completion. The original team made the test usable; later framework and evaluator work reduced the burden of continuing to use it. These contributions are visible, but their individual effects on popularity have not been measured.

What could improve?

Problem and current status	Possible improvement	How to check whether it helps
An automated check can miss a requirement. Verified already repairs documented evaluation problems. Audit	Build a collection of valid alternative solutions and convincing failures. Include cases where a request was sent but the intended change did not persist.	Compare evaluator decisions with human inspection and saved state; measure both false passes and false failures. These are proposed audit cases, not claims that Verified currently fails them.
Related tasks can appear on both sides of a split. Magentic-One used a split by template to separate development from testing. Study, §5.1	Keep entire templates together; add held-out workflows and sites when testing broader generalization.	Compare performance on new details within familiar templates, new templates, and new sites. These test different kinds of transfer.
Stable sites cover only part of web use. WebArena-Pro, reported in July 2026, already broadens coverage to 300 tasks across 20 applications, including audio and video. Pro publication	Pair fixed-version tests with controlled changes to layouts, data, and permissions. Broader coverage and robustness to change deserve separate measurements.	Hold task goals constant and measure the drop after each change. Check new tasks remain solvable; do not assume a newer benchmark has inherited the original’s adoption.
Resetting and running sites adds experimental overhead. The original repository requires reset procedures; Verified also provides environment tooling. Original setup, Verified tools	Record initial-state checks and isolate runs that modify the same data. Report environment failures separately.	Repeat selected tasks in different orders and compare sequential with parallel runs. Investigate changes caused by shared state before attributing them to the model.

What to borrow: build the task, the environment, and the evidence for success together. WebArena-Verified adds a second lesson: keeping a useful benchmark trustworthy requires revisiting how it assigns scores.

6. HLE: difficult questions require expert coordination and review

How was it designed and built?

Humanity’s Last Exam was built by CAIS, Scale, and subject experts, with contributors including Long Phan, Dan Hendrycks, and Summer Yue. It targets difficult academic questions with relatively definite answers. Paper

Contributors submit questions, answers, and explanations. Questions are screened for difficulty using several contemporary models, reviewed and revised by people with relevant expertise, and approved by organizers or reviewers. Prizes and authorship opportunities help recruit expert contributions. Paper, §3

Feedback after release also matters. In April 2025, the finalized set was reduced to 2,500 questions after confirmed errors were addressed and readily searchable questions were removed. Final-version account

How is it scored?

Accuracy is the main outcome, while reported confidence helps assess whether models remain confident when wrong. The official method uses a model to extract and judge answers. Even with reference answers, wording, precision, and judge choice can affect edge cases. Tool access and the inclusion of images need separate reporting. Evaluation method

Strengths and adoption

Expert recruitment supplies difficult questions across subjects, while a common format makes integration easier. HLE offers new comparison material when older tests struggle to distinguish strong models. Claude, Gemini, Qwen, and Kimi reports include HLE results. Claude, Gemini, Qwen, Kimi

My interpretation is that demand, expert coordination, common scoring, and distribution all contributed. Expert review can improve quality, but expert authorship does not make errors impossible.

Problems and possible improvements

Problem and evidence	Possible improvement	How to check whether it helps
Errors and directly searchable questions required revisions. Final-version account	Continue accepting evidence-backed challenges, use independent subject review, and retain change records with affected results.	Sample questions for reviewer disagreement, errors, and score changes caused by corrections.
Filtering on model failure may favor the screening models’ weaknesses. The size of this effect needs testing. Construction method	Keep a smaller comparison set of expert questions not selected by model performance, and evaluate models that were not used during screening.	Check whether discrimination remains and whether a few subjects or unusual question types dominate the result.
Closed-answer success does not demonstrate autonomous research. The project states this boundary. Scope	Audit reasoning and evidence on a subset. If the target is research, add separate research or experimental tasks and report them separately.	Measure correct answers with faulty reasoning, and test whether knowledge scores predict performance on the independent tasks. Review adds cost.

What to borrow: check difficulty, correctness, and intended use separately. Making questions harder does not automatically broaden what the score means.

7. GDPval / GDPval-AA: compare work products without treating scores as productivity

How was it designed and built?

Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, and colleagues built GDPval around knowledge work in the U.S. economy. They selected sectors and occupations and asked experienced professionals to create requests, reference materials, and deliverables based on their work. The full set has 1,320 tasks; the public gold subset has 220. Paper, §2

Tasks pass through model-assisted screening and multiple rounds of human review. The original study uses blinded comparisons by occupational experts, who explain their judgments, and also introduces an experimental automated grader. Paper, §2.4–2.5

How is it run and scored?

The model receives a request and materials, then produces documents, spreadsheets, or other files. Reviewers compare those products with reference work. The deliverables resemble real work, but the experiment still supplies particular materials and conditions. Release account

GDPval-AA has a different execution and scoring procedure. Artificial Analysis uses the 220 public tasks, tools provided through Stirrup, and Elo ratings derived from blind pairwise comparisons. This must be distinguished from the original GDPval expert comparisons and win or tie rates. AA methodology

Strengths and adoption

Readers can connect the result to a recognizable work product. Experts help align tasks and quality judgments with occupational expectations. The AA platform adds a cross-model evaluation service; Claude and Gemini reports use GDPval-AA. Claude, Gemini

My interpretation is that, as agents began producing useful files, this evaluation addressed a practical question: is the output usable? The personnel requirements are concrete—professional experience, task creation, review, and grading time.

Problems and possible improvements

Problem and evidence	Possible improvement	How to check whether it helps
The initial tasks generally supply the context upfront instead of testing repeated clarification. Paper, §5	Preserve the original evaluation and add an interactive version with questions, additional materials, and revision requests.	Compare initial quality, revised quality, and user time. Publish separate scores when the task scope changes.
Human judgments differ, and automated grading does not fully replace experts. Paper, §5	Review factual accuracy, calculations, requirement coverage, and presentation separately. Use multiple blinded reviewers and resolve important disagreements.	Include polished but incorrect deliverables and accurate but plain ones, checking that the evaluator distinguishes the relevant qualities.
The covered tasks are not an entire occupation. The study has explicit limits on work types and conditions. Release account	Report human review and correction time, broken down by occupation and task type.	On independent work tasks, test whether total human effort falls while quality stays acceptable. Model generation time alone is not a productivity measure.

What to borrow: work products can make an evaluation more useful, but economic implications need their own validation.

8. ARC-AGI-2: keep the format simple and invest in task design and human testing

How is it designed and scored?

François Chollet, Mike Knoop, Gregory Kamradt, and colleagues retain the input–output grid format: show examples, provide a new input, and ask the system to construct the corresponding output. Discrete colors and bounded grids make the output directly checkable. Paper, §3

The project tests tasks with human participants, using completion rates and times to examine difficulty and calibrate subsets. Public tasks, held-out evaluation tasks, competition rules, and resource conditions form the broader project. Attempt limits and budgets should be stated for the particular competition or leaderboard being cited. Paper, §4, Release and competition

Strengths and adoption

The interface is simple and the question is clear: can a system use a few examples to handle a new rule? Retaining the format also supports reuse of existing tools. Competitions and public resources provide ways to participate; Gemini and Sonnet 4.6 report ARC-AGI-2 results. Gemini, Sonnet

My interpretation is that a clear research target and sustained organization helped build a shared test. Grid performance and competition activity do not, on their own, establish general intelligence.

Problems and possible improvements

Problem	Possible improvement	How to check whether it helps
Extensive search can raise scores under different costs. The authors explicitly discuss brute-force search and efficiency. Paper	Compare matched cost, time, and attempt limits alongside best-achieved scores.	Check whether a method remains better across budgets, rather than only in its most expensive configuration.
Human-solvable grids remain a limited task family.	Test independently designed rule families and other interaction tasks, stating which abilities are expected to transfer.	Check whether improvement extends beyond familiar grid patterns and interfaces.
Repeated feedback on a held-out set can encourage adaptation to that set. The paper raises this concern about the earlier benchmark. Paper	Limit repeated probing and periodically introduce independently created, difficulty-calibrated tasks.	Compare old and new sets with difficulty references. Maintenance and historical comparison become more expensive.

ARC-AGI-3 already moves toward interactive environments. That is an existing extension, but its different tasks need separate analysis; a new release does not itself prove transfer. ARC-AGI-3 release

9. LiveCodeBench: make task dates part of the experiment

How is it designed and scored?

Naman Jain and colleagues collect contest problems from platforms including LeetCode, AtCoder, and Codeforces and record their publication dates. In code generation, models receive a problem statement and examples, and unseen tests check their programs. The project also includes repair, execution, and test-output prediction scenarios. A single result in a model report should not be assumed to cover all of them. Paper

The v6 release used by Qwen3.5 and Kimi K2.5 is documented as containing 1,055 problems from May 2023 through April 2025. The repository also distinguishes a faster, reduced-test configuration from the full tests. Repository, Qwen, Kimi

LiveCodeBench Pro is a separate work. Zihan Zheng, Zerui Cheng, and colleagues involve programming-contest experts in analyzing tasks and failed submissions. Gemini 3.1 Pro reports Pro results. Authors, tasks, and scores should not be merged with v6. Pro paper, Gemini

Strengths and adoption

New contests provide a continuing source of tasks, and timestamps make contamination questions easier to examine. Executable tests also make comparison practical. My interpretation is that this addresses concern about models seeing old coding questions while remaining easy to integrate into established evaluation workflows.

Problems and possible improvements

Problem	Possible improvement	How to check whether it helps
A recently released model may have seen older problems. The v6 window ends in April 2025; “Live” is not a guarantee against contamination. Repository	Report task dates and, where possible, evaluate on problems known to postdate training data or a frozen model. Mark unknown training coverage explicitly.	Compare difficulty-matched periods and check duplicates or reformulations. A gap between old and new tasks can also reflect difficulty, so it is not proof of contamination by itself.
Passing depends on test coverage, and reduced and full test sets differ. Repository	Add boundary tests to a subset and use deliberately altered, incorrect programs to probe the tests.	Compare disagreements between full and reduced testing, while checking that added tests do not reject valid solutions.
Contest performance does not directly measure maintenance, requirements discussion, or deployment.	Report repository-level work separately rather than treating all coding tasks as one ability.	Check whether gains survive on independent software-engineering tasks. More contest problems alone do not expand the scope.

What to borrow: dates, sources, and test settings belong in the score description. “New” and “live” cannot substitute for validation.

What these designs have in common

These benchmarks did not succeed through one identical design. BrowseComp makes answers easier to judge; τ² builds a controlled interaction world; Terminal-Bench, OSWorld, and WebArena supply task environments; HLE and GDPval rely on expert coordination. Each turns an important question into a test other teams can use.

Each useful choice also creates a cost:

Design choice	What it enables	What still needs checking
Short answers or automated tests	Lower-cost comparison	Incomplete references, mistaken grading, and missed process requirements
Real websites and applications	Tasks closer to actual use	Environment changes, initialization, and reproducibility
Selecting questions that defeat current models	Better separation of contemporary models	Model-specific weaknesses and task representativeness
Public data, code, and reference solutions	Research, reproduction, and adoption	Exposure during training or browsing
Frequent task and grader revisions	More reliable results and longer useful life	Versions, rerun costs, and comparability with old results

The useful question is specific: which design choice solved which problem, what did it cost, and how can that cost be measured? Labels such as “realistic,” “hard,” and “comprehensive” need this explanation behind them.

Three directions worth testing

These are research suggestions drawn from the cases, not claims of new methods or demonstrated improvements.

Audit evaluators systematically. Assemble alternative valid solutions, superficially convincing failures, and trajectories that finish through invalid actions. Have experts label them, then measure false rejections, false acceptances, review cost, and performance across task types.
Separate models, tools, and execution conditions. Keep tasks fixed while changing the agent framework for one model, or the model inside one framework. Control budgets and environments to explain where differences come from.
Evaluate controlled conditions and real change together. For example, test a fixed corpus and the live web separately, then examine whether the results agree. Report both stability and the range of real situations the evaluation covers.

A small check of the grading code

Examining a pinned BrowseComp reference implementation reveals a concrete mismatch. At commit 652c89d, the parser returns correct: yes, while its caller compares the result with yes. Along that path, even a positive mock judgment is not counted as correct. Pinned source, lines 92–108

The function was extracted from that source and checked with mock grader outputs. This involved no model calls, benchmark questions, or full benchmark run. The essential string comparison is:

import re

reply = "correct: yes"                 # Synthetic grader output
parsed = re.search(r"correct: (yes|no)", reply).group(0)
print(parsed == "yes")                 # False: parsed still includes "correct: "

This finding concerns that reference-code version. It does not establish that any paper or model provider used the same path, or that its published scores are wrong. The practical lesson is to test evaluation code with known positive, negative, and unparseable outputs alongside reviewing the questions themselves.

Sources and scope

Methods are described for the cited versions, with later revisions identified separately. Explanations of adoption are based on public records rather than author interviews or causal comparisons. Proposed improvements include ways to test them, but have not been validated here. Apart from the local parser check, no full benchmark was reproduced, and no benchmark questions or answers are reproduced in this article.

The main sources are grouped below; links within each case point to the evidence for individual claims.

Terminal-Bench: paper, timeline, tasks, Harbor, 2.1 revisions, 4.0 release, continuous maintenance, infrastructure experiment.
BrowseComp: paper, authors’ introduction, pinned evaluation code, BrowseComp-Plus paper, follow-up repository.
τ-bench family: τ² paper, environment evaluator, metrics, repository and version notes, τ³ release.
OSWorld: original framework, Verified revision, 2.0 paper.
WebArena: project, original paper, repository, pinned evaluator, CUA adoption, Magentic-One and its split.
WebArena extensions: Verified workshop publication, pinned Verified repository, response format, normalization, network evaluation, Pro publication.
HLE: original paper, final-version and evaluation details.
GDPval: paper, release, GDPval-AA methodology.
ARC: ARC-AGI-2 paper, release and competition, ARC-AGI-3 release.
LiveCodeBench: original paper, repository and versions, Pro paper.
Adoption records: GPT-5.2, GPT-5.4, GPT-5.6, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro, Qwen3.5, Kimi K2.5, Artificial Analysis index.

The AI says “done.” How do we check?

2026-09-09T23:14:53-07:00

Suppose an agent is asked to research a topic, compare the options, and produce a report. A few minutes later, it replies: “Done.”

That is a delivery notification. To find out how well the work was done, someone still needs to open the file and check the sources and conclusions.

“Done” is not a passing grade.

This is a useful starting point for talking about LLM and LLM-agent benchmarks: the tasks, environments, and grading rules used to measure capability. Two terms come up often: verifiable and unverifiable.

They sound abstract. A simpler way in is to ask how the assignment gets graded.

Verifiable: there is a reasonably clear, repeatable way to check the result.

A math answer can be checked against a known answer. Code can be run against functional tests. If an agent changes an order, the result can be checked in the database.

The scope of that check matters. Passing the available code tests means those checks passed. It does not guarantee correctness in every situation.

Unverifiable: within the available time and budget, there is no reliable way to automatically assess the overall result.

Does a report offer useful insight? Is an explanation right for its audience? Has a proposal considered the important constraints? People can judge these things, and explicit criteria can help. But a cheap, consistent overall assessment is harder to obtain than checking a number.

“Unverifiable” does not mean impossible to assess forever. It describes the verification conditions currently available.

And how hard an assignment is and how easy it is to grade are different questions. A difficult programming problem can have clear tests. People may disagree about a single line of copy.

Different parts of the same assignment can also be graded separately. “Write three paragraphs” can be checked automatically. Whether those paragraphs explain the topic clearly needs another kind of evaluation. IFEval uses checkable instruction constraints to evaluate models. Meeting length and formatting requirements still leaves content quality to assess. IFEval

Before labeling an entire task, it helps to break it down: which parts can be checked directly, which require evidence, and which need informed judgment?

Several recent developments follow from this distinction.

First, checks are becoming training feedback.

DeepSeek-R1-Zero uses rule-based rewards, including math-answer checks and code tests. Models can make repeated attempts and adjust their behavior using that feedback. A verifier can therefore serve both evaluation and training. DeepSeek-R1

Good task designs and verification mechanisms may support both practice and exams. The practice questions and exam questions still need to be separate.

A high score on repeatedly practiced questions does not, by itself, establish performance on unseen tasks. Knowing the exam questions and transferring a skill require different evidence.

Second, agent evaluation needs to inspect what actually happened.

If an agent says a record was updated, check the record. If it says a bug was fixed, inspect the patch and test results.

WebArena evaluates completion of web tasks. τ-bench brings together simulated users, tools, and domain rules, and compares the final database state with the target state. These approaches give evaluation an observable result in the environment. WebArena, τ-bench

If a task also restricts how the work may be done, the action record needs checking. Updating the intended record while breaking other records should show up in the evaluation.

Another easy detail to miss: an agent’s score reflects the model, tools, prompts, memory, and retry strategy together. Model comparisons should control these conditions where possible. Complete-system comparisons should disclose the configuration and cost.

Otherwise, one system might have search, code execution, and many retries while another gets a single response. Crediting the entire difference to the model would misattribute the result.

Third, open-ended tasks are moving from impression-based scores toward specific grading criteria.

A rubric is essentially an acceptance checklist. For a research report, it might ask:

Did it answer the user’s key questions?
Do sources support the important conclusions?
Did it explain conflicts between sources?
Can the reader understand the conclusions and trade-offs?

QuRL, published at ICLR 2026, explores constructing question-specific rubrics from web material and using them as reinforcement-learning feedback for open-ended question answering. This offers more specific learning signals for open-ended tasks. QuRL

But a checklist can omit something important. A judge can also award points that an answer has not earned.

Having an LLM grade another LLM’s work is convenient, but it still needs calibration. Research on LLM-as-a-judge has documented biases involving answer position and length. Fluent language does not automatically establish factual correctness. LLM-as-a-Judge

A useful follow-up is to change the tasks, change the judge, and bring in independent human evaluation. Does the improvement hold up?

There is one more exam to administer: the grader’s.

This applies to programmatic tests, too.

In 2026, OpenAI’s audits of SWE-bench Verified and SWE-Bench Pro reported mismatches between task descriptions and tests. Some tests required functionality the task had not specified; others depended too heavily on a particular implementation. A rejected solution may be wrong, but the acceptance criteria may also be at fault. Verified audit, Pro audit

Building a benchmark can therefore begin by testing its ability to grade.

A practical development process looks like this:

Write the acceptance criteria before collecting a large question bank. Narrow “Can this agent do research?” into a concrete task, such as checking information and producing a comparison report using fixed materials and a fixed budget. Specify the inputs, tools, deliverables, and success conditions. Start with a small pilot, have someone else run it independently, and fix the problems before expanding.
Grade different parts of the assignment separately. File creation and formatting can be checked automatically. Whether a citation supports a conclusion requires consulting the source. The quality of an analysis may need a rubric calibrated against human judgments. Preserve the component scores; attractive formatting should not hide a critical factual error.
Test the grader first. Give it correct answers expressed in different ways, as well as polished answers with substantive errors. Check whether it rejects valid solutions or accepts invalid ones. Independently review reference answers, allow reasonable alternative solutions, and record disagreement among human evaluators.
Give the agent a reproducible exam environment. Reset each run to the same initial state, record the tools and resource conditions, and isolate hidden tests from the workspace the agent can modify. Record environment failures separately. A power outage in the exam room is not evidence that the student cannot solve the problem.
Keep practice and exam questions separate. Group related tasks derived from the same document, repository, or template according to the generalization being measured. LiveCodeBench’s ongoing collection of new problems offers one approach to reducing contamination. Fresh questions still need deduplication and source checks. LiveCodeBench
Publish more than the highlight reel. Run multiple trials and report success rates, variation, error categories, and total cost. Disclose model versions, tool configurations, and retry rules. Succeeding once and succeeding reliably are different user experiences.

That last point directly affects whether an agent can be trusted to complete everyday tasks.

τ-bench’s pass^k measures success across all of several trials, which differs from succeeding at least once across many attempts. A highlight reel shows what the system has managed to do. Everyday use also depends on how consistently it can do it. τ-bench

A useful next research question is: can an agent recognize when it should double-check?

If a tool response is ambiguous, inspect the actual state. If sources conflict, investigate further. If the evidence is insufficient, preserve that uncertainty. These actions may improve the deliverable, but they also take time and money. Whether they are worth it needs to be measured.

One starting point is to apply automatic checks, model judges, and combinations of evidence to the same deliverables under comparable evaluation budgets. Independent human evaluations and held-out tasks can then test which scores better predict actual deliverable quality.

A useful benchmark should make clear what the system accomplished, what evidence supports that judgment, how consistently it succeeds, and what it cost.

Then, when an agent next says “Done,” there is somewhere to look for the proof.

Turning a thought into an article

2026-09-09T00:00:00-07:00

Some ideas arrive as little more than a loose outline. They might begin with a sentence you read, a question you haven’t resolved, or a small detail worth remembering. Writing them down doesn’t have to mean finding an answer right away. It can simply be a way to slow down and take a closer look.

This is a sample article for previewing the layout. Use it to explore paragraphs, images, quotations, lists, and code blocks before replacing them with your own writing.

Start with a small question

An article can start somewhere small: what made you pause today? Which assumption feels a little less certain now that you have looked at it again?

You don’t have to explain everything in the opening paragraph. Start with the question and follow it forward. Often, what a reader needs is a line of thought they can follow.

Writing gives a thought that hasn’t taken shape a little room to grow.

Figure 1 · A thought can unfold a little at a time.

Give paragraphs room to breathe

In a longer article, headings act as signposts. They help readers find their way, and they give the writer a chance to ask: what is this section really trying to say?

Three simple habits can help:

Keep a concrete observation. A small example is often easier to understand than an abstract conclusion.
State the question clearly. It is worth recording even before you have an answer.
Leave yourself a way back in. The next time you return, you can pick up the thought and keep going.

Occasionally, a note moves between languages. Both versions share the same reading width and spacing, so quotations and technical explanations feel at home in either language.

When an article includes code

Code has its own space, preserving indentation and syntax colors. Longer lines scroll inside the code block.

notes = ["Observe", "Ask", "Write", "Reconsider"]

for step, note in enumerate(notes, start=1):
    print(f"{step}. {note}")

You can also include inline code in a sentence to describe a variable, a filename, or a short command.

Don’t rush toward a conclusion

A note does not have to answer every question. Sometimes it helps you ask a question more clearly. Sometimes it leaves you with a direction worth exploring.

That is already a beginning. When something new comes along, return to the page and add another paragraph.

This sample content demonstrates the blog’s reading layout.