The AI says “done.” How do we check?

Suppose an agent is asked to research a topic, compare the options, and produce a report. A few minutes later, it replies: “Done.”

That is a delivery notification. To find out how well the work was done, someone still needs to open the file and check the sources and conclusions.

“Done” is not a passing grade.

This is a useful starting point for talking about LLM and LLM-agent benchmarks: the tasks, environments, and grading rules used to measure capability. Two terms come up often: verifiable and unverifiable.

They sound abstract. A simpler way in is to ask how the assignment gets graded.

Verifiable: there is a reasonably clear, repeatable way to check the result.

A math answer can be checked against a known answer. Code can be run against functional tests. If an agent changes an order, the result can be checked in the database.

The scope of that check matters. Passing the available code tests means those checks passed. It does not guarantee correctness in every situation.

Unverifiable: within the available time and budget, there is no reliable way to automatically assess the overall result.

Does a report offer useful insight? Is an explanation right for its audience? Has a proposal considered the important constraints? People can judge these things, and explicit criteria can help. But a cheap, consistent overall assessment is harder to obtain than checking a number.

“Unverifiable” does not mean impossible to assess forever. It describes the verification conditions currently available.

And how hard an assignment is and how easy it is to grade are different questions. A difficult programming problem can have clear tests. People may disagree about a single line of copy.

Different parts of the same assignment can also be graded separately. “Write three paragraphs” can be checked automatically. Whether those paragraphs explain the topic clearly needs another kind of evaluation. IFEval uses checkable instruction constraints to evaluate models. Meeting length and formatting requirements still leaves content quality to assess. IFEval

Before labeling an entire task, it helps to break it down: which parts can be checked directly, which require evidence, and which need informed judgment?

Several recent developments follow from this distinction.

First, checks are becoming training feedback.

DeepSeek-R1-Zero uses rule-based rewards, including math-answer checks and code tests. Models can make repeated attempts and adjust their behavior using that feedback. A verifier can therefore serve both evaluation and training. DeepSeek-R1

Good task designs and verification mechanisms may support both practice and exams. The practice questions and exam questions still need to be separate.

A high score on repeatedly practiced questions does not, by itself, establish performance on unseen tasks. Knowing the exam questions and transferring a skill require different evidence.

Second, agent evaluation needs to inspect what actually happened.

If an agent says a record was updated, check the record. If it says a bug was fixed, inspect the patch and test results.

WebArena evaluates completion of web tasks. τ-bench brings together simulated users, tools, and domain rules, and compares the final database state with the target state. These approaches give evaluation an observable result in the environment. WebArena, τ-bench

If a task also restricts how the work may be done, the action record needs checking. Updating the intended record while breaking other records should show up in the evaluation.

Another easy detail to miss: an agent’s score reflects the model, tools, prompts, memory, and retry strategy together. Model comparisons should control these conditions where possible. Complete-system comparisons should disclose the configuration and cost.

Otherwise, one system might have search, code execution, and many retries while another gets a single response. Crediting the entire difference to the model would misattribute the result.

Third, open-ended tasks are moving from impression-based scores toward specific grading criteria.

A rubric is essentially an acceptance checklist. For a research report, it might ask:

Did it answer the user’s key questions?
Do sources support the important conclusions?
Did it explain conflicts between sources?
Can the reader understand the conclusions and trade-offs?

QuRL, published at ICLR 2026, explores constructing question-specific rubrics from web material and using them as reinforcement-learning feedback for open-ended question answering. This offers more specific learning signals for open-ended tasks. QuRL

But a checklist can omit something important. A judge can also award points that an answer has not earned.

Having an LLM grade another LLM’s work is convenient, but it still needs calibration. Research on LLM-as-a-judge has documented biases involving answer position and length. Fluent language does not automatically establish factual correctness. LLM-as-a-Judge

A useful follow-up is to change the tasks, change the judge, and bring in independent human evaluation. Does the improvement hold up?

There is one more exam to administer: the grader’s.

This applies to programmatic tests, too.

In 2026, OpenAI’s audits of SWE-bench Verified and SWE-Bench Pro reported mismatches between task descriptions and tests. Some tests required functionality the task had not specified; others depended too heavily on a particular implementation. A rejected solution may be wrong, but the acceptance criteria may also be at fault. Verified audit, Pro audit

Building a benchmark can therefore begin by testing its ability to grade.

A practical development process looks like this:

Write the acceptance criteria before collecting a large question bank. Narrow “Can this agent do research?” into a concrete task, such as checking information and producing a comparison report using fixed materials and a fixed budget. Specify the inputs, tools, deliverables, and success conditions. Start with a small pilot, have someone else run it independently, and fix the problems before expanding.
Grade different parts of the assignment separately. File creation and formatting can be checked automatically. Whether a citation supports a conclusion requires consulting the source. The quality of an analysis may need a rubric calibrated against human judgments. Preserve the component scores; attractive formatting should not hide a critical factual error.
Test the grader first. Give it correct answers expressed in different ways, as well as polished answers with substantive errors. Check whether it rejects valid solutions or accepts invalid ones. Independently review reference answers, allow reasonable alternative solutions, and record disagreement among human evaluators.
Give the agent a reproducible exam environment. Reset each run to the same initial state, record the tools and resource conditions, and isolate hidden tests from the workspace the agent can modify. Record environment failures separately. A power outage in the exam room is not evidence that the student cannot solve the problem.
Keep practice and exam questions separate. Group related tasks derived from the same document, repository, or template according to the generalization being measured. LiveCodeBench’s ongoing collection of new problems offers one approach to reducing contamination. Fresh questions still need deduplication and source checks. LiveCodeBench
Publish more than the highlight reel. Run multiple trials and report success rates, variation, error categories, and total cost. Disclose model versions, tool configurations, and retry rules. Succeeding once and succeeding reliably are different user experiences.

That last point directly affects whether an agent can be trusted to complete everyday tasks.

τ-bench’s pass^k measures success across all of several trials, which differs from succeeding at least once across many attempts. A highlight reel shows what the system has managed to do. Everyday use also depends on how consistently it can do it. τ-bench

A useful next research question is: can an agent recognize when it should double-check?

If a tool response is ambiguous, inspect the actual state. If sources conflict, investigate further. If the evidence is insufficient, preserve that uncertainty. These actions may improve the deliverable, but they also take time and money. Whether they are worth it needs to be measured.

One starting point is to apply automatic checks, model judges, and combinations of evidence to the same deliverables under comparable evaluation budgets. Independent human evaluations and held-out tasks can then test which scores better predict actual deliverable quality.

A useful benchmark should make clear what the system accomplished, what evidence supports that judgment, how consistently it succeeds, and what it cost.

Then, when an agent next says “Done,” there is somewhere to look for the proof.