最前線と中心：評価を誰が評価するのか？（12 分読了）

Editor’s note: Some of the most interesting questions in AI are being asked by information theoreticians, around how to provide context to an emerging class of AI agents. A few weeks ago, we waded into those waters with a blog about the Open Knowledge Format, a specification that formalizes the LLM-wiki pattern into a portable, interoperable format to represent the metadata, context, and curated knowledge that modern AI systems need to operate. That blog generated a ton of interest, so we’ve decided to bring you more of the same, as part of our new “Frontier and Center” series. Today, we hear from two members of Google Data Cloud’s frontier AI team on the recurring challenge of how to systematically evaluate whether or not an agent is able to answer questions effectively based on its context. Read on for more, and watch this space for more blogs from this team.

A passing grade is the least interesting thing an exam can tell you. It says the student cleared the bar; leaving you entirely in the dark about how narrow their failures were, how effortless their passes were, or what to teach next. Yet this is exactly how we evaluate AI agents. We run a fixed benchmark, calculate a score, and declare progress. In doing so, we are handing our agents a pass/fail exam when what we actually need is a map of the agent’s capabilities: a picture of the terrain that shows exactly where capability falls off, and by how much.

For data agents, this map matters a lot for data discovery in search and retrieval — the unglamorous first step where an agent, handed a vague human question and a warehouse or data lake of thousands of tables and files, has to find the right datasets before it can reason over anything. Discovery is a "needle in a haystack" problem. Real users phrase their questions imperfectly, and inferring what datasets to retrieve presents a real challenge to agents. So the interesting question in evaluations is never "can the agent pass?" It is "how vague can the question get before the agent breaks?" An exam cannot easily answer that, but a map can.

Today, we share an approach rooted in information theory that we’ve been leveraging to add detail and nuance, i.e., fidelity, to benchmarks, so we can better understand agents’ performance as a part of their evaluations. Along the way, the added fidelity exposed some deeper issues with the quality of emergent evaluation cases themselves.

Difficulty, measured

When it comes to retrieval, evaluation cases are often stratified into tiers of difficulty. This can happen organically, e.g., pervasive and enduring failure scenarios are deemed difficult. Or it can be from labels applied by humans or machines categorizing some questions as "easy" or "hard" for an agent to answer correctly, e.g., based on the context provided in the query. While this kind of sentiment-based labeling is not the only way to label test cases, it’s frequently used despite its imperfections, such as being challenging to reproduce.

Despite being an industry staple, the approach of assessing every evaluation case by hand is unrealistic at scale. What we need is a rigorous approach that can modulate the difficulty of evaluation cases. We’re iterating on a meta-benchmark we call Discovery Bench: a framework that modulates an evaluation case by generating “easy” and “hard” variations of every case. This allows us to audit how close or how far an agent is from succeeding in those cases.

The lever for modulating the difficulty of an input query comes via a tried-and-trusted concept that’s present across information theory and machine learning: surprisal, or the likelihood of an output given a set of inputs. In our case, a query’s surprisal represents the uncertainty that remains about the correct dataset given the query.

The thinking behind our approach is simple: A term or a phrase in an evaluation query has high informative power when it sharply distinguishes the target from everything else in the corpus. Therefore, we can adjust the difficulty of evaluation cases by adding or removing terms with varying levels of informative power.

Let’s work through a real example from KramaBench, a publicly available benchmark. One of KramaBench’s datasets has information about orbiting satellites, and the example query from the suite includes the following text: "…the total count of satellite major altitude changes for satellite 48445 during 2024 using TLE history."

The token "TLE" is sharply distinguishing; it points almost uniquely at the TLE_____48445 table from the dataset. Strip it, and the query degrades to "the count of satellite altitudes for satellite 48445," whose vague phrasing now matches density tables, precise-orbit files, and decay logs alike. Surprisal makes this quantitative: rare, pointed terms carry more bits than common ones.

The remaining surprisal of a query is how much uncertainty is left about its answer. As surprisal approaches zero, the query has become specific enough to pinpoint exactly one dataset.

The heart of the idea behind Discovery Bench is this refinement loop, which we call iterative surprisal-based query refinement, or iSQR, which generates cases with higher or lower informative power to test where an agent can start successfully answering the query:

Figure 2: The iSQR refinement loop.

The crux is being able to control the challenge embedded into the evaluation case by making adjustments: Instead of one fixed phrasing per question, we generate the same question at three levels of calibrated ambiguity [high, medium, low], with each grounded in bits (not subjective opinion). We can even justify, term by term, why a word was added or removed. Difficulty stops being a property that is attributed by sentiment or classification, and becomes one we engineer.

The cliff you couldn't see

Here is what Discovery Bench’s difficulty dial reveals — and what a single-phrasing benchmark structurally cannot.

We have an F1 agent that's built for recall (on Gemini 3.1 Pro). Running it against KramaBench and across the full sweep of ambiguity levels traces a curve: 0.34 at high ambiguity, 0.76 at neutral, 0.81 at medium, 0.78 at low.

Figure 3: F1 swept across ambiguity — the dot versus the curve.

Two findings fall out immediately (and neither were visible to a conventional eval).

First, the cliffs. This query scores a perfect F1 = 1.00 at neutral phrasing — and 0.00 at high ambiguity. It is the satellite-48445 case from above: drop the distinguishing token "TLE" and the agent loses the table entirely. Same query, same agent, same ground truth; one notch vaguer and it falls off a cliff. A static benchmark tests the neutral phrasing, stamps "solved," and reports flat ground where there is a precipice. Pass/fail was particularly misleading in that it did not just miss the cliff, but it told us the terrain was level.

Second, the sweet spot. For Discovery Agent, medium ambiguity beat neutral, and low ambiguity sometimes underperformed it. More specificity is not monotonically better for the system being evaluated; there is an optimal amount of steering. That is a graded, actionable signal. This is the "how close, how hard" texture we were missing from a scalar. It tells you where to hill-climb, or improve, the agent: in our case, straight at concrete failure modes like time-sharded tables (precision collapsing to ~8% as the agent over-retrieves 21 near-identical shards for a two-table answer) and context blow-up (F1 dropping from 0.75 to 0.32 once a query triggers long search chains). The map did not just say that the agent failed, but it said where, and why. Note that our hypothesis that less ambiguity and more context (via steering terms) should improve retrieval generally holds true, but for the specific Discovery Agent being exercised, the idiosyncratic “sweet spot” meaningfully highlighted trade-offs in its implementation.

We're not alone

The field is converging on meta-benchmarking and exerting greater control of how we challenge and evaluate our agents. A growing body of work uses item response theory, the latent-ability model behind standardized testing, to treat difficulty as a measured quantity rather than a label: tinyBenchmarks and metabench show that a handful of informative items reproduce a model's full score, and PSN-IRT turns the same lens on benchmark quality itself. Others audit the ground truth directly: MMLU-Redux found that 6.49% of Massive Multitask Language Understanding (MMLU) questions are mislabeled, and Platinum Benchmarks re-cleaned ten datasets to minimize both label errors and ambiguity — the same two axes we sweep for. And ambiguity is increasingly treated as intrinsic rather than noise: AmbigQA showed that a large fraction of real questions admit multiple readings, and later work finds that apparent hallucinations often stem from query ambiguity rather than model failure. What we have not seen elsewhere is the combination: information-theoretic ambiguity sweeping applied as a meta-benchmark over live enterprise data.

A benchmark we trusted turned out to be broken

We built our first evaluation on kramabench-astronomy, a benchmark established in the field, and one which other teams had already leaned on for their own evals. Teams derived benchmarks from this dataset, and we hypothesized subtle issues may have been introduced over time. When we actually read the benchmarks used by teams, with Gemini's help, we found it was wrong in meaningful ways: ground-truth tables that did not answer their query, a question whose 124 sharded tables exceeded what some teams’ retrieval APIs could even return, months specified where exact dates were required. Quietly broken ground truth means quietly wrong conclusions not just for us, but for every prior analysis built on it.

This is the generalized crux of the matter: an evaluation is itself an artifact that can be defective, and almost nobody evaluates it. We instrument the agent and trust the ruler, but where do we validate that the measuring stick makes sense?

When two maps disagree

Now the recursive turn: If difficulty is something we generate, then we need to evaluate the generator itself; we should not trust it blindly either.

So we built the same ambiguity sweep two ways: steering terms from a pure-LLM guess, versus terms grounded in TF-IDF surprisal. The two disagreed violently. At high ambiguity, the LLM-built sweep scored the agent at F1 ≈ 0.34; the grounded sweep, ≈ 0.85. One of these maps is badly distorted. The grounded one, predictably, is the more robust: surprisal gives it a footing the free-running LLM lacks.

This is "evaluate your evals," made concrete. The information-theoretic lens does not only grade the agent along a continuous axis; it grades the benchmark's own construction, and adjudicates between the two.

Evaluate your evals

We have spent years optimizing agents against rulers we never measured. The bitter irony is that better models make this worse: as agents clear coarse benchmarks, the score saturates near the top and the exam loses its ability to highlight where the agent can be improved.

So the call to action is uncomfortable and overdue: evaluate your evals. Read your ground truth. Treat difficulty as a measured quantity, not a label: sweep it, plot it, find the bit-width where your system breaks. Ask not just "did it pass?" but "how close was the miss, how hard was the pass, and would a slightly vaguer question have sent it off a cliff?" Build evaluations that produce signals; not just verdicts.

There is a genuine tension to sit with here. Difficulty-as-entropy is only as reliable as the model that estimates the entropy. There's a risk that if we push too hard on a measurable proxy, we optimize the ruler instead of the agent. That is not a reason to retreat to pass/fail; it is a reason to keep the evaluator under the same scrutiny as what it is evaluating. The moment we stop asking who evaluates the evaluators is the moment our maps stop being useful again.

*1. Maia Polo, F. et al. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML 2024. arxiv.org/abs/2402.14992

**2. Kipnis, A. et al. metabench: A Sparse Benchmark of Reasoning and Knowledge in Large Language Models. ICLR 2025. arxiv.org/abs/2407.12844

**3. Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory (PSN-IRT). AAAI 2026. arxiv.org/abs/2505.15055

**4. Gema, A. P. et al. Are We Done with MMLU? (MMLU-Redux). 2024. arxiv.org/abs/2406.04127

**5. Vendrow, J. et al. Do Large Language Model Benchmarks Test Reliability? (Platinum Benchmarks). 2025. arxiv.org/abs/2502.03461

**6. White, C., Dooley, S. et al. LiveBench: A Challenging, Contamination-Limited LLM Benchmark. 2024. arxiv.org/abs/2406.19314

**7. Min, S. et al. AmbigQA: Answering Ambiguous Open-domain Questions. EMNLP 2020. aclanthology.org/2020.emnlp-main.466

**8. Lai, E., Vitagliano, G. et al. KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes. 2025. arxiv.org/abs/2506.06541*

Posted in- Data Analytics

• AI & Machine Learning
• Application Development