Arena、ファクトアレーナにおける事実性の検証と評価手法を公開

Back to All Articles

Research

Product

Arena

Team

• Published on 14 Jul 2026
• Last updated on 31 Jul 2026
• Reading Time: 8 minutes

Factuality remains one of the most persistent questions users face when using AI models. Today, we are launching a leaderboard that ranks models not only by human preference, but also by the factual accuracy of their responses.

Human preference has been at the core of Arena’s model rankings from the beginning. But some aspects of model quality are difficult for humans to evaluate. One of them is factuality—the correctness of claims in a model’s response. Manually fact-checking claims is slow and labor-intensive, so factuality may not always be reflected in human preference votes. That is why today we are adding factuality in the Arena: a new ranking of models according to a weighted combination of human preference and factuality, two complementary signals that tell only a partial story in isolation.

We are initially adding factuality into our Text and Search Arenas. This will start as a non-default toggle which can be selected in our leaderboard UI.

##

First, we randomly sample battles for a factuality audit. Then, we extract the atomic claims from each model’s response. We filter for claims that are reasonably web-verifiable. Each atomic claim is then verified by a system of search agents that gives calibrated truth probabilities. Calibration is achieved by post-processing raw probabilities based on a high-quality dataset of verified factuality annotations. We then compare the average probability of claim correctness between each response. The model with higher average claim truthfulness wins the battle; the higher the margin the stronger the win.

Using the labeled “factuality battles” we construct a unified leaderboard that reflects both human preference and factuality with our composite Bradley-Terry model. The model gives us control over the weight between factuality labels and pure human preference labels. Our default factuality weight is 25%. This means models which are not factual will be essentially unable to top the leaderboard. Likewise, models that never produce false claims, but are otherwise unhelpful to humans, will also be unable to top the leaderboard.

##

To power these rankings, we’ve labeled over 2 million claims made by LLMs in real-world conversations, 1.3+ million from Text Arena, and 700k+ from Search Arena. These span roughly 130k Text Arena battles and 40k Search Arena battles. In Text Arena battles, a claim was found in at least one response 76% of the time; in Search Arena 88% of the time. In Text Arena, models averaged 5 claims per response; in Search Arena models averaged nearly 10. The marginal true claim rate in Text Arena was 87%, and 89% in Search Arena. This is assisted by Arena’s AutoModality classifier, which automatically routes prompts that likely need web access to Search Arena.

We can plot each model’s score against the factuality weight. Weight 100% means that the score is purely factuality-based, 0% means it is purely human-preference-based. Different models have different slope. Most top models (by human preference score) have decreasing score as factuality weight is increased. On both Text and Search Arena we see similar trends: OpenAI's GPT and SpaceXAI's Grok models go up or stay flat as factuality weight increases. Anthropic's Claude and Google's Gemini models go down or stay flat. Finally, Meta's muse-spark-1.1 and Baidu's ernie-5.1 drop heavily.

Above we show factuality weight vs. score for all models broken down by four providers to visualize patterns in each provider's model suite. We see most OpenAI models move up in score while Anthropic models decrease or stay the same. In particular, the previous most factual Claude Opus version before the newest 4.8 release was 4.5—released back in November 2025. We find that while Anthropic models are generally more preferred by humans, OpenAI's newest models are generally more factual.

SpaceXAI's Grok-4-0709 was a solidly factual model; however, the subsequent Grok-4.1 series heavily regressed in factuality. The newer Grok-4.20, Grok-4.3, and Grok-4.5 versions have improved their factuality, though Grok-4.3 is far less preferred by humans.

Google models have less variance in factuality scores OpenAI and SpaceXAI models. Notably, the Gemini-2.5 model series still remain the most factual to this day—Gemini seems to be getting less factual over time.

These trends remain similar when models have access to search tools in the Search Arena.

We find most open source models decline in score under higher factuality weight, with the exceptions of mistral-medium-3.5 and Tencent's huyuan-hy3-preview.

While human preference across nearly all providers has been climbing over time, plotting pure factuality score against flagship model release date shows OpenAI is the only provider steadily improving factuality at the same time. SpacexAI has also been improving with their recent releases; this is similarly true in Search Arena. Notably, Google and Anthropic models have stayed at relatively similar levels of factuality across the past year, especially when given access to search tools. Meta showed a large improvement in factuality between muse-spark and muse-spark-1.1 but still remains behind other top closed and open source labs.

We can also observe the relationship between human preference score and factuality score. We notice a weak positive correlation. Ultimately, the signals are largely orthogonal—this is why adding a factuality signal is a necessary subcomponent of evaluation. For example, a model response is perfectly factual if the model refuses to provide an informative response; such responses perform poorly in terms of human preference. At the same time, a long response with many facts may seem comprehensive and win a human preference vote, but it may also introduce factual errors.

Looking at average claim correctness for various industry categories, we find slight, but intuitive, variance in model factuality in different industries. Models are largely factual in mathematical tasks. They are also reasonably factual in Software, Medical, and Scientific tasks. The least factual industry area was Legal and Government tasks.

##

Below we provide a viewer to show a sample of extracted claims.

##

Rewarding style is not an incorrect thing to do. Communicating effectively with humans is essential in chat, so formatting and prose are important factors in evaluation. The deeper worry is that a more “stylish” response will obfuscate undeniable errors in a model's response.

With the new factuality leaderboard, we address this concern head-on: rather than using style as a proxy for such undesirable behavior, we attempt to measure and detect the existence of such behavior directly. Factuality is, of course, only one notion of response correctness, but it is a particularly important and appropriate one to prioritize because: (1) humans are increasingly trusting models to answer questions outside their area of knowledge; (2) increasingly models themselves are better at checking facts than humans.

##

Below we provide a more detailed explanation of the methods used to fairly audit claims and ingest them into a principled leaderboard calculation.**

Composite Bradley-Terry model**

Rather than relying on either human preference or factuality alone, we use the composite Bradley-Terry model to combine both objectives. While factuality measures whether a response is factually sound, it alone does not capture whether it addresses what the user actually asked for. The human preference component preserves that signal, promoting responses that are both factually grounded and responsive to user intent.

Composite loss

We fit one rating vector θ∈RM(M\theta \in \mathbb{R}^M (M = number of models on the leaderboard) by minimizing a composite loss with two components plus a small ridge penalty:

• Lhuman(θ)L_\text{human}(\theta) is a standard Bradley-Terry negative log-likelihood evaluated on per-battle human-vote outcomes.
• Lfact(θ)L_\text{fact}(\theta) is a standard BT negative log-likelihood evaluated on per-battle factuality-derived soft outcomes.
• Note: whuman+wfact=1w_\text{human} + w_\text{fact} = 1. We will most often write the factuality weight as just w;whuman=1−ww; w_\text{human} = 1 - w.

The fitted rating θ\theta is the rating that best explains the joint distribution of human votes and factuality outcomes simultaneously, weighted by ww.

Factuality outcome per battle

For battles eligible for the factuality term, the factuality outcome is a soft outcome in (0,1)(0,1) produced by applying a sigmoid to the per-side aggregate truth-probability gap:

• avg_truth_probx\text{avg\_truth\_prob}_x is the mean of the per-claim aggregated truth probabilities across all verified claims emitted by side xx of the battle.
• TT is a fixed temperature chosen so that the variance of the factuality-vote distribution matches the variance of the human-vote distribution.

**

Which battles enter the factuality term**

A battle is eligible to contribute to LfactL_\text{fact} whenever at least one side emits a web verifiable claim. We find that roughly ~76% of Text Arena battles have relevant verifiable claims. The handling per case:

Case

Treatment

Both sides emit verifiable claims

Standard soft outcome via: σ(avg_truth_proba−avg_truth_probbT)\sigma\left(\frac{\text{avg\_truth\_prob}_a - \text{avg\_truth\_prob}_b}{T}\right)

Exactly one side emits verifiable claims, and that side's claims are truthful in aggregate

Tie (0.5) under LfactL_{\text{fact}}​. The abstaining side is not penalized for not making claims.

Exactly one side emits verifiable claims, and that side's claims are false in aggregate

Loss for the side that emitted false claims. Abstention is treated as a credible alternative to a false response.

Neither side emits verifiable claims

Battle is dropped from LfactL_{\text{fact}}​ entirely; it still contributes normally to LhumanL_{\text{human}}​.

Abstention is never penalized, but emitting false claims is — even when the opposing side abstained. We expect LhumanL_\text{human} to counterweight this signal correctly; the composite system inherently disciplines pure abstention via LhumanL_\text{human}. The factuality axis is purely meant to capture the harm of confident misstatements.

Notice we only care about *verifiable claims*. These are claims that are web verifiable, unambiguous, and objective. Claims that do not fall under this are not extracted and are skipped.

Examples:

Claim

Fact Checkable?

The current president is doing a good job.

No.

The current president of the US is 82 years old.

Yes.

In PyTorch, BCEWithLogitLoss’s default reduction is ‘none’.

Yes.

In Verilog, a function cannot be disabled using the disable statement.

Yes.

The conditional distribution of conformal coverage given the calibration data follows Beta(l, n+1−l), where l = ⌊(n+1)α⌋.

Yes.

Conformal prediction is a new and cool statistics research area.

No.

Katara should have been with Zuko, not Aang.

No.

Bob was a 8 year old boy with a fascination with UFOs, born in a random Kansas town.

No.

Confidence intervals

Per-model rating confidence intervals are computed via the sandwich variance estimator, as they are on the previous leaderboards. In particular, letting θ^=argmin⁡θL(θ)\hat{\theta} = \operatorname*{argmin}_{\theta} L(\theta), we have:

where H=∑kwk⋅HkH = \sum_k w_k \cdot H_k​ is the composite Hessian (HkH_k​ denotes either the human preference or factuality BT Hessian), B=∑k(wk2/Nk)⋅ΣkB = \sum_k \left( w_k^2 / N_k \right) \cdot \Sigma_k is the composite gradient covariance, and NkN_k​ denotes the corresponding total number of battles. Reported intervals on the leaderboard are 95% intervals derived from this estimator, namely (θ^m±1.96Var^(θ^)mm)(\hat{\theta}_m \pm 1.96\sqrt{\widehat{\text{Var}}(\hat{\theta})_{mm}}) for model mm, mapped into the standard Elo-like rating space.

Confidence interval widths shift as ww changes—in particular, intervals tend to widen at higher ww when factuality data is sparser than human-vote data.

Share this Article