Artificial Analysis、エンドポイント精度指数を発表

Announcing the Artificial Analysis Endpoint Accuracy Index, measuring how much of an open weights model's accuracy each serverless API endpoint preserves. We are initiating coverage with GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 coming soon

Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels and tune their inference stacks, and sometimes they simply ship bugs. We are bringing the rigor of our Artificial Analysis Intelligence Index to measuring endpoints, so developers can pick providers on accuracy, not just price and speed

We benchmark each serverless endpoint against our own self-hosted reference deployment of the official weights, where 100% represents matching the reference. An endpoint is at reference parity when its result falls within the 95% confidence interval of the reference. Coverage is live for GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 accuracy coverage launching soon

Key elements of the Endpoint Accuracy Index:

• Three areas, equally weighted: tool calling (BFCL-500, 500 questions, 3 repeats), scientific reasoning (HLE-250, 250 questions, 10 repeats) and long context recall (AA-LCR-25, 25 questions, 10 repeats). Each subset separates endpoints on the serving choices that drive accuracy differences, with repeats sized for tight confidence intervals

• Reference deployment: we self-host the official weights at the lab's recommended precision, following the lab's serving recipe, and publish the complete commands for each reference

• Inference parameters: we run the model's highest supported reasoning mode and each endpoint's highest supported output length and context window

• Confidence intervals: the parity test accounts for uncertainty in both the endpoint's runs and the reference's runs

• Rotating coverage: models enter once sufficient number of providers serve them and exit when a newer version in the same family supersedes them. We benchmark new endpoints as providers launch them and refresh all listed endpoints periodically

• Point in time: each result carries the date it was measured, with multi-day benchmarks dated to their final day

Key results for GLM-5.2

• Output token limits restrict accuracy. Restrictive limits cut responses off before the model finishes reasoning, and the most restrictive endpoints score half the reference or less on HLE-250

Key results for gpt-oss-120b

• Tool call handling separates endpoints. Providers parse and format tool calls differently, and some endpoints score 22% on BFCL-500 against 37% for the reference

• Serving configuration changes what the model does at the same requested settings. Some endpoints produce far fewer reasoning tokens at the same configured level, and restricted context windows truncate long context tasks

Key results for DeepSeek V4 Pro

• DeepSeek V4 Pro endpoints are more in line with the reference. Majority of the endpoints are at reference parity, and DeepSeek's own first-party endpoint scores slightly above the reference

How the Endpoint Accuracy Index is calculated: we run the same three evaluations against each endpoint and against the reference deployment, score each as a percentage of the reference result, and average the three with equal weight

Endpoints that score below the reference generally produce fewer output tokens per task. Output limits and reduced reasoning effort show up directly in token usage, and the lowest-scoring endpoints on both models produce roughly half the reference's output tokens

Full results:

• https://artificialanalysis.ai/models/glm-5-2/providers

• https://artificialanalysis.ai/models/gpt-oss-120b/providers

• https://artificialanalysis.ai/models/deepseek-v4-pro/providers

Methodology: https://artificialanalysis.ai/methodology/endpoint-accuracy-index