チェスを通じたモデル行動の評価

Meet Compass — Dagster’s new AI data analyst for Slack. Turn questions into trusted insights, instantly. Try Compass now →Discover What assets do best, an animated, narrated story about how data assets work together. Watch now →

Try Dagster+Sign InBenchmarks measure outcomes, not behavior. By letting AI models play chess in repeatable tournaments, we can observe how they handle risk, repetition, and long-term objectives, revealing patterns that static evals hide.

Evaluating AI models is hard. Not because we lack metrics, but because what we care about often doesn’t fit neatly into a single number. We want to understand how models behave over time, how they respond to complex situations, how they compare to one another, and where they fail in surprising ways.

So instead of another benchmark chart, let’s do something more fun. Let’s play chess.

Chess turns out to be a great playground for thinking about model evaluation. It is structured, adversarial, and stateful, with clear outcomes: win, lose, or draw. Better yet, it lets us simulate model behavior repeatedly and compare models head-to-head.

Before we go further, it’s worth clarifying what this evaluation is and isn’t. This isn’t about measuring chess skill or producing a leaderboard. It’s about observing behavior over time: how models handle repeated states, how they trade off risk versus safety, and which failure modes only appear after many interactions. Chess provides a compact, stateful environment where these dynamics are easy to surface and compare.

A Repeatable, Programmatic Evaluation Setup

We use the Python chess library to simulate games programmatically. Each “player” is simply a function that, given a board position and a list of legal moves, chooses its next move.

Dagster orchestrates everything: tournaments, players, configurations, results, and artifacts. A tournament job looks something like this:

ops: tournament_start: config: num_games: 100 player_1: megan player_2: thomasEach tournament:

Runs a configurable number of games

Alternates colors so each player plays White an equal number of times. White moves first and has a slight advantage.

Enforces a maximum number of moves to avoid infinite games

Records outcomes and final board states as Dagster artifacts

With that in place, we can start experimenting.

Baseline: No Objective, No Strategy

“Megan” and “Thomas” are random agents using the Python chess library and Python’s random module. On each turn, they select a move at random from the list of legal moves and hope for the best. After each move, we use the chess library to check the state of the game and determine whether either player has won.

Using Dagster, we pit Megan and Thomas against each other in a 100-game tournament. One thing becomes clear almost immediately: neither player is particularly good at chess. Of the 100 games played, 87 ended in draws due to the maximum limit of 200 moves.

In the remaining 13 games, there was a decisive result. Thomas managed to defeat Megan by a narrow margin, winning 7 games to Megan’s 6.

We can also store the final board state of each game as a Dagster artifact. Below is the board position from one of Thomas’s rare victories.

The takeaway here is not that randomness is good at chess. It is that we now have a baseline. If a model cannot outperform random play, that tells us something immediately.

Baseline: Explicit Objectives and Long-Horizon Optimization

Now we can jump to the other extreme. Stockfish is an AI chess engine. Unlike our first set of players randomly flinging pieces at each other, Stockfish is very good at chess. Elo, the numerical rating system used in chess, estimates Stockfish at around 3600. For context, the highest Elo rating ever achieved by a human is 2882, held by Magnus Carlsen.

Thomas now has the honor of playing against Stockfish.

ops: tournament_start: config: num_games: 100 player_1: stockfish player_2: thomasAs expected, Stockfish defeats Thomas in every game. It also wins quickly, with an average game length of just 26 moves, compared to the excruciating 189 moves it took Thomas and Megan to finish a game on average.

This result is not surprising. Human players cannot reasonably expect to beat Stockfish. To make things more interesting, we can adjust Stockfish to play at a lower level. In the next tournament, Stockfish behaves like a player with an Elo rating of 1400.

ops: tournament_start: config: num_games: 100 player_1: stockfish:elo=1400 player_2: thomasA rating of 1400 represents a solid intermediate player. This is still more than enough to beat Thomas in every game, although Stockfish now needs a few more moves to do so, averaging 46 moves per game.

Now that we understand the two extremes of our simulated chess environment, it is time for the interesting part.

Evaluating General-Purpose Models Under Identical Constraints

We are now ready to introduce general-purpose AI models into our tournament. The goal is to let any model play ch