エージェント間相互作用中のトークンIDの記録による強化学習の改善

Reinforcement learning (RL) is one of the techniques we use to make language models better at sustained, multistep tasks like writing code, navigating a website, or carrying out a research workflow. The model doesn't act alone in those settings; it's wrapped in a piece of software we call a harness, which lets it call tools, observe the results of using them, and decide what to do next. To improve such a model with RL, we let it attempt many tasks inside the harness, score how well each attempt went, and use the score to nudge the model's parameters toward the choices that worked. The hard part turns out to be the bookkeeping. To turn a scored attempt into a parameter update, the trainer needs an exact record of what the model produced — not a summary, and not a transcript that looks as though it captures a complete exchange but drops vital information. Internally, models see text as a sequence of numbered units called tokens; an English sentence might have 10 or 20 tokens, each assigned an integer ID by a piece of software called a tokenizer. Two strings that look identical in a transcript can map to different token IDs after a small change of formatting, and that gap, however small, is enough to make the trainer optimize against a slightly different past than the one the model actually experienced. Today we're releasing Turnstile, a small proxy written in the Rust programming language that sits between any agent harness and the backend system that runs the model. Turnstile records the exact token-level history of every request as it happens, at the only point where that history is unambiguously correct: the moment of generation. It then exports a generic, framework-neutral trajectory that can feed into whichever RL training stack you already use. We use Turnstile to drive real RL training runs. In the validations we report here, two different agents — a text-only coding agent and a multimodal computer-use agent — improved steadily over the course of their RL runs. In both cases the agent harness was left unchanged, and the data Turnstile records flowed directly into the training stack and produced the expected learning signal end to end. Tokens, rollouts, and why agent transcripts can lie Three pieces of vocabulary do most of the work in the rest of this post, so it's worth grounding them up front. A tokenizer is a deterministic function that turns text into a list of integer token IDs and token IDs back into text. Each model is paired with a specific tokenizer; you can't substitute one for another. The tokenizer is unforgiving: a stray space, a different way of writing a tool call as JSON, or a slightly different chat template (the format string a serving system uses to wrap roles and messages into a single text input the model can read) can change the token IDs even when the rendered text looks the same to a human. We'll call this kind of mismatch retokenization drift when it comes from rerunning a tokenizer over text we’ve already seen and chat template drift when it comes from the surrounding format changing under us. A rollout is one recorded attempt at a task: the prompt, every tool call, tool feedback, the model’s responses, and the final outcome. The version of the model that produced the rollout is called the behavior policy. The mathematics of policy-gradient RL works cleanly only when the trainer optimizes the model's behavior against the context the behavior policy actually saw. If we rerender the prompt and end up with a slightly different token sequence, we're now training the model against a context unfamiliar to the behavior policy. The training signal degrades, sometimes invisibly, since the model still appears to be learning. That is why agent harnesses make the problem worse rather than better. A harness is not a static prompt; during a single rollout it may compact older messages to save context, retry a malformed tool call, branch into subagents, merge their results back, or summarize history. All of that is normal, useful agent behavior. But each rewrite is another chance for the next request's token sequence to drift away from what the model actually generated last turn. The transcript a harness produces is a faithful record of the conversation; it is not, in general, a faithful record of the tokens, and it is the tokens the trainer needs. Capturing tokens at the proxy boundary Turnstile's central design choice is to stop trying to reconstruct token-level state from text after the rollout is over. We capture it at the moment of generation, where it is already correct, and we do that without changing the harness. The proxy speaks the same HTTP API every modern agent harness already speaks: the OpenAI Chat Completions API, which has become the de facto standard for "send a list of messages, get back a response." The harness creates a rollout group with Turnstile, points its Chat Completions client at Turnstile's address instead of the real backend, and runs unchanged. Behind the scenes, every request flows through Turnstile to the inference backend (today SGLang, with vLLM planned). Turnstile records the exact token IDs the model sampled, the per-token log probabilities (the model's own confidence values for each token, expressed as logarithms; the trainer needs these to compute its update), and a loss mask that marks which tokens were generated by the model and should contribute to training versus which came from the user, tools, or the system prompt and should not. When the rollout is finished, the harness asks Turnstile for the recorded trajectories. Each trajectory is a TrainingSequence object containing the token IDs, log probabilities, the loss masks for the full sequence, and a record of which version of the model's weights was active during which spans of the sequence. (The trainer needs these weight version boundaries to know if the model parameters changed mid-rollout because of an asynchronous update.) Turning that into the specific batch shape your trainer wants is straightforward adapter work: attach the reward, expand the mask, and hand it off. Existing harnesses can stay black boxes There are a number of agent harnesses, such as OpenHands, Codex, and Terminus, that are already useful but were not designed as training runtimes. Without Turnstile, using one of these to drive an RL training run requires it to record token IDs, log probabilities, masks, and routing traces; in practice, that work often falls to a separate harness-shaped component built inside the training system. Either way, a harness ends up acting as a token-level RL data pipeline, and that is the wrong abstraction. The harness knows the information it intended to send to the model, but it does not, in general, know the exact token sequence, cache state, routing trace, or processed multimodal inputs the model actually used. Those live in the backend. With Turnstile, the production harness doesn't have to log training data. It points its Chat Completions client at Turnstile instead of the inference backend and otherwise runs unchanged. Turnstile records the model-facing rollout state: it does not need to understand the harness's private control flow or the semantic reason a context changed. If the next request is a faithful token-level extension of an earlier one, Turnstile merges it into the same trajectory. If the harness compressed memory, rewrote history, merged a subagent result, or otherwise changed the prefix in a way that cannot be proven equivalent, Turnstile starts a new sequence and keeps the trainable suffix honest. All this applies in the strict black-box case: a proprietary harness whose internals are closed to the training system can still drive an RL training run, with no source-level integration at all. Below are two examples of using Turnstile with open-source harnesses in a black-box fashion. Multiturn agents and prefix-aware trajectories A naïve way to store these recordings would be to treat every request as an independent training example. That’s wasteful, because each new request includes the whole conversation so far, so the same token strings would end up being duplicated over and over. It’s also subtly wrong, because it loses the relationship between turns. Instead, Turnstile stores a multiturn rollout as a single growing token path — so long as the path is faithful to what the model actually saw. When a later request to the model is just the previous request plus a few new tokens at the end (the new user message, a tool result, an LLM response), Turnstile recognizes the overlap at the token level — not by comparing rendered strings but by checking that the previously captured token IDs really do appear unchanged at the start of the new request. If they do, the two turns become one continuous trainable sequence with the loss mask correctly identifying which spans were the model's outputs. When the next request cannot be safely extended from a previous one — because the harness rewrote earlier messages, or the tokens drifted for some reason — Turnstile does not pretend otherwise. It starts a new training sequence. We call this "exploding the trajectory". It costs more training tokens than the optimistic alternative, but it ensures that every token string used for RL training is one the behavior policy actually saw. The point is not to maximize compactness; the point is never to lie to the trainer about what happened. Mixture-of-experts routing adds a hidden dimension Some modern models use a mixture-of-experts (MoE) architecture, in which only a small subset of the model's parameters — called experts — are activated for any given input token. The choice of experts is itself part of the computation, made by a small router network at every layer. The routing decision depends on the activations at each layer, and tiny differences in how the previous tokens were processed can change which experts are picked. This matters for RL because, even if two requests have the same token IDs, the tokens might get routed to different experts on different runs. Last fall, researchers at Peking University and their colleagues characterized this discrepancy and proposed recording the MoE model’s routing decisions so the trainer can replay them. We adopt the same principle. When MoE capture is enabled, Turnstile asks the inference backend for the routing trace and records it alongside the tokens. Every time it extends the token path, it checks that the routing for the shared prefix matches what was recorded last turn. If it doesn't — for example, because a key-value cache miss forced the backend to recompute the prefix and pick different experts — Turnstile splits the trajectory rather than train under the wrong routing. Multimodal rollouts When the model also takes images as input — a vision-language model — the rollout has another piece of state to keep track of. The model doesn't see the raw bytes of the uploaded image; it sees the output of an image processor, a fixed pipeline that resizes and crops the image, converts it into a tensor of pixel values, and inserts placeholder tokens into the text to mark where the image goes. The same image bytes can produce different tensors as the result of a different processor version, a different resize policy, or a different patch geometry, and the placeholder count can change with the input dimensions. If the RL trainer has to reprocess the image from scratch, the visual prefix it trains under may not be the visual prefix the behavior policy saw. Turnstile treats image processing as part of the rollout. When a request includes an image, Turnstile decodes it, hashes and stores the original bytes for audit, runs the model's configured processor, and records the processed pixel features in the same trajectory as the token IDs, in the order in which the placeholders appear. The exported sequence carries both the raw and processed visual data, so the trainer can use whichever it needs. Where this is going Turnstile is early. The current implementation has a Rust core, an SGLang backend, Python bindings for in-process training scripts, prefix-aware multiturn capture, optional MoE routing capture, and multimodal support. Near-term work is broader: a vLLM backend, more training-framework adapters, and more multimodal-model coverage. The long-term shape is unchanged from the design we started with: agent harnesses should not have to become RL data pipelines, and trainers should not have to guess what happened from rendered text. The model sampled the tokens. We record them. References Turnstile README Turnstile data design Turnstile multimodal image support design SGLang multimodal compatibility notes OSWorld and its open-source agent harness OpenHands THUDM/slime Prime Intellect renderers blog Prime Intellect renderers GitHub prime-rl algorithms (extension property, multi-turn trajectory merging) prime-rl inference (router replay) Polar: Agentic RL on Any Harness at Scale NVIDIA-NeMo/ProRL-Agent-Server Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers rLLM Agent Lightning Strands Agents SGLang provider