トークンを流し続ける：16のオープンソースRLライブラリからの教訓

Back to Articles Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

Upvote 2

TL;DR -- For those of you who don't have time to read 5,000 words about async RL plumbing (we get it, you have models to train):

The problem: In synchronous RL (reinforcement learning) training, data generation (model inference to create data samples) dominates wall-clock time -- a single batch of 32K-token rollouts on a 32B (32-billion parameter) model can take hours, while the GPUs used for training remain idle.

The solution everyone converged on: Disaggregate (separate) inference and training onto different GPU pools, connect them with a rollout buffer (temporary storage for model outputs), and transfer weights asynchronously (without waiting), so neither side waits for the other.

We surveyed 16 open-source libraries that implement this pattern and compared them across 7 axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends.

Key findings: Ray dominates orchestration (8/16 surveyed distributed computing libraries). The NCCL (NVIDIA Collective Communications Library) broadcast is the default method for transferring model weights. Staleness management refers to how outdated data samples are handled, ranging from simply dropping old samples to using advanced importance-sampling correction. LoRA (Low-Rank Adaptation) training is sparsely supported. Distributed MoE (Mixture of Experts) support is the emerging differentiator.

If you'd rather skip straight to the good part, here's the full comparison table (no reading required, we won't judge).

But seriously, if you stick around, you might learn a thing or two about why your GPUs are idle 60% of the time.

1. Motivation: From synchronous RL training to async architectures 1.1 How TRL Does RL Training Today

1.2 Colocated vs. Disaggregated Training

1.3 The Generation Bottleneck

1.4 The Core Insight

1. Libraries Surveyed

1. The Comparison Framework: Seven Axes Axis 1: Orchestration & Concurrency Primitive

Axis 2: Rollout Buffer Design

Axis 3: Weight Synchronisation Protocol

Axis 4: Staleness Management

Axis 5: Partial Rollout Handling

Axis 6: LoRA Training Support

Axis 7: Distributed Training Backend & Parallelism

1. Global Overview: Sixteen Libraries at a Glance

1. The Next Wave: Design Implications 5.1 Critic-Free Algorithms: Memory Freed, But Weight Sync Pressure Increases

5.2 Process Rewards: A New Synchronisation Barrier

5.3 Multi-Agent Co-Evolution: The Straggler Problem Compounds

5.4 Training-Inference Mismatch: The Deepseek v3.2 MoE Case Study

5.5 Distillation: The Same Async Problem Under a Different Name

1. Design Choices for TRL's Async Trainer Design Principle: Keep Orchestration Lightweight

1. Bounded Queue with Per-Token model_version

1. NCCL Weight Sync with Packed Transfers

1. Partial Rollout Support for Agentic Workloads

1. Motivation: From synchronous RL training to async architectures

Async RL training has emerged as the dominant paradigm for post-training at scale. Several trends in modern post-training have made synchronous training loops nearly impossible to scale:

Long rollouts from reasoning models. Chain-of-thought training produces very long rollouts, and a single synchronous generation batch can take hours to complete on a single GPU. During all of that time, training GPUs sit completely idle.

Value-function-free trainers like GRPO use group-relative advantages. This means generating up to G times more rollouts per prompt, and the entire batch is gated by the slowest completion in the group.

The rise of agentic RL training. When models interact with tools, sandboxes, and external environments across multi-turn trajectories, rollout lengths and latencies become highly variable. A simple API call might return in seconds, while a complex reasoning chain with tool use can run for minutes or hours. MiniMax's Forge framework, used to train MiniMax-M2.5, illustrates the scale this reaches in practice: context lengths up to 200K tokens, over a hundred thousand distinct agent scaffolds and environments, and daily throughput on the order of millions of samples. At this scale, any synchronous barrier between generation and training becomes a severe bottleneck. The straggler problem alone (where a handful of slow rollouts block an entire batch) can idle hundreds of GPUs.

The open-source ecosystem has converged on a common architectural response: disaggregate inference from training onto separate GPU pools, connect them with a rollout buffer, and let both sides run concurrently.

We are developing a new async trainer for TRL, one of the most widely used libraries for model post-training. To guide our design, we surveyed sixteen open-source libraries that were built from the ground up around asynchronous training and compared them across seven axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends. This article distills the design principles we extracted from that survey.

Beyond RL, the need for async infrastructure is increasingly evident. For example, on-policy distillation, where a student generates sequences and a teacher scores them, mirrors GRPO but swaps the reward function for a teacher forward pass. Recognizing this structural similarity, everything in this survey applies equally to async distillation. We'll return to this broader point in Section 5.

1.1 How TRL Does RL Training Today

TRL's current GRPOTrainer

training_step()

Looking at the GRPOTrainer

Prompt sampling: draw a batch of prompts from the dataset. Nothing crazy here, let's continue.

Generation, calls model.generate()

Reward scoring: evaluate each completion against one or many reward functions.

Advantage computation

Forward and backward passes: compute the clipped policy gradient loss and backpropagate.

Optimizer step, update model weights.

Weight sync, push updated weights to the inference engine (vLLM) so the next generation uses the new policy.

Each phase blocks until completion before the next begins. The timeline looks like this:

TRL offers the steps_per_generation

The library also supports running vLLM in server

1.2 Colocated vs. Disaggregated Training

Before discussing async training, it is essential to understand the two deployment topologies for RL training with a separate inference engine:

Colocated mode places inference and training on the same set of GPUs. A single GPU (or TP group) holds both the training model (under FSDP or ZeRO) and the inference engine (vLLM or SGLang). Only one role is active at a time: during generation, the training model's parameters may be offloaded or resharded into an inference-friendly layout (e.g., from FSDP shards to vLLM's tensor-parallel layout); during training, the inference engine is paused or put to sleep. Weight "sync" is essentially free; it is at most an in-place resharding on the same GPU, not a network transfer. The advantage of the colocated mode is simplicity and cost; you need fewer total GPUs. The fundamental limitation is that inference and training cannot overlap. For example, here is the Trl with vllm in colocate_mode

Disaggregated mode places inference and training on separate GPU pools. The inference pool runs vLLM or SGLang continuously; the training pool runs the optimizer continuously. The two pools communicate via a weight synchronisation protocol (NCCL broadcast, filesystem checkpoint, HTTP, etc.) and a data transfer mechanism (Ray object store, Redis streams, shared memory, etc.)

The biggest advantage of the disaggregated mode is that inference and training can run concurrently. While the trainer computes gradients on batch N, the inference pool is already generating rollouts for batch N+K, enabling async training. However, this benefit comes at a cost: additional GPUs are required.

Concurrency, asynchronicity, and parallelism are distinct concepts that often get conflated. In this article, when we say "async training," we mean something specific: generation and training running in parallel, with effective overlap; the inference pool is producing the next batch of rollouts while the training pool is computing gradients on the current batch. This is fundamentally a disaggregated-mode capability. Colocated mode can benefit from optimisations like sleep/wake memory management or fast in-place resharding to speed up inference, but it cannot achieve true simultaneous overlap; inference and training still take turns on the same GPUs. Every library in this survey that implements meaningful async overlap uses disaggregated mode as the foundation.

1.3 The Generation Bottleneck

In RL training for reasoning models, autoregressive generation dominates wall-clock time. A single rollout for a math or coding task can produce 8K–64K tokens of chain-of-thought reasoning (see QED-Nano rollout lengths).

To ground this concretely, consider vLLM benchmarks on a single H100 80GB GPU (bf16, no quantisation, offline throughput mode). A 7B model (DeepSeek-R1-Distill-Qwen-7B) achieves ~6,300 output tokens/s aggregate throughput; a 32B model (DeepSeek-R1-Distill-Qwen-32B) drops to ~1,200 output tokens/s. These are total throughput across all concurrent requests, the number the inference engine can push through per second, regardless of how many sequences share the GPU.

Now consider a typical GRPO training step: G=8 completions per prompt × 64 prompts/batch = 512 rollouts. How long does generation take?

Output length per rollout

Total output tokens (512 rollouts)

Time on 1×H100 (7B @ ~6K tok/s)

Time on 1×H100 (32B @ ~1.2K tok/s)

2K tokens (short CoT)

8K tokens (medium CoT)

32K tokens (long CoT)

Even at the short end (2K tokens generated with a 7B model), generation alone consumes several minutes per training step. At the long end, where frontier reasoning models increasingly operate, a single generation phase can take hours on one GPU. Scaling to 8 inference GPUs divides these times by roughly 8× (assuming linear throughput scaling), but even then, 32K-token rollouts on a 32B model still take ~28 minutes per step.

The straggler problem compounds this further. In group-based algorithms like GRPO, you sample G completions per prompt. The batch cannot proceed until the slowest completion finishes. Chain-of-thought output lengths are highly variable; a single prompt might produce completions ranging from 1K to 32K tokens. The batch is gated by the longest completion, and continuous batching only partially mitigates this: shorter sequences free up slots for new work, but the last sequence in a GRPO group still blocks the group's reward computation and training step.

1.4 The Core Insight

Every library in this survey has independently converged on the same architectural principle: physically separate inference GPUs from training GPUs, and push weights asynchronously, so generation never stops and training never waits.

The inference pool runs continuously, feeding completed rollouts into a buffer. The training pool pulls from the buffer, computes gradient updates, and periodically pushes new weights back to the inference pool to keep it in sync. The two loops run at their own pace, decoupled by the buffer.

This setup is highly scalable, but it introduces a new class of problems: staleness (rollouts generated under an old policy), weight synchronisation overhead, partial rollout handling, etc. The rest of this article dissects in detail how current open-source libraries address these issues.

1. Libraries Surveyed

GitHub ⭐ (Mar. '26)

inclusionAI/Ant Group

github.com/inclusionAI/AReaL

github.com/OpenPipe/ART

github.com/NousResearch/atropos

github.com/radixark/miles

github.com/NVIDIA-NeMo/RL

github.com/sail-sg/oat

github.com/allenai/open-instruct

github.com/ServiceNow/PipelineRL

github.com/PrimeIntellect-ai/prime-rl

github.com/alibaba/ROLL

github.com/NovaSky-AI/SkyRL

github.com/THUDM/slime

github.com/meta-pytorch/torchforge

github.com/google/tunix

github.com/verl-project/verl

github.com/PrimeIntellect-ai/verifiers

1. The Comparison Framework: Seven Axes

To make sense of the rapidly expanding ecosystem of async RL libraries, we propose seven orthogonal axes of comparison. Each axis captures a fundamental design decision that shapes the library's performance, complexity, and trade-offs.

Axis 1 – Orchestration & Concurrency Primitive: how distributed components are coordinated (Ray actors, asyncio, pub/sub, HTTP).

Axis 2 – Rollout Buffer Design: how rollouts flow from inference to training.

Axis 3 – Weight Synchronisation Protocol: how updated weights reach inference servers, and whether the system must pause to accept them or continue generating.

Axis 4 – Staleness Management: how off-policy rollouts are handled : version rejection, depth bounding, or importance-sampling correction.

Axis 5 – Partial Rollout Handling: what happens to in-flight generations when a weight update arrives mid-sequence.

Axis 6 – LoRA Training Support: General LoRA support and whether adapter-only parameters can be trained and synced, enabling sub-millisecond weight transfers.

Axis 7 – Distributed Training Backend & Parallelism: what parallelism strategy is used for training, constraining max model size.

Axis 1: Orchestration & Concurrency Primitive

How does the system coordinate its distributed components?

The choice of orchestration framework determines the programming model, failure semantics, and scalability ceiling. Rather than listing per-library implementation details, the landscape decomposes cleanly into four orchestration types, fundamental coordination paradigms that differ in abstraction level, failure model, and deployment requirements:

Orchestration Type

Distributed Actor Model

Components are actors, isolated stateful processes with mailboxes, managed by a runtime that handles scheduling, resource placement, fault tolerance, and object transfer. Communication is via asynchronous RPC / futures / object store.

Ray: verl, SkyRL, NeMo-RL, SLIME, MILES, ROLL, OAT, open-instruct. Monarch: TorchForge.

Richest abstraction; solves scheduling and fault tolerance out-of-the-box. Adds a non-trivial runtime dependency and framework-specific debugging overhead.

Native Python Concurrency

Components are threads, coroutines (asyncio

multiprocessing

verifiers-rl, PipelineRL (intra-pool), ART (asyncio

Minimal dependencies, easy to debug, full control. Limited to single-node unless paired with additional IPC (Redis, HTTP, NCCL) for multi-node communication.

Pub/Sub Message Bus

Components are decoupled producers and consumers communicating through append-only streams or message queues. Not orchestration per se, a data transport layer between independently running pools.

PipelineRL (inter-pool: Redis XADD

Clean decoupling across pool boundaries without RPC. Does not manage process lifecycle, scheduling, or fault recovery; must be paired with another orchestration type.

HTTP Microservices

Components are independent services communicating via REST APIs. Language-agnostic, maximum decoupling.

Any inference server, any language, zero shared state. Highest latency (if NCCL); no shared object store; fault tolerance is the user's responsibility.

Note on Tunix: Tunix (Google) uses a JAX-native mesh model with ThreadPoolExecutor

The table above reveals a striking pattern: eight of the sixteen libraries surveyed use Ray as their orchestration backbone. This is not a coincidence; it reflects a deep architectural fit between the actor model and the structure of RL training. A survey by Anyscale (the company behind Ray) of open-source RL libraries for LLMs confirms this convergence. RL training at large scales involves fundamentally heterogeneous components (inference engines, training engines, environments, reward models) that must be orchestrated across a cluster, often on different hardware types, with different scaling requirements and failure modes. Ray's actor model maps directly onto this:

Actor isolation and heterogeneous resources. Each RL component (vLLM inference server, FSDP trainer, reward model, environment pool) becomes a Ray actor with its own resource requirements (num_gpus

Scheduling and autoscaling. Ray's scheduler handles the combinatorial problem of placing heterogeneous actors across a cluster. When generation requires 8× more GPU-hours than training, you can just tell Ray to scale your inference actors independently.

Fault tolerance. Long RL training runs (days to weeks) are vulnerable to GPU failures, OOM kills, and network partitions. Ray's actor restart policies and object store replication provide resilience that would require significant custom infrastructure with raw asyncio

multiprocessing

Object store for zero-copy data transfer. Rollout data can be large, tens of GB per batch for very long-context reasoning. Ray's shared-memory object store enables zero-copy transfer between actors on the same node, avoiding serialization overhead that usually comes with multiprocessing.Queue

Ecosystem maturity. Ray has been battle-tested at scale since 2017, with production deployments on thousands of GPUs. The debugging overhead is real (Ray Dashboard, distributed stack traces, placement group failures), but the alternative, building equivalent coordination from scratch, is worse at the multi-node scale. That said, Ray is a heavy dependency: it pulls in its own scheduler, object store, and dashboard, adding operational complexity that not every team needs. This is exactly why libraries like PRIME-RL, PipelineRL, and AReaL opted for lightweight native-Python coordination (asyncio, threading, Redis streams) instead --- when you control the full stack and your deployment topology is fixed, the simplicity and debuggability of vanilla Python often outweigh the conveniences Ray provides.

The cost is a hard dependency on a non-trivial runtime. This trade-off can be worthwhile, especially for production-scale training (64+ GPUs, multi-day runs, complicated reward computation).

While Ray's actor model is the main player on the field, Monarch emerged as a new PyTorch-native distributed actor framework from Meta, purpose-built for GPU workloads. Like Ray, Monarch is based on the actor model; components are independent actors with mailboxes communicating via messages, but it is designed from the ground up for the PyTorch/CUDA ecosystem rather than being a general-purpose distributed runtime.

Monarch offers several capabilities particularly relevant to async RL. An example implementation of async RL with Monarch (from the GPU Mode lecture series) demonstrates the architecture: generators, a replay buffer, and a trainer are modelled as Monarch actors, with the replay buffer absorbing latency variance from straggler rollouts and RDMA weight sync pushing updated parameters to generators without blocking training. The pattern is structurally identical to Ray-based designs (verl, SkyRL, open-instruct) but implemented with pure PyTorch-native primitives.

Axis 2: Rollout Buffer Design

How do generated rollouts flow from inference to training, and how deep is the pipeline?

The buffer is the data structure sitting between generation and training. Its depth controls the maximum degree of asynchrony, and therefore the maximum staleness.

No buffer (synchronous)

TRL (current), ART (gather-all-then-train)

Generation and training alternate strictly; zero staleness, maximum idle time

Double-buffer (one-step-ahead)

verifiers-rl, SLIME (async mode), MILES, OAT

Submit generation N+1 at the start of training step N; overlap exactly one batch

Bounded async queue

SkyRL, verl (fully async), NeMo-RL, ROLL, PRIME-RL, TorchForge, Tunix, open-instruct (async_steps

max_head_offpolicyness

Multiple batches in flight; staleness bounded by queue capacity

Unbounded / stream

PipelineRL (Redis streams), SLIME (fully async mode), Atropos

Continuous generation; staleness bounded only by explicit version control

The double-buffer pattern is the simplest upgrade from synchronous to asynchronous training: it overlaps exactly one generation with one training step and introduces at most one step of policy lag !

Deeper queues, on the other hand, improve throughput but require staleness management.

The buffer controls how much data is in flight. But data is only half the equation. The other half is getting updated weights back to the inference servers before those rollouts go stale. That's where weight sync comes in!

Axis 3: Weight Synchronisation Protocol

How do new model weights reach the inference servers after a gradient update?

Scope note: This axis focuses on disaggregated mode, where inference and training run on separate GPU pools, since that is the deployment topology where async overlap (and therefore weight sync design) actually matters. Colocated setups (same GPUs for both roles) are inherently synchronous and do not face the transport/interrupt trade-offs discussed below.

This is the most architecturally consequential axis. The protocol determines sync latency, interrupt granularity, and whether partial rollouts are possible.

There is a critical distinction to make here: the transport mechanism and the interrupt model. Most libraries pause generation at a coarse boundary, an HTTP request, a full batch, or even a full training step, before initiating weight transfer. PipelineRL is the outlier: it never stops generating at all.

Transport mechanism:

PipelineRL, SkyRL, SLIME, MILES, ROLL, OAT, NeMo-RL, PRIME-RL, open-instruct, AReaL

NCCL + Bucketing

KV + Shared Memory

Filesystem + HTTP

PRIME-RL, AReaL, ART

CUDA IPC (Zero-copy)

Filesystem + Restart

In the interrupt model, when does the generation pause to accept new weights?

This is where PipelineRL fundamentally diverges from every other library. Rather than listing each library individually, the landscape collapses into five conceptual tiers, ordered from finest to coarsest interrupt granularity:

Interrupt Granularity

Never (In-flight per-forward-pass)

Sequences never stop. The weight swap happens between token decode steps (~1-10ms gap). Running sequences seamlessly continue with new weights.

PipelineRL, open-instruct (opt-in)

Per HTTP Request (Abort + Resync)

In-flight HTTP requests are aborted. Partial tokens are resubmitted with a prefix-resume mechanism or recycled for retry.

SkyRL, SLIME, MILES

Soft Pause (Drain in-flight)

No new generation requests are accepted while in-progress ones finish naturally. Once drained, weights are synced and generation resumes.

PRIME-RL, AReaL, open-instruct (default), verl (async)

Per Training Step / Batch (Blocking)

Generation must fully complete. The trainer and inference engine take turns blocking each other.

NeMo-RL, ROLL, OAT, TorchForge, Tunix, verifiers-rl, Atropos

The "never-stop" tier is qualitatively different from all others: PipelineRL, for example, hooks into the inference engine so that the lock is acquired and released per transformer forward pass (one token step for one sequence). A weight update waits at most one forward pass (~few ms), swaps all parameters, and generation resumes immediately. Every other library stops generation at a coarser boundary, from one HTTP request (~hundreds of ms) up to a full batch boundary (~seconds).

Weight sync controls when new weights arrive. But async training means rollouts are always being generated under some policy version, and that generating policy might be several gradient steps behind the trainer. How libraries handle this policy lag is staleness management.

Axis 4: Staleness Management

How does the system handle the fact that generated rollouts may come from an older policy than the one being trained?

Once generation and training overlap, samples become off-policy. Three orthogonal strategies have emerged for managing this staleness, and most production systems combine more than one:

Strategy 1: Per-sample version rejection. Every sample is tagged with the integer policy version that generated it. At training time, samples whose version falls behind the current policy by more than a threshold are hard-dropped before entering the loss computation. Simple and correct, but wastes the precious compute spent generating discarded samples.

Strategy 2, Depth Bounding. The queue or buffer between generation and training has a bounded capacity (or an explicit staleness gate), which architecturally limits how far behind any sample can be. This ranges from depth=1 (one-step-ahead double buffering, where staleness is impossible by construction) to explicit capacity formulas tied to version gaps. No per-sample version tracking is required; the bound is enf