再帰型言語モデルの強化：強化学習による効率的な実装

*We RL fine-tune small (4B) models to behave as native recursive language models (RLMs) by training parent and child RLMs under a single, shared policy. With RL, small models can learn task-specific, RLM behavior that cannot be elicited through prompting or even SFT. This blog assumes a basic level of familiarity with RLMs. A great resource for learning about them is the original RLM blog post.*

Contents

We investigate RL fine-tuning 4B models to be used as RLMs in production settings. While RLMs are a powerful inference strategy, they can have unpredictable latency and can require extensive prompt tuning to elicit consistent behavior. RL fine-tuning allows us to train purpose-built RLMs that are cheap to deploy.

Rather than training separate policy models for parent and child RLMs, we train one model to play both roles of a parent decomposer and child sub-agent. We use a simple RL training setup where rollouts from children RLMs inherit the advantages of rollouts of the parent RLMs that spawned them, which allows us to train a single policy. This eliminates the need for additional reward signals for individual child RLM trajectories.

On an evidence selection task over several scientific documents, we show that an RL fine-tuned 4B model performs just as well as Claude Sonnet 4.6 with an identical RLM harness and REPL environment, all while being a fraction of the size and cost.

Our code, which includes training scripts, our implementation of the RLM scaffold, and our evidence selection environment, is available on SkyRL.

RLMs spawn language models (LMs) inside a programmatic environment that stores long user prompts (context), which are traditionally fed directly into the context window of an LM. In this environment, the context is an external object that the LM can inspect through programmatic operations and decompose by recursively calling itself with the ultimate goal of answering some user query about the context.

Like the original RLM paper, we use a Python Read-Eval-Print-Loop (REPL) as our environment. Rather than treating code execution as just another tool, RLMs in a REPL make code the primary interface through which the model inspects and transforms data. Every turn, the model writes code it wants to execute, the REPL executes the code, and the RLM orchestrator returns the results of the executed code (primarily print() statements) as a user message back to the model for the next turn.

![RLMs interact with their context inside of a Python REPL environment. They can recursively call themselves to decompose large prompts. Figure from [[1]](https://www.alphaxiv.org/abs/2512.24601).](https://www.alphaxiv.org/assets/rlm-overview-Bir1v2Vm.png)

The REPL exposes a set of built-in functions to the model:

• FINAL(answer) / FINAL_VAR(variable_name) — Marks the end of the rollout: FINAL(...) returns the literal string as the final answer, while FINAL_VAR(...) looks up an existing REPL variable by name and returns its value.

• rlm_query(prompt, context=None) — Spawns a single child RLM rollout with the given prompt (and optional context override), running a fresh agent loop under the same policy, and returns the child's final answer string back into the parent's REPL.

• rlm_query_batched(prompts, context_list=None) — Same as rlm_query but dispatches multiple children in parallel (one per prompt, paired with the corresponding context) and returns a list of their final answers in order.

The ability to interact with context programmatically and spawn sub-RLMs or sub-LMs makes RLMs a very powerful inference strategy for tackling long context problems.

For this blog, we'll focus on the task of evidence selection from scientific documents. Given a question and a set of arXiv papers, the objective is to return snippets from the papers that answer the question. The context that is stored in the REPL for this task is the full text of all of the papers in the set for a given question.

In the single paper case, consider the following question about the original RLM paper: *What baselines are used for this paper?* A simple strategy that the RLM can use is to call a keyword search function with the term "baseline" over the context, analyze the matches, and return the most relevant snippets.

When extending this to multiple papers, it's clear the task can be decomposed and parallelized with sub-RLMs. An obvious approach is to have the root RLM identify which papers are worth further exploring and spawn sub-RLMs to extract snippets from individual papers. One point worth highlighting here is that the benefits of RLMs extend beyond just long-context tasks. This particular task doesn't have an incredibly long context to begin with (experimenting with several hundred papers is something we want to try), but it is highly parallelizable. Not only does embracing the parallelizable nature of the task improve wall-clock time, but it actually also improves task performance. Several works show that sequential reasoning throws LLMs into the "prefix" trap, which is where the model ends up exploring whatever it first explores.

Another point to make clear is that RAG is a poor fit for this task. The model needs to dynamically return verbatim spans of varying length and count, not a fixed-size top-k. RAG and RLMs aren't mutually exclusive though, indexed chunks could be exposed as just another REPL tool alongside search and extract_section.

The original RLM paper SFTs Qwen3-8B with RLM-like reasoning trajectories from Qwen3-Coder-480B-A35B-Instruct. The purpose of SFT is to teach the model how to navigate a REPL environment, including RLM-specific syntax for submitting an answer and making sub-LM calls. However, SFT was not used to train RLMs for a specific task or dataset.

For reasoning models and traditional agent harnesses, RL has proven to be a robust way to improve the capabilities of a model for a specific task without the catastrophic forgetting that accompanies pure SFT-based approaches. How can we extend this to RLMs?

RLM (no sub-calls)

Before considering the recursive element of RLMs, we want to first confirm we can RL fine-tune Qwen3.5-4B on the single-paper variant of our evidence selection task, which doesn't involve spawning child RLMs. There are a couple of takeaways from these initial runs that are worth sharing.

Task Strategy. The specific task strategy has to be in the prompt. This includes descriptions of predefined functions to guide the RLM and expedite rollout generation. For the single-paper task, the strategy is searching the paper with multiple keyword variants using search, expanding the most promising snippets with refined search calls, and then submitting final answers with extract_section, all across 5-7 turns. Without an explicit strategy, even a frontier model like Sonnet 4.6 will take 90 seconds to generate a rollout compared to 30 seconds when given a strategy.

Cold-start SFT. While this may not be needed with larger models, the RLM harness is tricky enough that even with a good prompt and predefined tools, Qwen3.5-4B will have 0 pass@16 scores. RLM-based tasks are outside the edge of competence for most small models.

When training without an SFT phase, we observed many of the same failure modes observed in the original RLM paper, which include using the wrong syntax to submit final answers and excessive reasoning and tool calls that would bloat the model's context window across turns.

For the SFT phase itself, we use the same methodology from . We generated teacher RLM rollouts with Qwen3.5-397B-A17B on the same evidence selection task and filtered out rollouts that either contained REPL errors or scored zero F1 on character-span overlap between retrieved evidence snippets and gold evidence snippets. Our SFT dataset was a small, held-out portion of the RL dataset of a few dozen examples. In initial runs, we found that doing SFT on traces produced over the entire RL training dataset (even though it isn't a large dataset) led to entropy collapse and subsequent instability .

Stepwise Training. Another nuance of the RLM scaffold is that successive turns do not share prefixes. The user prompt is not part of the persistent chat history and is instead appended before each turn. The point of this is to ensure the original user query is not buried deep into the context window of the model and that the RLM is reminded of its task.

Because the per-turn user prompt is rewritten rather than accumulated, you cannot use a rollout as a single training example. Each turn has to be a separate training sample, so a rollout of N turns produces N samples. For advantage calculation, only rollouts corresponding to the last step of trajectories are included in the GRPO group, and the advantage is broadcast to rollouts from previous turns. See more about step-wise RL training here.

LLM Judges. We use rubric-based LLM judges for reward assignment. We initially tried verifiable rewards like F1 of selected snippets, but this proved to be very noisy. Questions like "Which method scores the best on X baseline?" could be answered with several selections of text, some of which weren't included in our labels. We experienced a similar issue in our previous work on retrieval agents. To circumvent this, we used rubric-based LLM judges that were provided with the original query, the ground truth text, and the predicted text. Rubric-based judges have been shown to be more robust to reward hacking . A great overview of rubric-based rewards by Cameron Wolfe can be found here.

With these considerations, RL fine-tuning yields significant improvements on the single-paper task, with eval judge scores jumping from around 0.6 to 0.8 with Qwen3.5-4B.

Note that because we RL on top of an SFT model, training begins at 0.6 reward rather than zero, demonstrating the importance of a cold-start SFT phase.

RLM (with sub-calls)

Now let's consider the multi-paper case, which requires a true RLM with recursive sub-calls. We want the root RLM to identify which papers are worth dispatching child RLMs to, and child RLMs should extract the relevant passages from their assigned paper.

It might be tempting to train separate policies for root and children RLMs. However, this would require two sets of reward signals and an unwieldy training pipeline to train two policies. Alternatively, we can use a frozen model for sub-calls and only train a model for the root RLM, which is the approach taken by the original RLM paper. However, given that an SFT 4B model is unable to perform the child task successfully, some form of on-policy training is necessary. We devise a training objective and an RL training setup that can generalize across different RLM tasks and enables a single policy to effectively learn both root and child RLM roles.

Put simply. Advantages are computed for root RLM rollouts using standard GRPO.

Each child rollout inherits the advantage of its parent root, and child contributions to the loss

are averaged (divided by kgk_g, the number of children in that rollout) so no single root is

overweighted for spawning more sub-calls.

Or more formally. For each query xx in the batch, sample a group of GG root rollouts:

{yg}(g=1)G∼πθold(⋅∣x)\lbrace y_g \rbrace_{(g = 1)}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid x)

Each root RLM rollout ygy_g deterministically induces a set of child prompts ϕ(yg)={xg,i}i=1kg\phi(y_g) = \lbrace x_{g,i} \rbrace_{i=1}^{k_g} via its rlm_query or rlm_query_batched calls, and children are sampled from the same policy:

yg,i∼πθold(⋅∣xg,i),i=1,…,kgy_{g,i} \sim \pi_{\theta_{\text{old}}}(\cdot \mid x_{g,i}), \quad i = 1, \dots, k_g

Advantage Calculation. Only root RLM rollouts receive a verifiable reward rg=R(x,yg)r_g = R(x, y_g). Group-relative advantages are computed over the GG root rollouts:

Ag=rg−mean ⁣({rg′}g′=1G)std ⁣({rg′}g′=1G)A_g = \dfrac{r_g - \text{mean}\!\left(\lbrace r_{g'} \rbrace_{g'=1}^G\right)}{\text{std}\!\left(\lbrace r_{g'} \rbrace_{g'=1}^G\right)}

Advantage Inheritance. For simplicity, we have every child of ygy_g inherit ygy_g's advantage.

Ag,i:=Agfor all i=1,…,kgA_{g,i} := {A_g} \quad \text{for all } i = 1, \dots, k_g

GRPO Objective. Let

ρθ(y∣x)=πθ(y∣x)πθold(y∣x)\rho_\theta(y \mid x) = \dfrac{\pi_\theta(y \mid x)}{\pi_{\theta_{\text{old}}}(y \mid x)}

denote the per-token importance ratio (token index suppressed for clarity).

The GRPO objective extended over the RLM tree is:

J(θ)=Ex,{yg},{yg,i}[1G∑g=1G(Lgroot(θ)⏟root rollout+1kg∑i=1kgLg,ichild(θ)⏟child rollouts)]−β DKL[πθ∥πref]\mathcal{J}(\theta) = \mathbb{E}_{x, \lbrace y_g \rbrace, \lbrace y_{g,i} \rbrace} \Bigg[ \dfrac{1}{G} \sum\limits_{g=1}^{G} \Bigg( \underbrace{\mathcal{L}_g^{\text{root}}(\theta)}_{\text{root rollout}} + {\color{red}\dfrac{1}{k_g} \sum\limits_{i=1}^{k_g} \underbrace{\mathcal{L}_{g,i}^{\text{child}}(\theta)}_{\text{child rollouts}}} \Bigg) \Bigg] - \beta \, \mathbb{D}_{\text{KL}}[\pi_\theta \| \pi_{\text{ref}}]

where each per-rollout clipped surrogate is

Lgroot(θ)=1∣yg∣∑t=1∣yg∣min⁡(ρθ(yg(t))Ag,  clip(ρθ(yg(t)),1−ϵ,1+ϵ)Ag)\mathcal{L}_g^{\text{root}}(\theta) = \dfrac{1}{|y_g|} \sum\limits_{t=1}^{|y_g|} \min\Big( \rho_\theta(y_g^{(t)}) A_g, \; \text{clip}(\rho_\theta(y_g^{(t)}), 1-\epsilon, 1+\epsilon) A_g \Big)

Lg,ichild(θ)=1∣yg,i∣∑t=1∣yg,i∣min⁡(ρθ(yg,i(t))Ag,i,  clip(ρθ(yg,i(t)),1−ϵ,1+ϵ)Ag,i)\mathcal{L}_{g,i}^{\text{child}}(\theta) = \dfrac{1}{|y_{g,i}|} \sum\limits_{t=1}^{|y_{g,i}|} \min\Big( \rho_\theta(y_{g,i}^{(t)}) A_{g,i}, \; \text{clip}(\rho_\theta(y_{g,i}^{(t)}), 1-\epsilon, 1+\epsilon) A_{g,i} \Big)

Note that we add a normalization term 1kg\frac{1}{k_{g}} when summing the loss contributions of the child RLM rollouts. This is to ensure that the contribution across all depths of the RLM is balanced. Without normalization, child rollouts dominate the gradient update when kg≫1k_g \gg 1.

While the training objective above assumes a max RLM depth of 1, this objective can be expanded to any RLM depth with a nice recursive structure. For a given RLM trajectory yy, the recursive subtree loss is:

Lsubtree(y,A)=Lnode(y,A)+1ky∑i=1kyLsubtree(yi,A)\mathcal{L}_{\text{subtree}}(y, A) = \mathcal{L}_{\text{node}}(y, A) + \frac{1}{k_y} \sum_{i=1}^{k_y} \mathcal{L}_{\text{subtree}}(y_i, A)

where {xi}i=1ky\lbrace x_i \rbrace_{i=1}^{k_y} is the set of child RLM prompts dispatched by yy, AA is the advantage assigned to yy, and each child RLM rollout is sampled by yi∼πθold(⋅∣xi)y_i \sim \pi_{\theta_{\text{old}}}(\cdot \mid x_i).

The rationale behind having children inherit their parent trajectory's final advantage is the same rationale behind assigning every token the same sequence-level advantage in GRPO, even though different tokens make different contributions to the sequence-level advantage. This is an unbiased estimator of the true gradient that, with sufficient training steps, yields stable results.

Alternatively, if the child RLM rollouts are highly repetitive, you can randomly sample one to contribute to the gradient instead of averaging across all kgk_g. While this approach can speed up training, it can also be very noisy.

j∼Uniform(1,…,kg)j \sim \text{Uniform}(1, \dots, k_g)

J(θ)=Ex,{yg},{yg,i}[1G∑g=1G(Lgroot(θ)⏟root rollout+Lg,jchild(θ)⏟sampled child)]−β DKL[πθ∥πref]\mathcal{J}(\theta) = \mathbb{E}_{x, \lbrace y_g \rbrace, \lbrace y_{g,i} \rbrace} \Bigg[ \dfrac{1}{G} \sum\limits_{g=1}^{G} \Bigg( \underbrace{\mathcal{L}_g^{\text{root}}(\theta)}_{\text{root rollout}} + {\color{red}\underbrace{\mathcal{L}_{g,j}^{\text{child}}(\theta)}_{\text{sampled child}}} \Bigg) \Bigg] - \beta \, \mathbb{D}_{\text{KL}}[\pi_\theta \| \pi_{\text{ref}}]

For the final multi-paper RLM training runs, we train on an SFT Qwen3.5-4B model. Training is done on a single 8xH200 node with a batch size of 16 and 8 samples per prompt. With up to 4 child RLMs per parent, generation can hit 512 concurrent rollouts, which initially surfaced race conditions around REPL timeouts for child RLMs. After fixing these, we observe a consistent and stable reward curve.

Through RL post-training the 4B model, the average rubric score jumps from 0.3 to 0.6 on the training dataset. On the eval dataset, the fine-tuned RLM stacks up well against other frontier models using the same RLM scaffold.

Claude Sonnet 4.6<text x="0" y="0" dy="12" text-anchor="middle" fill="#3a3e45" font-family=""IBM