RL エージェントのスキルを活用した自己研究ワークフローの実行方法と NVIDIA NeMo の活用

Coding AI agents are becoming practical operators for long-running machine learning (ML) workflows. They can inspect repositories, set up runtimes, resolve build issues, launch experiments, monitor execution, analyze metrics, and summarize results.

For reinforcement learning (RL) research, this matters because meaningful metrics often appear only after the essential experiment infrastructure is in place. Autoresearch is an open source Python project by Andrej Karpathy for automating AI and ML model training.

With this approach, autonomous AI agents translate high-level goals into hypotheses, edit and test a real codebase, keep metric-improving ideas, and hand results back to the human researcher. This enables teams to build specialized domain agents starting from a performant open model, such as NVIDIA Nemotron, then improve it with RL against measurable domain tasks while maintaining control over data, IP, deployment, and the training workflow.

This post walks through how to run a lightweight, no-code-from-the-user-side, skill-based autoresearch workflow. It was tested using a frontier coding agent (Codex with GPT 5.5) on NVIDIA NeMo RL and NVIDIA NeMo Gym, using an NVIDIA Brev GPU instance.

Building an autonomous RL research workflow

The workflow demonstrates the following three agent capabilities:

• Full-stack autonomy: Setting up the software stack; resolving dependencies; managing GPU memory, disk space, and checkpoints; launching experiments; monitoring runs; and debugging issues.

• Goal-driven autoresearch: Profiling a baseline, proposing or following hypotheses, launching experiments, analyzing metrics, and iterating toward a research objective.

• Paper-to-code: Reading a paper, forming an implementation plan, translating the algorithm into code, adding tests, and starting validation training.

In the examples provided in this post, Codex first brings up a full NeMo RL and NeMo Gym stack for a vision language model (VLM) RL training smoke test. It then conceptualizes and creates a novel NeMo Gym visual counting environment from scratch, and trains the Qwen3-VL-2B-Instruct model, increasing its accuracy from 25.0% to 96.9% on the task. Finally, it implements an off-policy RL algorithm from a research paper and begins a 10-hour validation training campaign.

The goal of autoresearch is not to remove the researcher from the loop, but instead to hand off the repetitive setup and iteration work to the agent. Researchers are still responsible for setting goals, reviewing milestones, steering strategy, and making the final decision.

Using NeMo RL, NeMo Gym, and agent skills for autoresearch

NeMo RL and NeMo Gym are open source libraries that are part of the NVIDIA NeMo framework. NeMo RL is built on AutoModel, Megatron-Bridge, and vLLM for post-training LLMs and VLMs, orchestrated by Ray. It supports workflows such as GRPO, DPO, SFT, and reward model training, with recipe-driven configuration that can scale from small validation runs to distributed training. NeMo Gym provides environments where models can interact with tasks, receive rewards, and learn through live-generated experience.

Together, NeMo RL and NeMo Gym provide a useful substrate for agent-led RL research. An agent can inspect recipes, wire up a new environment, run a baseline, adjust training parameters, and compare results in the same repository.

Codex with GPT 5.5 offers strong reasoning, code navigation, and tool-use capabilities, but it does not automatically know every local operating convention. It may not know where checkpoints should live, which metrics are authoritative, or how to recover the campaign objective after a long session, context compaction, or disconnect.

For this reason, three complementary agent skills were also used:

• Brev-etiquette: Operating guidance for NVIDIA Brev GPU instances. It keeps the repository clean, stores large artifacts such as checkpoints and caches on the intended volume, and handles secrets safely.

• Session-memory: A durable session diary for long-running work. It records the overall goal, subtasks, loaded skills, important files, decisions, progress, and handoff notes.

• Autoresearch: The experiment loop. It preserves the user’s objective, establishes baselines, creates a branch for each hypothesis, logs attempts in a ledger, monitors stop rules, and summarizes results for human review.

These skills act as structured, reusable workflow instructions. They encode operating context and institutional knowledge so the agent can execute the research loop more reproducibly.

Prerequisites

This walkthrough uses the following setup:

• Visual Studio Code

• Codex plugin with the GPT 5.5 model

• A Brev instance with one NVIDIA L40S 48 GB GPU

• NeMo RL repository (for the Brev launchable; precloned under /home/ubuntu/RL)

• Hugging Face and Weights & Biases credentials, if the workflow requires authenticated model, dataset, or logging access

The following sections validate the three agent capabilities. First, set up the machine and validate the stack. Next, run a goal-driven autoresearch campaign. Finally, use the same agent workflow to implement a research paper and begin a longer validation training run.

Full-stack autonomy

This section demonstrates the agent’s ability to automanage the full hardware and software stack. Before starting an autoresearch loop, first validate that the machine, repository, dependencies, model access, and training loop work end to end. This setup validation is the foundation for longer agent-led research runs.

Step 1: Launch a Brev instance

Launch a Brev instance with one NVIDIA L40S 48 GB GPU using the NeMo-RL Autoresearch Launchable.

Step 2: Connect VS Code to the remote instance

On Windows, open the SSH config file, typically under C:\Users\<YOUR_USER_NAME>\.ssh\config, and add the following:

Host Brev-Ubuntu

HostName <BREV_IP_ADDRESS>

User ubuntu

IdentityFile C:/Users/<YOUR_USER_NAME>/.brev/brev.pem

Find BREV_IP_ADDRESS listed at the top of the Brev instance page. In VS Code, select Connect to Host and choose the configured host name, such as Brev-Ubuntu. On macOS and Linux, the Brev key is typically under ~/.brev/brev.pem, and the SSH configuration is similar.

Step 3: Install and configure Codex

From VS Code, open the NeMo RL directory under /home/ubuntu/RL. Then install the Codex IDE extension from the VS Code extensions marketplace. After installation, Codex appears in the chat window. Sign in using an API key or ChatGPT account.

Codex can run in several permission modes: Default, Auto-Review, and Full Access. For this workflow, Auto-Review is a reasonable starting point. It allows Codex to run sandboxed commands while adding review for elevated actions. Full access gives broader control over the instance and should only be used when you are comfortable with the security settings.

Step 4: Prepare the NeMo RL workspace

Open a VS Code terminal in /home/ubuntu/RL. If needed, create a .env file for credentials:

export WANDB_API_KEY=wandb_v1xxx

export HF_TOKEN=hf_mqyyy

Familiarize yourself with the Brev environment. For example, type df -h to find out more about the storage configuration on the virtual instance.

Step 5: Run a NeMo RL smoke test

Now prompt Codex to set up NeMo RL and run a small RL training smoke test:

/goal Use the nemo-rl-brev-etiquette skill under ./skills, set up NemoRL to

do a smoke test RL training of the Qwen3-VL-2B-Instruct model

The /goal command sets a persistent objective for Codex to work toward. In this run, Codex should load the Brev operating skill, initialize missing NeMo RL submodules, download the Qwen3-VL model from Hugging Face, route large assets such as Hugging Face, Torch, Triton, uv, Ray, logs, and worker caches to /ephemeral, and resolve dependency or configuration issues until a training step completes.

The elapsed time is approximately 40 minutes to an hour. The goal is not the most optimized NeMo RL setup, but a validated end-to-end path through repository setup, model access, dependency resolution, and training execution.

Step 6: Persist the setup with session memory

After the stack is working, persist the setup state with the following prompt:

Now use the nemo-rl-session-memory skill under ./skills, persist this setup information

To resume after a network disconnect or VS Code restart, you can load the latest session with the following prompt:

Use the nemo-rl-session-memory skill under ./skills, load the latest session, and provide me with a summary

This provides Codex with the previous goal, environment assumptions, loaded skills, important paths, and recent command outcomes before continuing.

Goal-driven autoresearch

With the stack validated, the next step is to move from setup to a goal-driven campaign. In this mode, the human provides the objective and budget, and Codex manages the loop: baseline, hypotheses, experiments, metrics, and summary.

Step 7: Create a new NeMo Gym environment

Before launching the campaign, ask Codex to implement a new RL environment. The example below creates a visual counting task for a VLM:

/goal Now using the Nemo gym circle click environment as a reference,

implement a new Gym environment named "star count", that teaches

a model to count the number of stars of different colors on a canvas of

different resolutions. Show me several examples

The resulting environment generates images with colored stars at varying canvas resolutions and asks the model to count the number of stars by color.

Step 8: Start the autoresearch campaign

Next, start a time-bounded campaign using the following prompt:

Great. /goal Now use the nemo-rl-autoresearch and nemo-rl-session-memory skill under ./skills, start a new

auto research session, goal: Train the Qwen3-VL-2B-Instruct model to high accuracy, using SFT algorithm

on the star count gym environment. Start with profiling the model out of the box accuracy. Time budget: 5h

The prompt must contain the objective, method, task environment, baseline requirement, and time budget. The autoresearch skill translates these into a campaign with measurable stop rules and an experiment ledger.

In this run, Codex wires up the synthetically generated data from the star count environment into the NeMo RL SFT path. SFT is used in this case because a prior autoresearch session found that GRPO did not improve accuracy on this task, while the Codex proposed SFT route later proved effective.

In one experiment, Codex achieved the best result in under an hour. The Codex final report when prompted to provide a brief summary is shown below:

The star_count SFT campaign trained Qwen/Qwen3-VL-2B-Instruct on the new NeMo Gym visual

counting environment using LoRA SFT. We first profiled the model out of the box on a fixed 64-example

held-out eval split, where it reached 25.0% exact accuracy with mean reward 0.705208. After setting up the

environment, data generation, evaluator, and SFT recipe, the first LoRA smoke run improved accuracy to 78.125%.

Scaling the SFT data and training steps produced the main gains. A 2048-train / 256-val, 160-step run

reached 95.3125% exact accuracy, and the best 4096-train / 512-val, 320-step run reached 96.875% exact

accuracy with mean reward 0.992188. The best checkpoint was step_240, which slightly beat the final checkpoint

on partial-credit reward while tying exact accuracy.

Overall, accuracy improved from 25.0% to 96.875%, a gain of 71.875 percentage points. The remaining errors

were not formatting failures; they were near-miss off-by-one count mistakes on individual colors, suggesting the

next best improvement would be targeted hard-case augmentation for dense or visually ambiguous scenes.

The remaining errors were near-miss off-by-one count mistakes on individual colors. This suggests targeted hard-case augmentation for dense or visually ambiguous scenes as the next experiment.

This captures the core value of the agent-led research loop. Specifically, Codex did not run only one command. Instead, it profiled a baseline, built the environment and data path, ran a validation experiment, scaled data and training steps, evaluated checkpoints, analyzed errors, and proposed a plausible next direction for human review.

Paper-to-code

Paper-to-code is an idea pioneered by OpenAI PaperBench to auto-implement research ideas from papers. Speeding this process, Codex reads a paper, derives an implementation plan, writes and tests the code, and then kicks off training.

Step 9: Give Codex a paper and an implementation target

For this example, the target paper is LLMs Can Learn to Reason via Off-Policy RL. The paper introduces Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL), a long-lag off-policy RL method. Instead of treating the training-inference policy mismatch as a bug to eliminate, OAPL uses the lagged generation policy as part of the objective, enabling stable and efficient training even when the training and generation models are substantially out of sync.

This is a useful test case because it requires both algorithmic implementation and system-level judgment. The loss computation changes, and the long-lag setup also affects how training and generation are coordinated.

The composite prompt is:

/goal Start a new session with nemo-rl-session-memory skill under ./skills. First, read

Write and run unit tests to satisfaction. Write document. Then start a new autoresearch

session with the nemo-rl-autoresearch skill, train a Qwen3-1.7B model with this algorithm

on the DAPO-math-17K dataset. Time budget: 10h

Codex breaks the request into the following stages:

• Read the PDF

• Extract the algorithm

• Inspect NeMo RL hook points

• Produce an implementation plan

• Translate the algorithm into code

• Add unit tests

• Write documentation

• Start a validation campaign on Qwen3-1.7B with DAPO-math-17K dataset

This prompt is intentionally demanding. In practice, the safer workflow is staged. First use Codex for reading and planning with human review, then approve implementation, documentation, and unit tests, and only then start a longer autoresearch validation campaign.

Additional validation can include self-consistency, where Codex implements the algorithm twice on separate branches and compares decisions, or an agentic cross-check, where another frontier coding agent reviews the paper, implementation, tests, and results.

Step 10: Training validation and follow-up research question

Figure 3 shows the training progress of the OAPL algorithm compared with a GRPO variant, DAPO, on the Qwen3-1.7B model and the DAPO-math-17K dataset.

In this run, OAPL reached higher accuracy with fewer training steps, consistent with the paper’s central claim about effective long-lag off-policy training. The right panel also shows that OAPL used significantly more tokens than DAPO, which creates a natural follow-up research question: Why does OAPL become more verbose, and what algorithmic changes could promote brevity without sacrificing accuracy?

Tips and best practices for autoresearch workflows

While increasingly powerful, frontier agents still exhibit a few failure modes to watch out for, including:

• Context drift: Long sessions can pass through multiple context compaction cycles. Without durable memory, the agent may forget the objective, loaded skills, or stop rules.

• Local housekeeping: Logs, checkpoints, datasets, and caches can scatter across the filesystem unless the machine-specific etiquette is explicit.

• Steering drift: Mid-run instructions are useful, but follow-up questions sometimes derail an autoresearch session, leading to early termination.

• Low-signal loops: Sometimes Codex gets stuck in a trial loop. For example, training for a small number of steps and/or using a tiny batch, which cannot provide enough signal. A human researcher should easily spot this unproductive behavior and steer Codex away.

Several practices we have found useful during the building and testing of the autoresearch skills for NeMo RL and NeMo Gym include the following:

• Treat autoresearch skills as evolving workflow instructions. Codex can help update them over time to reflect institutional conventions, best practices, and lessons learned from previous runs.

• While Codex is resourceful and helpful, for tough research questions, human guidance and judgment are still mission-critical.

• It’s useful to go through an explicit brainstorming and planning process to validate ideas before a long-running autoresearch execution phase.

• Think of Codex as a capable intern or junior colleague. This implies that the human researcher doesn’t necessarily need to maintain a deep, single-line-of-code-level understanding of every implementation, but still has methodologies to validate to be confident in the work carried out.

• Set explicit budget constraints before starting a long-running autoresearch session, such as a time limit, GPU-hour budget, or maximum number of experiments. This keeps the agent focused on the highest-value hypotheses and prevents overly broad experiment sweeps.

Get started with autoresearch

Agents are not replacing researchers, but rather supporting a productive division of labor. The agent handles much of the setup, debugging, logging, and experiment loop while also providing insights and proposals for the next steps. The researcher defines the goal, provides domain judgment, steers the campaign, and decides which results are meaningful.

With NeMo RL, NeMo Gym, Brev, Codex, and a small set of operating skills, researchers can reduce the time and compute cost spent on setup, debugging, and repeated experiment execution. This shifts more of the workflow toward deciding which experiments are worth running next and how to interpret the results.

To get started with autoresearch for reinforcement learning, check out these related resources:

• Try the environment from the tutorial with the Brev Launchable.

• Review the NVIDIA-verified Agent Skills documentation and browse the NVIDIA/skills GitHub repo, including the NeMo RL Auto Research skill.

• Explore NVIDIA NeMo RL and NVIDIA NeMo Gym for RL training and gym environments.