Mistral AI、単一RGBカメラで複雑な環境をナビゲートする8Bモデル「Robostral Navigate」を発表

Mistral AI has released Robostral Navigate, its first model built for embodied navigation. The 8B model takes RGB images and a plain-language instruction, then moves a robot. Notably, it reaches 76.6% success on R2R-CE validation unseen using only a single RGB camera.

What is Robostral Navigate?

Robostral Navigate is an 8B model for robotic navigation through complex environments. These environments include offices, residential buildings, commercial buildings, and outdoor settings. You give it one instruction, and it completes the whole task alone. For example: ‘Leave the lobby, walk through the corridor, enter the supply room, and stop to face the second shelf.‘ Importantly, it moves through live spaces full of people and obstacles never seen in training.

Moreover, most navigation systems use depth sensors, LiDAR, or several cameras together. By contrast, Robostral Navigate uses one ordinary RGB camera and no depth sensors. Fewer sensors also make the system more efficient than multi-sensor approaches.

Navigation via Pointing: How It Works

Beyond hardware, the next question is how the model decides where to move. To choose each move, the model uses a method called pointing. Given a task and observation history, it predicts where the robot moves next. Specifically, it infers the image coordinates of the target in the current camera view. It also predicts the desired orientation upon arrival.

This choice has a clear benefit. Unlike commands based on metric displacements, pointing stays robust to camera-intrinsic and world-scale changes. However, pointing fails when the target sits outside the current field of view. In that case, the model falls back to displacements in the robot’s local coordinate frame. For example: ‘Move 2 meters forward, 1.5 meters to the left, and turn 25 degrees left.‘

(function(){

window.addEventListener('message', function(e){

if(e && e.data && e.data.type === 'rbs-resize'){

var f = document.getElementById('rbs-frame');

if(f && e.data.height){ f.style.height = e.data.height + 'px'; }

}

});

})();

Built From the Ground Up

Notably, Robostral Navigate does not rely on existing open-source VLMs. Instead, it starts from Mistral’s vision-language model built for grounding tasks. Those tasks include pointing, counting, and object localization. Navigation then emerges as a natural extension of grounding. Once the model knows where things are, it learns how to move.

For training data, Mistral built a generation pipeline entirely in simulation. This pipeline produced roughly 400,000 trajectories collected across 6,000 scenes.

Efficient Training and Online Reinforcement Learning

Beyond data, training efficiency was core element to the design. The method uses an algorithm based on prefix-caching. A tree-based attention-masking strategy compresses an entire episode into one sequence. Consequently, the model trains on all time steps in a single forward pass. It also prevents information leakage between time steps.

As a result, this approach cuts training tokens by 22× while keeping all learning signals. In practice, runs that once took months now finish in days. After supervised training, Mistral applied CISPO, an online reinforcement learning algorithm. This stage lets the model learn from trial and error, recover from failures, and acquire exploratory behaviors. It also mitigates the distribution shift issue of vanilla behavior cloning. On its own, this step raised success rate by 3.2%.

Benchmark and Performance

On evaluation, Robostral Navigate reaches state-of-the-art results on R2R-CE. R2R-CE means Room-to-Room in Continuous Environments, a standard instruction-following benchmark. It is built on Matterport3D, where an agent follows language instructions in a continuous 3D environment. The ‘validation unseen’ split holds out environments to measure generalization. The benchmark scores agents with Success Rate, Oracle Success Rate, Success weighted by Path Length, and Navigation Error.

It scores 79.4% success rate on validation seen. It scores 76.6% on validation unseen. Furthermore, it beats the best single-camera approach by 9.7 points. It also beats the best depth or multi-camera system by 4.5 points.

Comparison Table

To place these results in context, the table below contrasts Robostral Navigate with typical multi-sensor systems.

AttributeRobostral NavigateTypical multi-sensor VLN systems

SensorsSingle RGB cameraDepth sensors, LiDAR, or multiple cameras

Model size8B, built in-houseVaries by approach

Base modelIn-house grounding VLMOften built on open-source VLMs

Training data~400,000 sim trajectories, 6,000 scenesMix of simulation and real data

Decision methodPointing + local-displacement fallbackMetric displacements or map-based planning

R2R-CE val-unseen76.6%4.5 pts lower (best depth/multi-camera)

Robot supportWheeled, legged, flying; multiple sizesFrequently platform-specific

The right column describes common practice in vision-and-language navigation, not one named system.

Use Cases With Examples

Given these traits, several applications become practical. In manufacturing, a robot can carry parts between stations from one instruction. In delivery and logistics, a wheeled robot can move packages across a warehouse. In hospitality, a robot can guide a guest from lobby to room. Because the model runs on wheeled, legged, and flying robots, one fleet can share it. Additionally, it stays robust to differences in camera intrinsics across those robots.

A Simplified Code View

To make the pointing loop concrete, the pseudo-code below shows the decision path. It is illustrative, not the official API.

Copy CodeCopiedUse a different Browser

obs_history = [] # past RGB frames

instruction = "Leave the lobby, walk to the second shelf, stop."

done = False

while not done:

frame = camera.read() # single RGB image

obs_history.append(frame)

action = model.predict(instruction, obs_history)

if action.type == "point": # target visible in view

robot.move_to_pixel(action.x, action.y, action.heading)

else: # target out of field of view

robot.move_local(action.forward_m, action.left_m, action.turn_deg)

done = action.stop

Key Takeaways

Robostral Navigate is an 8B embodied navigation model from Mistral AI.

It reaches 76.6% on R2R-CE validation unseen with one RGB camera.

Pointing predicts target pixels; a local-displacement fallback covers out-of-view goals.

Prefix-caching cuts training tokens 22×, turning months of training into days.

CISPO online reinforcement learning added a further 3.2% success rate, framed as a step toward a unified embodied agent.

Check out the Technical details. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Mistral AI Releases Robostral Navigate: An 8B Model Enabling Robots to Navigate Complex Environments Using a Single RGB Camera appeared first on MarkTechPost.