Scale Labs、ロボットデータ品質管理の測定手法を動画レビューから構築

Before a robot demonstration video becomes useful training data, someone has to decide whether it's trustworthy enough to learn from. That decision is the Quality Check (QC) problem, and at scale, no human can make it alone. The goal of an automated QC system is to surface clips that are broken, ambiguous, or risky enough to need human review before they enter the dataset.

Small scale quality control is a human task: open the video, watch carefully, and mark whether the data should be used. Large scale QC determines what the model is allowed to learn from.

We have been studying this problem across two data streams:

• Robot-mounted manipulation videos, where the task is to decide whether a demonstration is valid before annotation
• Ego-camera collector videos, where the task is to catch first-person recording failures, motion failures, environment violations, and incomplete tasks

The surprising lesson is that "just ask a VLM if the video is good" is the wrong recipe. The most useful systems work more like measurement pipelines, they make the evidence explicit, break the rules down into smaller parts, calibrate for each type of failure, and send uncertain clips to humans along with the exact visual evidence that triggered the model.

The Problem With “Pass” and “Reject”

The QC rulebook starts with what sounds like a simple question: should this clip pass?

But a rejected clip can be rejected for many different reasons. The camera might turn green. The task might stop early. An object might fall out of reach. Two actions might overlap. A human hand might enter the scene. A camera might be missing. The clip might contain private information.

Those are very different problems. They need different evidence, and they create different risks if the model gets them wrong.

Data Distribution

Most clips pass. In one sample robot QC snapshot, only a small share of labeled demonstrations were rejected. A model can look decent by mostly predicting “passed,” while still missing the rare failures we care about. The opposite is also bad: a model that predicts “rejected” too often floods reviewers with false alarms and removes useful clips from the dataset.

So we cannot judge the model by accuracy alone. We need to ask three simpler questions:

• How many rejected clips does it catch?
• How many passed clips does it preserve?
• And when it makes a mistake, is that mistake acceptable for the review workflow?

The task worked well because it was narrow and concrete. The VLM was not trying to decide whether the entire clip was good or bad. It was helping reviewers flag one specific, visible camera issue.

The Broad VLM Judge

We started with the simplest version of the idea: give a VLM the rulebook, examples, and the video, then ask it to decide whether the clip should pass or be rejected.

The benchmark was controlled across runs. We used the same task mix, left-camera streams only, the full QC rules in the prompt, and compact contact sheets with rule examples. We also tried several ways of giving the model evidence: full videos, low-fps transcodes, split videos, sampled frames, and a second verification pass using the same model.

The result was not a clean win; each setup failed in a different way. Whole-video inputs caught more rejected clips, but they also rejected too many clips that had passed. Sparse-frame inputs preserved more passed clips, but missed most of the failures. The verifier made the model more conservative, but not more useful.

So the problem was not simply “the model cannot see,” but that we were asking one model to answer too broad a question.

Breaking QC into Small Checks

Instead of asking, “Should this whole video pass?” we moved to focused checks. Each check looks for one failure mode and gets the evidence needed to judge that failure.

• Is there a sensor issue?
• Did the demonstration fail to complete?
• Did an object leave the workspace in an unrecoverable way?
• Do two actions overlap when they should be separate?

This is closer to how a human reviewer thinks. A reviewer doesn't just decide that a clip "feels bad"; they look for a reason: the camera failed, the task stopped early, the object fell out of reach, the action labels overlap. Each reason needs different evidence.

For the VLM checks, we decode the video, sample frames across time, add explicit final-tail frames, and pack everything into labeled contact sheets. That changed the output from a vague pass/reject decision into a specific claim tied to visible evidence.

When Failure Only Appears at the End

Sensor Issue was the first place where focused checks with explicit evidence clearly helped.

In the human labels, many rejected clips were marked for camera artifacts: green frames, pixelation, blur, or other visual corruption. One pattern showed up repeatedly: most of the video looked fine, but the final frame, or final few frames, turned green.

A broad video judge can easily miss that. If the model samples the video internally, it may never see the final bad frame. And if the prompt says not to reject a clip for one isolated artifact, it may ignore the exact frame that explains the label.

The policy needed to be specific: an isolated green tail frame does not always mean the entire demonstration should be rejected, but it should be surfaced as Sensor Issue evidence on the affected frames or segment.

Once we made those tail frames explicit in the contact sheet and aligned the prompt with the policy, the task became much easier. The model was no longer guessing from the whole video. It was checking the exact evidence the label depended on.

See video for animation:

The First Useful Slice

The first useful target was not all robot QC. It was narrower: green frames, pixelation, and color corruption in the camera stream.

This is the kind of visible issue that can be useful in a review pipeline. The VLM contact-sheet linter preserved more passed clips than the CV-only checker, a simpler computer vision script that only looks for visual artifacts, while still catching most green, pixel, and color failures. The CV-only checker caught slightly more rejected clips, but produced more false positives.

The broader Sensor Issue category was harder. Blur and smudge are difficult to separate from normal motion, lighting, and viewpoint changes. When we widened the rule too much, both approaches became less useful.

The takeaway is not that VLMs solve QC on their own. When the failure mode is specific, the evidence is visible, and the policy is clear, a VLM can help reviewers find the right clips faster.

Other Robot Linters

We also built focused linters for incomplete demonstrations, unrecoverable mistakes, and action overlap.

A Broad VLM Judge Was Hard to Tune

video metadataffmpeg 0.5 fpssplit video6 frames6 frames + verifier0%25%50%75%100%% of Videos55%55%50%15%5%47%