動画モデルにおけるビジュアルプロンプトの活用方法について解説

**

*Core team

TL;DR:** Just as text-based prompt engineering systematically improves language model performance, visual prompt engineering (VIPE) serves as a simple, compute-efficient approach to elicit better visual reasoning performance from video models by transforming the input task image (e.g., making abstract sketches photorealistic).

Physics VPCT

Conjunctive Search

Connect the Dots

Sort 3 Numbers

Maze

RushHour

Physics VPCT

Conjunctive Search

Connect the Dots

Sort 3 Numbers

Maze

RushHour

Abstract

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from *visual* prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models.

Podcast

Listen to the NotebookLM-generated podcast overview of our paper:

Your browser does not support the audio element.

Task Examples

Below, we show examples for each task and split. We show a solution attempt on the original image (which often fails), a solution attempt on the most successful VIPE variant for the task and split, and other VIPE variants obtained by freeform VIPE or atomic concept edit (ACE) VIPE.

Physics VPCT

Predict which bucket the ball will land in after rolling down the ramps.

Text Prompt: "The ball moves down in a physically plausible way, sliding down the obstacles, and ends up in one of the three containers at the bottom. [...]"

Original Solution Attempt Failed

Industrial factory chutes

Success

Other Variants

Ancient stone ruins

Baking kitchen prep

Billiard trick shot

Candy sweets landscape

Claymation stop motion

Construction site steel

Garage rube goldberg

Macro forest nature

Marble run toy

Mini golf course

Neon cyberpunk arcade

Office desk supplies

Retro pachinko machine

Science chemistry lab

Steampunk brass mechanism

Water park slides

Winter ice slopes

Wooden physics puzzle

Zen garden bamboo

Conjunctive Search

Locate the target object defined by a conjunction of features (e.g., color and shape).

Text Prompt: "The two blue balls begin to glow."

Original Solution Attempt Failed

Globe lightbulbs case

Success

Other Variants (Freeform)

Arcade joystick panel

Billiard table felt

Candy assortment countertop

Deep sea bioluminescence

Dj mixer interface

Fantasy crystal orbs

Gemstone jewelry velvet

Gym equipment wood floor

Holiday ornaments snow

Lab plasma balls

Macro marbles and dice

Neon mini golf

Night pool led floats

Photorealistic 3d shapes

Pinball arcade bumpers

Sci fi energy cores

Synthwave neon grid

Twilight led bocce

Vintage toys hardwood

Other Variants (ACE)

REPLACE flat color of the two blue circles with a 3D glossy material

REPLACE flat rendering of blue circles with 3D glossy volumetric material

SET border outline of the two blue balls as a thick white stroke

REPLACE flat color of blue balls with 3D glossy material

REPLACE flat material of blue circles with glossy 3D rendering

SET material of the two blue circles as glossy 3D glass

SET rendering material of the blue circles as 3D glossy

SET material of the blue balls as glossy 3D spheres

SET materiality of the two blue circles as 3D glossy spheres

REPLACE size of the blue circles with a significantly larger scale

SET size of the blue circles as significantly enlarged

SET material of the blue circles as glossy 3D

SET color saturation of non-target shapes as completely desaturated grayscale

REMOVE color saturation from all non-target objects

REPLACE colors of all red circles and blue squares with grayscale

SET material of the blue circles as glossy 3D spheres

SET the material of the blue circles as glossy 3D spheres

REPLACE flat shading of the two blue circles with a glossy 3D material

SET material of the blue circles as glossy 3D

Connect the Dots

Connect same-colored dots.

Text Prompt: "Connect each pair of same-colored circles with a line."

Original Solution Attempt Failed

Thick black stroke

Success

Other Variants (Freeform)

Aerial flower beds

Building block baseplate

Chalk street art

Corkboard pushpins

Embroidery hoop

Frosted cookies baking sheet

Glowing neon nodes

Gym floor vinyl spots

Mobile puzzle ui

Pcb terminal nodes

Retro arcade crt

Round stickers cardboard

Sewing buttons fabric

Synth patchbay jacks

Wax seals parchment

Wet acrylic dollops

Whiteboard magnets

Wooden board game tokens

Yarn pegboard

Zen garden stones

Other Variants (ACE)

REPLACE flat 2D shapes with volumetric 3D spheres

REPLACE standard hues with highly saturated neon colors

SET border of each circle as thick outline

REPLACE solid white background with dark black background

SET outer border of the circles as a thick black stroke

REPLACE current background color with a high-contrast dark gray

SET internal content of the circles as matching text labels

REPLACE bright white background with a dark slate background

SET border of each circle as a thick black stroke

SET border of each circle as a thick white outline

SET color saturation of the circles as ultra-vibrant

SET color saturation of the circles as maximally saturated neon

SET size of the circles as significantly larger

REPLACE current rendering style with glossy 3D spheres

REPLACE current background color with solid black

REPLACE current circle sizes with 40% enlarged versions

SET color saturation of the circles as highly increased

SET size of all circles as 30 percent larger

REPLACE plain white background with a solid black background

Sort 3 Numbers

Sort three numbers in ascending order.

Text Prompt: "The numbers pop and disappear one at a time, in numeric order, starting from the smallest one."

Original Solution Attempt Failed

Retro arcade sprites

Success

Other Variants (Freeform)

Ar notification badges

Bubblegum bubbles

Chalk shooting targets

Dry ice smoke bubbles

Giant bubble wrap

Glass lightbulbs

Glossy candy blobs

Glowing fireworks

Holographic data nodes

Inflatable pool floats

Latte foam bubbles

Numbered balloons

Papier mache pinatas

Pop art comic splats

Popping boba pearls

Soap bubbles

Squishy jelly cubes

Vr aim trainer targets

Water balloons

Other Variants (ACE)

SET rendering of the numbers as 3D embossed

REPLACE current plain rendering with a neon sign aesthetic

REPLACE current background color with a high-contrast dark solid

REPLACE plain white background with a solid high-contrast dark color

REPLACE sterile environment with a cohesive chalkboard texture

SET color of each number as a unique vivid hue

SET color of each number as a distinct saturated hue

REPLACE 2D flat typography with 3D volumetric geometry

REPLACE current flat typography with 3D metallic rendered text

REPLACE uniform black text color with distinct saturated colors

SET spatial boundaries of each number as distinct geometric boxes

SET overall scale of the numbers as uniformly enlarged

REPLACE white background with a dark high-contrast background

REPLACE current font weight with an ultra-bold heavy weight

REPLACE current uniform black text color with distinct vibrant colors

SET rendering of the numbers as subtle volumetric 3D geometry

REPLACE current white background with a dark solid color

SET surface texture of the scene as a green chalkboard

SET dimensionality of the numbers as subtly embossed 3D

Maze Solving

Find a path from start to finish in various maze sizes.

Text Prompt: "Create a 2D animation based on the provided image of a maze. The red circle slides smoothly along the white path, stopping perfectly on the green circle. [...]"

Split: 5×5 Maze

Original Solution Attempt Failed

Classic 2d arcade game

Success

Other Variants

Algorithmic pathfinding visualization

Animated transit map infographic

Classic flash web animation

Dark mode gps tracking ui

Dark mode ui motion

Digital circuit routing

Digital rhythm game interface

Flat 2d physics sandbox

Flat felt stop motion

Glossy mobile puzzle interface

Industrial cam simulation

Layered 2d motion graphics

Matte papercraft 3d

Neon motion graphics

Precise vector math animation

Robot vacuum app map

Sleek 2d vector puzzle

Smooth arcade tabletop

Vintage celluloid animation

Split: 7×7 Maze

Original Solution Attempt Failed

Classic 2d arcade game

Success

Other Variants

Algorithmic pathfinding visualization

Animated transit map infographic

Classic flash web animation

Dark mode gps tracking ui

Dark mode ui motion

Digital circuit routing

Digital rhythm game interface

Flat 2d physics sandbox

Flat felt stop motion

Glossy mobile puzzle interface

Industrial cam simulation

Layered 2d motion graphics

Matte papercraft 3d

Neon motion graphics

Precise vector math animation

Robot vacuum app map

Sleek 2d vector puzzle

Smooth arcade tabletop

Vintage celluloid animation

Split: 9×9 Maze

Original Solution Attempt Failed

Neon motion graphics

Success

Other Variants

Algorithmic pathfinding visualization

Animated transit map infographic

Classic 2d arcade game

Classic flash web animation

Dark mode gps tracking ui

Dark mode ui motion

Digital circuit routing

Digital rhythm game interface

Flat 2d physics sandbox

Flat felt stop motion

Glossy mobile puzzle interface

Industrial cam simulation

Layered 2d motion graphics

Matte papercraft 3d

Precise vector math animation

Robot vacuum app map

Sleek 2d vector puzzle

Smooth arcade tabletop

Vintage celluloid animation

Split: Irregular Maze

Original Solution Attempt Failed

Digital rhythm game interface

Success

Other Variants

Algorithmic pathfinding visualization