NVIDIA、既存アプリにRTXセンサーシミュレーション統合

Developers building 3D, design, simulation, robotics, and industrial digital twin applications need ways to bring physical AI capabilities into the tools and services they already use. Many of these workflows already depend on OpenUSD scenes, simulation-ready (SimReady) assets, Blender-based workflows, CAD pipelines, or domain-specific app stacks. The challenge is how to provide applications and AI agents with callable tools to inspect scenes, generate sensor outputs, test changes, and return evidence developers can trust.

NVIDIA Omniverse libraries, now part of NVIDIA Agent Toolkit, provide modular building blocks for adding these capabilities into existing applications. ovrtx is the RTX sensor simulation library in that stack, now available as prerelease software on GitHub. It provides developers with a lightweight C and Python SDK for generating camera, lidar, radar, and related sensor outputs from OpenUSD content while keeping control inside the host application.

Within the Omniverse libraries stack, ovstage provides a shared runtime stage for OpenUSD scene data, enabling libraries such as ovphysx and ovrtx to exchange simulation state, scene changes, and prim identities through a common data substrate.

This post walks through how ovrtx integrates into existing workflows as part of the Omniverse libraries model. Steps include loading OpenUSD content, configuring render and sensor outputs, stepping from an application loop, reading results back into the app, and applying the same model in a PTC Onshape Render Studio example and a SimReady Blender sample workflow.

How do NVIDIA Omniverse libraries help application developers?

Simulation and rendering are becoming part of the application layer for a wide range of workflows including design, robotics, autonomous vehicles (AVs), industrial digital twins, and physical AI.

Developers and teams need to visualize OpenUSD scenes, generate physically grounded data, test perception systems, validate behavior, and view results inside the tools they use.

These include CAD systems, Blender-based workflows, PLM or PDM workflows, OpenUSD asset pipelines, internal simulation stacks, and domain-specific applications.

A previous post, Integrate Physical AI Capabilities into Existing Apps with NVIDIA Omniverse Libraries, introduces a library-first approach for adding Omniverse capabilities to existing application stacks. That post covers how standalone libraries can expose sensors, physics, and storage capabilities through focused APIs, giving teams more control over where those capabilities run in their own tools and services. ovrtx applies that approach specifically to Omniverse RTX sensor simulation and visualization.

As announced at SIGGRAPH 2026, NVIDIA Agent Toolkit now includes Omniverse libraries, with ovrtx serving as the RTX sensor simulation layer. It helps applications and agents generate camera, lidar, radar, semantic segmentation, and visual preflight outputs from OpenUSD scenes. Used with other Omniverse libraries, it becomes part of a tool-driven workflow for preparing and validating simulation-ready 3D content.

That application control matters whenever sensor simulation needs to fit inside an existing workflow. For physical AI teams, those outputs can support synthetic data generation (SDG), perception testing, robotics workflows, simulation-ready digital twins, and design review.

With ovrtx now available as pre-release software on GitHub, you can inspect the C and Python APIs, run the examples, test OpenUSD scene loading, and evaluate how RTX sensor outputs fit into your own application loop.

How to integrate ovrtx into existing workflows

The core principle behind ovrtx is straightforward. As a developer, you can bring NVIDIA Omniverse RTX sensor simulation into your own applications while maintaining control of the application architecture, UI, data model, and workflow. The lightweight C and Python SDK is designed for projects that need a real-time 3D viewport, RTX rendering, or sensor simulation, including camera, lidar, radar, ultrasonic, and related outputs.

A typical integration follows five steps:

• Create and configure an ovrtx renderer.

• Load or reference OpenUSD content from the application’s existing pipeline.

• Define RenderProducts and RenderVars for the camera, lidar, radar, semantic segmentation, or rendered outputs the workflow needs.

• Step rendering or sensor simulation from the application loop.

• Map the results back to CPU or GPU memory so the app can display, save, analyze, or pass them to another process.

A pseudo-code example is shown below—the host application owns the workflow.

renderer = ovrtx.Renderer()

renderer.open_usd(app_scene_or_usd_layer)

render_products = {"/Render/Camera"}

while app.is_running():

app.update_scene_state()

products = renderer.step(

render_products=render_products,

delta_time=app.frame_delta_seconds,

)

for product in products.values():

for frame in product.frames:

color = frame.render_vars["LdrColor"].map(device=ovrtx.Device.CPU)

app.use_render_output(color)

Control remains in the host application. The app owns the loop; ovrtx produces the requested render or sensor outputs; and the results return to the workflow for display, storage, analysis, or downstream processing.

Core components in the ovrtx integration flow

The host application uses a few core pieces when it calls ovrtx: the Omniverse RTX renderer, an OpenUSD stage, and RenderProducts and RenderVars that define the outputs to generate.

Omniverse RTX Renderer

Omniverse RTX Renderer is the core technology exposed by the ovrtx library. It takes advantage of NVIDIA RTX, combining “on chip” RT Cores in NVIDIA RTX GPUs with AI acceleration enabled by Tensor Cores. NVIDIA Omniverse RTX enables real-time, physically-based rendering and simulation of light transport in 3D environments, using flexible physically-based lighting and materials to model cameras, lidar, radar and related sensor outputs.

ovstage

ovstage is the shared runtime stage for Omniverse libraries. It provides C and Python APIs for reading, writing, querying, and managing OpenUSD scene data such as transforms, materials, hierarchy, metadata, and simulation state across CPU and GPU memory. In a multilibrary workflow, ovstage allows ovphysx to write updated simulation states and ovrtx to read changed scene data for sensor simulation—without each library maintaining a separate scene representation.

OpenUSD

NVIDIA has adopted OpenUSD as the foundational data layer for NVIDIA Omniverse and simulation workflows. OpenUSD defines a standard data representation for 3D scenes and provides C++ and Python APIs to read and write those scenes. Data in OpenUSD is organized into a scene graph that ovrtx can render. USD stages can define and handle complex geometry, photometric lights, OpenPBR materials, cameras, and other sensors.

RenderProducts and RenderVars

RenderProducts and RenderVars provide a USD-native way to define the outputs to obtain from a renderer or sensor simulator.They provide a lot of flexibility to specify output types, resolutions, formats and other parameters. A RenderProduct is associated with one or more sensors, and ovrtx generates the outputs defined by the RenderVars associated with it. RenderProducts and RenderVars can be defined programmatically and stored in the USD scene or defined at runtime using the ovrtx C or Python APIs.

Application flow

Think of ovrtx as an on-demand service your app calls when it needs to visualize an OpenUSD scene or generate render and sensor outputs. The first steps, creating a renderer, loading OpenUSD content, and configuring RenderProducts and RenderVars, usually happen once or infrequently. The final steps, stepping rendering or sensor simulation and reading outputs, typically run inside the application loop.

Create and configure an ovrtx renderer

Your existing app, whether Python, C, or C++, starts by creating a renderer instance and setting basic options it needs.

• Initialize ovrtx and construct a renderer object, pointing it to the correct GPUs and any global configuration options.

• If you have multiple views or sensors, you still have a single renderer instance coordinating them. Multiple RenderProducts will be attached later.

renderer = ovrtx.Renderer()

Load or reference OpenUSD content

Next, the renderer needs to know which USD stage to draw from.

• Point the renderer at an OpenUSD file or URL with open_usd, load USDA directly from a string with open_usd_from_string, or compose additional referenced content with add_usd_reference.

• If your app already has an Omniverse or USD pipeline, ovrtx can read that OpenUSD content directly instead of requiring the app to rebuild the scene in another format.

renderer.open_usd("warehouse_scene.usd")

Configure render and sensor outputs

With a renderer and stage in place, the app displays which outputs it needs.

Define RenderProducts

• Example: An RGB camera on a robot, a top‑down surveillance camera, a spinning lidar sensor on a vehicle.

• Each RenderProduct is tied to a camera or sensor prim path in the USD stage and a resolution.

Attach RenderVars. For each RenderProduct, specify which outputs ovrtx should generate. Examples include:

• Color, such as LdrColor or HdrColor

• Depth or distance

• Surface normals

• Semantic segmentation

• Lidar point clouds

• Radar detections

• Other sensor‑specific tensors

Step rendering or sensor simulation from the application loop

The app calls ovrtx from its main loop, which may already manage physics, animation, user input, networking, or other simulation state.

After the app updates the OpenUSD stage, it advances rendering or sensor simulation by passing a time increment. ovrtx then generates the requested outputs for the configured RenderProducts and RenderVars.

Returning the output to the application

For each generated frame, the app reads the requested outputs as tensors, either on CPU or GPU memory. It consumes these in other parts of the application logic, as needed.

Getting GPU and CPU views of the data:

• ovrtx exposes each RenderVar as a tensor handle. Wrap it using DLPack, NumPy, or PyTorch utilities.

• To stay on GPU (to feed a model, for example), keep the buffer on device and pass it directly to PyTorch or Warp without copies.

• To save images or point clouds, request these outputs on CPU memory, and use other logic to write PNGs, NumPy arrays, and so on.

op = renderer.step_async(

render_products={"/Render/Camera"},

delta_time=1.0 / 60,

)

pending = op.wait()

products = pending.fetch()

for _name, product in products.items():

for frame in product.frames:

var = frame.render_vars["LdrColor"].map(

device=ovrtx.Device.CPU

)

When to use ovrtx with other Omniverse libraries

ovrtx can be used on its own or alongside other NVIDIA Omniverse libraries, depending on what the host application needs to do. Start with ovrtx when the application needs RTX sensor simulation, viewport output, or visualization. Add other libraries when the workflow also needs physics, streaming, storage, or another application service.

Use this table as a starting point for deciding what the host application needs. Start with ovrtx when the workflow needs rendered frames, viewport output, or simulated sensor outputs from OpenUSD content. Add ovphysx, ovstream, or ovstorage only when the same workflow also needs physics behavior, interactive streaming, or OpenUSD content management. The following examples show how those choices map to practical integration patterns, from a minimal ovrtx render to interactive viewing and multi-library workflows.

Examples of ovrtx integration in practice

Start with the NVIDIA-Omniverse/ovrtx GitHub repo for direct SDK examples, setup guidance, and API-focused skills. For agent guidance, reference NVIDIA Agent Skills. To see how Omniverse libraries fit together in early workflows, explore the NVIDIA-Omniverse/omniverse-labs GitHub repo, which includes experimental samples, integration experiments, reference patterns, and proof-of-concept apps from the NVIDIA Omniverse team.

Example: Minimal “hello world”

Start with the minimal ovrtx example in the ovrtx repository. This “hello world” example is a simple smoke test for validating the direct SDK path before adding interaction, sensor outputs, or a larger application shell.

*Video 1. Learn how to run the minimal example, connect Python, C, and C++ applications to Omniverse RTX rendering processes, and generate sensor outputs for robotics and synthetic data workflows*

The example shows how to create a renderer, load an OpenUSD scene, render a single image, and copy the results back to the CPU for writing as a PNG. Pass the --png flag and the output is written to _output/render.png, which can be inspected with any image viewer. Without the flag, the example opens the rendered image directly in your default image viewer.

Use this for: Adding RTX sensor simulation capabilities to an existing application without building a full Omniverse Kit app.

Example: Build an interactive USD viewer

Use the Omniverse Realtime Viewer skill when the goal is to build a viewer application around OpenUSD content, especially when the workflow needs browser-streamed access to a GPU-backed renderer. ovrtx remains the rendering layer, while the viewer adds the surrounding app structure: scene loading, camera controls, selection, render settings, UI, and local desktop or browser-streamed delivery. For lower-level C++ viewport integration, the vulkan-interop example shows how to display ovrtx output in a real-time viewport using Vulkan and CUDA.

For browser-streamed workflows, a server owns the ovrtx renderer and OpenUSD stage access, then uses ovstream to send rendered frames to a browser UI. For a hands-on path, use the Brev launchable to deploy and test a real-time viewer application built with the ovrtx and ovstream libraries.

To learn more, read the agent-assisted integration walkthrough on the NVIDIA-Omniverse/omniverse-labs GitHib repo.

Use this for: Developing a streaming USD viewer with RTX rendering, browser-based camera controls, and a GPU-backed deployment path using NVIDIA Omniverse libraries and an AI coding agent.

Example: Adding physics with ovphysx

After scene visualization, the next step is often making it behave physically, with gravity, collisions, friction, and other simulation behavior. That is where ovphysx comes in. Like ovrtx, ovphysx can be called from an application workflow so the app can simulate behavior and visualize the result in a real-time viewport.

The Omniverse-Labs ovrtx plus ovphysx sample shows this combined workflow in practice. The sample uses a two-process architecture: the viewer process is owned by ovrtx, while a separate ovphysx worker streams pose data back to the application loop.

Use this for: Adding physics simulation to a rendered or sensor-aware OpenUSD workflow.

Example: SimReady Blender sample workflow

The SimReady Blender sample workflow, on display at SIGGRAPH 2026, shows how software makers can use Omniverse libraries, now part of NVIDIA Agent Toolkit, to add agent-ready simulation capabilities—including NVIDIA RTX sensor simulation, physics, and validation—into existing 3D applications while keeping creators in control. This is now openly available as a blueprint for integrating Omniverse libraries into Blender.

Use this for: Understanding how Omniverse libraries can help software makers add agent-ready simulation capabilities into existing 3D applications while keeping creators and developers in control.

Designers and engineers who want to run this kind of agent workflow locally can deploy it on NVIDIA DGX Station using NVIDIA NemoClaw. NemoClaw packages the Nemotron 3 Ultra open frontier model, a secure agent runtime, and Omniverse libraries into a complete local stack—a personal AI workstation with access to Blender and simulation tools.

How to integrate ovrtx into cloud-native design workflows

PTC Onshape Render Studio team is exploring how ovrtx can fit into cloud-native CAD and PDM workflows, where product data, collaboration tools, rendering needs, and established user experiences already come together. That makes it a useful validation point for showing how NVIDIA Omniverse RTX rendering can fit into an existing application workflow while preserving the architecture, data model, and product experience users already rely on.

This early work uses Onshape content to explore how ovrtx can support cloud-native rendering workflows with more complex product design geometry. ovrtx enables integration into existing systems, so teams can evaluate how it fits into the services and application layers they already use, instead of treating rendering as a separate environment.

For the PTC Onshape Render Studio team, ovrtx helps address three practical integration needs:

• Application-controlled rendering: The host application controls when rendering happens, how outputs are configured, and how results return to the workflow.

• Cloud-native integration: Rendering capabilities can be exposed through the application or service layer teams already use, including cloud-native service workflows.

• OpenUSD-based scene exchange: ovrtx renders from OpenUSD scene data, giving teams a practical way to connect product design geometry, visualization, and downstream simulation workflows.

For application teams, the takeaway is that ovrtx can make RTX sensor simulation and visualization part of the workflow they already own. The PTC work shows how a lower-level API can support production-oriented exploration inside cloud-native design systems, where rendering must stay connected to product data, collaboration workflows, and the application experience.

According to Brendan Hack, Senior Director of Software Development at PTC Onshape, “ovrtx gives us a lower-level API path for bringing NVIDIA RTX into systems cloud-native design tools while keeping the application architecture and product experience in our control.”

Get started with NVIDIA ovrtx

NVIDIA Omniverse libraries provide modular building blocks for integrating RTX sensor simulation, physics, storage, and related physical AI capabilities into existing application stacks. With ovrtx, you can add camera, lidar, radar, semantic segmentation, and visual preflight outputs to OpenUSD-based workflows.

For teams developing design, simulation, robotics, industrial digital twin, synthetic data generation, or physical AI applications, ovrtx provides a practical path to evaluate RTX sensor simulation inside the tools, pipelines, and application loops they already own.

Ready to get started?

• Explore Omniverse libraries: Visit the NVIDIA-Omniverse GitHub repo to find tools and skills that help agents and developers add RTX sensor simulation, physics, OpenUSD workflows, storage, and validation to existing applications.

• Run the examples: Review Omniverse Labs for early experimental samples and proof-of-concept projects that show how Omniverse libraries fit together.

• Use NVIDIA physical AI agent skills: Use agent skills to turn repeatable setup, simulation, and evaluation tasks into workflows that developers can inspect, adapt, and run with an AI coding agent.

• Run locally on DGX Station: Deploy NemoClaw on DGX Station in just three steps, in under 30 minutes.