ByteDance、音声・映像・テキストを統合したリアルタイム LLM「SeedRealtime」を発表

Tech Blog

Overview

Today, we are officially introducing SeedRealtime, a native audio-visual full-duplex LLM.

As a key step toward omni-modal interaction, SeedRealtime natively unifies audio, video, and text within a single architecture, enabling real-time interaction over continuous multimodal streams and a new experience of watching, listening, and speaking simultaneously. The model advances audio-visual joint understanding, proactive interaction, and conversational timing.

End-to-end human evaluations show that, compared with cascaded models, SeedRealtime reduces conversational pacing issues by half, significantly decreases interruptions, latency, and false triggers, and also markedly improves the usability rate of single-turn conversations in terms of smoothness and completeness.

Currently, SeedRealtime has been fully rolled out, pioneering large-scale deployment of audio-visual full-duplex technology in the industry.

Technology Implementation

SeedRealtime adopts an end-to-end unified audio-visual modeling framework in which perception, understanding, decision-making, and response generation happen within one model, reducing the information loss and error accumulation introduced by multi-stage cascaded systems. It also natively supports full-duplex interaction, continuously modeling conversational state and timing to enable smoother and more natural real-time exchanges.

On the engineering side, we optimize continuously for low-latency real-time interaction through chunked audio-visual input and streaming generation, while improving serving efficiency with efficient quantization and inference optimization.

Joint Audio-Visual Understanding

SeedRealtime natively integrates visual, audio, and temporal information. It not only uses the live scene to resolve homophones and ambiguous speech, but also continuously tracks visual changes and references, enabling more accurate understanding of user intent and a tighter fusion of what is seen, heard, and said.

In overlapping multi-speaker conversations, SeedRealtime can simultaneously identify people, distinguish voices, and understand content, allowing it to track who is speaking, capture the key points of the discussion, and step in with suggestions at the right moment.

Proactive Interaction

SeedRealtime combines persistent environmental awareness with proactive communication. When it detects changes in the scene, such as the appearance of a key object, it can respond proactively and invoke tools as part of the interaction, turning passive response into active collaboration.

In mobile scenarios such as museum visits, SeedRealtime can keep track of the user's goal, provide timely reminders, and naturally explain what is happening in the live scene, while filtering out irrelevant distractions to maintain smooth interaction.

Natural Conversational Timing

SeedRealtime can perceive conversational state and rhythm in real time, naturally deciding when to join in, pause, or respond. It is also resilient to interference, distinguishing side conversations and background noise to avoid false triggers and keep communication smooth and coherent.

In noisy multi-speaker environments such as stations and airports, SeedRealtime can filter out irrelevant interference, continuously understand and retain the key points of a conversation, and recall them accurately when needed.

Outlook

Audio-visual interaction in the real world is complex and continuous: backgrounds are noisy, voices overlap, scenes change, and users may pause, add information, or interrupt at any time. To meet these real-world challenges, we will continue pushing forward in several directions:

Lower latency and more natural timing: Further reducing end-to-end delay from hearing to understanding to responding, so that interruptions, overlaps, backchannels, and pauses in real conversations can be handled more naturally—not just faster, but with better timing.

More proactive perception and decision-making: Continuously understanding what is happening in both audio and video, and deciding when to remind, when to supplement, and when to provide exactly the information the user needs—so the model not only speaks, but also acts when appropriate.

More robust performance in complex multi-party scenarios: Reliably identifying who is speaking, what they are looking at, and who should be addressed in noisy environments, multi-person scenes, and multi-party conversations, bringing these capabilities into more real-life and work settings.

From "conversing" to "acting": Connecting tool use with the real world, so that the model can do more than communicate and observe—it can help users search, book, and complete tasks, turning real-time multimodal understanding into concrete action.

We hope AI interaction will no longer be limited to turn-based question answering, but instead understand context in continuously changing real-world situations, grasp timing sensitively, and offer help at the right moment.