2026-11-18 10:00
Introduction: Autonomous speaking or listening every 0.7 seconds, 4-step distillation enables real-time generation. What is Duo v1? Duo v1 is a fully duplex audio-video interaction model jointly open-sourced by Ant Tech Institute and West Lake Xinchen, and the first implementation of a Reciprocal World Model (RWM).
What is Duo v1?
Duo v1 is a fully duplex audio-video interaction model jointly open-sourced by Ant Tech Institute and West Lake Xinchen, and the first realization of a Reciprocal World Model (RWM). Duo v1 consists of two coupled modules: Intent and Realization. The Intent module acts as the "brain" of the virtual agent, autonomously deciding to speak or listen approximately every 0.7 seconds and generating corresponding content; the Realization module causally generates consistent audio and video based on its hidden state. Three-stage training combined with Dual Rollout addresses inference distribution shift, ultimately distilled into a 4-step real-time generation system.

Key Features of Duo v1
- Fully Duplex Real-Time Interaction: Speak and listen simultaneously—models autonomously decide when to speak or listen while receiving audio-video input.
- Autonomous Participation Decision-Making: Not passively executing instructions, but natively determining
<speak> or <wait> based on conversational context.
- Joint Audio-Video Generation: Unified generation of audio and video ensures lip sync, facial expressions, and speech content are synchronized.
- Interrupt Response Capability: Handles natural user interruptions and adjusts responses accordingly, rather than playing back pre-scripted sequences.
- Identity and Emotional Consistency: Maintains stable visual appearance and consistent audio-visual emotional expression throughout interactions.
- Multi-Agent, Multi-Scenario Support: Supports diverse characters including animals and avatars, covering four interaction scenarios: conversation, guidance, collaboration, and companionship.
Technical Principles of Duo v1
- Reciprocal World Model (RWM): Traditional world models simulate physical environments with external user control; RWM internalizes intelligent agents as part of the world itself. Virtual agents share the same interactive temporal space with the environment, possessing perception, decision-making, and expressive capabilities—not merely controlled NPCs.
- Intent Module (Cognitive Agent): Operates in ~0.7-second time blocks, continuously reasoning over user input and interaction history to output
<speak>/<wait> decisions and language content. Outputs are hidden states directly serving as conditional signals for the Realization module.
- Realization Module (Audio-Visual Expression): Jointly generates audio and video based on Intent's hidden state, user input, and its own generation history. Uses module-specific causal attention masks to ensure no access to future inputs at any point, maintaining consistency between training and inference.
- Three-Stage Training: Phase 1 trains Intent for participation decisions and text prediction, using action-aware re-balancing to mitigate “over-waiting” bias; Phase 2 freezes Intent and trains Realization for causal joint audio-visual generation; Phase 3 applies distribution matching distillation to compress the model into a 4-step real-time generator for latency efficiency.
- Dual Rollout: Addresses distribution shift between training (using recorded media) and inference (using self-generated history). Freezes Intent to autoregressively generate text, while Realization is trained on its own synthesized audio-visual history.
- Data Construction: Synthetic data is generated via LLMs following control cards to produce scripts, then rendered frame-by-frame into videos, with a fixed opening video used as reference to suppress error accumulation. Recorded data undergoes ASR, VAD, speaker separation, and temporal alignment before being unified into causal interaction format, followed by four types of automated quality checks.

Follow WeChat and reply "Open Source" to join the AI Open Source Project Community
How to Use Duo v1
- Environment Setup: Prepare Linux + Python 3.10+ + FFmpeg + 2 CUDA GPUs. After cloning the repository, create and install two virtual environments:
.venv-model (install PyTorch matching your CUDA version first, then pip install -e '.[realization]') and .venv-speech (pip install -e '.[speech]').
- Download Weights: Run
hf download inclusionAI/Duo-1.2-causal-base --local-dir weights to download the full model package (including ASR and alignment models), or first download Duo-1.0-causal-base and then overlay Duo-1.0-distill to achieve 4-step real-time generation.
- Copy Configuration:
cp configs/inference.example.json configs/inference.json.
- Fill Configuration: In
inference.json, specify three executable Python paths (Realization / Intent / Speech, each belonging to one of the two virtual environments) and directories for each model’s weights.
- Start Service: Run
.venv-model/bin/python run_demo.py --config configs/inference.json --port 4173.
- Begin Usage: Open
http://127.0.0.1:4173/video/index.html in your browser. Upload or record an MP4/WebM video to generate and play Duo’s real-time audio-video response.
Core Advantages of Duo v1
- Reciprocal World Model Paradigm: First implementation of a Reciprocal World Model where intelligent agents are internalized as part of the world—virtual agents are no longer externally controlled NPCs.
- Native Fully Duplex Capability: The Intent module autonomously decides
<speak>/<wait> every ~0.7 seconds, naturally enabling simultaneous speaking/listening and interrupt responsiveness.
- Strict Causal Consistency: Module-specific attention masks enforce causal dependencies across the shared timeline—no module can access future inputs, ensuring consistent behavior between training and inference.
- End-to-End Joint Generation: Audio and video are jointly generated by the Realization module, inherently guaranteeing consistency in lip movements, facial expressions, and speech content.
Project Links for Duo v1
- Official Website: https://duo-v1.github.io/
- GitHub Repository: https://github.com/inclusionAI/Duo
- HuggingFace Model Hub: https://huggingface.co/collections/inclusionAI/duo-v1
- arXiv Technical Paper: https://arxiv.org/pdf/2611.15131
Competitive Comparison with Similar Projects
| Comparison Dimension |
Duo v1 |
Gemini Live |
| Core Paradigm |
Reciprocal World Model (RWM) |
Real-time Multimodal Voice/Video Interaction API |
| Output Format |
Joint audio-video output (generating both visual and audio of the virtual agent) |
Voice-only output (can understand video but does not generate visual representation) |
| Interaction Decision-Making |
Intent module autonomously decides speak/wait every 0.7 seconds; decisions are intrinsic to the model |
Native full-duplex, but participation rhythm is controlled by server-side policies |
| World Simulation Capability |
Scenes are jointly modeled as part of the interactive temporal space |
None (focuses on dialogue understanding, does not simulate physical world) |
| Openness Level |
Fully open-source: report + code + weights (Apache 2.0), supports self-hosting |
Closed-source API (~$0.01/min, billed per token) |
| Latency |
4-step distillation + ~0.7-second chunk rhythm |
End-to-end ~300–500 ms |
| Deployment Barrier |
High: requires 2 CUDA GPUs for self-hosting |
Low: API call only |
| Language Support |
Primarily Chinese and English (bilingual training data) |
90+ languages—widest in industry |
Application Scenarios for Duo v1
- AI Companionship & Emotional Support: Generate virtual companions such as cats or anime girls, engaging in expressive, emotionally dynamic casual conversations and companionship.
- Online Education & Tutoring: Virtual teachers provide real-time explanations while observing student reactions, autonomously adjusting teaching pace based on questions and confusion.
- Intelligent Customer Service & Shopping Assistant: Digital human agents watch user demonstrations and respond in real time, allowing users to interrupt and ask follow-up questions, mimicking real-life counter service experience.
- Remote Collaboration & Joint Planning: Virtual colleagues participate in proposal discussions and co-edit plans (e.g., “let’s revise the birthday plan together” or “design around materials” type collaborative dialogues).
- Accessibility & Inclusive Interaction: Provides real-time bidirectional communication interfaces with both voice and visual avatars for visually or hearing-impaired users, using emotionally consistent audio-visual outputs to lower communication barriers.
Source: AI Tools Collection