Every 0.7 seconds decide to speak or listen — AntChain opens source Duo v1

Every 0.7 seconds decide to speak or listen — AntChain opens source Duo v1

2026-11-18 10:00

Introduction: Autonomous speaking or listening every 0.7 seconds, 4-step distillation enables real-time generation. What is Duo v1? Duo v1 is a fully duplex audio-video interaction model jointly open-sourced by Ant Tech Institute and West Lake Xinchen, and the first implementation of a Reciprocal World Model (RWM).

What is Duo v1?

Duo v1 is a fully duplex audio-video interaction model jointly open-sourced by Ant Tech Institute and West Lake Xinchen, and the first realization of a Reciprocal World Model (RWM). Duo v1 consists of two coupled modules: Intent and Realization. The Intent module acts as the "brain" of the virtual agent, autonomously deciding to speak or listen approximately every 0.7 seconds and generating corresponding content; the Realization module causally generates consistent audio and video based on its hidden state. Three-stage training combined with Dual Rollout addresses inference distribution shift, ultimately distilled into a 4-step real-time generation system.

Duo v1

Key Features of Duo v1

  • Fully Duplex Real-Time Interaction: Speak and listen simultaneously—models autonomously decide when to speak or listen while receiving audio-video input.
  • Autonomous Participation Decision-Making: Not passively executing instructions, but natively determining <speak> or <wait> based on conversational context.
  • Joint Audio-Video Generation: Unified generation of audio and video ensures lip sync, facial expressions, and speech content are synchronized.
  • Interrupt Response Capability: Handles natural user interruptions and adjusts responses accordingly, rather than playing back pre-scripted sequences.
  • Identity and Emotional Consistency: Maintains stable visual appearance and consistent audio-visual emotional expression throughout interactions.
  • Multi-Agent, Multi-Scenario Support: Supports diverse characters including animals and avatars, covering four interaction scenarios: conversation, guidance, collaboration, and companionship.

Technical Principles of Duo v1

  • Reciprocal World Model (RWM): Traditional world models simulate physical environments with external user control; RWM internalizes intelligent agents as part of the world itself. Virtual agents share the same interactive temporal space with the environment, possessing perception, decision-making, and expressive capabilities—not merely controlled NPCs.
  • Intent Module (Cognitive Agent): Operates in ~0.7-second time blocks, continuously reasoning over user input and interaction history to output <speak>/<wait> decisions and language content. Outputs are hidden states directly serving as conditional signals for the Realization module.
  • Realization Module (Audio-Visual Expression): Jointly generates audio and video based on Intent's hidden state, user input, and its own generation history. Uses module-specific causal attention masks to ensure no access to future inputs at any point, maintaining consistency between training and inference.
  • Three-Stage Training: Phase 1 trains Intent for participation decisions and text prediction, using action-aware re-balancing to mitigate “over-waiting” bias; Phase 2 freezes Intent and trains Realization for causal joint audio-visual generation; Phase 3 applies distribution matching distillation to compress the model into a 4-step real-time generator for latency efficiency.
  • Dual Rollout: Addresses distribution shift between training (using recorded media) and inference (using self-generated history). Freezes Intent to autoregressively generate text, while Realization is trained on its own synthesized audio-visual history.
  • Data Construction: Synthetic data is generated via LLMs following control cards to produce scripts, then rendered frame-by-frame into videos, with a fixed opening video used as reference to suppress error accumulation. Recorded data undergoes ASR, VAD, speaker separation, and temporal alignment before being unified into causal interaction format, followed by four types of automated quality checks.

Dig GitHub

Follow WeChat and reply "Open Source" to join the AI Open Source Project Community

How to Use Duo v1

  • Environment Setup: Prepare Linux + Python 3.10+ + FFmpeg + 2 CUDA GPUs. After cloning the repository, create and install two virtual environments: .venv-model (install PyTorch matching your CUDA version first, then pip install -e '.[realization]') and .venv-speech (pip install -e '.[speech]').
  • Download Weights: Run hf download inclusionAI/Duo-1.2-causal-base --local-dir weights to download the full model package (including ASR and alignment models), or first download Duo-1.0-causal-base and then overlay Duo-1.0-distill to achieve 4-step real-time generation.
  • Copy Configuration: cp configs/inference.example.json configs/inference.json.
  • Fill Configuration: In inference.json, specify three executable Python paths (Realization / Intent / Speech, each belonging to one of the two virtual environments) and directories for each model’s weights.
  • Start Service: Run .venv-model/bin/python run_demo.py --config configs/inference.json --port 4173.
  • Begin Usage: Open http://127.0.0.1:4173/video/index.html in your browser. Upload or record an MP4/WebM video to generate and play Duo’s real-time audio-video response.

Core Advantages of Duo v1

  • Reciprocal World Model Paradigm: First implementation of a Reciprocal World Model where intelligent agents are internalized as part of the world—virtual agents are no longer externally controlled NPCs.
  • Native Fully Duplex Capability: The Intent module autonomously decides <speak>/<wait> every ~0.7 seconds, naturally enabling simultaneous speaking/listening and interrupt responsiveness.
  • Strict Causal Consistency: Module-specific attention masks enforce causal dependencies across the shared timeline—no module can access future inputs, ensuring consistent behavior between training and inference.
  • End-to-End Joint Generation: Audio and video are jointly generated by the Realization module, inherently guaranteeing consistency in lip movements, facial expressions, and speech content.

Project Links for Duo v1

  • Official Website: https://duo-v1.github.io/
  • GitHub Repository: https://github.com/inclusionAI/Duo
  • HuggingFace Model Hub: https://huggingface.co/collections/inclusionAI/duo-v1
  • arXiv Technical Paper: https://arxiv.org/pdf/2611.15131

Competitive Comparison with Similar Projects

Comparison Dimension Duo v1 Gemini Live
Core Paradigm Reciprocal World Model (RWM) Real-time Multimodal Voice/Video Interaction API
Output Format Joint audio-video output (generating both visual and audio of the virtual agent) Voice-only output (can understand video but does not generate visual representation)
Interaction Decision-Making Intent module autonomously decides speak/wait every 0.7 seconds; decisions are intrinsic to the model Native full-duplex, but participation rhythm is controlled by server-side policies
World Simulation Capability Scenes are jointly modeled as part of the interactive temporal space None (focuses on dialogue understanding, does not simulate physical world)
Openness Level Fully open-source: report + code + weights (Apache 2.0), supports self-hosting Closed-source API (~$0.01/min, billed per token)
Latency 4-step distillation + ~0.7-second chunk rhythm End-to-end ~300–500 ms
Deployment Barrier High: requires 2 CUDA GPUs for self-hosting Low: API call only
Language Support Primarily Chinese and English (bilingual training data) 90+ languages—widest in industry

Application Scenarios for Duo v1

  • AI Companionship & Emotional Support: Generate virtual companions such as cats or anime girls, engaging in expressive, emotionally dynamic casual conversations and companionship.
  • Online Education & Tutoring: Virtual teachers provide real-time explanations while observing student reactions, autonomously adjusting teaching pace based on questions and confusion.
  • Intelligent Customer Service & Shopping Assistant: Digital human agents watch user demonstrations and respond in real time, allowing users to interrupt and ask follow-up questions, mimicking real-life counter service experience.
  • Remote Collaboration & Joint Planning: Virtual colleagues participate in proposal discussions and co-edit plans (e.g., “let’s revise the birthday plan together” or “design around materials” type collaborative dialogues).
  • Accessibility & Inclusive Interaction: Provides real-time bidirectional communication interfaces with both voice and visual avatars for visually or hearing-impaired users, using emotionally consistent audio-visual outputs to lower communication barriers.

Source: AI Tools Collection

#AI Tools

Disclaimer: Contains third-party opinions, does not constitute financial advice

Share To
X
Telegram
WeChat
QQ
Link
Recommended Reading

Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

09-24
Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

09-02
Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

09-02
Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

Open-source Retool is here, drag-and-drop backend with 60+ components

08-19
Open-source Retool is here, drag-and-drop backend with 60+ components

Claude Opus 5 is Here: Half the Price Approaching Fable 5

07-25
Claude Opus 5 is Here: Half the Price Approaching Fable 5

Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

07-25
Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks

07-25
Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks