2026-09-11 10:44

Introduction: EchoWM scores 81.7 on WBench navigation benchmark, leading the field with synchronized multi-round exploration. What is EchoWM? EchoWM is an open-source interactive audio-visual world model developed by JD Explore Research Institute.

What is EchoWM?

EchoWM is an open-source interactive audio-visual world model from JD Explore Research Institute. Building upon the native audio-video generation capabilities of JoyAI-Echo, it enables users to explore AI-generated environments in real time from first-person or third-person perspectives, with consistent synchronization between visual content, spatial relationships, environmental audio, and speech over extended durations without drift. Trained using a unified camera intent interface and world data engine, EchoWM achieves an average score of 81.7 on the WBench navigation benchmark, ranking first among all models.

Core Features of EchoWM

• Real-time Explorable World Generation: Users can freely move, turn, and backtrack in AI-generated scenes using WASD keys, with the environment dynamically unfolding in real time based on input.
• Native Audio-Visual Joint Generation: Synchronized generation of 720p video and environmental sounds, music, and character speech within a single framework, with audio naturally evolving alongside visual changes.
• Dual-Perspective Control (First- & Third-Person): In first-person mode, the camera acts as the observer’s viewpoint; in third-person mode, the camera automatically follows subjects such as characters or vehicles.
• Multi-Round Long-Term Continuity: Using the audio-visual output from the previous round as context for the next, scene layout, subject appearance, and sound remain coherent across multiple interaction cycles.
• Top Ranking on WBench Navigation Benchmark: Achieves an average score of 81.7 across 158 test cases, leading in both consistency and interactivity metrics.

Technical Principles of EchoWM

• Unified Camera Intent Interface: Maps discrete keyboard commands into continuous 6-DoF trajectories relative to initial pose. This single interface uniformly describes camera motion—directly representing observer movement in first-person mode, while in third-person mode, the model learns the coupling between "subject motion → camera follow" from training data. No additional role trajectory or camera preset parameters are required. The trajectory is injected only into the video branch, while audio remains synchronized with visual content.
• World Data Engine: Integrates four complementary data sources: gameplay recordings, human-played videos, UE simulations, and general internet videos. Uses the 90th percentile of maximum translational displacement across trajectories as a global normalization reference ("ruler"), preserving relative displacements between different trajectories and avoiding velocity discontinuities caused by segment-wise calibration.
• Progressive Four-Stage Training: Stage 1 performs audio-visual rich pre-training to establish foundational generation capability. Stage 2 freezes the backbone and trains a lightweight trajectory branch to enhance control responsiveness. Stage 3 conducts low-learning-rate joint fine-tuning using data that balances control reliability and audio-visual quality. Stage 4 applies autoregressive post-training, transforming the model into a causal block-wise generator via Teacher Forcing, combined with short-term Self-Gradient Forcing to help the model adapt to its own generated history. Finally, distribution matching distillation compresses inference into a four-step sampling process, paired with sink-plus-FIFO caching to maintain long-term streaming interaction while controlling KV memory cost.

Follow WeChat and reply “Open Source” to join the AI Open Source Project Community

How to Use EchoWM

• Clone Repository: git clone the repository github.com/jd-opensource/JoyAI-Echo, then navigate into the echo_wm/ directory (independent from the long-video project echo_longvideo/).
• Create Environment: Run conda create -n echo-wm python=3.11 to set up an isolated environment and activate it.
• Install Dependencies: First install PyTorch 2.9.1 (cu128), then run pip install -r requirements.txt. Verify GPU availability using torch.cuda.is_available().
• Download Model Weights: Use hf download Echo-Team/Echo-WM to fetch the model checkpoint. Additionally, download the Gemma 3 text encoder (gated repository—requires accepting license on HF website and running hf auth login first).
• Run Official Examples: Execute scripts/run_wm_case.py --case examples/wm_cases/0010 and other built-in full-case examples to verify correct environment setup.

• Custom Generation: Run inference_wm.py, passing in an initial frame image, six-field prompt, and action sequence (Action DSL) to generate a video with synchronized audio.
• Write Action Commands: Use Action DSL to describe camera motion, formatted as <key>-<frames>, separated by commas. For example, w-60,a-60 means moving forward for 60 frames, then left for 60 frames. Supports combinations of w/s/a/d/i/j/k.

Core Advantages of EchoWM

• Audio-Visual Integration: Natively generates 720p video and environmental audio, music, and voice within a single framework, with sound dynamically synchronized to visuals and events—solving the common issue in many world models where only visuals are generated without accompanying audio.
• Unified Control Interface: A single relative 6-DoF trajectory interface simultaneously drives first-person observer movement and third-person camera following, eliminating the need for separate controllers per perspective.
• Long-Term Coherence: Through autoregressive post-training and sink-plus-FIFO caching, users can continue multi-round exploration using the audio-visual context from prior rounds, maintaining scene integrity, subject consistency, and audio alignment over extended periods.
• Benchmark Leadership: Achieves a top score of 81.7 on the WBench Navigation benchmark, with user studies showing significantly higher preference compared to LingBot-World-v2 and HappyOyster.

Project Links for JoyAI-EchoWM

• Official Website: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/
• GitHub Repository: https://github.com/jd-opensource/JoyAI-Echo
• arXiv Technical Paper: https://arxiv.org/pdf/2608.23189

Competitive Comparison with Similar Models

Application Scenarios of EchoWM

• Game Prototyping: Creators enter generated scenes for playtesting to validate spatial layout, path design, and camera experience before finalizing development direction.
• Interactive Storytelling: Narratives branch based on user choices and continuously generate content, replacing traditional pre-rendered assets for interactive dramas and novels.
• Immersive Content: Transforms film and advertising from passive viewing to active immersion—audiences can freely explore the world within the frame.
• Virtual Tours: Enables low-cost digital walkthroughs of museums, scenic spots, real estate, and more—without frame-by-frame modeling.
• Digital Human Performances: Synthesizes character speech, motion, dynamic surroundings, camera movement, and audio in sync—ideal for virtual streamers and digital employees.

Source: AI Tools Hub