2026-11-18 14:50
Introduction: Seven-dimensional physical description benchmarking identifies gaps, boosting robot success rate by 7.5%. What is Wujie·PhysCapBench? Wujie·PhysCapBench is
What is Wujie·PhysCapBench?
Wujie·PhysCapBench is the first cross-domain video physical description evaluation benchmark jointly developed by Inverse Matrix, Zhiyuan Institute, Peking University, and others. It encompasses 1,500 video segments across three data categories: third-person scene, first-person manipulation, and handheld gripper demonstration. Evaluated from seven dimensions, it measures the coverage and fidelity of captions in capturing physical information such as motion, contact, deformation, and state changes. Validated through video generation and robotic policy experiments, it provides a unified "ruler" for data optimization of physical AI foundation models.

Core Features of Wujie·PhysCapBench
- Cross-domain Unified Evaluation: Integrates third-person scenes, first-person operations, and handheld gripper demonstrations into a single framework, enabling consistent measurement across diverse data sources using a unified scale.
- Physical Information Gap Localization: Evaluates caption coverage and fidelity across seven dimensions—motion, contact, state transitions, and their temporal dynamics—exposing shortcomings in description strategies.
- Annotation Quality Assessment: Uses reference descriptions built upon continuous video evidence as a gold standard to verify whether annotations are supported by observable content.
- Training Data Optimization Guidance: Validates that more comprehensive physical descriptions directly improve model performance via downstream video generation and robotic policy experiments, guiding data curation efforts.
Technical Principles of Wujie·PhysCapBench
- Cross-domain Data Organization: The benchmark organizes data by physical processes: third-person scenes emphasize object motion, deformation, and environmental evolution; first-person operations focus on hand-object contact and state transitions; handheld gripper demonstrations highlight gripper interaction and execution outcomes. All three data types share a common quality objective—accuracy and completeness of physical information—and each contributes 500 video sequences, totaling 1,500, integrated into a unified evaluation system, enabling comparative analysis across different observational perspectives.
- Seven-Dimensional Evaluation Framework: To address the limitation of current captions in omitting fine-grained physical details despite covering high-level actions, the evaluation is decomposed into seven dimensions, examining action processes, contact relationships, state changes, and their temporal ordering. Each dimension is scored independently and aggregated into a composite score. This granular breakdown enables precise identification of strengths and weaknesses across different description strategies, allowing direct comparison in leaderboards.
- Reference Description Construction Based on Video Evidence: Physical processes often span multiple stages—pre-contact motion, post-contact deformation, and final state—making single-frame observations insufficient for accurate description. Thus, the annotation process begins with global observation and event localization, systematically identifying involved entities, actions, contacts, and state changes, followed by item-by-item verification against continuous video evidence. Mechanistic and causal claims are grounded solely in observable facts; complex procedures are rigorously checked for consistency in phase segmentation and outcome states. The entire annotation and quality control workflow is executed by seven professional annotators.

Follow WeChat and reply “open source” to join the AI Open Source Project Community
How to Use Wujie·PhysCapBench
- Access Resources: Download the evaluation code and 1,500 video clips with reference descriptions from the GitHub repository (code) and Hugging Face dataset page.
- Select Evaluation Target: Identify the caption source to evaluate—could be human-annotated, original descriptions from existing datasets, or model-generated captions.
- Generate/Prepare Candidate Descriptions: Generate candidate captions for target videos and organize them into the three domain categories.
- Score Against Reference Descriptions: Compare candidate captions with reference descriptions built from video evidence provided by the benchmark, assessing coverage and fidelity across the seven physical information dimensions.
- Analyze Dimensional Scores: Review overall and per-dimension scores to pinpoint information gaps in the description strategy.
- Optimize Description Strategy: Improve annotation templates or generation prompts based on identified gaps; when necessary, reorganize annotation workflows following the reference description construction process.
- Validate Downstream Benefits: Apply optimized physical descriptions as conditioning inputs for video generation or enhance policy training instructions, confirming real-world gains via metrics such as LPIPS/PVCS or task success rate.
Core Advantages of Wujie·PhysCapBench
- Cross-domain Unification: First benchmark to integrate third-person scenes, first-person operations, and handheld gripper demonstrations under a single evaluation framework, filling the gap in cross-domain physical description assessment.
- Fine-grained Evaluation Dimensions: The seven-dimensional scoring system breaks down “description quality” into specific physical information types—motion, contact, state change—enabling precise identification of each strategy’s weaknesses.
- Verifiable Annotations: Reference descriptions are constructed from continuous video evidence and manually verified, ensuring both comprehensive coverage and factual accuracy.
- Comparable Results: Composite and per-dimension scores are presented in leaderboard format, making relative performance among different strategies immediately apparent for easy comparison and selection.
- Quality-to-Utility Correlation: Demonstrated through real experiments—video generation (LPIPS reduced by up to 30.88%) and robotic policy (success rate increased by 7.5 percentage points)—more complete physical descriptions directly translate into tangible model improvements.
Project Links for Wujie·PhysCapBench
- Official Website: https://hfwang2001.github.io/physcapbench-site/
- GitHub Repository: https://github.com/hfwang2001/PhysCapBench
- Hugging Face Dataset Hub: https://huggingface.co/datasets/hfwang01/PhysCapBench
- arXiv Technical Paper: https://arxiv.org/pdf/2611.16424
Competitive Comparison with Similar Benchmarks
| Comparison Criteria |
Wujie·PhysCapBench |
PhysBench |
| Core Objective |
Evaluate caption fidelity and coverage of physical information in videos |
Assess multimodal models’ ability to answer physics commonsense questions |
| Evaluation Target |
Textual descriptions (data-side) |
Model outputs (model-side) |
| Data Domains |
Third-person scenes, first-person operations, handheld gripper demonstrations (cross-domain unified) |
Primarily physics commonsense questions; not specifically designed for operational/robotic data |
| Evaluation Method |
Seven-dimensional scoring by physical information type + comparison with reference descriptions |
Answer accuracy rate (objective question format) |
| Reference Standard |
Reference descriptions built from video evidence and manually verified |
Standard answer options |
| Downstream Validation |
Yes: Video generation and robotic policy experiments |
Focuses on understanding capability assessment; does not directly validate generation or policy gains |
| Core Value |
Guides data annotation and training supervision optimization |
Measures model-level physical knowledge proficiency |
Application Scenarios of Wujie·PhysCapBench
- World Model Pre-training Data Optimization: Select and rewrite video captions to fully convert physical experience—motion, contact, state change—into training supervision signals.
- Video Generation Data Annotation: Provide high-fidelity physical condition descriptions for video generation models, enhancing consistency between generated content and real-world physical dynamics.
- Robot Imitation Learning Data QA: Assess the linguistic quality of real-world or gripper demonstration data, enriching descriptions of procedural steps and final states.
- Embodied Intelligence Training Instruction Enhancement: Leverage the Training Instruction Upsampler paradigm to provide richer process supervision for policy learning, improving task success rates.
Source: AI Tools Suite