Peking University and Tianjin University Open Source 3D Spatial Foundation: Enabling Models to Locate Objects Independently

Peking University and Tianjin University Open Source 3D Spatial Foundation: Enabling Models to Locate Objects Independently

2026-11-16 18:00

Introduction: Empowering visual models to actively explore, triangulate, and construct online 3D object maps. What is SpatialScout? SpatialScout is an open-source benchmark for 3D spatial intelligence developed by Peking University in collaboration with Tianjin University.

What is SpatialScout?

SpatialScout is an open-source benchmark for 3D spatial intelligence developed by Peking University in collaboration with Tianjin University, featuring 80 indoor scenes and 3,140 target objects. SpatialScout requires visual-language models to control a freely movable camera, autonomously explore the environment, invoke triangulation tools, and progressively build a unified-coordinate 3D map of object centers. Evaluation simultaneously assesses localization accuracy and target coverage.

SpatialScout

Core Features of SpatialScout

  • Active Exploration: Models autonomously control camera pan, tilt, and capture images, deciding where to observe next.
  • Cross-View Triangulation: Utilize corresponding pixels across different views to invoke tools and estimate the 3D position of surface points on objects.
  • Online Mapping & Correction: Submit and update object center coordinates incrementally within a fixed coordinate system defined by the initial camera.
  • Dual-Dimensional Scoring: Evaluate localization accuracy and target coverage using both center error and scale-aware 3D IoU.
  • Process Analysis: Preserve full trajectories of observations, tool invocations, and position updates, enabling analysis of model strategies in view selection and spatial correction.

Technical Principles of SpatialScout

  • Unified Coordinate Mapping Task Definition: The evaluation frames spatial understanding as constructing a map of object centers under a unified coordinate system—agents start from an initial viewpoint, receive RGB images, a list of target objects, camera intrinsics and pose, and a publicly accessible search range, continuously mapping observations from multiple views into 3D coordinates; true centers, bounding boxes, and evaluation scores are retained on the evaluator side, not returned during exploration.
  • Tool-Aware Active Perception Interface: All models interact via a unified tool interface and prompt template—mobile cameras can capture new images, and cross-view corresponding pixels can be invoked for triangulation; models must independently determine which visual features belong to the same object and how surface points correspond to geometric centers. Each task is limited to 100 model calls and 100 images (including the initial image), with each call capable of requesting multiple tools.
  • Scale-Aware IoU Scoring Mechanism: Models need only submit object center coordinates—no size prediction required. The evaluator translates the reference bounding box to the predicted center and computes 3D intersection-over-union (IoU) with the ground-truth box. Full-target average IoU includes unsubmitted targets as zero-score to penalize low coverage, while submitted-target average IoU reflects localization quality. For interchangeable instances within the same class, one-to-one matching within groups eliminates bias from object ID ordering.
  • Trajectory-Based Failure Attribution Analysis: Through quantitative and qualitative analysis of multi-model trajectories, it reveals that effective use of cross-view feature correspondence and geometric tools is critical for localization quality—small geometric residuals do not guarantee correct feature or center identification.

digging GitHub

Follow WeChat and reply "open source" to join the AI Open Source Project Community

How to Use SpatialScout

  • Obtain Code: Clone the repository from GitHub (github.com/JesseZZZZZ/SpatialScout) and configure the environment and dependencies according to instructions.
  • Integrate Target Model: Integrate the target visual-language model into the benchmark’s unified tool interface and prompt framework, equipping it with image input and tool invocation capabilities.
  • Load Task: The agent starts from an initial viewpoint, receiving RGB images, a list of target objects, camera intrinsics and pose, and the publicly accessible search range.
  • Autonomous Exploration: The model makes autonomous decisions to pan or tilt the camera and capture new images—each task is budgeted at 100 model calls and 100 images.
  • Geometric Estimation: Invoke triangulation tools on corresponding pixels across different views belonging to the same object to estimate the 3D position of surface points.
  • Submit & Update Map: Submit estimated object centers in the initial camera’s coordinate system, allowing incremental supplementation of missing targets or correction of prior estimates.
  • Automatic Scoring: The evaluator computes center error, target coverage, and scale-aware 3D IoU based on submitted coordinates, generating final evaluation results.

Core Advantages of SpatialScout

  • Interactive Real-World Task Design: Transforms spatial understanding from “image-based Q&A” to a full-cycle task involving free camera movement, autonomous evidence gathering, and 3D coordinate output—closer to real-world embodied intelligence requirements.
  • Uniform and Fair Evaluation Protocol: All models use the same tool interface, prompts, and interaction budget, enabling direct horizontal comparison of results.
  • Dual Metrics for Accuracy and Coverage: Combining full-target average IoU and submitted-target average IoU effectively detects opportunistic behaviors such as reporting only easily identifiable objects.
  • No Need to Predict Object Dimensions: The evaluator shifts the reference bounding box to the predicted center to compute IoU—models only need to submit coordinates, reducing scoring interference from size estimation errors.
  • Anti-Ordering Cheating Matching Mechanism: Interchangeable instances within the same class use intra-group one-to-one matching, eliminating bias from object ID ordering in evaluations.
  • Process-Level Analyzability: Full observation, tool invocation, and position update trajectories are preserved, enabling fine-grained attribution analysis of view selection, cross-view correspondence, and failure patterns.

Project Links for SpatialScout

  • GitHub Repository: https://github.com/JesseZZZZZ/SpatialScout
  • arXiv Technical Paper: https://arxiv.org/pdf/2610.26659

Competitive Comparison with Similar Benchmarks

Comparison Dimension SpatialScout VSI-Bench
Task Paradigm Active Exploration: Models autonomously move the camera and invoke tools to build maps Passive Question-Answering: Models answer directional, distance, and counting questions about images/videos
Output Format 3D object center coordinates in a unified coordinate system Text-based multiple-choice or numerical answers
Interaction Capability Supports tool invocation and multi-turn autonomous interaction (100-call budget) No interaction—single input of image or video
Evaluation Metrics Center error, target coverage, scale-aware 3D IoU Answer accuracy rate
Key Focus Areas Active perception + cross-view correspondence + geometric estimation + online mapping Spatial reasoning in static scenes
Data Source 80 physically simulated indoor scenes, 3,140 target objects First-person videos and panoramic images from real-world scenes
Process Analysis Full trajectory preserved—enables analysis of view selection and failure attribution Only final result—no process-level analysis available
Anti-Cheating Design Intra-group one-to-one matching, unsubmitted targets scored as zero Multiple-choice format inherently limits answer space

Application Scenarios for SpatialScout

  • Embodied Intelligence Robots: Evaluate and improve robots’ ability to autonomously locate target objects (e.g., “find the cup on the table”) in unfamiliar rooms.
  • AR/VR Spatial Perception: Assess whether models can accurately estimate 3D object positions, providing foundational support for persistent virtual object anchoring and occlusion rendering.
  • Smart Home & Service Robots: Enable vacuum cleaners and companion robots to build indoor object maps, supporting commands like “fetch the remote from the sofa.”
  • Warehouse & Logistics Inventory: Empower mobile robots to autonomously inspect shelves and output precise 3D coordinates for each item, aiding inventory counting and picking.
  • Automated 3D Reconstruction & Digital Twinning: Validate active perception-based mapping algorithms, offering evaluation and training benchmarks for rapidly generating accurate digital twins of indoor environments.

Source: AI Tools Hub

#AI Tools

Disclaimer: Contains third-party opinions, does not constitute financial advice

Share To
X
Telegram
WeChat
QQ
Link
Recommended Reading

Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

09-24
Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

09-02
Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

09-02
Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

Open-source Retool is here, drag-and-drop backend with 60+ components

08-19
Open-source Retool is here, drag-and-drop backend with 60+ components

Claude Opus 5 is Here: Half the Price Approaching Fable 5

07-25
Claude Opus 5 is Here: Half the Price Approaching Fable 5

Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

07-25
Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks

07-25
Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks