2026-11-12 08:00

Introduction: A unified tool system upgrade leads to a 29.49-point surge in robot performance. Robots are entering their own Computer Use era.

Robots are entering their own Computer Use era.

On the 10th, Anthropic released Fable 5.2, marking new progress. Compared to prior models, its physical robot control capabilities have significantly improved, even enabling it to complete tasks it had never been specifically trained for.

In RoboDojo Sim simulation evaluations, Fable 5.2 achieved a score of 64.9%, far surpassing Anthropic’s internal benchmark GPT-6 Astra (29.8%) under the same configuration.

As early as September, MIT researcher Phillip Isola proposed in "Robot-Use Agents" that future robots with internet connectivity and open sensor and actuator interfaces could become tools callable by general-purpose Agents.

This aligns with the System 1/System 2 division paradigm in embodied intelligence: general models handle high-level reasoning and task planning, while low-level policies manage specific action execution. Models like GPT-6 Astra and Fable 5.2 have already demonstrated the feasibility of this approach.

However, two critical challenges remain before this vision becomes fully operational.

First is the model. Closed-source frontier models have shown promising robot operation capabilities, but researchers cannot directly access their weights or leverage robot interaction data for continuous training.

Second is the system. Different robots vary significantly in hardware structure, sensors, and control interfaces, and simulation environments differ from real-world settings. If each robot integration requires rebuilding the entire tool system, cross-environment transfer and large-scale experimentation will incur prohibitively high costs.

Thus, RUA needs a unified interface layer that abstracts underlying hardware and environmental differences, allowing models to consistently receive perception inputs, invoke control tools, and receive execution feedback.

Recently, a research team led by Ding Ning, Liu Zhiyuan, Xu Mengdi, and Zhou Bowen from Tsinghua University, in collaboration with Professor Pieter Abbeel and the Berkeley Artificial Intelligence Research Lab (BAIR) at UC Berkeley, jointly launched and open-sourced uni-robot-agent, aiming to establish a unified, modular harness for Robot-Use Agents (RUA), enabling language models to control diverse robots and simulation environments via standardized tool interfaces.

Paper link: https://arxiv.org/pdf/2611.01560

Blog link: https://openrua.ai/uni-robot-agent

Code link: https://github.com/Open-RUA/uni-robot-agent

The framework performs impressively in practical benchmarks. On the RoboDojo benchmark, the open-weight model Qwen3.8-Flash-Next paired with this framework achieved 31.39 points—surpassing the previously reported official GPT-6 Astra + RoboProbe score of 28.97. Under the same framework, GPT-6 Astra further reached 51.90 points.

In real-robot evaluation across 19 tasks, Qwen without task-specific fine-tuning scored 53.50 points, leading both fine-tuned π₀.₅ and zero-shot Dream-1 by 5.75 and 10 points respectively. GPT-6 Astra achieved 80.50 points when combined with this framework.

Beyond scores, this work also analyzes how far models can push robot operations, uncovering failure causes, capability gaps, and runtime costs.

Even further, they aim to build a universal foundation enabling open-source models to interface with various robots, paving the way for future training and on-premise deployment.

Why is a Unified Interface the Foundation for Cross-Environment Generalization?

In the Coding Agent domain, the same set of tools can be reused across different programming tasks. Models read files, execute code, and inspect results through similar interfaces, enabling reuse of existing tool usage experience across environments.

Robot operation involves greater hardware heterogeneity. Even for simple object grasping, single-arm and dual-arm robots may have differing action interfaces. Models must understand new parameters, coordinate systems, and control rules—rendering prior tool invocation experience non-transferable.

uni-robot-agent aims to bridge these gaps through a unified tool interface.

For example, a Franka single-arm robot must position, grasp, and stack blocks; an ALOHA dual-arm robot must coordinate both arms for object handover; and a real YuMi dual-arm robot must precisely align and insert pins into holes. Despite differences in hardware structure, action space, and task goals, models can invoke consistent perception, motion, skill, and task control interfaces, with underlying adapters handling robot-specific control variations.

In other words, models can follow uniform tool invocation rules. When integrating new environments, developers mainly need to implement corresponding adapters, minimizing changes to upper-layer Agent logic.

Tool reusability across different environments.

A deeper implication of a unified interface lies in data consistency.

If different environments use identical tool semantics, parameter units, and feedback formats, the interaction trajectories generated during robot task execution can be easily aggregated. Experience accumulated from single-arm stacking, dual-arm handover, and real-robot insertion can thus be consolidated into a single data ecosystem.

This design enables cross-environment Agent training—researchers can train a single model using interaction data from diverse robots, learning transferable tool usage and operational skills.

How is uni-robot-agent Designed?

The design of uni-robot-agent revolves around two core objectives: providing a unified foundation for cross-environment Agent training, and objectively assessing the existing robot operation capabilities of general models.

To achieve this, the research team structured the system into five layers: Agent, Orchestration, Tools, Services, and Backends.

The Agent layer manages interactions with models, handles context and task decisions; the Orchestration layer schedules tool requests; the Tools layer provides a unified operation interface to models; the Services layer manages runtime resources; and the Backends layer connects actual robots, simulators, and model implementations.

This modular architecture allows independent replacement of models, tools, and robot backends. Researchers can compare model performances without rewriting the entire system, accumulating structurally consistent interaction data.

Objective evaluation demands conditions as close as possible to real-world deployment. Simulation environments can provide precise 3D object positions, CAD models, hidden scene states, and even automatic success detection—all information typically inaccessible on real robots. If models rely on such privileged data to succeed, it's impossible to gauge their true visual understanding, spatial reasoning, and manipulation abilities.

Therefore, in primary simulation experiments, uni-robot-agent only provides RGB images, camera calibration, robot state, and task instructions to models—excluding depth ground truth, object CAD assets, hidden scene annotations, task validators, or external network access. Additionally, researchers prohibited shortcuts tailored specifically to the test tasks, preventing models from achieving high scores via pre-written answers.

So, without these extra signals, how can general models perform complex robot operations? The key lies in properly organizing model-perceived information and ensuring reliable tool execution.

Key design choices. Each column presents two complementary designs aimed at ensuring physical state consistency, action execution precision, and context management.

For instance, when a robot grasps an object, the model must compute the grasp coordinates based on image and arm position. If the image comes from a previous time step while the arm state has already updated, the calculation may suffer from timing discrepancies.

uni-robot-agent binds camera images, calibration data, and robot state to the same timestamp, reducing positioning errors due to asynchronous states. During action execution, request deduplication prevents redundant operations, while monitoring arm movement detects stalls or anomalies.

After action completion, the system feeds back execution results and new environmental observations to the model, helping it assess task success and determine whether to adjust subsequent actions.

To address long-duration robot actions, the framework introduces asynchronous execution. Models can continue processing other tasks while certain tools run, and motion requests on the same arm are queued sequentially to avoid command conflicts.

These designs integrate robotics engineering best practices into the Harness, minimizing interference from misaligned information, repeated actions, and missing feedback on model performance.

By properly organizing information and providing reliable control tools, uni-robot-agent enables general language models to demonstrate competitive robot operation capabilities—even without relying on simulation privileges or task-specific shortcuts.

Experimental Validation: How Far Can Open Models Go?

To evaluate the practical impact of the unified Harness, the research team first conducted simulation experiments on RoboDojo.

This benchmark includes 42 robot operation tasks across five dimensions: generalization, fine manipulation, long-horizon execution, memory, and open-vocabulary instructions. Each main experiment runs 252 evaluations.

The impact of the unified Harness is striking.

RoboDojo benchmark results. Avg. denotes the unweighted average across the five capability dimensions. †GAP (Hybrid) indicates the hybrid solution combining π₀.₅.

Under the same Qwen3.8-Flash-Next model, RoboProbe scored only 1.90, RPent reached 7.75, and the hybrid GAP (π₀.₅) achieved 21.01.

Switching to uni-robot-agent, Qwen’s score rose to 31.39, with a success rate of 25%. Compared to RoboProbe, the overall score increased by 29.49 points.

This experiment shows that even with unchanged model weights, tool system design can significantly affect robot operation performance.

The framework also adapts to other models. DeepSeek-V4.1-Flash achieved 27.11 points, while GPT-6 Astra reached 51.90 points with a 46.67% success rate.

Different models exhibit distinct strengths. Qwen and DeepSeek perform relatively better in memory and open-vocabulary tasks than in fine manipulation and long-horizon execution; some specialized VLA and WAM models show advantages in long-horizon tasks.

Real-robot testing further validates model capabilities in physical environments.

The team conducted 19 operation tests using a dual-arm robot, covering five dimensions: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks. Specific tasks included inserting tubes into racks, passing bottles between arms, stacking blocks by color order, wiping whiteboards, and solving Tower of Hanoi puzzles.

Representative real-robot tasks across five dimensions.

GPT-6 Astra scored 80.50, achieving a perfect 20/20 in compositional generalization; Qwen scored 53.50, outperforming fine-tuned π₀.₅ and zero-shot Dream-1.

Real-robot evaluation results. Each dimension maxes at 20 points, total max at 100 points.

However, input condition differences exist in this comparison. Qwen and Astra used RGB images plus depth information from RealSense D405 sensors; π₀.₅ and Dream-1 used only RGB. Thus, score differences cannot be fully attributed to model capacity alone.

More crucially, which external information do models actually rely on?

With both depth and camera calibration provided, Qwen scored 35.89. Removing depth but keeping calibration dropped the score to 31.39. Further removing calibration and retaining only RGB and robot state reduced the score to 9.13.

Astra scored 55.83, 55.04, and 31.29 under the same settings.

Robot operation performance under reduced geometric information. The chart records model progress across 42 tasks via 771 tool calls and final scores when skill library is enabled.

This indicates models can leverage camera calibration, multi-view observation, and geometric computation to complete partial spatial localization without direct depth measurement. However, without calibration, performance drops significantly—reconstructing reliable spatial relationships from RGB alone remains challenging.

Whether pre-defined skill libraries affect evaluation results is another concern.

Disabling the skill library and skill authoring features reduced Qwen’s overall score from 31.39 to 30.31—retaining 96.6% of original performance—and success rate dropped only from 25% to 24.58%. This suggests performance does not heavily depend on pre-packaged task skills; standardized perception and motion tools, combined with model reasoning, already support substantial operation capability.

Impact of skill guidance on task performance.

Cost-wise, Qwen averages ~$1.09 per task, DeepSeek ~$0.94, and Astra ~$21.29.

Execution speed also impacts robot Agent efficiency. At the point where cumulative score first reaches 25, Astra consumed ~7.4 million tokens over 42.6 minutes; Qwen used 23.5 million tokens over 95.5 minutes; DeepSeek used 49.9 million tokens over 99.2 minutes.

To reduce waiting time, the team further tested asynchronous pipelines.

In selected five tasks, pipeline execution was faster than serial execution, with reductions ranging from 4.8% to 73.7%. For example, conveyor belt pick-up time dropped from ~59.7 minutes to 15.7 minutes.

When Do General Models Begin to Acquire Robot Operation Capabilities?

The unified Harness enables different models to be tested under comparable interaction conditions, allowing us to ask a more fundamental question:

What core capabilities must general models possess to perform robot tasks?

The research team compared 13 models from different families, generations, and scales, finding no simple correlation between robot operation ability and model size or release date.

Robot operation performance across different generations, model series, and parameter scales.

Take the Qwen series as an example: Qwen3.5, 3.6, and 3.7 all failed to achieve full task success. At Qwen3.8, the 27B model reached 8.06% success rate, while Flash-Next improved to 25%. Notably, despite smaller scale, the 27B model completed some tasks impossible for older large models.

The GPT series also shows generational improvement. GPT-5.6 Sol could complete some tasks with 19.17% success rate; GPT-6.1 Sol rose to 37.50%; GPT-6 Astra further reached 46.67%.

These results suggest robot operation capability may have already existed within general models, with improvements likely driven by training data distribution, capability composition, and training methodology.

By analyzing failure traces across models, the team identified four key capability types.

The figure shows the correlation between each model’s RoboDojo composite score and error rates in four capability dimensions. Lower error rates in these dimensions generally correspond to better robot task performance.

First is visual understanding. Robots must identify objects, judge orientations, and verify post-action states. Some earlier Qwen models misidentified digit 0 as 9 or interpreted upside-down cups as upright, leading to actions based on incorrect observations.

In examined trajectories, such errors occurred in 12.2% to 23.8% of cases for older models. No such errors were found in Qwen3.8-Flash-Next samples.

Second is spatial localization. Models must determine target 3D positions from images. Without depth, multi-view triangulation can help, but matching the same physical location across views remains difficult.

The paper found that even GPT-6.1 Sol had ~31.5% of relevant triangulation calls failing geometric consistency checks.

The third type is geometric computation. Robots must transform positions and poses between camera, world, and robotic arm coordinate systems. Older Qwen models showed error rates of 47.3% to 81.8% in such calculations; Qwen3.8-Flash-Next reduced this to 8.2%, and DeepSeek-V4.1-Flash further dropped to 1.5%.

Finally is embodied reasoning. Models must assess grasp posture validity, reachability of targets, and collision risks. Some models repeatedly failed to grasp despite finding correct locations due to improper wrist orientation.

The team also correlated robot operation scores with general benchmarks like ERQA, MMMU-Pro, ScreenSpot-Pro, and LiveCodeBench v6, finding varying degrees of correlation with visual understanding, embodied reasoning, spatial localization, and coding ability. Thus, robot operation capability may represent a generalized combination of foundational skills across domains.

Relationship between robot operation performance and general model capabilities.

Robot operation involves continuous visual judgment, spatial localization, geometric computation, and physical decision-making—any flaw in one component can lead to task failure. Future improvements in RUA require simultaneous advancement in these interrelated foundational capabilities.

Universal Framework: What Does uni-robot-agent Support?

Beyond direct tool invocation, uni-robot-agent supports Code as Policy, Hybrid mode (VLM and VLA/WAM collaboration), and Clean Mode (retaining only basic tools). These configurations can be tested within the same framework.

Code as Policy allows models to control robots via code generation. Models can perform computations, invoke actions, and decide continuation based on execution feedback.

The team compared code control versus direct tool invocation using Qwen across RoboDojo’s five capability dimensions. In 252 evaluations, Qwen scored 25.60. Compared to direct tool calling, code control achieved higher scores in fine manipulation (17.40), though overall performance still lags.

Qwen’s score comparison under Code as Policy mode.

The framework also supports Hybrid mode, enabling VLM and VLA collaboration. The VLM interprets instructions, analyzes scenes, and organizes tasks; the VLA generates action segments based on observations, with feedback informing subsequent decisions.

The team demonstrated this collaboration in simulation scenarios involving Panda arm with Allegro dexterous hand, Unitree Go2 quadruped robot, Unitree G1 humanoid robot, and R1Pro wheeled robot, integrating action models like π₀.₅, GR00T, and NaVILA—covering dexterous manipulation, humanoid operation, and mobile navigation.

As model capabilities grow, currently tool-encapsulated logic may eventually be delegated to models themselves, potentially thinning the Harness. To explore this direction, uni-robot-agent offers Clean Mode—retaining only four basic tools: move, hold, finish, capture_scene—enabling models to complete tasks through observation, computation, and action composition.

Under camera-calibration-enabled settings, Clean Mode retained 90.4% and 90.5% of Qwen and Astra’s full-tool performance respectively, with average token usage reduced by 35.7% and 78.8%. Compared to historical full-configurations, this shows that after tool and context simplification, models maintain most task performance—providing preliminary evidence for thinning the Harness and improving token efficiency.

Token consumption vs. score comparison for Qwen and Astra under Clean Mode.

The framework’s compatibility also extends to more real robots.

The team integrated ARX One and Songling Piper-X real machines. Hardware differences are handled by underlying adapters, while models interact with the environment and execute actions through a unified interface, adjusting next steps based on feedback. Additionally, Zhiyuan G2 and Unitree G1 are under integration.

Real-machine integration of ARX One and Songling Piper-X

Access code and deployment instructions are also open-sourced, enabling developers to run uni-robot-agent on compatible hardware, swap models, and test their own tasks.

Conclusion

Robot intelligence stands at a pivotal fork.

Traditional dedicated policy approaches are tightly coupled to hardware, suffering from high scalability costs and limitations in generalization and long-horizon execution. The uni-robot-agent path enables general models to directly control robots via a unified interface—a fundamentally different expansion logic.

uni-robot-agent is the first open-source outcome of the OpenRUA project. Initiated by researchers from top universities and cutting-edge labs, OpenRUA spans embodied intelligence and language modeling, with the goal of building an open infrastructure for Agents capable of acting in the physical world.

Team homepage: https://openrua.ai/

From this perspective, the significance of uni-robot-agent extends beyond providing a robot operation framework—it demonstrates that open-weight models can achieve competitive performance in both simulation and real-world settings when information is well-organized and tool feedback is reliable.

Yet this is just the beginning. The next critical challenge lies in leveraging interaction data to continuously update models. uni-robot-agent has laid the infrastructure; the evolution of training and adaptation has just started.

Source: Machine Heart