机器人与具身智能 颠覆级 有讲解视频
发表时间
2026-09-20
arXiv
2609.23863

收录解读

操作策略必须知道哪些物体重要以及它们在哪里,但当前机器人基础模型所依赖的预训练主干(如VLA中的语言、WAM中的视频生成)并不直接要求这种度量级3D接地,而是依赖从示教数据中隐式学习。

GAMs提出了一种以3D接地为构建基础的机器人基础模型策略,将语言、点或盒提示统一变换为所选物体的共享物体中心表示,融合目标视觉特征和度量物体几何,并通过多流Transformer与机器人状态历史混合来预测动作块;它既可自主运行,也可作为高层规划器的低级控制器用于长时程和记忆依赖任务。

实验给出了跨仿真和真实机器人的量化结果:RoboTwin 2.0上50任务平均成功率55.3% vs. 52.0%,场景随机化下47.6% vs. 30.4%;LIBERO-PRO上61% vs. 53%;双机械臂YAM视觉偏移下17/20 vs. 4/20;与Molmo2规划器组合在Franka上达到64.7% ID和49.8% OOD步骤完成率。

这些证据表明显式3D接地可以显著提升策略的视觉鲁棒性和目标重定位泛化,并支持与高层规划器组合。限制是绝对成功率仍不算很高,真实机器人试验次数有限,且未提及代码和权重开放,因此‘新范式’的说法需要更多社区复验与跨任务扩展。

原始摘要

Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for $π_{0.5}$) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for $π_{0.5}$, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.

解读视频

相关论文

链接