机器人与具身智能 颠覆级 有讲解视频
发表时间
2026-10-09
arXiv
2610.10787

收录解读

具身智能体在真实物理环境中需要同时具备高层推理与低层运动控制能力,但现有方法往往将两者割裂,导致反应延迟高、无法应对突发事件。该问题对于机器人导航、人机协作等应用至关重要,是当前具身智能与语言模型结合的核心挑战。

NavGPT-3提出了一种层次化导航运行时,将语言模型推理、动作策略执行与监控作为独立线程运行,由运行时调度器决定当前控制权,支持中断与线程切换。底层动作策略NavGPT VLA采用码本分配机制,根据场景变化动态分配视觉token,实现高效低延迟控制。

分离推理/动作/监测线程并加入调度与控制权版本;同8B VLA上RxR-CE成功率78.19→90.43,Go2匹配对照中重叠+优先级使p95停止1840→180ms、漏停7/30→0/30,体现可复用具身运行时而非只换大模型。

仍集中导航、依赖大型训练集与前沿规划器,缺跨操作技能和广泛独立复现;与人类DE演示的数值匹配不等于普遍人类水平,未达到paradigm。

原始摘要

Language models trained with long-horizon agentic reinforcement learning can generalize knowledge through reasoning, express precise actions, and pursue goals over many steps, raising the ceiling on what an embodied agent can understand and decide. Physical interaction, however, remains the domain of action policies, which provide dense, low-latency control. We present NavGPT-3, a harness that connects the two models, with an OS-like runtime built above it: reasoning, acting, and monitoring run as threads with their own context, tools, and permissions, while the runtime schedules them and decides which thread controls the robot's motion, so that the robot can react to sudden real-world events through interruption and thread switching. Beneath it, our action policy NavGPT VLA, trained on 19.28M examples, allocates visual tokens using codec allocation, in proportion to scene change; its 8B model alone reaches 74.51 SR on R2R-CE and leads RxR-CE with 78.19 SR. With the complete harness, NavGPT-3 sets the state of the art on R2R-CE (81.51 SR) and, for the first time, brings an autonomous agent to human level: on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW) at 1 min 22 s per episode, versus roughly 3 min for a human. We comprehensively ablate the harness design and the interaction between the two models, showing how tools and the action policy shape the path from language-model reasoning to physical control: when NavGPT VLA executes the route, the reasoning loop shortens and the system's minimum reaction time falls from 3-19 s per language-model decision to 0.5-1 s per action-policy step (1-2 Hz). These results show that designing this embodied interface is central to connecting frontier language-model intelligence with low-level physical control. We will release all models, code, and evaluation records.

解读视频

相关论文

链接