机器人与具身智能 颠覆级 暂无讲解视频
发表时间
2026-10-26
arXiv
2610.31762

收录解读

导航策略常直接接收视觉预训练目标决定的表示,难以选择最有用的细节。

RepresentNav让策略从多个冻结视觉模型的内部层构造自身表示,并以未来潜在状态预测训练导航动作。

VLNVerse细粒度测试成功率83.80%,较比较基线高20.05个百分点;同编码器消融中构造空间优于拼接,预测自身未来空间将碰撞率18.05%降至13.45%。

相对 paradigm,仍基于已有视觉骨干、JEPA和导航训练范式,未证明跨具身任务的普适替代;部分RxR指标和跟踪碰撞率不占优。

原始摘要

What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.

相关论文

链接