收录解读
GUI智能体有望成为数字设备的通用执行器,但现有方法在真实设备可靠性、跨平台操作、长程任务、主动服务和自主改进等方面仍不足。本研究提出Qwen-UI-Agent,一个面向真实世界的通用GUI智能体,覆盖移动、桌面、网页和DeepSearch环境,旨在解决这些挑战。
核心方法包括:统一动作空间,将GUI操作与CLI执行交错,并在单轮生成批量动作;结合多样沙箱环境与大规模真实设备移动运行时;提出AutoResearch式数据飞轮,利用智能体自主构造任务、诊断失败并规划迭代;使用在线强化学习支持超过100步的轨迹训练,配合超过一万个并发环境加速数据收集;轻量级harness层支持主动服务启动和有状态工作流。
在多个基准上取得领先结果:移动端MobileWorld 82.1%、MobileWorld-Real 92.2%、AndroidDaily 97.5%;计算机端OSWorld-Verified 79.5%、OSWorld-v2部分进度40.0%;浏览器端WebArena 73.6%、ScreenSpot-Pro 81.5%,与Claude Opus 4.8、Gemini 3.1 Pro、GPT-5.6 Sol等前沿模型相比具有竞争力。
该工作为GUI智能体提供了统一的训练框架和数据飞轮范式,具有跨平台通用性和自动化数据构造能力,但作为技术报告,缺少对各个组件贡献的消融分析,且未明确开源范围。若后续提供开放代码和深入分析,有望成为该领域的范式级工作。
原始摘要
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL support training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computerand browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorldReal, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorldVerified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.