Abstract
We present GestOS, a gesture-based operating system for high-level control of heterogeneous robot teams. Unlike prior systems that map gestures to fixed commands or single-agent actions, GestOS interprets hand gestures semantically and dynamically distributes tasks across multiple robots based on their capabilities, current state, and supported instruction sets. The system combines lightweight visual perception with large language model (LLM) reasoning: hand poses are converted into structured textual descriptions, which the LLM uses to infer intent and generate robot-specific commands. A robot selection module ensures that each gesture-triggered task is matched to the most suitable agent in real time. This architecture enables context-aware, adaptive control without requiring explicit user specification of targets or commands. By advancing gesture interaction from recognition to intelligent orchestration, GestOS supports scalable, flexible, and user-friendly collaboration with robotic systems in dynamic environments.
Abstract (translated)
我们介绍了GestOS，这是一种基于手势的操作系统，用于对异构机器人团队进行高级控制。与以前将手势映射到固定命令或单一代理动作的系统不同，GestOS 通过考虑机器人的能力、当前状态和可支持指令集来语义化地解释手部姿势，并动态分配任务给多个机器人。该系统结合了轻量级视觉感知技术和大型语言模型（LLM）推理：将手势转换为结构化的文本描述，然后由 LLM 推断意图并生成特定于机器人的命令。一个机器人选择模块确保每次通过手势触发的任务都能实时匹配最合适的代理。这种架构使上下文感知、自适应控制成为可能，而无需用户明确指定目标或命令。通过将手势交互从识别提升到智能编排，GestOS 支持在动态环境中与机器人系统进行可扩展的、灵活且用户友好的协作。
URL
https://arxiv.org/abs/2509.14412
