Abstract
In situated collaboration, speakers often use intentionally underspecified deictic commands (e.g., ``pass me \textit{that}''), whose referent becomes identifiable only by aligning speech with a brief co-speech pointing \emph{stroke}. However, many embodied benchmarks admit language-only shortcuts, allowing MLLMs to perform well without learning the \emph{audio--visual alignment} required by deictic interaction. To bridge this gap, we introduce \textbf{Egocentric Co-Speech Grounding (EcoG)}, where grounding is executable only if an agent jointly predicts \textit{What}, \textit{Where}, and \textit{When}. To operationalize this, we present \textbf{EcoG-Bench}, an evaluation-only bilingual (EN/ZH) diagnostic benchmark of \textbf{811} egocentric clips with dense spatial annotations and millisecond-level stroke supervision. It is organized under a \textbf{Progressive Cognitive Evaluation} protocol. Benchmarking state-of-the-art MLLMs reveals a severe executability gap: while human subjects achieve near-ceiling performance on EcoG-Bench (\textbf{96.9\%} strict Eco-Accuracy), the best native video-audio setting remains low (Gemini-3-Pro: \textbf{17.0\%}). Moreover, in a diagnostic ablation, replacing the native video--audio interface with timestamped frame samples and externally verified ASR (with word-level timing) substantially improves the same model (\textbf{17.0\%}$\to$\textbf{42.9\%}). Overall, EcoG-Bench provides a strict, executable testbed for event-level speech--gesture binding, and suggests that multimodal interfaces may bottleneck the observability of temporal alignment cues, independently of model reasoning.
Abstract (translated)
在情境合作中，说话者常常使用故意欠指代的指示性命令（例如，“把那个递给我”），其具体所指的是通过讲话与短暂的手势配合——即指示动作“笔画”来确定。然而，许多基于体感的评估标准允许只依赖语言的捷径，使得多模态大型语言模型（MLLMs）在不需要学习指示性互动所需的音视频对齐的情况下也能表现良好。为了弥补这一差距，我们引入了**第一人称视角伴随言语指代（Egocentric Co-Speech Grounding, EcoG）**，在这种机制下，只有当代理同时预测“什么”、“在哪里”和“何时”的时候，才允许执行指向操作。为实现这一点，我们提出了**EcoG-Bench**，这是一个针对**811个第一人称视角片段**的诊断评估基准，这些片段具有密集的空间注释以及毫秒级笔画监督，并且是双语（英语/中文）模式下进行的。该基准遵循**渐进认知评估**协议。 对最新多模态大型语言模型的评测揭示了执行能力上的显著差距：虽然人类在EcoG-Bench上接近满分地表现优异（严格第一人称视角准确率为96.9%），但在原生视频-音频设置中的最佳性能依然很低（通义千问3Pro: 17.0%）。此外，在一次诊断消融实验中，用带有时间戳的帧样本和外部验证过的自动语音识别（ASR，具有单词级定时）来代替原生音视频接口，极大地提升了同一模型的表现（从**17.0%**提升到**42.9%**）。 总体而言，EcoG-Bench为事件级别的言语-手势绑定提供了一个严格的可执行测试环境，并暗示多模态界面可能独立于模型推理能力之外限制了时间对齐线索的可观测性。
URL
https://arxiv.org/abs/2603.07966
