Abstract
Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.
Abstract (translated)
主动且实时的互动体验对于类人AI伴侣来说至关重要，但面临着三大挑战：(1) 在持续流式输入的情况下实现低延迟推理；(2) 自主决定何时响应；以及 (3) 控制生成内容的质量和数量以满足实时约束。在这项工作中，我们通过两个游戏场景——解说员和导游来实例化AI伴侣，并选择了这两个场景因其适合自动评估。我们引入了“Live Gaming Benchmark”，这是一个大规模的数据集，包含三个代表性场景：个人解说、联合解说以及用户指导。此外，我们提出了Proact-VL框架，该框架能够将多模态语言模型塑造成具备类人环境感知和互动能力的主动型实时交互代理。广泛的实验表明，Proact-VL在保持强大的视频理解能力的同时，实现了优越的响应延迟和质量，证明了其在实时交互应用中的实用性。
URL
https://arxiv.org/abs/2603.03447
