Abstract
Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at this https URL.
Abstract (translated)
人类对新概念的认知本质上是一个持续流动的过程：我们不断识别新对象或身份，并随时间更新记忆。然而，当前的多模态个性化方法大多局限于静态图像或离线视频，这割裂了连续视觉输入与即时现实反馈的联系，限制了其为未来AI助手提供实时、交互式个性化响应的能力。为弥合这一差距，我们首次提出并正式定义了新任务——个性化流式视频理解（PSVU）。为促进该新方向的研究，我们引入了PEARL-Bench，这是首个专为该挑战性场景设计的综合基准。它通过两种模式评估模型在精确时间戳对个性化概念的响应能力：（1）帧级模式，关注离散帧中的特定人物或物体；（2）新颖的视频级模式，关注跨连续帧展开的个性化动作。PEARL-Bench包含132个独特视频和2,173条带精确时间戳的细粒度标注。通过自动化生成与人工验证结合的流程，严格确保了概念多样性和标注质量。为应对这一挑战性新场景，我们进一步提出了PEARL——一种即插即用、无需训练的强基线策略。在8个离线与在线模型上的广泛评估表明，PEARL达到了最先进性能。值得注意的是，将其应用于3种不同架构时均能持续提升PSVU效果，证明了其高度有效性与鲁棒性。我们希望本研究能推动视觉-语言模型（VLM）个性化发展，并激发对流式个性化AI助手的进一步研究。代码已公开于此https链接。
URL
https://arxiv.org/abs/2603.20422
