Abstract
Anticipating future events is crucial for various application domains such as healthcare, smart home technology, and surveillance. Narrative event descriptions provide context-rich information, enhancing a system's future planning and decision-making capabilities. We propose a novel task: $\textit{long-term future narration generation}$, which extends beyond traditional action anticipation by generating detailed narrations of future daily activities. We introduce a visual-language model, ViNa, specifically designed to address this challenging task. ViNa integrates long-term videos and corresponding narrations to generate a sequence of future narrations that predict subsequent events and actions over extended time horizons. ViNa extends existing multimodal models that perform only short-term predictions or describe observed videos by generating long-term future narrations for a broader range of daily activities. We also present a novel downstream application that leverages the generated narrations called future video retrieval to help users improve planning for a task by visualizing the future. We evaluate future narration generation on the largest egocentric dataset Ego4D.
Abstract (translated)
预测未来事件对于医疗保健、智能家居技术和监控等各个应用领域至关重要。叙述性事件描述提供了富含背景信息的内容，增强了系统对未来规划和决策的能力。我们提出了一项新的任务：**长期未来叙述生成**（long-term future narration generation），这项任务超越了传统的行动预判，通过生成未来的日常活动的详细叙述来扩展其范围。为此，我们引入了一个视觉-语言模型ViNa，专门设计用于应对这一具有挑战性的任务。 ViNa结合了长时间视频及其对应的叙述，生成了一系列对未来事件和行为进行预测的未来叙述序列，这些预测覆盖较长的时间跨度。与现有的仅执行短期预测或描述观察到视频的多模态模型不同，ViNa能够为更广泛的日常活动生成长期未来的叙述。此外，我们还提出了一种新的下游应用，该应用利用生成的叙述来帮助用户通过可视化未来的方式提高任务规划能力，这种方法称为**未来视频检索**（future video retrieval）。 我们在最大的第一人称视角数据集Ego4D上评估了这项未来叙述生成技术。
URL
https://arxiv.org/abs/2503.01416
