Abstract
In the current era of Machine Learning, Transformers have become the de facto approach across a variety of domains, such as computer vision and natural language processing. Transformer-based solutions are the backbone of current state-of-the-art methods for language generation, image and video classification, segmentation, action and object recognition, among many others. Interestingly enough, while these state-of-the-art methods produce impressive results in their respective domains, the problem of understanding the relationship between vision and language is still beyond our reach. In this work, we propose a common ground between vision and language based on events in space and time in an explainable and programmatic way, to connect learning-based vision and language state of the art models and provide a solution to the long standing problem of describing videos in natural language. We validate that our algorithmic approach is able to generate coherent, rich and relevant textual descriptions on videos collected from a variety of datasets, using both standard metrics (e.g. Bleu, ROUGE) and the modern LLM-as-a-Jury approach.
Abstract (translated)
在当今的机器学习时代，Transformer模型已成为计算机视觉和自然语言处理等多个领域的标准方法。基于Transformer的解决方案是当前最先进的语言生成、图像和视频分类、分割、动作与物体识别等众多领域技术的基础。有趣的是，尽管这些最先进技术在其各自领域取得了令人瞩目的成果，但理解视觉与语言之间的关系仍然是我们尚未攻克的问题。 在这项工作中，我们提出了一种基于时间和空间中事件的解释性和程序化方法来连接以学习为基础的视觉和语言最新模型，并提供了解决长期以来存在的用自然语言描述视频问题的新方案。通过使用标准指标（如Bleu、ROUGE）以及现代的大规模语言模型作为评审的方法，我们验证了我们的算法能够生成连贯、丰富且相关的文本描述，适用于从不同数据集中收集的各种视频。
URL
https://arxiv.org/abs/2501.08460
