Abstract
In recent years, there has been a proliferation of spatiotemporal foundation models in different scientific disciplines. While promising, these models are often domain-specific and are only assessed within the particular applications for which they are designed. Given that many tasks can be represented as video modeling problems, video foundation models (ViFMs) hold considerable promise as general-purpose domain-agnostic approaches. However, it is not known whether the knowledge acquired on large-scale but potentially out-of-domain data can be effectively transferred across diverse scientific disciplines, and if a single, pretrained ViFM can be competitive with domain-specific baselines. To address this, we introduce SciVid, a comprehensive benchmark comprising five *Sci*entific *Vid*eo tasks, across medical computer vision, animal behavior, and weather forecasting. We adapt six leading ViFMs to SciVid using simple trainable readout modules, establishing strong baselines and demonstrating the potential for effective transfer learning. Specifically, we show that state-of-the-art results can be obtained in several applications by leveraging the general-purpose representations from ViFM backbones. Furthermore, our results reveal the limitations of existing ViFMs, and highlight opportunities for the development of generalizable models for high-impact scientific applications. We release our code at this https URL to facilitate further research in the development of ViFMs.
Abstract (translated)
近年来，不同科学领域中涌现出了许多时空基础模型。尽管这些模型前景广阔，但它们往往是特定领域的，并且仅在其设计的应用场景中进行评估。鉴于许多任务可以被表示为视频建模问题，视频基础模型（Video Foundation Models, ViFMs）作为通用的、不受限于特定领域的解决方案展现了巨大的潜力。然而，尚不清楚在大规模但可能超出领域范围的数据上获得的知识能否有效地跨多个科学学科传递，以及单一的预训练ViFM是否能与特定领域的基准方法相竞争。 为了解决这些问题，我们引入了SciVid，这是一个全面的基准测试集，涵盖了五个*科研视频任务*（Scientific Video Tasks），涉及医学计算机视觉、动物行为和天气预报。我们将六个领先的ViFMs适配到SciVid中，使用简单的可训练输出模块，并建立了强大的基线模型，展示了有效迁移学习的可能性。具体来说，我们证明了通过利用来自ViFM骨干网络的通用表示形式，在多个应用中可以获得最先进的结果。此外，我们的结果显示现有ViFMs的局限性，并强调为具有高影响力的应用开发泛化模型的机会。 为了促进进一步的研究和视频基础模型（Video Foundation Models, ViFMs）的发展，我们公开了代码，网址是[此处]（请将"this https URL"替换为实际的链接）。
URL
https://arxiv.org/abs/2507.03578
