Abstract
In recent years, large transformer-based video encoder models have greatly advanced state-of-the-art performance on video classification tasks. However, these large models typically process videos by averaging embedding outputs from multiple clips over time to produce fixed-length representations. This approach fails to account for a variety of time-related features, such as variable video durations, chronological order of events, and temporal variance in feature significance. While methods for temporal modeling do exist, they often require significant architectural changes and expensive retraining, making them impractical for off-the-shelf, fine-tuned large encoders. To overcome these limitations, we propose DejaVid, an encoder-agnostic method that enhances model performance without the need for retraining or altering the architecture. Our framework converts a video into a variable-length temporal sequence of embeddings, which we call a multivariate time series (MTS). An MTS naturally preserves temporal order and accommodates variable video durations. We then learn per-timestep, per-feature weights over the encoded MTS frames, allowing us to account for variations in feature importance over time. We introduce a new neural network architecture inspired by traditional time series alignment algorithms for this learning task. Our evaluation demonstrates that DejaVid substantially improves the performance of a state-of-the-art large encoder, achieving leading Top-1 accuracy of 77.2% on Something-Something V2, 89.1% on Kinetics-400, and 88.6% on HMDB51, while adding fewer than 1.8% additional learnable parameters and requiring less than 3 hours of training time. Our code is available at this https URL.
Abstract (translated)
近年来，基于大规模变压器的视频编码模型在视频分类任务中取得了显著的进步。然而，这些大型模型通常通过从多个片段中平均嵌入输出来处理视频，并生成固定长度的表示，这种方法未能考虑到多种时间相关特征，如不同视频时长、事件发生的顺序以及随时间变化的功能重要性差异。虽然存在一些用于时间建模的方法，但它们往往需要对架构进行重大调整和昂贵的重新训练，这使得这些方法对于现成的微调大型编码器来说不切实际。 为克服这些限制，我们提出了DejaVid——一种无需重新训练或修改架构即可提升模型性能的编码无关方法。我们的框架将视频转换为包含多变量时间序列（MTS）的可变长度的时间序列嵌入形式，这种格式自然保留了时间顺序并适应不同的视频时长。接下来，我们在编码后的MTS帧上学习每时刻、每个特征的权重，从而能够考虑随时间变化的功能重要性差异。为了执行这一学习任务，我们引入了一种受传统时间序列对齐算法启发的新神经网络架构。 我们的评估表明，DejaVid显著提升了最先进的大型编码器的表现，在Something-Something V2上实现了77.2%的领先Top-1准确率，在Kinetics-400上为89.1%，在HMDB51上达到88.6%，同时仅增加了不到1.8%的可学习参数，并且训练时间少于3小时。我们的代码可以在提供的链接中获取。
URL
https://arxiv.org/abs/2506.12585
