Abstract
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.
Abstract (translated)
视频模型中的 Tokenization 通常通过补丁化（patchification）实现，这会产生大量冗余的 token。这极大地限制了视频处理的效率和可扩展性。尽管最近基于轨迹的 tokenizer 提供了一种有前景的解决方案，能够将视频时长与 token 数量解耦，但它们依赖于复杂的外部分割和跟踪管道，这些管道运行缓慢且不针对特定任务。我们提出了 TrajTok，这是一种端到端的视频 tokenizer 模块，它完全集成并协同训练视频模型以适应下游目标，并根据语义复杂性动态调整其 token 粒度，不受视频时长的影响。 TrajTok 包含一个统一的分段器，它可以对空间和时间上的像素进行隐式聚类，直接在一个前向传递中生成对象轨迹。通过优先考虑下游任务适应性而不是像素级分割精度，TrajTok 既轻量又高效，并且在实践中提高了视频理解性能。 借助 TrajTok，我们实现了一个从头开始训练的视频 CLIP 模型（TrajViT2），它在分类和检索基准测试中实现了最佳的准确率扩展性，同时保持了与最好的 token 合并方法相当的效率。此外，TrajTok 也证明了其作为 tokenizer 超出角色之外的功能多样性。我们展示了它可以无缝地集成到预训练视觉特征的探测头（TrajAdapter）或视觉-语言模型中的对齐连接器（TrajVLM），尤其在长视频推理任务中表现出色。
URL
https://arxiv.org/abs/2602.22779
