Abstract
Test-Time Training (TTT) has recently emerged as a promising direction for efficient sequence modeling. TTT reformulates attention operation as an online learning problem, constructing a compact inner model from key-value pairs at test time. This reformulation opens a rich and flexible design space while achieving linear computational complexity. However, crafting a powerful visual TTT design remains challenging: fundamental choices for the inner module and inner training lack comprehensive understanding and practical guidelines. To bridge this critical gap, in this paper, we present a systematic empirical study of TTT designs for visual sequence modeling. From a series of experiments and analyses, we distill six practical insights that establish design principles for effective visual TTT and illuminate paths for future improvement. These findings culminate in the Vision Test-Time Training (ViT$^3$) model, a pure TTT architecture that achieves linear complexity and parallelizable computation. We evaluate ViT$^3$ across diverse visual tasks, including image classification, image generation, object detection, and semantic segmentation. Results show that ViT$^3$ consistently matches or outperforms advanced linear-complexity models (e.g., Mamba and linear attention variants) and effectively narrows the gap to highly optimized vision Transformers. We hope this study and the ViT$^3$ baseline can facilitate future work on visual TTT models. Code is available at this https URL.
Abstract (translated)
测试时间训练（TTT，Test-Time Training）最近作为高效序列建模的一个有前景的方向而出现。TTT将注意力操作重新定义为一个在线学习问题，在测试时从键值对构造出一个紧凑的内部模型。这种重新定义不仅开辟了一个丰富且灵活的设计空间，还实现了线性计算复杂度。然而，构建强大的视觉TTT设计仍然具有挑战性：对于内核模块和内核训练的基本选择缺乏全面的理解和实用指南。为弥合这一关键差距，本文中我们对视觉序列建模中的TTT设计进行了系统的实证研究。通过一系列实验和分析，我们提炼出了六项实践洞察，这些洞察确立了有效视觉TTT的设计原则，并指明了未来改进的道路。这些发现最终形成了Vision Test-Time Training（ViT$^3$）模型，这是一个纯粹的TTT架构，在实现线性复杂度的同时还支持并行计算。我们在包括图像分类、图像生成、目标检测和语义分割在内的多种视觉任务中评估了ViT$^3$。结果表明，ViT$^3$在性能上始终与最先进的线性复杂度模型（如Mamba和线性注意变体）匹配或超越，并有效地缩小了与高度优化的视觉Transformer之间的差距。我们希望这项研究以及ViT$^3$基准能促进未来关于视觉TTT模型的工作。代码可在此URL获取。 通过这一系统性的实证研究，我们不仅展示了如何设计高效的视觉TTT架构，还为后续的研究工作提供了宝贵的洞见和方向指引。
URL
https://arxiv.org/abs/2512.01643
