Abstract
Vision-transformers (ViTs) and large-scale convolution-neural-networks (CNNs) have reshaped computer vision through pretrained feature representations that enable strong transfer learning for diverse tasks. However, their efficiency as backbone architectures for geometric estimation tasks involving image deformations in low-data regimes remains an open question. This work considers two such tasks: 1) estimating 2D rigid transformations between pairs of images and 2) predicting the fundamental matrix for stereo image pairs, an important problem in various applications, such as autonomous mobility, robotics, and 3D scene reconstruction. Addressing this intriguing question, this work systematically compares large-scale CNNs (ResNet, EfficientNet, CLIP-ResNet) with ViT-based foundation models (CLIP-ViT variants and DINO) in various data size settings, including few-shot scenarios. These pretrained models are optimized for classification or contrastive learning, encouraging them to focus mostly on high-level semantics. The considered tasks require balancing local and global features differently, challenging the straightforward adoption of these models as the backbone. Empirical comparative analysis shows that, similar to training from scratch, ViTs outperform CNNs during refinement in large downstream-data scenarios. However, in small data scenarios, the inductive bias and smaller capacity of CNNs improve their performance, allowing them to match that of a ViT. Moreover, ViTs exhibit stronger generalization in cross-domain evaluation where the data distribution changes. These results emphasize the importance of carefully selecting model architectures for refinement, motivating future research towards hybrid architectures that balance local and global representations.
Abstract (translated)
视觉变换器（ViTs）和大规模卷积神经网络（CNNs）通过预训练的特征表示，彻底改变了计算机视觉领域，为各种任务提供了强大的迁移学习能力。然而，在数据量较少的情况下，作为涉及图像变形的几何估计任务的骨干架构，它们的效率仍然是一个开放性问题。这项工作考虑了两个这样的任务：1) 估计图像对之间的二维刚体变换；2) 预测立体图像对的基本矩阵，这是一个在自动驾驶、机器人技术和三维场景重建等多种应用中都很重要的问题。为了解决这个引人入胜的问题，本研究系统地比较了大规模CNN（如ResNet、EfficientNet和CLIP-ResNet）与基于ViT的基础模型（如CLIP-ViT变体和DINO），在各种数据规模设置下进行对比分析，包括少量样本场景。这些预训练的模型是为分类或对比学习优化的，鼓励它们主要关注高层次语义信息。然而，所考虑的任务需要平衡局部特征和全局特征，这挑战了直接采用这些模型作为骨干架构的做法。 实验性的比较分析表明，在大量下游数据场景中进行微调时，ViT的表现优于CNN，这一点与从头开始训练的情况相似。但在小规模数据场景下，CNN的归纳偏置以及较小的容量提高了它们的表现，使其能够匹敌ViT的表现。此外，在跨领域评估中（即数据分布发生变化），ViTs表现出更强的泛化能力。这些结果强调了在微调时谨慎选择模型架构的重要性，并激励未来的研究向平衡局部和全局表示的混合架构发展。
URL
https://arxiv.org/abs/2510.04794
