Abstract
Semantic correspondence made tremendous progress through the recent advancements of large vision models (LVM). While these LVMs have been shown to reliably capture local semantics, the same can currently not be said for capturing global geometric relationships between semantic object regions. This problem leads to unreliable performance for semantic correspondence between images with extreme view variation. In this work, we aim to leverage monocular depth estimates to capture these geometric relationships for more robust and data-efficient semantic correspondence. First, we introduce a simple but effective method to build 3D object-class representations from monocular depth estimates and LVM features using a sparsely annotated image correspondence dataset. Second, we formulate an alignment energy that can be minimized using gradient descent to obtain an alignment between the 3D object-class representation and the object-class instance in the input RGB-image. Our method achieves state-of-the-art matching accuracy in multiple categories on the challenging SPair-71k dataset, increasing the PCK@0.1 score by more than 10 points on three categories and overall by 3.3 points from 85.6% to 88.9%. Additional resources and code are available at this https URL.
Abstract (translated)
语义对应关系通过最近大型视觉模型（LVM）的进步取得了巨大进展。尽管这些LVM已被证明能够可靠地捕捉局部语义，但目前还不能说它们能有效地捕捉不同语义对象区域之间的全局几何关系。这个问题导致在处理视角变化极大的图像时，语义对应的性能变得不可靠。在这项工作中，我们旨在利用单目深度估计来捕捉这些几何关系，以实现更稳健且数据效率更高的语义对应。 首先，我们介绍了一种简单但有效的方法，可以使用稀疏标注的图像对应数据集从单目深度估计和LVM特征构建3D对象类别表示。其次，我们提出了一个对齐能量公式，该公式可以通过梯度下降法最小化来获得3D对象类表示与输入RGB图像中对象实例之间的对齐。 我们的方法在具有挑战性的SPair-71k数据集的多个类别上实现了最先进的匹配准确性，在三个类别上的PCK@0.1分数提高了超过10个百分点，并且整体从85.6%提高到了88.9%，增加了3.3个百分点。更多资源和代码可以在[这里](https://this_https_URL.com/)找到。 请注意，最后一个URL应被替换为实际提供的链接地址。
URL
https://arxiv.org/abs/2503.22462
