Abstract
The contour estimation of transparent fragments is very important for autonomous reassembly, especially in the fields of precision optical instrument repair, cultural relic restoration, and identification of other precious device broken accidents. Different from general intact transparent objects, the contour estimation of transparent fragments face greater challenges due to strict optical properties, irregular shapes and edges. To address this issue, a general transparent fragments contour estimation framework based on visual-tactile fusion is proposed in this paper. First, we construct the transparent fragment dataset named TransFrag27K, which includes a multiscene synthetic data of broken fragments from multiple types of transparent objects, and a scalable synthetic data generation pipeline. Secondly, we propose a visual grasping position detection network named TransFragNet to identify, locate and segment the sampling grasping position. And, we use a two-finger gripper with Gelsight Mini sensors to obtain reconstructed tactile information of the lateral edge of the fragments. By fusing this tactile information with visual cues, a visual-tactile fusion material classifier is proposed. Inspired by the way humans estimate a fragment's contour combining vision and touch, we introduce a general transparent fragment contour estimation framework based on visual-tactile fusion, demonstrates strong performance in real-world validation. Finally, a multi-dimensional similarity metrics based contour matching and reassembly algorithm is proposed, providing a reproducible benchmark for evaluating visual-tactile contour estimation and fragment reassembly. The experimental results demonstrate the validity of the proposed framework. The dataset and codes are available at this https URL.
Abstract (translated)
透明碎片的轮廓估计对于自主重组至关重要，尤其在精密光学仪器修复、文物复原及其他珍贵设备破损事故鉴定等领域。与一般完好的透明物体不同，透明碎片的轮廓估计面临更大挑战，因其具有严格的光学特性、不规则的形状与边缘。针对此问题，本文提出一种基于视觉-触觉融合的通用透明碎片轮廓估计框架。首先，我们构建了名为TransFrag27K的透明碎片数据集，该数据集包含多类型透明物体破碎碎片的多场景合成数据，以及一个可扩展的合成数据生成流程。其次，我们提出了名为TransFragNet的视觉抓取位置检测网络，用于识别、定位并分割采样抓取位置。同时，我们使用配备Gelsight Mini传感器的双指夹爪获取碎片侧边的重建触觉信息。通过融合该触觉信息与视觉线索，提出了一种视觉-触觉融合材料分类器。受人类结合视觉与触觉估计碎片轮廓的方式启发，我们引入了基于视觉-触觉融合的通用透明碎片轮廓估计框架，并在实际验证中展现出强劲性能。最后，提出了一种基于多维相似度度量的轮廓匹配与重组算法，为评估视觉-触觉轮廓估计与碎片重组提供了可复现的基准。实验结果表明了所提框架的有效性。数据集与代码已公开于此https URL。
URL
https://arxiv.org/abs/2603.20290
