Abstract
Accurate perception of object hardness is essential for safe and dexterous contact-rich robotic manipulation. Here, we present TactEx, an explainable multimodal robotic interaction framework that unifies vision, touch, and language for human-like hardness estimation and interactive guidance. We evaluate TactEx on fruit-ripeness assessment, a representative task that requires both tactile sensing and contextual understanding. The system fuses GelSight-Mini tactile streams with RGB observations and language prompts. A ResNet50+LSTM model estimates hardness from sequential tactile data, while a cross-modal alignment module combines visual cues with guidance from a large language model (LLM). This explainable multimodal interface allows users to distinguish ripeness levels with statistically significant class separation (p < 0.01 for all fruit pairs). For touch placement, we compare YOLO with Grounded-SAM (GSAM) and find GSAM to be more robust for fine-grained segmentation and contact-site selection. A lightweight LLM parses user instructions and produces grounded natural-language explanations linked to the tactile outputs. In end-to-end evaluations, TactEx attains 90% task success on simple user queries and generalises to novel tasks without large-scale tuning. These results highlight the promise of combining pretrained visual and tactile models with language grounding to advance explainable, human-like touch perception and decision-making in robotics.
Abstract (translated)
准确地感知物体的硬度对于安全和灵巧的操作密集型机器人操作至关重要。在这里，我们介绍了TactEx，这是一个可解释的多模态机器人交互框架，它将视觉、触觉和语言统一起来以实现人类级别的硬度估计和互动指导。我们在水果成熟度评估这一任务上对TactEx进行了评价，该任务需要触摸感应与上下文理解能力。系统融合了GelSight-Mini触觉流、RGB观察结果以及语言提示。 一个基于ResNet50+LSTM的模型从连续触觉数据中估计硬度，而跨模态对齐模块则结合视觉线索和大型语言模型（LLM）提供的指导信息。这种可解释的多模态界面允许用户通过显著的类别分离（所有水果配对的p值均小于0.01）来区分成熟度级别。 对于触摸放置的位置选择，我们将YOLO与Grounded-SAM (GSAM)进行了比较，并发现GSAM在细粒度分割和接触点选取方面更为稳健。一个轻量级的语言模型解析用户指令并生成与触觉输出相关联的自然语言解释。在端到端评估中，TactEx对简单用户查询的成功率为90%，并且能够泛化到新任务而无需大规模调整。 这些结果突显了结合预训练视觉和触觉模型以及语言接地以推进机器人领域可解释、人类级别的触摸感知与决策制定的潜力。
URL
https://arxiv.org/abs/2602.18967
