Abstract
This manuscript explores multimodal alignment, translation, fusion, and transference to enhance machine understanding of complex inputs. We organize the work into five chapters, each addressing unique challenges in multimodal machine learning. Chapter 3 introduces Spatial-Reasoning Bert for translating text-based spatial relations into 2D arrangements between clip-arts. This enables effective decoding of spatial language into visual representations, paving the way for automated scene generation aligned with human spatial understanding. Chapter 4 presents a method for translating medical texts into specific 3D locations within an anatomical atlas. We introduce a loss function leveraging spatial co-occurrences of medical terms to create interpretable mappings, significantly enhancing medical text navigability. Chapter 5 tackles translating structured text into canonical facts within knowledge graphs. We develop a benchmark for linking natural language to entities and predicates, addressing ambiguities in text extraction to provide clearer, actionable insights. Chapter 6 explores multimodal fusion methods for compositional action recognition. We propose a method fusing video frames and object detection representations, improving recognition robustness and accuracy. Chapter 7 investigates multimodal knowledge transference for egocentric action recognition. We demonstrate how multimodal knowledge distillation enables RGB-only models to mimic multimodal fusion-based capabilities, reducing computational requirements while maintaining performance. These contributions advance methodologies for spatial language understanding, medical text interpretation, knowledge graph enrichment, and action recognition, enhancing computational systems' ability to process complex, multimodal inputs across diverse applications.
Abstract (translated)
这份手稿探讨了多模态对齐、翻译、融合和迁移，以增强机器对复杂输入的理解。我们将这项工作分为五章，每章都针对多模态机器学习中的独特挑战。 第三章介绍了Spatial-Reasoning Bert，它能够将基于文本的空间关系转换为剪贴画之间的二维排列。这使得空间语言的有效解码成为可能，并转化为视觉表示，从而开启了与人类空间理解一致的自动场景生成之路。 第四章提出了一种方法，用于将医学文本翻译成解剖图谱中的特定三维位置。我们引入了一个损失函数，利用医学术语的空间共现来创建可解释映射，显著增强了对医学文本导航的理解能力。 第五章解决了将结构化文本转换为知识图中规范事实的问题。我们建立了一个基准，用以连接自然语言与实体和谓语，解决文本提取中的歧义问题，提供更清晰、可行的见解。 第六章探讨了用于组成性动作识别的多模态融合方法。我们提出了一种结合视频帧和对象检测表示的方法，提高了识别的鲁棒性和准确性。 第七章研究了面向第一人称视角动作识别的多模态知识迁移。我们展示了如何通过多模态知识蒸馏使RGB-only模型模仿基于多模态融合的能力，从而在减少计算需求的同时保持性能。 这些贡献推进了空间语言理解、医学文本解读、知识图谱丰富以及行动识别的方法论，增强了计算系统处理复杂多元输入的能力，涵盖各种应用。
URL
https://arxiv.org/abs/2512.20501
