Abstract
The recent advancements introduced by Large Language Models (LLMs) have transformed how Artificial Intelligence (AI) can support complex, real world tasks, pushing research outside the text boundaries towards multi modal contexts and leading to Multimodal Large Language Models (MLMs). Given the current adoption of LLM based assistants in solving technical or domain specific problems, the natural continuation of this trend is to extend the input domains of these assistants exploiting MLMs. Ideally, these MLMs should be used as real time assistants in procedural tasks, hopefully integrating a view of the environment where the user being assisted is, or even better sharing the same point of view via Virtual Reality (VR) or Augmented Reality (AR) supports, to reason over the same scenario the user is experiencing. With this work, we aim at evaluating the quality of currently openly available MLMs to provide this kind of assistance on technical tasks. To this end, we annotated a data set of furniture assembly with step by step labels and manual references: the Manual to Action Dataset (M2AD). We used this dataset to assess (1) to which extent the reasoning abilities of MLMs can be used to reduce the need for detailed labelling, allowing for more efficient, cost effective annotation practices, (2) whether MLMs are able to track the progression of assembly steps (3) and whether MLMs can refer correctly to the instruction manual pages. Our results showed that while some models understand procedural sequences, their performance is limited by architectural and hardware constraints, highlighting the need for multi image and interleaved text image reasoning.
Abstract (translated)
大语言模型（LLMs）近期取得的进展，已改变了人工智能（AI）支持复杂现实任务的方式，将研究推向文本边界之外，迈向多模态语境，并催生了多模态大语言模型（MLMs）。鉴于当前基于LLM的助手在解决技术或特定领域问题中的广泛应用，这一趋势的自然延续是利用MLMs扩展这些助手的输入领域。理想情况下，这些MLMs应能作为程序性任务中的实时助手，最好能整合被辅助用户所处的环境视图，甚至通过虚拟现实（VR）或增强现实（AR）设备共享同一视角，从而对用户正在经历的场景进行推理。本研究旨在评估当前开源MLMs在技术任务中提供此类辅助的质量。为此，我们标注了一个带有逐步标签和手动参考的家具组装数据集：即《手册到行动数据集》（Manual to Action Dataset, M2AD）。我们使用该数据集评估了以下三点：（1）MLMs的推理能力在多大程度上可减少对详细标注的需求，从而实现更高效、低成本的标注实践；（2）MLMs是否能够追踪组装步骤的进展；（3）MLMs能否正确引用说明书页面。结果表明，尽管部分模型能理解程序性序列，但其性能受限于架构和硬件限制，凸显了对多图像及图文交错推理的需求。
URL
https://arxiv.org/abs/2603.22321
