Abstract
This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data-including textual design requirements, CAD images, and engineering drawings-derived from the Formula SAE student competition. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different sources. The benchmark features automatic evaluation metrics and is divided into segments-Rule Comprehension, Rule Compliance, and Rule Extraction-based on tasks that engineers perform when designing according to requirements. We evaluate state-of-the-art models like GPT4 and LLaVA against the benchmark, and our study uncovers the existing gaps in MLLMs' abilities to interpret complex engineering documentation. Key findings suggest that while MLLMs demonstrate potential in navigating technical documents, substantial limitations exist, particularly in accurately extracting and applying detailed requirements to engineering designs. This benchmark sets a foundation for future advancements in AI-supported engineering design processes. DesignQA is publicly available at: this https URL.
Abstract (translated)
这项研究介绍了一种名为DesignQA的新基准，旨在评估多模态大型语言模型（MLLM）在理解和技术文档中应用工程要求的能力。该基准重点关注现实世界的工程挑战，将多模态数据（包括文本设计要求、CAD图像和工程图纸）来源于方程式SAE学生竞赛，与许多现有MLLM基准不同。DesignQA包含基于文档的视觉问题，其中输入图像和输入文档来自不同的来源。基准基于工程师根据要求进行设计时执行的任务进行划分-规则理解、规则遵守和规则提取。我们评估了最先进的GPT4和LLaVA模型与该基准的比较，我们的研究揭示了MLLM在解释复杂工程文档方面的能力所存在的现有缺口。研究发现，尽管MLLM表现出在导航技术文档方面的潜力，但仍然存在很大的局限性，特别是在准确提取和应用详细工程设计要求方面。这项基准为支持AI辅助工程设计过程的未来发展奠定了基础。DesignQA可以在以下链接公开使用：https://this URL。
URL
https://arxiv.org/abs/2404.07917
