Abstract
Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.
Abstract (translated)
理解并建模集体智能对于解决复杂的社会系统至关重要。被称为模糊认知图（FCMs）的指向图是一种强大的编码因果思维模型的工具，但从文本中提取高完整性的FCM具有挑战性。本研究介绍了一种使用大型语言模型（LLMs）自动提取FCM的方法。我们引入了新的基于图的相似度度量，并通过Elo评分系统将它们与人类评价进行相关性分析。结果表明，与人类评价之间存在积极的相关性，但最佳表现指标仍存在捕捉FCM细微之处的局限性。对LLMs的微调提高了性能，但现有的措施仍然存在不足之处。本研究强调了需要针对FCM提取的定制化软相似度度量，并通过自然语言处理推进集体智能建模。
URL
https://arxiv.org/abs/2409.18911
