Abstract
Large language models (LLMs) have dramatically advanced machine learning research including natural language processing, computer vision, data mining, etc., yet they still exhibit critical limitations in reasoning, factual consistency, and interpretability. In this paper, we introduce a novel learning paradigm -- Modular Machine Learning (MML) -- as an essential approach toward new-generation LLMs. MML decomposes the complex structure of LLMs into three interdependent components: modular representation, modular model, and modular reasoning, aiming to enhance LLMs' capability of counterfactual reasoning, mitigating hallucinations, as well as promoting fairness, safety, and transparency. Specifically, the proposed MML paradigm can: i) clarify the internal working mechanism of LLMs through the disentanglement of semantic components; ii) allow for flexible and task-adaptive model design; iii) enable interpretable and logic-driven decision-making process. We present a feasible implementation of MML-based LLMs via leveraging advanced techniques such as disentangled representation learning, neural architecture search and neuro-symbolic learning. We critically identify key challenges, such as the integration of continuous neural and discrete symbolic processes, joint optimization, and computational scalability, present promising future research directions that deserve further exploration. Ultimately, the integration of the MML paradigm with LLMs has the potential to bridge the gap between statistical (deep) learning and formal (logical) reasoning, thereby paving the way for robust, adaptable, and trustworthy AI systems across a wide range of real-world applications.
Abstract (translated)
大型语言模型（LLMs）已经在机器学习研究中取得了显著进展，包括自然语言处理、计算机视觉和数据挖掘等领域，但它们在推理能力、事实一致性以及可解释性方面仍然存在关键性的局限。本文介绍了一种新颖的学习范式——模块化机器学习（Modular Machine Learning, MML），作为下一代LLMs的重要方法。MML将复杂结构的LLMs分解为三个相互依存的部分：模块表示、模块模型和模块推理，旨在增强LLMs在反事实推理能力上的表现，减少幻觉现象，并促进公平性、安全性和透明度。具体而言，所提出的MML范式可以： 1) 通过分离语义组件来阐明LLMs的内部工作机制； 2) 允许灵活且任务适应性的模型设计； 3) 实现可解释和逻辑驱动的决策过程。 我们通过利用诸如解耦表示学习、神经架构搜索以及符号-神经混合学习等先进技术，提出了基于MML范式的LLMs可行实现方法。同时，我们也批判性地指出了关键挑战，如连续神经处理与离散符号处理的融合、联合优化及计算可扩展性等问题，并提出了一些值得进一步探索的未来研究方向。最终，将MML范式集成到LLMs中有可能弥合统计（深度）学习和形式（逻辑）推理之间的差距，从而为各种现实世界应用中的稳健、适应性和可信的人工智能系统铺平道路。
URL
https://arxiv.org/abs/2504.20020
