Abstract
In Chomsky's provocative critique "The False Promise of CHATGPT," Large Language Models (LLMs) are characterized as mere pattern predictors that do not acquire languages via intrinsic causal and self-correction structures like humans, therefore are not able to distinguish impossible languages. It stands as a representative in a fundamental challenge to the intellectual foundations of AI, for it integrally synthesizes major issues in methodologies within LLMs and possesses an iconic a priori rationalist perspective. We examine this famous critic from both the perspective in pre-existing literature of linguistics and psychology as well as a research based on an experiment inquiring the capacity of learning both possible and impossible languages among LLMs. We constructed a set of syntactically impossible languages by applying certain transformations to English. These include reversing whole sentences, and adding negation based on word-count parity. Two rounds of controlled experiments were each conducted on GPT-2 small models and long short-term memory (LSTM) models. Statistical analysis (Welch's t-test) shows GPT2 small models underperform in learning all of the impossible languages compared to their performance on the possible language (p<.001). On the other hand, LSTM models' performance tallies with Chomsky's argument, suggesting the irreplaceable role of the evolution of transformer architecture. Based on theoretical analysis and empirical findings, we propose a new vision within Chomsky's theory towards LLMs, and a shift of theoretical paradigm outside Chomsky, from his "rationalist-romantics" paradigm to functionalism and empiricism in LLMs research.
Abstract (translated)
在乔姆斯基（Noam Chomsky）的尖锐批评文章《CHATGPT 的虚假承诺》中，大型语言模型（LLMs）被描述为仅仅是模式预测器，它们不像人类那样通过内在因果和自我纠正机制来学习语言。因此，这些模型无法区分不可能存在的语言。这篇评论代表了对人工智能基础理论的一个根本性挑战，因为它综合了LLM方法论中的主要问题，并具有典型的先验理性主义视角。 我们从现有的语言学和心理学文献以及基于实验的研究两方面审视这一著名的批评。该实验研究调查了大型语言模型（LLMs）学习可能与不可能的语言的能力。我们通过应用特定的变换到英语上，构建了一组语法上不可能的语言。这些变换包括反向整个句子，并根据单词数量的奇偶性添加否定。我们在GPT-2小型模型和长短期记忆（LSTM）模型上分别进行了两轮受控实验。统计分析（Welch t检验）显示，在学习所有不可能语言方面，GPT-2小型模型的表现明显不如在可能的语言上的表现（p<.001）。另一方面，LSTM模型的表现支持了乔姆斯基的论点，表明变换架构演进的独特作用。 基于理论分析和实证发现，我们提出了乔姆斯基理论中关于LLMs的新视角，并建议从他的“理性主义浪漫”范式转向功能主义和经验主义在LLMs研究中的新理论框架。
URL
https://arxiv.org/abs/2602.08437
