Abstract
Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron's full functional behavior can be easily visualized as an image, thus revealing how the network's output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning.
Abstract (translated)
现代人工智能领域的许多兴奋之处在于，扩大现有系统的规模会导致性能提升。然而，更好的性能是否必然意味着内部表示也更好呢？虽然代表乐观主义者认为答案是肯定的，但本文对此观点提出了挑战。我们比较了通过开放式演化过程进化的神经网络与使用传统随机梯度下降（SGD）训练的神经网络在生成单个图像这一简单任务上的表现。这种最小设置提供了一个独特的优势：可以轻松地将每个隐藏神经元的完整功能行为可视化为一个图像，从而揭示出网络输出行为是如何通过一个个神经元构建起来的。 结果令人震惊：尽管两种类型的网络都产生了相同的行为输出，但它们的内部表示却大相径庭。SGD训练的网络表现出一种我们称之为“碎片化纠缠表示”（FER）的形式。有趣的是，演化出来的网络在很大程度上缺乏这种“碎片化纠缠表示”，甚至接近于统一因子表示（UFR）。在大规模模型中，“碎片化纠缠表示”可能会削弱核心功能能力，如泛化、创造力和持续学习等。 因此，理解和缓解“碎片化纠缠表示”的问题对于未来代表学习的发展可能至关重要。
URL
https://arxiv.org/abs/2505.11581
