Abstract
Generalization, the ability to perform well beyond the training context, is a hallmark of biological and artificial intelligence, yet anticipating unseen failures remains a central challenge. Conventional approaches often take a ``bottom-up'' mechanistic route by reverse-engineering interpretable features or circuits to build explanatory models. While insightful, these methods often struggle to provide the high-level, predictive signals for anticipating failure in real-world deployment. Here, we propose using a ``top-down'' approach to studying generalization failures inspired by medical biomarkers: identifying system-level measurements that serve as robust indicators of a model's future performance. Rather than mapping out detailed internal mechanisms, we systematically design and test network markers to probe structure, function links, identify prognostic indicators, and validate predictions in real-world settings. In image classification, we find that task-relevant geometric properties of in-distribution (ID) object manifolds consistently forecast poor out-of-distribution (OOD) generalization. In particular, reductions in two geometric measures, effective manifold dimensionality and utility, predict weaker OOD performance across diverse architectures, optimizers, and datasets. We apply this finding to transfer learning with ImageNet-pretrained models. We consistently find that the same geometric patterns predict OOD transfer performance more reliably than ID accuracy. This work demonstrates that representational geometry can expose hidden vulnerabilities, offering more robust guidance for model selection and AI interpretability.
Abstract (translated)
泛化能力，即在训练场景之外仍能表现良好，是生物智能和人工智能的一个显著特征。然而，预测未见的失败仍然是一个核心挑战。传统方法通常采用“自下而上”的机制主义路径，通过逆向工程可解释性特征或电路来构建解释模型。尽管这些方法颇具洞察力，但它们在提供预测性的高阶信号以预见实际部署中的失败方面往往表现不佳。我们提出了一种受医学生物标志物启发的“自上而下”方法，用于研究泛化失败：识别系统级别的测量指标，作为模型未来性能的稳健指示器。这种方法不关注详细的内部机制，而是系统地设计和测试网络标记以探测结构与功能之间的联系、识别预后指标，并在实际场景中验证预测。 在图像分类任务中，我们发现分布内（ID）对象流形的相关几何属性会一致地预测出分布外（OOD）泛化的表现不佳。具体而言，两种几何度量的减少——有效流形维度和效用——可以跨多种架构、优化器及数据集预示较差的OOD性能。我们将这一发现应用于基于ImageNet预训练模型的任务迁移学习中。我们一致地发现，相同的几何模式比分布内的准确性更可靠地预测OOD转移表现。 这项工作展示了表示几何学如何揭示隐藏的脆弱性，并为模型选择和AI可解释性提供了更加稳健的指导。
URL
https://arxiv.org/abs/2603.01879
