Abstract
Attention mechanisms have become integral in AI, significantly enhancing model performance and scalability by drawing inspiration from human cognition. Concurrently, the Attention Schema Theory (AST) in cognitive science posits that individuals manage their attention by creating a model of the attention itself, effectively allocating cognitive resources. Inspired by AST, we introduce ASAC (Attention Schema-based Attention Control), which integrates the attention schema concept into artificial neural networks. Our initial experiments focused on embedding the ASAC module within transformer architectures. This module employs a Vector-Quantized Variational AutoEncoder (VQVAE) as both an attention abstractor and controller, facilitating precise attention management. By explicitly modeling attention allocation, our approach aims to enhance system efficiency. We demonstrate ASAC's effectiveness in both the vision and NLP domains, highlighting its ability to improve classification accuracy and expedite the learning process. Our experiments with vision transformers across various datasets illustrate that the attention controller not only boosts classification accuracy but also accelerates learning. Furthermore, we have demonstrated the model's robustness and generalization capabilities across noisy and out-of-distribution datasets. In addition, we have showcased improved performance in multi-task settings. Quick experiments reveal that the attention schema-based module enhances resilience to adversarial attacks, optimizes attention to improve learning efficiency, and facilitates effective transfer learning and learning from fewer examples. These promising results establish a connection between cognitive science and machine learning, shedding light on the efficient utilization of attention mechanisms in AI systems.
Abstract (translated)
注意力机制在人工智能领域中变得至关重要，通过借鉴人类认知的灵感显著提高了模型的表现和可扩展性。同时，在认知科学中提出的注意模式理论（AST）认为个体通过创建注意力的模型来管理自己的注意力，从而有效地分配认知资源。受AST启发，我们引入了基于注意模式的注意力控制（ASAC），将注意模式概念融入人工神经网络之中。我们的初步实验重点在于在变压器架构内嵌入ASAC模块。该模块采用向量量化变分自动编码器（VQVAE）作为注意力抽象器和控制器，以便实现精确的注意力管理。通过显式地建模注意力分配，我们旨在提高系统的效率。 我们在视觉和自然语言处理（NLP）领域展示了ASAC的有效性，突出了其提高分类准确性和加快学习过程的能力。在不同数据集上的视觉变压器实验表明，注意控制器不仅提高了分类准确性，还加速了学习速度。此外，我们已经证明该模型在噪声和分布外数据集上具有鲁棒性和泛化能力。另外，在多任务设置中也展示了改进的性能。 快速实验揭示了基于注意力模式的模块增强了对对抗攻击的抵抗力、优化了注意力以提高学习效率，并促进了有效的迁移学习以及从少量样本中的学习。这些有前景的结果建立了认知科学和机器学习之间的联系，阐明了在AI系统中有效利用注意机制的方法。
URL
https://arxiv.org/abs/2509.16058
