Abstract
The evolution of Vision Transformers has led to their widespread adaptation to different domains. Despite large-scale success, there remain significant challenges including their reliance on extensive computational and memory resources for pre-training on huge datasets as well as difficulties in task-specific transfer learning. These limitations coupled with energy inefficiencies mainly arise due to the computation-intensive self-attention mechanism. To address these issues, we propose a novel Super-Pixel Based Patch Pooling (SPPP) technique that generates context-aware, semantically rich, patch embeddings to effectively reduce the architectural complexity and improve efficiency. Additionally, we introduce the Light Latent Attention (LLA) module in our pipeline by integrating latent tokens into the attention mechanism allowing cross-attention operations to significantly reduce the time and space complexity of the attention module. By leveraging the data-intuitive patch embeddings coupled with dynamic positional encodings, our approach adaptively modulates the cross-attention process to focus on informative regions while maintaining the global semantic structure. This targeted attention improves training efficiency and accelerates convergence. Notably, the SPPP module is lightweight and can be easily integrated into existing transformer architectures. Extensive experiments demonstrate that our proposed architecture provides significant improvements in terms of computational efficiency while achieving comparable results with the state-of-the-art approaches, highlighting its potential for energy-efficient transformers suitable for edge deployment. (The code is available on our GitHub repository: this https URL).
Abstract (translated)
视觉变换器（Vision Transformers）的演进已经促使其在不同领域的广泛应用。尽管取得了大规模的成功，但它们仍然面临着一些重要挑战，包括依赖于大量计算和内存资源进行预训练的巨大数据集，以及任务特定迁移学习的困难。这些限制主要源于计算密集型的自我注意机制，并且伴随着能源效率低下的问题。 为了解决这些问题，我们提出了一种基于超级像素的补丁池化（Super-Pixel Based Patch Pooling, SPPP）技术，该技术生成了上下文感知、语义丰富的补丁嵌入，从而有效地减少架构复杂性并提高效率。此外，我们在我们的管道中引入了一个轻量级潜注意（Light Latent Attention, LLA）模块，通过将潜在令牌整合到注意力机制中，允许跨注意力操作显著降低时间与空间的复杂度。 通过利用直观的数据补丁嵌入和动态位置编码，我们所提出的方法能够自适应地调节跨注意力过程，聚焦于信息丰富的区域的同时保持全局语义结构。这种目标化注意提高了训练效率并加速了收敛速度。值得注意的是，SPPP模块轻量级且易于集成到现有的变换器架构中。 广泛的实验表明，我们的提出的架构在计算效率方面提供了显著的改进，并能与最先进的方法实现相媲美的结果，突显了其用于边缘部署的能量高效型变压器的潜力。代码可在我们的GitHub存储库（此链接）上获得。
URL
https://arxiv.org/abs/2506.18791
