Abstract
In the past few years, channel-wise and spatial-wise attention blocks have been widely adopted as supplementary modules in deep neural networks, enhancing network representational abilities while introducing low complexity. Most attention modules follow a squeeze-and-excitation paradigm. However, to design such attention modules, requires a substantial amount of experiments and computational resources. Neural Architecture Search (NAS), meanwhile, is able to automate the design of neural networks and spares the numerous experiments required for an optimal architecture. This motivates us to design a search architecture that can automatically find near-optimal attention modules through NAS. We propose SASE, a Searching Architecture for Squeeze and Excitation operations, to form a plug-and-play attention block by searching within certain search space. The search space is separated into 4 different sets, each corresponds to the squeeze or excitation operation along the channel or spatial dimension. Additionally, the search sets include not only existing attention blocks but also other operations that have not been utilized in attention mechanisms before. To the best of our knowledge, SASE is the first attempt to subdivide the attention search space and search for architectures beyond currently known attention modules. The searched attention module is tested with extensive experiments across a range of visual tasks. Experimental results indicate that visual backbone networks (ResNet-50/101) using the SASE attention module achieved the best performance compared to those using the current state-of-the-art attention modules. Codes are included in the supplementary material, and they will be made public later.
Abstract (translated)
在过去几年里，通道注意力模块和空间注意力模块已被广泛用作深度神经网络中的补充模块，在引入较低复杂度的同时增强了网络的表示能力。大多数注意力模块遵循挤压与激励（squeeze-and-excitation）范式。然而，设计此类注意力模块需要大量的实验和计算资源。同时，神经架构搜索（Neural Architecture Search, NAS）能够自动设计神经网络，并省去了寻找最优架构所需的大量实验。这促使我们设计一种通过NAS自动找到接近最优的注意力模块的搜索架构。为此，我们提出了SASE（Searching Architecture for Squeeze and Excitation operations），它通过在特定的搜索空间内进行搜索形成一个即插即用的注意力块。搜索空间被分为4个不同的集合，每个集合对应于通道或空间维度上的挤压或激励操作。此外，搜索集不仅包括现有的注意力模块，还包括之前未用于注意力机制中的其他操作。据我们所知，SASE是首次尝试细分注意力搜索空间并搜索超出当前已知注意力模块的架构。通过广泛的实验，在一系列视觉任务上测试了搜寻到的注意力模块。实验结果表明，使用SASE注意力模块的视觉骨干网络（ResNet-50/101）相较于那些使用现有最先进注意力模块的网络取得了最佳性能。代码包括在补充材料中，并将在稍后公开。
URL
https://arxiv.org/abs/2411.08333
