Abstract
Balancing accuracy and latency on high-resolution images is a critical challenge for lightweight models, particularly for Transformer-based architectures that often suffer from excessive latency. To address this issue, we introduce \textbf{ReGLA}, a series of lightweight hybrid networks, which integrates efficient convolutions for local feature extraction with ReLU-based gated linear attention for global modeling. The design incorporates three key innovations: the Efficient Large Receptive Field (ELRF) module for enhancing convolutional efficiency while preserving a large receptive field; the ReLU Gated Modulated Attention (RGMA) module for maintaining linear complexity while enhancing local feature representation; and a multi-teacher distillation strategy to boost performance on downstream tasks. Extensive experiments validate the superiority of ReGLA; particularly the ReGLA-M achieves \textbf{80.85\%} Top-1 accuracy on ImageNet-1K at $224px$, with only \textbf{4.98 ms} latency at $512px$. Furthermore, ReGLA outperforms similarly scaled iFormer models in downstream tasks, achieving gains of \textbf{3.1\%} AP on COCO object detection and \textbf{3.6\%} mIoU on ADE20K semantic segmentation, establishing it as a state-of-the-art solution for high-resolution visual applications.
Abstract (translated)
在高分辨率图像上平衡准确性和延迟对于轻量级模型来说是一个关键挑战，特别是基于Transformer的架构经常会出现过高的延迟问题。为了解决这个问题，我们引入了**ReGLA（ReLU-Gated Lightweight Architecture）**，一系列轻量级混合网络。这些网络结合了高效的卷积用于局部特征提取和基于ReLU的门控线性注意机制用于全局建模。 设计中的三个关键创新点包括： 1. **Efficient Large Receptive Field (ELRF) 模块**：通过保持大感受野的同时提升卷积效率。 2. **ReLU Gated Modulated Attention (RGMA) 模块**：在维持线性复杂度的同时增强局部特征表示能力。 3. **多教师蒸馏策略**：提高下游任务的性能。 广泛的实验验证了ReGLA的优势。特别地，ReGLA-M模型在ImageNet-1K数据集上实现了80.85%的Top-1准确率（输入尺寸为224px），并且仅需4.98毫秒延迟即可处理分辨率为512px的图像。此外，在下游任务中，与同样规模的iFormer模型相比，ReGLA在COCO物体检测上的AP指标提高了3.1%，ADE20K语义分割上的mIoU指标提升了3.6%。这些结果确立了ReGLA作为高分辨率视觉应用领域内的一种领先解决方案的地位。
URL
https://arxiv.org/abs/2602.05262
