Abstract
Deformable Attention Transformers (DAT) have shown remarkable performance in computer vision tasks by adaptively focusing on informative image regions. However, their data-dependent sampling mechanism introduces irregular memory access patterns, posing significant challenges for efficient hardware deployment. Existing acceleration methods either incur high hardware overhead or compromise model accuracy. To address these issues, this paper proposes a hardware-friendly optimization framework for DAT. First, a neural architecture search (NAS)-based method with a new slicing strategy is proposed to automatically divide the input feature into uniform patches during the inference process, avoiding memory conflicts without modifying model architecture. The method explores the optimal slice configuration by jointly optimizing hardware cost and inference accuracy. Secondly, an FPGA-based verification system is designed to test the performance of this framework on edge-side hardware. Algorithm experiments on the ImageNet-1K dataset demonstrate that our hardware-friendly framework can maintain have only 0.2% accuracy drop compared to the baseline DAT. Hardware experiments on Xilinx FPGA show the proposed method reduces DRAM access times to 18% compared with existing DAT acceleration methods.
Abstract (translated)
变形注意力变换器（DAT）在计算机视觉任务中表现出色，能够通过自适应聚焦于图像的显著区域来提升性能。然而，它们的数据依赖采样机制引入了不规则的记忆访问模式，这对高效的硬件部署提出了重大挑战。现有的加速方法要么导致较高的硬件开销，要么牺牲模型精度。为解决这些问题，本文提出了一种针对DAT友好的硬件优化框架。 首先，该研究提出了一种基于神经架构搜索（NAS）的新切片策略的方法，在推断过程中自动将输入特征划分为均匀的补丁，从而避免了内存冲突，且无需修改模型结构。此方法通过同时优化硬件成本和推理精度来探索最佳切片配置。其次，设计了一个基于FPGA的验证系统，用于测试该框架在边缘侧硬件上的性能。 在ImageNet-1K数据集上的算法实验表明，与基准DAT相比，我们的硬件友好型框架仅导致0.2%的准确率下降。而在Xilinx FPGA上的硬件实验显示，所提出的方法将DRAM访问时间减少了18%，相比于现有的DAT加速方法。
URL
https://arxiv.org/abs/2507.11549
