Abstract
As sketch research has collectively matured over time, its adaptation for at-mass commercialisation emerges on the immediate horizon. Despite an already mature research endeavour for photos, there is no research on the efficient inference specifically designed for sketch data. In this paper, we first demonstrate existing state-of-the-art efficient light-weight models designed for photos do not work on sketches. We then propose two sketch-specific components which work in a plug-n-play manner on any photo efficient network to adapt them to work on sketch data. We specifically chose fine-grained sketch-based image retrieval (FG-SBIR) as a demonstrator as the most recognised sketch problem with immediate commercial value. Technically speaking, we first propose a cross-modal knowledge distillation network to transfer existing photo efficient networks to be compatible with sketch, which brings down number of FLOPs and model parameters by 97.96% percent and 84.89% respectively. We then exploit the abstract trait of sketch to introduce a RL-based canvas selector that dynamically adjusts to the abstraction level which further cuts down number of FLOPs by two thirds. The end result is an overall reduction of 99.37% of FLOPs (from 40.18G to 0.254G) when compared with a full network, while retaining the accuracy (33.03% vs 32.77%) -- finally making an efficient network for the sparse sketch data that exhibit even fewer FLOPs than the best photo counterpart.
Abstract (translated)
随着时间的推移，草图研究作为一个整体已经成熟起来，并且它适应大规模商业化应用的前景即将来临。尽管针对照片的研究已相对成熟，但尚未有专门设计用于草图数据的有效推理方法的研究。在这篇论文中，我们首先展示了现有的最先进的高效轻量级模型在处理照片时表现良好，但在处理草图时却无法使用。接着，我们提出了两个专为草图定制的组件，这些组件可以在任何高效的图片网络上以即插即用的方式工作，从而使它们能够适应草图数据的应用需求。 具体而言，我们选择了细粒度基于草图的图像检索（Fine-Grained Sketch-Based Image Retrieval, FG-SBIR）作为演示案例，因为这是目前公认具有即时商业价值的草图问题。从技术上讲，我们首先提出了一种跨模态知识蒸馏网络，以将现有的高效照片模型转化为与草图兼容的形式，这使得计算量（FLOPs）和模型参数分别减少了97.96%和84.89%。然后，利用草图的抽象特性引入了一个基于强化学习的画布选择器，该选择器能够根据抽象程度动态调整，进一步将计算量减少了一半以上。 最终结果是在与完整网络相比时，FLOPs减少了99.37%，即从40.18G下降到了0.254G，同时保持了准确性（分别为33.03%和32.77%）。这使得对于稀疏的草图数据来说，这个高效的网络在计算量上甚至低于最佳的照片模型，并最终为处理这种类型的草图数据提供了一个高效的解决方案。
URL
https://arxiv.org/abs/2505.23763
