Abstract
Representation-based Siamese networks have risen to popularity in lightweight text matching due to their low deployment and inference costs. While word-level attention mechanisms have been implemented within Siamese networks to improve performance, we propose Feature Attention (FA), a novel downstream block designed to enrich the modeling of dependencies among embedding features. Employing "squeeze-and-excitation" techniques, the FA block dynamically adjusts the emphasis on individual features, enabling the network to concentrate more on features that significantly contribute to the final classification. Building upon FA, we introduce a dynamic "selection" mechanism called Selective Feature Attention (SFA), which leverages a stacked BiGRU Inception structure. The SFA block facilitates multi-scale semantic extraction by traversing different stacked BiGRU layers, encouraging the network to selectively concentrate on semantic information and embedding features across varying levels of abstraction. Both the FA and SFA blocks offer a seamless integration capability with various Siamese networks, showcasing a plug-and-play characteristic. Experimental evaluations conducted across diverse text matching baselines and benchmarks underscore the indispensability of modeling feature attention and the superiority of the "selection" mechanism.
Abstract (translated)
以表示为基础的孪生网络因低部署和推理成本而在轻量文本匹配中备受欢迎。尽管在孪生网络中已经实现了词级关注机制以提高性能，但我们提出了特征关注（FA）这一新型的下游块，旨在丰富嵌入特征之间的建模。通过采用“收缩和激发”技术，FA块动态地调整对单个特征的强调，使得网络能更关注对最终分类具有重要影响的特征。基于FA，我们引入了一种动态选择机制，称为选择性特征关注（SFA），并利用堆叠BiGRU Inception结构。SFA块通过穿越不同的堆叠BiGRU层，促使网络在不同的抽象层次上集中关注语义信息和嵌入特征。FA和SFA块都具有与各种孪生网络的无缝集成能力，展示了可插拔和 Play 的特点。在多样文本匹配基准和挑战中进行实验评估，结果表明建模特征关注和选择机制至关重要，而“选择”机制的优越性得到了充分证实。
URL
https://arxiv.org/abs/2404.16776
