Abstract
Domain-specific embedding models have shown promise for applications that require specialized semantic understanding, such as coding agents and financial retrieval systems, often achieving higher performance gains than general models. However, state-of-the-art embedding models are typically based on LLMs, which contain billions of parameters, making deployment challenging in resource-constrained environments. Model compression through pruning offers a promising solution, but existing pruning methods treat all parameters uniformly, failing to distinguish between general semantic representations and domain-specific patterns, leading to suboptimal pruning decisions. Thus, we propose GAPrune, a pruning framework that addresses this challenge by considering both domain importance and preserving general linguistic foundation. Our method uses Fisher Information to measure importance and general-domain gradient alignment to assess parameter behavior, then combines these signals using our Domain Alignment Importance (DAI) scoring. Lower DAI scores indicate that the parameter is either less important for the domain task or creates conflicts between domain and general objectives. Experiments on two domain benchmarks, FinMTEB and ChemTEB, show that GAPrune maintains performance within 2.5% of dense models in one-shot pruning at 50% sparsity, while outperforming all baselines. With retraining in 100 steps, GAPrune achieves +4.51% improvement on FinMTEB and +1.73% on ChemTEB, demonstrating that our pruning strategy not only preserves but enhances domain-specific capabilities. Our findings demonstrate that principled pruning strategies can achieve model compression and enhanced domain specialization, providing the research community with a new approach for development.
Abstract (translated)
领域特定嵌入模型在需要专门语义理解的应用程序中表现出巨大潜力，例如代码代理和金融检索系统，在这些应用中通常能比通用模型实现更高的性能提升。然而，最先进的嵌入模型往往基于大规模语言模型（LLMs），这类模型包含数十亿参数，在资源受限的环境中部署十分具有挑战性。通过剪枝进行模型压缩提供了一种有前景的解决方案，但现有的剪枝方法对所有参数采取同等对待的方式，未能区分一般语义表示和领域特定模式之间的差异，导致了次优的剪枝决策。因此，我们提出了GAPrune框架，该框架旨在同时考虑领域重要性和保留通用语言基础来应对这一挑战。 我们的方法使用Fisher信息度量重要性，并通过领域-通用梯度对齐评估参数行为，然后结合这些信号利用我们的域对齐重要性（DAI）评分。较低的DAI分数意味着该参数要么在领域任务中不那么重要，要么与领域和通用目标之间存在冲突。 实验结果表明，在FinMTEB和ChemTEB两个领域的基准测试上，GAPrune通过一次性50%稀疏度剪枝，性能保持接近稠密模型的97.5%，并且超过了所有基线。经过100步重新训练后，GAPrune在FinMTEB上的表现提高了4.51%，在ChemTEB上提高了1.73%，这证明了我们的剪枝策略不仅能保留领域特定能力，还能进一步增强这些能力。 我们的研究结果表明，基于原则的剪枝策略能够实现模型压缩和领域的专业化提升，并为研发社区提供了一种新的发展方向。
URL
https://arxiv.org/abs/2509.10844
