Abstract
Embedding fusion has emerged as an effective approach for enhancing performance across various NLP tasks. However, systematic guidelines for selecting optimal layers and developing effective fusion strategies for the integration of LLMs remain underexplored. In this study, we propose a layer-aware embedding selection method and investigate how to quantitatively evaluate different layers to identify the most important ones for downstream NLP tasks, showing that the critical layers vary depending on the dataset. We also explore how combining embeddings from multiple LLMs, without requiring model fine-tuning, can improve performance. Experiments on four English text classification datasets (SST-2, MR, R8, and R52) demonstrate that different layers in LLMs exhibit varying degrees of representational strength for classification, and that combining embeddings from different models can enhance performance if the models exhibit complementary characteristics. Additionally, we discuss resources overhead (memory and inference time) to provide a balanced perspective on the real world feasibility of embedding fusion. Future work will explore multilingual and domain specific datasets, as well as techniques for automating layer selection, to improve both performance and scalability.
Abstract (translated)
嵌入融合已成为增强各种自然语言处理（NLP）任务性能的有效方法。然而，关于如何选择最佳层次以及为大规模语言模型（LLMs）的集成开发有效的融合策略，系统性指导原则尚未得到充分探索。在这项研究中，我们提出了一种基于层的嵌入选择方法，并探讨了如何定量评估不同层次以识别对下游NLP任务最重要的那些层次，研究表明关键层次会因数据集的不同而变化。此外，我们还探索了在无需微调模型的情况下，通过结合来自多个LLMs的嵌入来改进性能的方法。我们在四个英文文本分类数据集（SST-2、MR、R8和R52）上的实验表明，LLMs中的不同层次对于分类任务表现出不同的表征能力，并且如果这些模型具有互补特征，则结合来自不同模型的嵌入可以提高性能。我们还讨论了资源开销（内存和推理时间），以提供关于嵌入融合在现实世界可行性方面的平衡视角。未来的研究将探索多语言及特定领域的数据集，以及自动选择层次的技术，从而进一步提升性能和可扩展性。
URL
https://arxiv.org/abs/2504.05764
