Abstract
This study explores strategies for efficiently classifying scientific full texts using both small, BERT-based models and local large language models like Llama-3.1 8B. We focus on developing methods for selecting subsets of input sentences to reduce input size while simultaneously enhancing classification performance. To this end, we compile a novel dataset consisting of full-text scientific papers from the field of invasion biology, specifically addressing the impacts of invasive species. These papers are aligned with publicly available impact assessments created by researchers for the International Union for Conservation of Nature (IUCN). Through extensive experimentation, we demonstrate that various sources like human evidence annotations, LLM-generated annotations or explainability scores can be used to train sentence selection models that improve the performance of both encoder- and decoder-based language models while optimizing efficiency through the reduction in input length, leading to improved results even if compared to models like ModernBERT that are able to handle the complete text as input. Additionally, we find that repeated sampling of shorter inputs proves to be a very effective strategy that, at a slightly increased cost, can further improve classification performance.
Abstract (translated)
这项研究探讨了利用基于BERT的小型模型和本地大型语言模型（如Llama-3.1 8B）高效分类科学全文的方法。我们专注于开发选择输入句子子集的策略，以减少输入大小同时提升分类性能。为此，我们编译了一个新颖的数据集，其中包括来自入侵生物学领域的完整文本科学论文，特别关注外来物种的影响。这些论文与国际自然保护联盟（IUCN）的研究人员为公众提供的影响评估相一致。 通过广泛的实验，我们展示了人类证据注释、大型语言模型生成的注释或可解释性评分等各种来源可以用于训练句子选择模型，从而在减少输入长度的同时提高基于编码器和解码器的语言模型的性能。即使与能够处理完整文本输入的现代BERT等模型相比，这种方法也能带来更好的结果。此外，我们发现重复采样较短的输入是一种非常有效的策略，在略微增加成本的情况下可以进一步提升分类性能。
URL
https://arxiv.org/abs/2502.06551
