Abstract
Scientific multi-label text classification suffers from extreme class imbalance, where specialized terminology exhibits severe power-law distributions that challenge standard classification approaches. Existing scientific corpora lack comprehensive controlled vocabularies, focusing instead on broad categories and limiting systematic study of extreme imbalance. We introduce AstroConcepts, a corpus of English abstracts from 21,702 published astrophysics papers, labeled with 2,367 concepts from the Unified Astronomy Thesaurus. The corpus exhibits severe label imbalance, with 76% of concepts having fewer than 50 training examples. By releasing this resource, we enable systematic study of extreme class imbalance in scientific domains and establish strong baselines across traditional, neural, and vocabulary-constrained LLM methods. Our evaluation reveals three key patterns that provide new insights into scientific text classification. First, vocabulary-constrained LLMs achieve competitive performance relative to domain-adapted models in astrophysics classification, suggesting a potential for parameter-efficient approaches. Second, domain adaptation yields relatively larger improvements for rare, specialized terminology, although absolute performance remains limited across all methods. Third, we propose frequency-stratified evaluation to reveal performance patterns that are hidden by aggregate scores, thereby making robustness assessment central to scientific multi-label evaluation. These results offer actionable insights for scientific NLP and establish benchmarks for research on extreme imbalance.
Abstract (translated)
科学多标签文本分类面临严重的类别不平衡问题，其中专业术语呈现极端的幂律分布，这对标准分类方法构成挑战。现有的科学语料库缺乏全面的受控词汇表，转而聚焦于宽泛类别，限制了针对极端不平衡的系统性研究。我们推出了AstroConcepts语料库，该库包含21,702篇已发表天体物理学论文的英文摘要，并采用《统一天文学术语表》中的2,367个概念进行标注。该语料库表现出严重的标签不平衡，76%的概念仅有不到50个训练样本。通过发布这一资源，我们能够对科学领域的极端类别不平衡进行系统性研究，并为传统方法、神经网络方法以及词汇约束大语言模型方法建立强基准。我们的评估揭示了三个关键模式，为科学文本分类提供了新的见解。首先，在天体物理学分类任务中，词汇约束大语言模型相对于领域适应模型能取得具有竞争力的性能，这表明参数高效方法具有潜力。其次，领域适应对罕见专业术语的提升相对更大，但所有方法的绝对性能仍有限。第三，我们提出频率分层评估方法，以揭示被聚合分数掩盖的性能模式，从而将鲁棒性评估置于科学多标签评估的核心。这些结果为科学自然语言处理提供了可操作的见解，并为极端不平衡研究建立了基准。
URL
https://arxiv.org/abs/2604.02156
