Abstract
Hierarchical Text Classification (HTC) aims to assign texts to structured label hierarchies; however, it faces challenges due to data scarcity and model complexity. This study explores the feasibility of using black box Large Language Models (LLMs) accessed via APIs for HTC, as an alternative to traditional machine learning methods that require extensive labeled data and computational resources. We evaluate three prompting strategies -- Direct Leaf Label Prediction (DL), Direct Hierarchical Label Prediction (DH), and Top-down Multi-step Hierarchical Label Prediction (TMH) -- in both zero-shot and few-shot settings, comparing the accuracy and cost-effectiveness of these strategies. Experiments on two datasets show that a few-shot setting consistently improves classification accuracy compared to a zero-shot setting. While a traditional machine learning model achieves high accuracy on a dataset with a shallow hierarchy, LLMs, especially DH strategy, tend to outperform the machine learning model on a dataset with a deeper hierarchy. API costs increase significantly due to the higher input tokens required for deeper label hierarchies on DH strategy. These results emphasize the trade-off between accuracy improvement and the computational cost of prompt strategy. These findings highlight the potential of black box LLMs for HTC while underscoring the need to carefully select a prompt strategy to balance performance and cost.
Abstract (translated)
层次文本分类（HTC）旨在将文本分配到结构化的标签层级体系中；然而，由于数据稀缺和模型复杂性的问题，HTC面临着挑战。这项研究探讨了使用通过API访问的黑盒大规模语言模型（LLMs）进行HTC的可行性，作为传统机器学习方法的一种替代方案，后者需要大量的标注数据和计算资源。我们评估了三种提示策略——直接叶标签预测（DL）、直接层级标签预测（DH）以及自顶向下的多步层级标签预测（TMH），在零样本和少量样本设置下进行性能对比，比较这些策略的准确性和成本效益。 实验使用两个数据集表明，在少数样本设置中，分类准确性相比零样本设置始终有所提高。传统机器学习模型在一个浅层层级的数据集中表现出了高精度；然而，LLMs（尤其是DH策略）在深层层级结构的数据集中往往优于传统的机器学习模型。API费用由于更深层次的标签体系所需的输入令牌更多而在DH策略下显著增加。这些结果强调了准确性改进与提示策略计算成本之间的权衡。 这项研究突显了黑盒大规模语言模型在HTC中的潜力，同时也强调了选择合适的提示策略以平衡性能和成本的重要性。
URL
https://arxiv.org/abs/2508.04219
