Abstract
This case study investigates the task of job classification in a real-world setting, where the goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position. We explore multiple approaches to text classification, including supervised approaches such as traditional models like Support Vector Machines (SVMs) and state-of-the-art deep learning methods such as DeBERTa. We compare them with Large Language Models (LLMs) used in both few-shot and zero-shot classification settings. To accomplish this task, we employ prompt engineering, a technique that involves designing prompts to guide the LLMs towards the desired output. Specifically, we evaluate the performance of two commercially available state-of-the-art GPT-3.5-based language models, text-davinci-003 and gpt-3.5-turbo. We also conduct a detailed analysis of the impact of different aspects of prompt engineering on the model's performance. Our results show that, with a well-designed prompt, a zero-shot gpt-3.5-turbo classifier outperforms all other models, achieving a 6% increase in Precision@95% Recall compared to the best supervised approach. Furthermore, we observe that the wording of the prompt is a critical factor in eliciting the appropriate "reasoning" in the model, and that seemingly minor aspects of the prompt significantly affect the model's performance.
Abstract (translated)
本案例研究探讨了在现实世界 setting 中进行 job classification 的任务,该任务的目标是确定对于 graduate 或 entry-level 职位来说,英语职位发布是否合适。我们探索了多种文本分类方法,包括监督方法,如传统的模型,如支持向量机 (SVMs) 和先进的深度学习方法,如 DeBERTa。我们将这些方法和 small 和 zero-shot 分类设置中的大型语言模型(LLM)进行比较。为了实现这一任务,我们采用了prompt engineering,这是一种技术,涉及设计prompts,以指导LLM 向预期输出方向移动。具体来说,我们评估了两种商业上最先进的基于 GPT-3.5 的语言模型,text-davinci-003 和 gpt-3.5-Turbo,以及prompt engineering 不同方面的对模型性能的影响。我们的结果表明,通过设计良好的prompt,零-shot gpt-3.5-Turbo分类器在所有其他模型上都表现更好,相较于最好的监督方法,提高了6%的Precision@95%Recall。此外,我们观察到prompt 的词法是提取模型中适当的“推理”的关键因素,并且似乎prompt 的一些方面对模型性能产生了显著影响。
URL
https://arxiv.org/abs/2303.07142
