Abstract
SemEval-2024 Task 8 provides a challenge to detect human-written and machine-generated text. There are 3 subtasks for different detection scenarios. This paper proposes a system that mainly deals with Subtask B. It aims to detect if given full text is written by human or is generated by a specific Large Language Model (LLM), which is actually a multi-class text classification task. Our team AISPACE conducted a systematic study of fine-tuning transformer-based models, including encoderonly, decoder-only and encoder-decoder models. We compared their performance on this task and identified that encoder-only models performed exceptionally well. We also applied a weighted Cross Entropy loss function to address the issue of data imbalance of different class samples. Additionally, we employed softvoting strategy over multi-models ensemble to enhance the reliability of our predictions. Our system ranked top 1 in Subtask B, which sets a state-of-the-art benchmark for this new challenge.
Abstract (translated)
SemEval-2024 任务 8 提出了一种检测人类撰写的和机器生成的文本的挑战。有三个子任务，用于不同的检测场景。本文提出了一种主要针对子任务 B 的系统。其旨在检测给定的完整文本是由人类撰写的，还是由特定的大型语言模型（LLM）生成的，实际上是一个多类文本分类任务。我们的团队 AISPACE 对基于变换器的模型进行了系统性的研究，包括仅编码器、仅解码器模型和编码器-解码器模型。我们比较了它们在这个任务上的表现，并发现仅编码器模型的表现尤为出色。我们还采用了一种加权交叉熵损失函数来解决不同类样本数据不平衡的问题。此外，我们还使用软投票策略来增强我们对预测的可靠性。我们的系统在子任务 B 上排名 top 1，为这个新挑战设定了最先进的基准。
URL
https://arxiv.org/abs/2404.00950
