Abstract
Text preprocessing is a fundamental component of Natural Language Processing, involving techniques such as stopword removal, stemming, and lemmatization to prepare text as input for further processing and analysis. Despite the context-dependent nature of the above techniques, traditional methods usually ignore contextual information. In this paper, we investigate the idea of using Large Language Models (LLMs) to perform various preprocessing tasks, due to their ability to take context into account without requiring extensive language-specific annotated resources. Through a comprehensive evaluation on web-sourced data, we compare LLM-based preprocessing (specifically stopword removal, lemmatization and stemming) to traditional algorithms across multiple text classification tasks in six European languages. Our analysis indicates that LLMs are capable of replicating traditional stopword removal, lemmatization, and stemming methods with accuracies reaching 97%, 82%, and 74%, respectively. Additionally, we show that ML algorithms trained on texts preprocessed by LLMs achieve an improvement of up to 6% with respect to the $F_1$ measure compared to traditional techniques. Our code, prompts, and results are publicly available at this https URL.
Abstract (translated)
文本预处理是自然语言处理中的一个基本组成部分，涉及诸如停用词去除、词干提取和词形还原等技术，以准备作为进一步处理和分析输入的文本。尽管上述技术具有上下文依赖性，传统方法通常会忽略上下文信息。在本文中，我们探讨了使用大型语言模型（LLM）执行各种预处理任务的想法，因为它们能够在不需大量特定语言注释资源的情况下考虑到上下文。通过在六种欧洲语言的多个文本分类任务上的网络数据进行全面评估，我们将基于LLM的预处理（特别是停用词去除、词形还原和词干提取）与传统算法进行了比较。我们的分析表明，LLMs能够以高达97%、82%和74%的准确率复制传统的停用词去除、词形还原和词干提取方法。此外，我们还展示了在使用LLM预处理文本上进行训练的机器学习算法，在$F_1$度量方面相比传统技术提高了最多6%。 我们的代码、提示以及结果可在以下网址公开获取：[此链接](https://example.com)（请注意实际发表时请替换为正确的URL）。
URL
https://arxiv.org/abs/2510.11482
