Abstract
This study enhances Jiang et al.'s compression-based classification algorithm by addressing its limitations in detecting semantic similarities between text documents. The proposed improvements focus on unigram extraction and optimized concatenation, eliminating reliance on entire document compression. By compressing extracted unigrams, the algorithm mitigates sliding window limitations inherent to gzip, improving compression efficiency and similarity detection. The optimized concatenation strategy replaces direct concatenation with the union of unigrams, reducing redundancy and enhancing the accuracy of Normalized Compression Distance (NCD) calculations. Experimental results across datasets of varying sizes and complexities demonstrate an average accuracy improvement of 5.73%, with gains of up to 11% on datasets containing longer documents. Notably, these improvements are more pronounced in datasets with high-label diversity and complex text structures. The methodology achieves these results while maintaining computational efficiency, making it suitable for resource-constrained environments. This study provides a robust, scalable solution for text classification, emphasizing lightweight preprocessing techniques to achieve efficient compression, which in turn enables more accurate classification.
Abstract (translated)
这项研究通过解决江等人基于压缩的分类算法在检测文本文档语义相似性方面的局限性，对其进行了改进。所提出的改进主要集中在单字符词（unigram）提取和优化拼接策略上，从而消除了对整个文档进行压缩的依赖。通过对提取出的单字符词进行压缩，该算法缓解了gzip滑动窗口固有的限制，提高了压缩效率并增强了相似性检测能力。优化后的拼接策略通过使用单字符词的并集取代直接拼接来减少冗余，并提升了标准化压缩距离（NCD）计算的准确性。 实验结果显示，在不同大小和复杂度的数据集中，平均准确率提高了5.73%，在包含较长文档的数据集中，这一改进幅度最高可达11%。值得注意的是，这种改进在标签多样性高且文本结构复杂的数据集上表现得更为显著。该方法在保持计算效率的同时实现了这些成果，使其适用于资源受限的环境。 这项研究提供了一种稳健、可扩展的解决方案来实现高效的文本分类，强调了通过轻量级预处理技术获得高效压缩的重要性，从而提高了分类准确性。
URL
https://arxiv.org/abs/2502.14444
