Abstract
Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at this https URL.
Abstract (translated)
前沿语言模型的质量越来越依赖于我们组织网络规模文本语料库的能力进行训练。当前主导的工具在速度和灵活性之间做出权衡：例如，FastText 等词汇分类器速度快但仅限于生成分类输出分数；而变换器文本嵌入模型则可以灵活支持多种工作流程（如聚类、分类及检索），但由于计算成本高昂，并不具备高效性。我们推出了 Luxical 库，该库用于高速“词汇密集”型文本嵌入，旨在结合这两种方法的最佳特性来实现大规模网络文本组织。 Luxical 结合了稀疏的 TF-IDF 特征、一个小的 ReLU 网络和知识蒸馏训练程序，以较低的成本近似大型变换器模型。在这份技术报告中，我们将描述 Luxical 的架构及其培训目标，并在两个不同的应用中评估具体的 Luxical 模型：定向网络爬取文档检索测试及基于文本分类的语言模型数据管理任务。 在这两项任务中，相对于不同大小的神经基线模型，Luxical 能提供从 3 倍到 100 倍的速度提升，并且在数据管理工作流程中接近 FastText 模型推理速度。根据这些评估结果，在大规模文本组织方面，测试过的 Luxical 模型展示出有利的计算/质量权衡，与神经基线模型的质量相当。 Luxical 可以作为开源软件通过此链接访问（请将“this https URL”替换为实际链接地址）。
URL
https://arxiv.org/abs/2512.09015
