Text_Classification
-
Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning
2025-10-11 13:16:28Bo Yuan, Yulin Chen, Yin ZhangAbstract
Parameter-efficient fine-tuning (PEFT) large language models (LLMs) have shown impressive performance in various downstream tasks. However, in many real-world scenarios, the collected training data inevitably contains noisy labels. To learn from noisy labels, most solutions select samples with small losses for model training. However, the selected samples, in turn, impact the loss computation in the next iteration. An inaccurate initial selection can create a vicious cycle, leading to suboptimal performance. To break this cycle, we propose Delora, a novel framework that decouples the sample selection from model training. For sample selection, Delora establishes a noisy label detector by introducing clean and noisy LoRA. Benefiting from the memory effect, the clean LoRA is encouraged to memorize clean data, while the noisy LoRA is constrained to memorize mislabeled data, which serves as a learnable threshold for selecting clean and noisy samples. For model training, Delora can use carefully selected samples to fine-tune language models seamlessly. Experimental results on synthetic and real-world noisy datasets demonstrate the effectiveness of Delora in noisy label detection and text classification.
Abstract (translated)
参数高效的微调（PEFT）大型语言模型（LLMs）在各种下游任务中表现出令人印象深刻的性能。然而，在许多现实场景中，收集的训练数据不可避免地包含噪声标签。为从噪声标签中学习，大多数解决方案选择具有小损失值的数据样本进行模型训练。然而，所选样本反过来会影响下一次迭代中的损失计算。初始选择不准确可能会产生恶性循环，导致次优性能。为了打破这个循环，我们提出了 Delora，这是一个将样本选择与模型训练解耦的新型框架。 对于样本选择，Delora 通过引入干净和噪声LoRA建立了噪声标签检测器。得益于记忆效应，干净的 LoRA 被鼓励记住清洁数据，而噪声的 LoRA 则被限制记住错误标记的数据，这作为可学习的阈值用于区分清洁与噪声样本的选择。 对于模型训练，Delora 可以使用精心挑选的样本无缝地微调语言模型。在合成和真实世界的噪声数据集上的实验结果证明了 Delora 在噪声标签检测和文本分类方面的有效性。
URL
https://arxiv.org/abs/2510.10208
PDF
-
One Sentence, Two Embeddings: Contrastive Learning of Explicit and Implicit Semantic Representations
2025-10-10 11:36:19Kohei Oda, Po-Min Chuang, Kiyoaki Shirai, Natthawut KertkeidkachornAbstract
Sentence embedding methods have made remarkable progress, yet they still struggle to capture the implicit semantics within sentences. This can be attributed to the inherent limitations of conventional sentence embedding methods that assign only a single vector per sentence. To overcome this limitation, we propose DualCSE, a sentence embedding method that assigns two embeddings to each sentence: one representing the explicit semantics and the other representing the implicit semantics. These embeddings coexist in the shared space, enabling the selection of the desired semantics for specific purposes such as information retrieval and text classification. Experimental results demonstrate that DualCSE can effectively encode both explicit and implicit meanings and improve the performance of the downstream task.
Abstract (translated)
句子嵌入方法已经取得了显著的进步，但仍难以捕捉句子中的隐含语义。这可以归因于传统句子嵌入方法固有的限制，即每句话仅分配一个向量。为了克服这一局限性，我们提出了一种名为DualCSE的句子嵌入方法，该方法为每个句子分配两个嵌入：一个是表示显式语义的嵌入，另一个是表示隐含语义的嵌入。这些嵌入共存于共享空间中，从而可以在诸如信息检索和文本分类等特定用途下选择所需的语义。实验结果表明，DualCSE能够有效地编码显性和隐性含义，并提高下游任务的表现。
URL
https://arxiv.org/abs/2510.09293
PDF
-
Efficient Prompt Optimisation for Legal Text Classification with Proxy Prompt Evaluator
2025-10-09 17:49:53Hyunji Lee, Kevin Chenhao Li, Matthias Grabmair, Shanshan XuAbstract
Prompt optimization aims to systematically refine prompts to enhance a language model's performance on specific tasks. Fairness detection in Terms of Service (ToS) clauses is a challenging legal NLP task that demands carefully crafted prompts to ensure reliable results. However, existing prompt optimization methods are often computationally expensive due to inefficient search strategies and costly prompt candidate scoring. In this paper, we propose a framework that combines Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to more effectively explore the prompt space while reducing evaluation costs. Experiments demonstrate that our approach achieves higher classification accuracy and efficiency than baseline methods under a constrained computation budget.
Abstract (translated)
提示优化的目标是系统地改进提示，以增强语言模型在特定任务上的表现。服务条款（ToS）中的公平性检测是一项具有挑战性的法律自然语言处理（NLP）任务，需要精心设计的提示来确保可靠的结果。然而，现有的提示优化方法通常由于搜索策略低效和候选提示评分成本高而计算开销大。在这篇论文中，我们提出了一种结合蒙特卡洛树搜索（MCTS）与代理提示评估器的框架，旨在更有效地探索提示空间同时减少评价成本。实验表明，在给定的计算预算下，我们的方法比基线方法实现了更高的分类准确率和效率。
URL
https://arxiv.org/abs/2510.08524
PDF
-
Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER
2025-10-08 21:21:09Junyi Zhu, Savas Ozkan, Andrea Maracani, Sinan Mutlu, Cho Jung Min, Mete OzayAbstract
Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that naïve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8% for NER and +8.8% for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.
Abstract (translated)
在移动平台上部署自然语言处理（NLP）模型需要能够在多种应用场景中适应并保持内存和计算效率的模型。我们研究了预微调策略，以增强轻量级BERT类似编码器对两种基本NLP任务类别：命名实体识别（NER）和文本分类的适用性。虽然单独的任务预微调可以改善每个任务类别的下游性能，但我们发现单纯的多任务预微调会引入相互冲突的优化信号，从而降低整体表现。 为了解决这个问题，我们提出了一种基于主要任务LoRA模块的简单而有效的多任务预微调框架，这使得单一共享编码器主干能够通过模块化适配器实现。我们的方法在满足实际部署约束的同时，实现了与单独预微调相当的表现水平。在21个下游任务上的实验表明，对于NER任务改进了+0.8%，对于文本分类改进了+8.8%，展示了我们方法在移动NLP应用中的通用性及有效性。
URL
https://arxiv.org/abs/2510.07566
PDF
-
Reasoning for Hierarchical Text Classification: The Case of Patents
2025-10-08 16:06:04Lekang Jiang, Wenjun Sun, Stephan GoetzAbstract
Hierarchical text classification (HTC) assigns documents to multiple levels of a pre-defined taxonomy. Automated patent subject classification represents one of the hardest HTC scenarios because of domain knowledge difficulty and a huge number of labels. Prior approaches only output a flat label set, which offers little insight into the reason behind predictions. Therefore, we propose Reasoning for Hierarchical Classification (RHC), a novel framework that reformulates HTC as a step-by-step reasoning task to sequentially deduce hierarchical labels. RHC trains large language models (LLMs) in two stages: a cold-start stage that aligns outputs with chain-of-thought (CoT) reasoning format and a reinforcement learning (RL) stage to enhance multi-step reasoning ability. RHC demonstrates four advantages in our experiments. (1) Effectiveness: RHC surpasses previous baselines and outperforms the supervised fine-tuning counterparts by approximately 3% in accuracy and macro F1. (2) Explainability: RHC produces natural-language justifications before prediction to facilitate human inspection. (3) Scalability: RHC scales favorably with model size with larger gains compared to standard fine-tuning. (4) Applicability: Beyond patents, we further demonstrate that RHC achieves state-of-the-art performance on other widely used HTC benchmarks, which highlights its broad applicability.
Abstract (translated)
层次文本分类（HTC）将文档分配到预定义的分类体系结构中的多个层级。自动化专利主题分类代表了最复杂的HTC场景之一，因为其需要专业的领域知识，并且标签数量巨大。之前的方法只输出扁平化的标签集合，这无法提供预测背后的理由。因此，我们提出了一种新的框架——用于层次分类的理由推理（RHC），它将HTC重新表述为一个逐步推理任务，以顺序推导出层级标签。RHC通过两个阶段训练大型语言模型（LLM）：第一阶段是对齐链式思维（CoT）推理格式的冷启动阶段；第二阶段是增强多步推理能力的强化学习（RL）阶段。 在我们的实验中，RHC展示了四个优势： 1. **有效性**：与之前的基准方法相比，RHC的表现更佳，并且其监督微调版本的准确性提高了大约3%。 2. **解释性**：在预测前生成自然语言的理由，有助于人工审核和理解模型的决策过程。 3. **可扩展性**：随着模型大小的增长，RHC表现出更好的性能提升效果，相比标准微调方法有更大的收益。 4. **适用性**：除了专利分类之外，我们在其他广泛使用的HTC基准上也展示了RHC能达到最先进的表现水平，这突显了其广泛的适用性和通用性。
URL
https://arxiv.org/abs/2510.07167
PDF
-
CLAQS: Compact Learnable All-Quantum Token Mixer with Shared-ansatz for Text Classification
2025-10-08 00:20:08Junhao Chen, Yifan Zhou, Hanqi Jiang, Yi Pan, Yiwei Li, Huaqin Zhao, Wei Zhang, Yingfeng Wang, Tianming LiuAbstract
Quantum compute is scaling fast, from cloud QPUs to high throughput GPU simulators, making it timely to prototype quantum NLP beyond toy tasks. However, devices remain qubit limited and depth limited, training can be unstable, and classical attention is compute and memory heavy. This motivates compact, phase aware quantum token mixers that stabilize amplitudes and scale to long sequences. We present CLAQS, a compact, fully quantum token mixer for text classification that jointly learns complex-valued mixing and nonlinear transformations within a unified quantum circuit. To enable stable end-to-end optimization, we apply l1 normalization to regulate amplitude scaling and introduce a two-stage parameterized quantum architecture that decouples shared token embeddings from a window-level quantum feed-forward module. Operating under a sliding-window regime with document-level aggregation, CLAQS requires only eight data qubits and shallow circuits, yet achieves 91.64% accuracy on SST-2 and 87.08% on IMDB, outperforming both classical Transformer baselines and strong hybrid quantum-classical counterparts.
Abstract (translated)
量子计算的发展速度非常快，从云端的量子处理单元（QPUs）到高吞吐量的GPU模拟器，这使得在玩具任务之外开发量子自然语言处理原型变得尤为紧迫。然而，设备仍然受到量子比特数量和深度的限制，训练过程可能不稳定，并且经典的注意力机制对计算资源和内存消耗较大。因此，设计紧凑、相位感知型的量子令牌混合器来稳定振幅并扩展到长序列的任务显得尤为重要。 我们提出了CLAQS（Compact Layer-Aware Quantum Sequence），这是一种用于文本分类任务的小巧而完全量子化的令牌混合器，在统一的量子电路内同时学习复值混合和非线性变换。为了实现端到端优化的稳定性，我们在模型中应用了L1正则化来调控振幅缩放，并引入了一种两阶段参数化量子架构，这种架构将共享令牌嵌入与窗口级的量子前馈模块分离。 在基于滑动窗口机制和文档级别聚合操作的前提下，CLAQS仅需要8个数据量子比特和浅层电路结构。尽管如此，在SST-2数据集上，它依然能够达到91.64%的准确率；而在IMDB数据集上，则达到了87.08%的准确率，这些结果优于经典的Transformer基线模型以及强大的混合型量子—经典模型。
URL
https://arxiv.org/abs/2510.06532
PDF
-
NASP-T: A Fuzzy Neuro-Symbolic Transformer for Logic-Constrained Aviation Safety Report Classification
2025-10-06 23:33:09Fadi Al Machot, Fidaa Al MachotAbstract
Deep transformer models excel at multi-label text classification but often violate domain logic that experts consider essential, an issue of particular concern in safety-critical applications. We propose a hybrid neuro-symbolic framework that integrates Answer Set Programming (ASP) with transformer-based learning on the Aviation Safety Reporting System (ASRS) corpus. Domain knowledge is formalized as weighted ASP rules and validated using the Clingo solver. These rules are incorporated in two complementary ways: (i) as rule-based data augmentation, generating logically consistent synthetic samples that improve label diversity and coverage; and (ii) as a fuzzy-logic regularizer, enforcing rule satisfaction in a differentiable form during fine-tuning. This design preserves the interpretability of symbolic reasoning while leveraging the scalability of deep neural architectures. We further tune per-class thresholds and report both standard classification metrics and logic-consistency rates. Compared to a strong Binary Cross-Entropy (BCE) baseline, our approach improves micro- and macro-F1 scores and achieves up to an 86% reduction in rule violations on the ASRS test set. To the best of our knowledge, this constitutes the first large-scale neuro-symbolic application to ASRS reports that unifies ASP-based reasoning, rule-driven augmentation, and differentiable transformer training for trustworthy, safety-critical NLP.
Abstract (translated)
深度变换器模型在多标签文本分类方面表现出色，但常常违反专家认为至关重要的领域逻辑，这对安全关键型应用尤为令人担忧。我们提出了一种将回答集编程（ASP）与基于变压器的学习相结合的混合神经符号框架，并应用于航空安全管理报告系统（ASRS）语料库。领域知识被形式化为加权ASP规则并通过Clingo求解器进行验证。这些规则以两种互补方式整合：(i) 作为基于规则的数据增强，生成逻辑一致的人工合成样本，增加标签多样性和覆盖率；以及 (ii) 作为一种模糊逻辑正则项，在微调过程中以可微分形式强制执行规则满足性。这种设计在利用深度神经架构的可扩展性的同时，保留了符号推理的解释能力。我们进一步调整每个类别的阈值，并报告标准分类指标和逻辑一致性率。与强大的二元交叉熵（BCE）基准相比，我们的方法提高了微平均F1分数和宏平均F1分数，在ASRS测试集上实现了高达86%的规则违反减少。据我们所知，这是第一个将ASP推理、基于规则的数据增强以及可微分变换器训练相结合的大规模神经符号应用于ASRS报告的安全关键型NLP应用。
URL
https://arxiv.org/abs/2510.05451
PDF
-
AWARE, Beyond Sentence Boundaries: A Contextual Transformer Framework for Identifying Cultural Capital in STEM Narratives
2025-10-06 16:19:57Khalid Mehtab Khan, Anagha KulkarniAbstract
Identifying cultural capital (CC) themes in student reflections can offer valuable insights that help foster equitable learning environments in classrooms. However, themes such as aspirational goals or family support are often woven into narratives, rather than appearing as direct keywords. This makes them difficult to detect for standard NLP models that process sentences in isolation. The core challenge stems from a lack of awareness, as standard models are pre-trained on general corpora, leaving them blind to the domain-specific language and narrative context inherent to the data. To address this, we introduce AWARE, a framework that systematically attempts to improve a transformer model's awareness for this nuanced task. AWARE has three core components: 1) Domain Awareness, adapting the model's vocabulary to the linguistic style of student reflections; 2) Context Awareness, generating sentence embeddings that are aware of the full essay context; and 3) Class Overlap Awareness, employing a multi-label strategy to recognize the coexistence of themes in a single sentence. Our results show that by making the model explicitly aware of the properties of the input, AWARE outperforms a strong baseline by 2.1 percentage points in Macro-F1 and shows considerable improvements across all themes. This work provides a robust and generalizable methodology for any text classification task in which meaning depends on the context of the narrative.
Abstract (translated)
在学生反思中识别文化资本（CC）主题可以为促进课堂中的公平学习环境提供有价值的见解。然而，诸如抱负目标或家庭支持等主题通常交织于叙述之中，并未以直接关键词的形式出现，这使得标准自然语言处理（NLP）模型难以检测这些主题，因为它们在处理句子时孤立地工作。这一核心挑战源于现有模型缺乏领域意识：标准模型是基于通用语料库预训练的，因而无法识别数据中特有的领域特定语言和叙述上下文。 为了解决这个问题，我们提出了AWARE框架，该框架系统性地尝试提高转换器模型在这项细致任务中的认知能力。AWARE包括三个核心组成部分： 1. **领域意识**：调整模型词汇以适应学生反思的语体风格。 2. **情境意识**：生成能够反映整篇文章上下文的句子嵌入。 3. **类别重叠意识**：采用多标签策略，识别单个句子中主题共存的情况。 我们的实验结果表明，通过使模型明确地了解输入的特点，AWARE在宏平均F1分数上超越了强大的基线模型2.1个百分点，并且在所有主题上都取得了显著的改进。这项工作为任何文本分类任务提供了一种稳健且通用的方法论，其中意义取决于叙述上下文。
URL
https://arxiv.org/abs/2510.04983
PDF
-
LLM Based Bayesian Optimization for Prompt Search
2025-10-05 22:32:50Adam Ballew, Jingbo Wang, Shaogang RenAbstract
Bayesian Optimization (BO) has been widely used to efficiently optimize expensive black-box functions with limited evaluations. In this paper, we investigate the use of BO for prompt engineering to enhance text classification with Large Language Models (LLMs). We employ an LLM-powered Gaussian Process (GP) as the surrogate model to estimate the performance of different prompt candidates. These candidates are generated by an LLM through the expansion of a set of seed prompts and are subsequently evaluated using an Upper Confidence Bound (UCB) acquisition function in conjunction with the GP posterior. The optimization process iteratively refines the prompts based on a subset of the data, aiming to improve classification accuracy while reducing the number of API calls by leveraging the prediction uncertainty of the LLM-based GP. The proposed BO-LLM algorithm is evaluated on two datasets, and its advantages are discussed in detail in this paper.
Abstract (translated)
贝叶斯优化（BO）已被广泛用于高效地优化昂贵的黑盒函数，同时限制评估次数。本文研究了使用贝叶斯优化进行提示工程以增强大型语言模型（LLMs）在文本分类中的效果。我们采用由LLM驱动的高斯过程（GP）作为代理模型来估计不同提示候选方案的表现。这些候选方案通过扩展一组种子提示并通过LLM生成，随后利用与GP后验概率结合的上置信界（UCB）获取函数进行评估。优化过程基于数据子集迭代地改进提示，旨在提高分类准确性的同时减少API调用次数，并通过LLM基GP预测不确定性实现这一目标。本文提出的BO-LLM算法在两个数据集上进行了评估，其优点也在文中详细讨论。
URL
https://arxiv.org/abs/2510.04384
PDF
-
Unmasking Backdoors: An Explainable Defense via Gradient-Attention Anomaly Scoring for Pre-trained Language Models
2025-10-05 20:15:56Anindya Sundar Das, Kangjie Chen, Monowar BhuyanAbstract
Pre-trained language models have achieved remarkable success across a wide range of natural language processing (NLP) tasks, particularly when fine-tuned on large, domain-relevant datasets. However, they remain vulnerable to backdoor attacks, where adversaries embed malicious behaviors using trigger patterns in the training data. These triggers remain dormant during normal usage, but, when activated, can cause targeted misclassifications. In this work, we investigate the internal behavior of backdoored pre-trained encoder-based language models, focusing on the consistent shift in attention and gradient attribution when processing poisoned inputs; where the trigger token dominates both attention and gradient signals, overriding the surrounding context. We propose an inference-time defense that constructs anomaly scores by combining token-level attention and gradient information. Extensive experiments on text classification tasks across diverse backdoor attack scenarios demonstrate that our method significantly reduces attack success rates compared to existing baselines. Furthermore, we provide an interpretability-driven analysis of the scoring mechanism, shedding light on trigger localization and the robustness of the proposed defense.
Abstract (translated)
预训练语言模型在广泛的自然语言处理（NLP）任务中取得了显著的成功，特别是在大规模、领域相关的数据集上进行微调时。然而，它们仍然容易受到后门攻击的威胁，在这种攻击中，对手会在训练数据中嵌入恶意行为使用触发模式。这些触发器在正常使用期间处于休眠状态，但在被激活时会导致目标误分类。 在这项工作中，我们调查了带有后门预训练编码器语言模型内部的行为，重点关注处理受污染输入时注意力和梯度归因的一致性变化；在这种情况下，触发令牌主导了注意力和梯度信号，覆盖了周围的上下文。我们提出了一种在推理时间进行防御的方法，通过结合标记级别的注意力和梯度信息来构建异常分数。我们在各种后门攻击场景下的文本分类任务中进行了广泛的实验，结果表明我们的方法显著降低了现有基准的攻击成功率。 此外，我们提供了对评分机制的可解释性驱动分析，揭示了触发器定位和所提议防御措施鲁棒性的新见解。
URL
https://arxiv.org/abs/2510.04347
PDF
-
SliceMoE: Routing Embedding Slices Instead of Tokens for Fine-Grained and Balanced Transformer Scaling
2025-10-05 16:57:32Harshil VejendlaAbstract
Mixture-of-Experts (MoE) layers scale transformers by routing tokens to a sparse subset of feed-forward experts. Token-level routing, however, assigns an entire semantic spectrum to each expert, creating capacity bottlenecks, load-balancing pathologies, and limited specialization. We introduce SliceMoE, an architecture that routes contiguous slices of a token's hidden vector. A d-dimensional embedding is partitioned into S slices, and for each slice, a lightweight shared router predicts the top-k experts. Experts operate on their assigned slices independently, and outputs are reassembled, maintaining per-token FLOP efficiency. Because slices from different tokens interleave within an expert, utilization is naturally smoother. We propose a slice-level capacity loss, cross-slice dropout, and efficient fused batched GEMM kernels. Experiments on WikiText-103 language modeling, WMT En-De translation, and three text-classification datasets show SliceMoE attains up to 1.7x faster inference than dense baselines, 12 to 18 percent lower perplexity than parameter-matched token-MoE, and improved expert balance, with interpretable expertise over syntactic versus semantic subspaces.
Abstract (translated)
混合专家（Mixture-of-Experts，MoE）层通过将令牌路由到一组稀疏的前馈专家子集来扩展转换器模型。然而，基于令牌级别的路由为每个专家分配了一个完整的语义频谱，这会导致容量瓶颈、负载均衡问题以及专业化的限制。为了应对这些问题，我们引入了SliceMoE架构，该架构根据连续的片段将令牌隐藏向量路由到不同的专家。 在SliceMoE中，一个d维嵌入被划分为S个片段（slices）。对于每一个片段，一个轻量级共享路由器预测出前k个专家。每个专家独立地操作它们分配到的片段，并且输出会被重新组装起来以保持每令牌的计算效率不变。由于来自不同令牌的片段在一个专家内交错进行处理，因此利用率会更加平滑。 我们提出了一种基于切片级别的容量损失、跨切片dropout策略以及高效的批矩阵乘法（GEMM）核来改进模型性能和效率。实验结果表明，在WikiText-103语言建模任务、WMT英德翻译任务及三个文本分类数据集上，与密集基线相比，SliceMoE实现了高达1.7倍的推理加速；与参数匹配的令牌MoE相比，困惑度降低了12%至18%，同时改进了专家间的平衡性，并且使得每个专家专注于语法或语义子空间中更具可解释性的专业知识。
URL
https://arxiv.org/abs/2510.04286
PDF
-
Replacing Softmax Similarity with a Sharpened Angular Similarity: Theory and Practice of Scaling To Billion-Context Attention
2025-10-05 02:57:40Sahil Joshi, Agniva Chowdhury, Amar Kanakamedala, Ekam Singh, Evan Tu, Anshumali ShrivastavaAbstract
Softmax Attention has a quadratic time complexity, which becomes prohibitive to run at long contexts, even with highly optimized GPU kernels. For example, FlashAttention (an exact, GPU-optimized implementation of Softmax Attention) cannot complete a single forward-backward pass of a multi-head attention layer once the context exceeds ~4 million tokens on an NVIDIA GH200 (96 GB). We introduce RACE Attention, a kernel-inspired alternative to Softmax Attention that is linear in sequence length and embedding dimension. RACE Attention replaces the exponential kernel with a sharpened angular (cosine) similarity, and approximates attention outputs via randomized projections and soft Locality-Sensitive Hashing (LSH). Across language modeling, masked language modeling, and text classification, RACE Attention matches the accuracy of strong baselines while reducing runtime and memory. In a controlled scale test, it processes up to 12 million tokens during a single forward-backward pass on an NVIDIA GH200 GPU and 75 million tokens on an Intel Xeon Gold 5220R CPU, well beyond the practical limits of the current state-of-the-art attention implementations. RACE Attention thus offers a practical, theoretically grounded mechanism for outrageously long context windows on today's hardware. We hope that it gets adopted in practice.
Abstract (translated)
Softmax注意力机制的时间复杂度为二次方，这在处理长上下文时变得非常不可行，即使使用了高度优化的GPU内核也是如此。例如，FlashAttention（一种精确且经过GPU优化的Softmax Attention实现）在NVIDIA GH200 GPU上一旦上下文超过约4百万令牌，就无法完成多头注意力层的一次前向-反向传递。 我们引入了一种名为RACE Attention的新方法，这是一种受内核启发的替代方案，它的时间复杂度与序列长度和嵌入维度呈线性关系。RACE Attention通过使用锐化的余弦相似度（即角相似度）来替换指数核，并且通过随机投影和软局部敏感哈希（LSH）近似注意力输出。 在语言建模、遮蔽语言建模以及文本分类等多种任务中，RACE Attention能够在减少运行时间和内存消耗的同时与强大的基准模型达到相同的准确性。在一个受控的规模测试中，在NVIDIA GH200 GPU上处理多达1.2亿个令牌，在Intel Xeon Gold 5220R CPU上处理多达7.5亿个令牌的一次前向-反向传递，远远超过了当前最先进的注意力机制实现的实际限制。 因此，RACE Attention为当今硬件上的极长上下文窗口提供了理论上合理且实用的解决方案。我们希望它能在实践中被采用。
URL
https://arxiv.org/abs/2510.04008
PDF
-
Cross-View Open-Vocabulary Object Detection in Aerial Imagery
2025-10-04 16:12:03Jyoti Kini, Rohit Gupta, Mubarak ShahAbstract
Traditional object detection models are typically trained on a fixed set of classes, limiting their flexibility and making it costly to incorporate new categories. Open-vocabulary object detection addresses this limitation by enabling models to identify unseen classes without explicit training. Leveraging pretrained models contrastively trained on abundantly available ground-view image-text classification pairs provides a strong foundation for open-vocabulary object detection in aerial imagery. Domain shifts, viewpoint variations, and extreme scale differences make direct knowledge transfer across domains ineffective, requiring specialized adaptation strategies. In this paper, we propose a novel framework for adapting open-vocabulary representations from ground-view images to solve object detection in aerial imagery through structured domain alignment. The method introduces contrastive image-to-image alignment to enhance the similarity between aerial and ground-view embeddings and employs multi-instance vocabulary associations to align aerial images with text embeddings. Extensive experiments on the xView, DOTAv2, VisDrone, DIOR, and HRRSD datasets are used to validate our approach. Our open-vocabulary model achieves improvements of +6.32 mAP on DOTAv2, +4.16 mAP on VisDrone (Images), and +3.46 mAP on HRRSD in the zero-shot setting when compared to finetuned closed-vocabulary dataset-specific model performance, thus paving the way for more flexible and scalable object detection systems in aerial applications.
Abstract (translated)
传统的对象检测模型通常在固定的一组类别上进行训练，这限制了其灵活性，并且增加了将新类别纳入系统的成本。开放词汇对象检测通过使模型能够在没有显式训练的情况下识别未见过的类来解决这一限制。利用预先训练好的模型，这些模型是在大量可用的地平视图像-文本分类对上对比性训练得到的，为在航拍图像中进行开放词汇对象检测提供了坚实的基础。 然而，领域变化、视角差异和极端尺度不同使得直接跨域知识迁移无效，需要专门的适应策略。在这篇论文中，我们提出了一种新颖的框架，用于将地平视图图像中的开放词汇表示适应到解决航拍图像中的对象检测问题，通过结构化的领域对齐来实现这一目标。该方法引入了对比性的图像对齐以增强航拍与地平视图嵌入之间的相似性，并采用多实例词汇关联来使航拍图像与文本嵌入对齐。 我们在xView、DOTAv2、VisDrone、DIOR和HRRSD数据集上进行了广泛的实验，用以验证我们的方法。在零样本设置中，我们的开放词汇模型相比经过微调的封闭词汇特定数据集模型，在DOTAv2上的mAP提升了+6.32%，在VisDrone（图像）上的mAP提升了+4.16%，在HRRSD上提升为+3.46%。这表明了更灵活和可扩展的对象检测系统在航拍应用中的潜力。
URL
https://arxiv.org/abs/2510.03858
PDF
-
What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification
2025-10-03 22:19:16Andrew Halterman, Katherine A. KeithAbstract
Generative large language models (LLMs) are now used extensively for text classification in computational social science (CSS). In this work, focus on the steps before and after LLM prompting -- conceptualization of concepts to be classified and using LLM predictions in downstream statistical inference -- which we argue have been overlooked in much of LLM-era CSS. We claim LLMs can tempt analysts to skip the conceptualization step, creating conceptualization errors that bias downstream estimates. Using simulations, we show that this conceptualization-induced bias cannot be corrected for solely by increasing LLM accuracy or post-hoc bias correction methods. We conclude by reminding CSS analysts that conceptualization is still a first-order concern in the LLM-era and provide concrete advice on how to pursue low-cost, unbiased, low-variance downstream estimates.
Abstract (translated)
生成式大型语言模型（LLM）现在广泛用于计算社会科学（CSS）中的文本分类。在这项工作中，我们重点关注在LLM提示之前和之后的步骤——即被分类概念的概念化过程以及将LLM预测应用于下游统计推断的过程，我们认为这些环节在大多数使用LLM的时代CSS研究中被忽略了。我们主张，LLM可能会诱使分析人员跳过概念化的步骤，从而产生概念错误，进而导致后续估计结果出现偏差。通过模拟实验，我们展示了仅靠提高LLM的准确性或采用事后偏置校正方法是不足以纠正这种由概念化引起的偏差的。最后，我们提醒CSS分析师，在LLM时代，概念化仍然是一个首要关注点，并提供了如何追求低成本、无偏且方差低的下游估计的具体建议。
URL
https://arxiv.org/abs/2510.03541
PDF
-
Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction
2025-09-30 20:42:13Zhexiong Liu, Diane LitmanAbstract
Large Language Models (LLMs) have shown extraordinary success across various text generation tasks; however, their potential for simple yet essential text classification remains underexplored, as LLM pre-training tends to emphasize generation over classification. While LLMs with instruction tuning can transform classification into a generation task, they often struggle to categorize nuanced texts. One such example is text revision, which involves nuanced edits between pairs of texts. Although simply fine-tuning LLMs for revision classification seems plausible, it requires a large amount of revision annotations, which are exceptionally expensive and scarce in the community. To address this issue, we introduce a plug-and-play layer-wise parameter-efficient fine-tuning (PEFT) framework, i.e., IR-Tuning, which fine-tunes a subset of important LLM layers that are dynamically selected based on their gradient norm distribution, while freezing those of redundant layers. Extensive experiments suggest that IR-Tuning surpasses several layer-wise PEFT baselines over diverse text revisions, while achieving fast convergence, low GPU memory consumption, and effectiveness on small revision corpora.
Abstract (translated)
大型语言模型（LLMs）在各种文本生成任务中展示了非凡的成功；然而，它们对于简单而基本的文本分类的潜力尚未得到充分探索，因为大规模预训练倾向于强调生成而非分类。虽然通过指令微调可以使LLM将分类问题转化为生成任务，但它们常常难以对细微差别较大的文本进行准确分类。其中一个例子就是文本修订，这涉及到两段文本之间的微妙编辑差异。尽管简单地为LLMs的修订分类进行微调似乎是可行的，但这需要大量的修订注释，而这些数据在社区中极其昂贵且稀缺。 为了应对这一问题，我们引入了一种即插即用层级参数高效微调（PEFT）框架，即IR-Tuning。该方法选择性地对一组重要层级进行细调，这些层级是基于它们的梯度范数分布动态选定的，并冻结那些冗余层级。通过广泛的实验表明，IR-Tuning在多种文本修订任务中超越了几种层级PEFT基线模型，同时实现了快速收敛、低GPU内存消耗以及对于小型修订语料库的有效性。
URL
https://arxiv.org/abs/2510.00268
PDF
-
QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
2025-09-28 08:35:31Qian Chen, Linxin Yang, Akang Wang, Xiaodong Luo, Yin ZhangAbstract
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase nonlinearity in neural networks, with the aim of enhancing the performance of existing architectures. To reduce parameter complexity and computational complexity, we propose a lightweight quadratic enhancer that uses low-rankness, weight sharing, and sparsification techniques. For a fixed architecture, the proposed approach introduces quadratic interactions between features at every layer, while only adding negligible amounts of additional model parameters and forward computations. We conduct a set of proof-of-concept experiments for the proposed method across three tasks: image classification, text classification, and fine-tuning large-language models. In all tasks, the proposed approach demonstrates clear and substantial performance gains.
Abstract (translated)
线性变换与非线性激活函数的结合构成了大多数现代深度神经网络的基础，使它们能够近似复杂度极高的函数。本文探讨了引入二次变换以进一步增加神经网络中的非线性程度，旨在提升现有架构的表现能力。为了降低参数复杂性和计算复杂性，我们提出了一种轻量级的二次增强器，该增强器利用低秩、权重共享和稀疏化技术。对于固定的架构，在每一层中我们的方法引入了特征之间的二次交互作用，并且仅添加了几乎可以忽略不计的额外模型参数和前向计算。 我们在三个任务上进行了概念验证实验：图像分类、文本分类以及大型语言模型的微调。在所有这些任务中，所提出的方法均展示了明显的性能提升。
URL
https://arxiv.org/abs/2510.03276
PDF
-
The Impact of Role Design in In-Context Learning for Large Language Models
2025-09-27 21:15:30Hamidreza Rouzegar, Masoud MakrehchiAbstract
In-context learning (ICL) enables Large Language Models (LLMs) to generate predictions based on prompts without additional fine-tuning. While prompt engineering has been widely studied, the impact of role design within prompts remains underexplored. This study examines the influence of role configurations in zero-shot and few-shot learning scenarios using GPT-3.5 and GPT-4o from OpenAI and Llama2-7b and Llama2-13b from Meta. We evaluate the models' performance across datasets, focusing on tasks like sentiment analysis, text classification, question answering, and math reasoning. Our findings suggest the potential of role-based prompt structuring to enhance LLM performance.
Abstract (translated)
上下文学习（ICL）使大型语言模型（LLM）能够在没有额外微调的情况下根据提示生成预测。虽然提示工程已经得到了广泛研究，但提示中角色设计的影响仍然鲜为人知。本研究通过使用来自OpenAI的GPT-3.5和GPT-4o以及来自Meta的Llama2-7b和Llama2-13b模型，在零样本和少量样本学习场景下考察了角色配置对性能的影响。我们评估了这些模型在情感分析、文本分类、问答和数学推理等任务上的表现。我们的研究结果表明，基于角色的提示结构有可能提升LLM的表现。 注意：原文中提到了“GPT-4o”，假设这是一个笔误或者是特定版本的名称，在翻译过程中保持了一致性处理，请根据实际情况进行调整。如果"GPT-4o"是专有名词或者具体指某个版本，则不需要更改。
URL
https://arxiv.org/abs/2509.23501
PDF
-
Language, Culture, and Ideology: Personalizing Offensiveness Detection in Political Tweets with Reasoning LLMs
2025-09-27 15:20:44Dzmitry Pihulski, Jan Koco\'nAbstract
We explore how large language models (LLMs) assess offensiveness in political discourse when prompted to adopt specific political and cultural perspectives. Using a multilingual subset of the MD-Agreement dataset centered on tweets from the 2020 US elections, we evaluate several recent LLMs - including DeepSeek-R1, o4-mini, GPT-4.1-mini, Qwen3, Gemma, and Mistral - tasked with judging tweets as offensive or non-offensive from the viewpoints of varied political personas (far-right, conservative, centrist, progressive) across English, Polish, and Russian contexts. Our results show that larger models with explicit reasoning abilities (e.g., DeepSeek-R1, o4-mini) are more consistent and sensitive to ideological and cultural variation, while smaller models often fail to capture subtle distinctions. We find that reasoning capabilities significantly improve both the personalization and interpretability of offensiveness judgments, suggesting that such mechanisms are key to adapting LLMs for nuanced sociopolitical text classification across languages and ideologies.
Abstract (translated)
我们探讨了大型语言模型（LLMs）在被提示采用特定政治和文化视角时，如何评估政治话语中的冒犯性。使用一个多语种的MD-Agreement数据集子集，该数据集以2020年美国大选期间的推特为主，我们评估了几种最近的大型语言模型——包括DeepSeek-R1、o4-mini、GPT-4.1-mini、Qwen3、Gemma和Mistral——这些模型被要求从英语、波兰语和俄语中的不同政治人物视角（极右翼、保守派、中间派、进步派）来判断推特内容是否冒犯。我们的结果显示，具备明确推理能力的大型模型（如DeepSeek-R1、o4-mini），在捕捉意识形态和文化差异方面更为一致且敏感，而较小的模型往往无法捕捉到细微的区别。我们发现，推理能力显著提高了冒犯性评判的个性化和可解释性，这表明此类机制对于跨语言和意识形态进行复杂社会政治文本分类来说至关重要。
URL
https://arxiv.org/abs/2510.02351
PDF
-
Universal Legal Article Prediction via Tight Collaboration between Supervised Classification Model and LLM
2025-09-26 09:42:20Xiao Chi, Wenlin Zhong, Yiquan Wu, Wei Wang, Kun Kuang, Fei Wu, Minghui XiongAbstract
Legal Article Prediction (LAP) is a critical task in legal text classification, leveraging natural language processing (NLP) techniques to automatically predict relevant legal articles based on the fact descriptions of cases. As a foundational step in legal decision-making, LAP plays a pivotal role in determining subsequent judgments, such as charges and penalties. Despite its importance, existing methods face significant challenges in addressing the complexities of LAP. Supervised classification models (SCMs), such as CNN and BERT, struggle to fully capture intricate fact patterns due to their inherent limitations. Conversely, large language models (LLMs), while excelling in generative tasks, perform suboptimally in predictive scenarios due to the abstract and ID-based nature of legal articles. Furthermore, the diversity of legal systems across jurisdictions exacerbates the issue, as most approaches are tailored to specific countries and lack broader applicability. To address these limitations, we propose Uni-LAP, a universal framework for legal article prediction that integrates the strengths of SCMs and LLMs through tight collaboration. Specifically, in Uni-LAP, the SCM is enhanced with a novel Top-K loss function to generate accurate candidate articles, while the LLM employs syllogism-inspired reasoning to refine the final predictions. We evaluated Uni-LAP on datasets from multiple jurisdictions, and empirical results demonstrate that our approach consistently outperforms existing baselines, showcasing its effectiveness and generalizability.
Abstract (translated)
法律文章预测（LAP）是法律文本分类中的关键任务，通过运用自然语言处理（NLP）技术来自动预测与案件事实描述相关的法律条文。作为法律决策的基础步骤，LAP在确定后续判决如指控和处罚方面发挥着核心作用。尽管它的重要性不言而喻，现有的方法在解决LAP的复杂性方面面临重大挑战。 监督分类模型（SCMs），例如卷积神经网络（CNN）和BERT，在捕捉复杂的事实模式时往往力有未逮，这主要是由于它们自身的局限性所致。另一方面，大型语言模型（LLMs）虽然在生成任务中表现出色，但在预测场景中的表现却不如人意，这是由法律条文的抽象性和基于ID的特点决定的。此外，不同法域之间法律系统的多样性进一步加剧了这一问题，因为大多数方法都是针对特定国家设计的，并不具备广泛的适用性。 为了解决这些局限性，我们提出了Uni-LAP，这是一个用于法律文章预测的通用框架，通过紧密合作整合了SCMs和LLMs的优势。具体而言，在Uni-LAP中，监督分类模型利用一种新颖的Top-K损失函数来生成准确的候选条文，而大型语言模型则运用类比于三段论的推理方法来优化最终预测结果。 我们在来自多个法域的数据集上评估了Uni-LAP，并通过实验证明我们的方法在各种基准测试中均表现优异，展示了其有效性和普适性。
URL
https://arxiv.org/abs/2509.22119
PDF
-
AutoIntent: AutoML for Text Classification
2025-09-25 13:27:52Ilya Alekseev, Roman Solomatin, Darina Rustamova, Denis KuznetsovAbstract
AutoIntent is an automated machine learning tool for text classification tasks. Unlike existing solutions, AutoIntent offers end-to-end automation with embedding model selection, classifier optimization, and decision threshold tuning, all within a modular, sklearn-like interface. The framework is designed to support multi-label classification and out-of-scope detection. AutoIntent demonstrates superior performance compared to existing AutoML tools on standard intent classification datasets and enables users to balance effectiveness and resource consumption.
Abstract (translated)
AutoIntent 是一种用于文本分类任务的自动化机器学习工具。与现有的解决方案不同，AutoIntent 提供了端到端的自动化功能，包括嵌入模型选择、分类器优化和决策阈值调整，并且在模块化界面中操作，类似于 sklearn 接口。该框架设计支持多标签分类和未定义范围检测。在标准意图分类数据集上，AutoIntent 的表现优于现有的 AutoML 工具，并使用户能够平衡效率与资源消耗。
URL
https://arxiv.org/abs/2509.21138
PDF
