Text_Generation
-
TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation
2026-03-09 10:03:17Toms Bergmanis, Martins Kronis, Ingus J\=anis Pretkalni\c{n}\v{s}, D\=avis Nicmanis, Je\c{l}izaveta Je\c{l}inska, Roberts Rozis, Rinalds V\=iksna, M\=arcis PinnisAbstract
Large language models often underperform in many European languages due to the dominance of English and a few high-resource languages in training data. This paper presents TildeOpen LLM, a 30-billion-parameter open-weight foundational model trained for 34 European languages to promote linguistic equity and improve performance for low-resource languages. To address the data imbalance, we combine dataset upsampling with a curriculum-based training schedule that alternates between uniform and natural language distributions. The resulting model performs favorably compared to other multilingual LLMs despite being trained with significantly fewer computing resources. Evaluation across multiple multilingual benchmarks shows that TildeOpen surpasses existing open-weight models in text generation and comprehension, particularly for Baltic, Finno-Ugric, and Slavic languages. Human evaluations confirm an up to tenfold reduction in linguistic errors relative to leading baselines. The model and associated resources are fully open-weight and publicly available at this http URL. These outcomes demonstrate that careful data curation and balanced training strategies can substantially enhance multilingual model quality without increasing model size or training volume.
Abstract (translated)
大型语言模型在许多欧洲语言中的表现通常不如人意，这是因为训练数据中英语和其他几种高资源语言占主导地位。本文介绍了TildeOpen LLM，这是一个为34种欧洲语言训练的300亿参数的基础开放权重模型，旨在促进语言平等并提高低资源语言的表现。为了应对数据不平衡问题，我们结合了数据集上采样与基于课程的学习时间表，该时间表在均匀分布和自然语言分布之间交替进行。 尽管使用计算资源显著较少，但最终生成的模型与其他多语种LLM相比表现良好。多项跨多种语言基准测试表明，TildeOpen 在文本生成和理解方面超越了现有的开放权重模型，特别是在波罗的语系、乌拉尔-芬兰语系和斯拉夫语系语言中的性能尤为突出。人工评估确认，在与领先基线相比时，语言错误减少了多达十倍。 该模型及相关资源完全为开放权重，并且公开可用（可在此网址访问：[this http URL]）。这些成果表明，通过仔细的数据整理和平衡的训练策略，可以在不增加模型大小或训练量的情况下显著提高多语种模型的质量。
URL
https://arxiv.org/abs/2603.08182
PDF
-
AgriPath: A Systematic Exploration of Architectural Trade-offs for Crop Disease Classification
2026-03-08 17:28:01Hamza Mooraj, George Pantazopoulos, Alessandro SugliaAbstract
Reliable crop disease detection requires models that perform consistently across diverse acquisition conditions, yet existing evaluations often focus on single architectural families or lab-generated datasets. This work presents a systematic empirical comparison of three model paradigms for fine-grained crop disease classification: Convolutional Neural Networks (CNNs), contrastive Vision-Language Models (VLMs), and generative VLMs. To enable controlled analysis of domain effects, we introduce AgriPath-LF16, a benchmark containing 111k images spanning 16 crops and 41 diseases with explicit separation between laboratory and field imagery, alongside a balanced 30k subset for standardized training and evaluation. All models are trained and evaluated under unified protocols across full, lab-only, and field-only training regimes using macro-F1 and Parse Success Rate (PSR) to account for generative reliability. The results reveal distinct performance profiles. CNNs achieve the highest accuracy on lab imagery but degrade under domain shift. Contrastive VLMs provide a robust and parameter-efficient alternative with competitive cross-domain performance. Generative VLMs demonstrate the strongest resilience to distributional variation, albeit with additional failure modes stemming from free-text generation. These findings highlight that architectural choice should be guided by deployment context rather than aggregate accuracy alone.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13354
PDF
-
Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement
2026-03-06 16:48:27Yakov Pyotr ShkolnikovAbstract
Vision-language models encode continuous geometry that their text pathway fails to express: a 6,000-parameter linear probe extracts hand joint angles at 6.1 degrees MAE from frozen features, while the best text output achieves only 20.0 degrees -- a 3.3x bottleneck. LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees, providing evidence for a pathway-training deficit rather than a representational one. Training objective determines accuracy more than architecture: five encoders spanning self-supervised, contrastive, and hybrid paradigms converge to statistically equivalent accuracy (R^2 approximately 0.55, TOST-equivalent at delta=0.03) despite sharing as little as CKA=0.41 representational similarity -- functional convergence without representational convergence. Autoregressive generation damages geometric fidelity, but the damage originates in the generation process, not in language alignment: Qwen2.5-VL's LLM layers actually improve probe accuracy over its raw vision encoder. Layer-wise analysis reveals a universal mid-network accuracy peak across all architectures, with attention heads in layers 18-22 carrying disproportionate geometric signal. These findings enable a single frozen backbone to function as a multi-task geometric sensor through lightweight probes, without fine-tuning or text generation.
Abstract (translated)
视觉语言模型编码了连续的几何信息，这是其文本路径无法表达的：一个包含6,000个参数的线性探针从冻结特征中提取出手关节角度，在MAE（均方误差）下达到了6.1度，而最佳的文字输出仅达到20.0度——相差3.3倍。LoRA微调（r=16，2,000张图片）将这一差距缩小至6.5度，这表明路径训练不足而不是表示不足的问题。 训练目标比架构更能决定准确性：五个跨自监督、对比和混合范式的编码器在统计上达到等效的精度（R^2约为0.55，在delta=0.03时具有TOST等价性），尽管它们之间共享的表征相似度低至CKA（基于核的方法）为0.41——功能收敛而非表示收敛。自回归生成损害了几何保真度，但这种损害源于生成过程本身，并非语言对齐：Qwen2.5-VL的LLM层实际上比其原始视觉编码器提高了探针精度。 逐层分析揭示了所有架构中普遍存在的一致网络中间准确性峰值，在第18-22层中的注意力头携带了不成比例的几何信号。这些发现使得单个冻结骨干可以通过轻量级探针充当多任务几何传感器，而无需微调或文本生成。
URL
https://arxiv.org/abs/2603.06459
PDF
-
HART: Data-Driven Hallucination Attribution and Evidence-Based Tracing for Large Language Models
2026-03-06 02:23:55Shize Liang, Hongzhi WangAbstract
Large language models (LLMs) have demonstrated remarkable performance in text generation and knowledge-intensive question answering. Nevertheless, they are prone to producing hallucinated content, which severely undermines their reliability in high-stakes application domains. Existing hallucination attribution approaches, based on either external knowledge retrieval or internal model mechanisms, primarily focus on semantic similarity matching or representation-level discrimination. As a result, they have difficulty establishing structured correspondences at the span level between hallucination types, underlying error generation mechanisms, and external factual evidence, thereby limiting the interpretability of hallucinated fragments and the traceability of supporting or opposing evidence. To address these limitations, we propose HART, a fine-grained hallucination attribution and evidence retrieval framework for large language models. HART formalizes hallucination tracing as a structured modeling task comprising four stages: span localization, mechanism attribution, evidence retrieval, and causal tracing. Based upon this formulation, we develop the first structured dataset tailored for hallucination tracing, in which hallucination types, error mechanisms, and sets of counterfactual evidence are jointly annotated to enable causal-level interpretability evaluation. Experimental results on the proposed dataset demonstrate that HART substantially outperforms strong retrieval baselines, including BM25 and DPR, validating the effectiveness and generalization capability of the proposed tracing paradigm for hallucination analysis and evidence alignment.
Abstract (translated)
大型语言模型（LLMs）在文本生成和知识密集型问答方面展示了卓越的性能，但它们容易产生幻觉内容，这严重削弱了其在高风险应用场景中的可靠性。现有的基于外部知识检索或内部模型机制的幻觉归因方法主要集中在语义相似度匹配或表征层面的区别上。因此，这些方法难以建立幻觉类型、潜在错误生成机制和外部事实证据之间的结构化对应关系，从而限制了对幻觉片段的理解性和支持或反对证据的可追溯性。 为了解决这些问题，我们提出了HART（Hallucination Attribution and Retrieval for Tracing），这是一种针对大型语言模型进行细粒度幻觉归因与证据检索的框架。HART将幻觉追踪形式化为一个包含四个阶段的结构化建模任务：片段定位、机制归因、证据检索和因果追溯。基于这一公式，我们开发了第一个专门用于幻觉追踪的结构化数据集，在该数据集中共同标注幻觉类型、错误机制及反事实证据集合以实现因果层面的理解性评估。 在提出的数据集上的实验结果表明，HART显著优于包括BM25和DPR在内的强大的检索基线模型，验证了所提出追踪范式在幻觉分析和证据对齐方面的有效性和泛化能力。
URL
https://arxiv.org/abs/2603.05828
PDF
-
HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token
2026-03-05 18:36:31Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun, Jiawei ZhouAbstract
Hallucinations remain a persistent challenge for vision-language models (VLMs), which often describe nonexistent objects or fabricate facts. Existing detection methods typically operate after text generation, making intervention both costly and untimely. We investigate whether hallucination risk can instead be predicted before any token is generated by probing a model's internal representations in a single forward pass. Across a diverse set of vision-language tasks and eight modern VLMs, including Llama-3.2-Vision, Gemma-3, Phi-4-VL, and Qwen2.5-VL, we examine three families of internal representations: (i) visual-only features without multimodal fusion, (ii) vision-token representations within the text decoder, and (iii) query-token representations that integrate visual and textual information before generation. Probes trained on these representations achieve strong hallucination-detection performance without decoding, reaching up to 0.93 AUROC on Gemma-3-12B, Phi-4-VL 5.6B, and Molmo 7B. Late query-token states are the most predictive for most models, while visual or mid-layer features dominate in a few architectures (e.g., ~0.79 AUROC for Qwen2.5-VL-7B using visual-only features). These results demonstrate that (1) hallucination risk is detectable pre-generation, (2) the most informative layer and modality vary across architectures, and (3) lightweight probes have the potential to enable early abstention, selective routing, and adaptive decoding to improve both safety and efficiency.
Abstract (translated)
幻觉仍然是视觉-语言模型（VLMs）面临的一个持久挑战，这些模型常常描述不存在的对象或捏造事实。现有的检测方法通常在文本生成之后操作，这使得干预既费时又不够及时。我们研究了是否可以在任何标记生成之前通过探测模型的内部表示来预测幻觉风险，在一次前向传递中完成这一任务。我们对一系列视觉-语言任务和八种现代VLM进行了调查，包括Llama-3.2-Vision、Gemma-3、Phi-4-VL 和 Qwen2.5-VL。我们研究了三种内部表示类型：(i) 仅视觉特征而不进行多模态融合；(ii) 文本解码器内的视觉标记表示；以及 (iii) 在生成前整合视觉和文本信息的查询标记表示。基于这些表示训练的探测器在不进行解码的情况下达到了很强的幻觉检测性能，例如，在Gemma-3-12B、Phi-4-VL 5.6B 和 Molmo 7B 上实现了高达0.93的AUROC（受试者工作特征曲线下的面积）。对于大多数模型来说，晚期查询标记状态最为预测性，而视觉或中间层特性在某些架构中占主导地位（例如，在使用仅视觉特征时，Qwen2.5-VL-7B 实现了约0.79的AUROC）。 这些结果表明：(1) 幻觉风险可以在生成之前被检测到；(2) 最有信息量的层和模态在不同架构之间有所不同；以及 (3) 轻量级探测器具有潜力，能够实现早期放弃、选择性路由和自适应解码，以提高模型的安全性和效率。
URL
https://arxiv.org/abs/2603.05465
PDF
-
Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixes
2026-03-05 18:25:26Pengxiang Li, Joey Tsai, Hongwei Xue, Kunyu Shi, Shilin YanAbstract
Diffusion Language Models (DLMs) promise highly parallel text generation, yet their practical inference speed is often bottlenecked by suboptimal decoding schedulers. Standard approaches rely on 'scattered acceptance'-committing high confidence tokens at disjoint positions throughout the sequence. This approach inadvertently fractures the Key-Value (KV) cache, destroys memory locality, and forces the model into costly, repeated repairs across unstable token boundaries. To resolve this, we present the Longest Stable Prefix (LSP) scheduler, a training-free and model-agnostic inference paradigm based on monolithic prefix absorption. In each denoising step, LSP evaluates token stability via a single forward pass, dynamically identifies a contiguous left-aligned block of stable predictions, and snaps its boundary to natural linguistic or structural delimiters before an atomic commitment. This prefix-first topology yields dual benefits: systemically, it converts fragmented KV cache updates into efficient, contiguous appends; algorithmically, it preserves bidirectional lookahead over a geometrically shrinking active suffix, drastically reducing token flip rates and denoiser calls. Extensive evaluations on LLaDA-8B and Dream-7B demonstrate that LSP accelerates inference by up to 3.4x across rigorous benchmarks including mathematical reasoning, code generation, multilingual (CJK) tasks, and creative writing while matching or slightly improving output quality. By fundamentally restructuring the commitment topology, LSP bridges the gap between the theoretical parallelism of DLMs and practical hardware efficiency.
Abstract (translated)
扩散语言模型（DLM）承诺了高度并行的文本生成，但它们的实际推理速度常常因为不理想的解码调度程序而受限。标准方法依赖于“散列接受”——在整个序列中分散确定高置信度的令牌。这种方法无意中破坏了键值（KV）缓存的一致性，摧毁了内存局部性，并迫使模型在不稳定令牌边界处进行昂贵且重复的修复操作。 为了解决这一问题，我们提出了最长稳定前缀（LSP）调度程序，这是一种无训练、模型无关的推理范式，基于单体前缀吸收。在每次去噪步骤中，LSP通过一次正向传递评估令牌稳定性，动态识别一系列连续左对齐的稳定预测，并将其边界调整到自然语言或结构分隔符处后进行原子承诺。 这种以前缀为主的拓扑结构带来了双重好处：系统层面，它将碎片化的KV缓存更新转化为高效的连续追加；算法方面，它在整个活跃后缀几何收缩的过程中保留了双向前瞻的能力，大大减少了令牌翻转率和去噪器调用次数。在LLaDA-8B和Dream-7B上的广泛评估表明，LSP能够通过严格的基准测试（包括数学推理、代码生成、多语言（中日韩）任务以及创意写作），将推理速度提高多达3.4倍，并且输出质量与现有方法相当或略有提升。 通过从根本上重新构建承诺拓扑结构，LSP弥合了DLM理论并行性与实际硬件效率之间的差距。
URL
https://arxiv.org/abs/2603.05454
PDF
-
Ensembling Language Models with Sequential Monte Carlo
2026-03-05 17:54:31Robin Shing Moon Chan, Tianyu Liu, Samuel Kiegeland, Clemente Pasti, Jacob Hoover Vigly, Timothy J. O'Donnell, Ryan Cotterell, Tim VieiraAbstract
Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$-ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$. To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$-ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.
Abstract (translated)
实践者可以访问大量语言模型和提示策略来解决许多语言建模任务；然而，先前的工作表明，模型性能对这两方面的选择非常敏感。经典机器学习的集成技术提供了一种原则性的方法：通过汇集多个来源的预测以实现比单一来源更好的性能。但是，在解码过程中将集成应用到语言模型上是具有挑战性的：简单地聚合下一个标记的概率会产生从局部归一化的、有偏差的近似分布中抽取的样本，这与通常难以处理的整体集成分布不匹配。 在这项工作中，我们介绍了一个统一框架，用于将$K$个语言模型组合成$f$-集成分布（对于一系列函数$f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$）。为了从这些分布中采样，我们提出了一种字节级别的顺序蒙特卡洛（SMC）算法，在共享字符空间上运行，这使得词汇表不匹配的模型集成和一致的采样成为可能。我们在各种结构化的文本生成任务中评估了一系列$f$-集成，并突出了与传统的概率平均策略相比，其他聚合策略的优势。我们还展示了更好的后验估计可以提高集成性能。 简单来说，这项研究提出了一种新的方法来将多个语言模型合并成一个更强大的系统，通过这种方法可以改善文本生成的质量和多样性。该框架不仅能够处理不同词汇表的模型，还可以灵活地根据不同的目标调整采样策略，从而在多种任务上表现出优越的结果。
URL
https://arxiv.org/abs/2603.05432
PDF
-
Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
2026-03-05 17:35:59Jielin Qiu, Zixiang Chen, Liangwei Yang, Ming Zhu, Zhiwei Liu, Juntao Tan, Wenting Zhao, Rithesh Murthy, Roshan Ram, Akshara Prabhakar, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Huan WangAbstract
We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While over 25 open-source speech-to-speech models and numerous voice agent frameworks exist, no single resource explains the complete pipeline from individual components to a working streaming voice agent with function calling capabilities. Through systematic investigation, we find that (1) native speech-to-speech models like Qwen2.5-Omni, while capable of high-quality audio generation, are too slow for realtime interaction ($\sim$13s time-to-first-audio); (2) the industry-standard approach uses a cascaded streaming pipeline: STT $\rightarrow$ LLM $\rightarrow$ TTS, where each component streams its output to the next; and (3) the key to ``realtime'' is not any single fast model but rather \textit{streaming and pipelining} across components. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured P50 time-to-first-audio of 947ms (best case 729ms) with cloud LLM APIs, and comparable latency with self-hosted vLLM on NVIDIA A10G GPU. We release the full codebase as a tutorial with working, tested code for every component.
Abstract (translated)
我们提供了一个从基础原理构建企业级实时语音代理的技术教程。尽管存在超过25个开源的语音到语音模型和许多语音代理框架，但没有单一资源能够解释从组件到具有功能调用能力的工作流式语音代理的完整流程。通过系统性研究，我们发现： 1. 本机语音转语音模型（如Qwen2.5-Omni）虽然能生成高质量音频，但由于处理速度过慢而不适合实时交互（约需13秒才能产生首个音频）； 2. 行业标准的方法采用级联流式管道：STT → LLM → TTS，其中每个组件将其输出流传输给下一个组件；以及 3. 实现“实时”的关键不在于单一的快速模型，而在于各组件之间的\textit{流处理和流水线}技术。 我们使用Deepgram（流式语音转文本），由vLLM服务的语言模型（具有功能调用的流式文本生成）和ElevenLabs（流式文本到语音转换），构建了一个完整的语音代理。在使用云端语言模型API时，我们的方案实现了中位数为947毫秒、最佳情况为729毫秒的时间至首次音频输出，并且与自托管vLLM在NVIDIA A10G GPU上的延迟相近。 我们将整个代码库作为教程发布，包括每个组件经过测试的工作代码。
URL
https://arxiv.org/abs/2603.05413
PDF
-
HiFlow: Hierarchical Feedback-Driven Optimization for Constrained Long-Form Text Generation
2026-03-05 09:41:39Yifan Zhu, Guanting Chen, Bing Wei, Haoran LuoAbstract
Large language models perform well in short text generation but still struggle with long text generation, particularly under complex constraints. Such tasks involve multiple tightly coupled objectives, including global structural consistency, local semantic coherence, and constraint feasibility, forming a challenging constrained optimization problem. Existing approaches mainly rely on static planning or offline supervision, limiting effective coordination between global and local objectives during generation. To address these challenges, we propose HiFlow, a hierarchical feedback-driven optimization framework for constrained long text generation. HiFlow formulates generation as a two-level optimization process, consisting of a planning layer for global structure and constraint modeling, and a generation layer for conditioned text generation. By incorporating constraint-aware plan screening and closed-loop feedback at both levels, HiFlow enables joint optimization of planning quality and generation behavior, progressively guiding the model toward high-quality, constraint-satisfying outputs. Experiments on multiple backbones confirm HiFlow's effectiveness over baseline methods.
Abstract (translated)
大型语言模型在短文本生成方面表现出色，但在长文本生成方面仍然面临挑战，尤其是在存在复杂约束的情况下。这类任务涉及多个紧密耦合的目标，包括全局结构一致性、局部语义连贯性和约束可行性，形成了一个具有挑战性的受限优化问题。现有的方法主要依赖于静态规划或离线监督，这限制了在生成过程中全局和局部目标的有效协调。为了解决这些挑战，我们提出了HiFlow，这是一种分层反馈驱动的优化框架，用于受约束的长文本生成。HiFlow将生成过程表述为一个两层次的优化过程，包括用于全局结构和约束建模的规划层以及用于条件文本生成的生成层。通过在两个层面中加入考虑约束的计划筛选与闭环反馈机制，HiFlow能够同时优化规划质量和生成行为，逐步引导模型产生高质量且符合约束要求的输出。实验证明了HiFlow相对于基线方法的有效性。
URL
https://arxiv.org/abs/2603.04996
PDF
-
Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models
2026-03-05 07:35:07Sean Lamont, Christian Walder, Paul Montague, Amir Dezfouli, Michael NorrishAbstract
Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@$k$ problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead, while ensuring that each sample contributes a unique perspective to the batch. We evaluate our method on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model. Our results demonstrate significantly improved diversity and Pass@$k$ performance across various temperature settings. As a simple modification to the sampling process, our method offers an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search. We make our code available at this https URL.
Abstract (translated)
在文本生成中，多样化的输出对于复杂的推理任务（如代码生成和数学问题解决）的有效探索是必要的。这类Pass@$k$问题从覆盖解决方案空间的多个候选方案中受益。然而，传统的采样方法常常浪费计算资源在重复失败模式上。尽管扩散语言模型作为主流自回归范式的一个竞争性替代方案已出现，但它们仍然容易受到冗余的影响，独立样本往往会收敛到相似的状态模式。为了解决这一问题，我们提出了一种无需训练、低成本的干预措施，以增强扩散语言模型在生成过程中的多样性。我们的方法对一批中间样本进行顺序修改，每个样本都从之前样本特征空间中被排斥，从而主动惩罚冗余现象。与需要重新训练或使用束搜索的方法不同，我们的策略几乎不增加计算开销，并确保每一份样本都能为批量贡献独特的视角。 我们在LLaDA-8B-Instruct模型上对HumanEval和GSM8K基准进行了方法评估，结果显示在各种温度设置下显著提高了多样性和Pass@$k$性能。作为对采样过程的一个简单修改，我们的方法能够立即且低成本地改善当前及未来扩散语言模型在这类需要多样化解决方案搜索的任务中的表现。 我们将在[此处提供URL]分享代码以供访问。
URL
https://arxiv.org/abs/2603.04893
PDF
-
Agentics 2.0: Logical Transduction Algebra for Agentic Data Workflows
2026-03-04 16:30:01Alfio Massimiliano Gliozzo, Junkyu Lee, Nahuel DefosseAbstract
Agentic AI is rapidly transitioning from research prototypes to enterprise deployments, where requirements extend to meet the software quality attributes of reliability, scalability, and observability beyond plausible text generation. We present Agentics 2.0, a lightweight, Python-native framework for building high-quality, structured, explainable, and type-safe agentic data workflows. At the core of Agentics 2.0, the logical transduction algebra formalizes a large language model inference call as a typed semantic transformation, which we call a transducible function that enforces schema validity and the locality of evidence. The transducible functions compose into larger programs via algebraically grounded operators and execute as stateless asynchronous calls in parallel in asynchronous Map-Reduce programs. The proposed framework provides semantic reliability through strong typing, semantic observability through evidence tracing between slots of the input and output types, and scalability through stateless parallel execution. We instantiate reusable design patterns and evaluate the programs in Agentics 2.0 on challenging benchmarks, including DiscoveryBench for data-driven discovery and Archer for NL-to-SQL semantic parsing, demonstrating state-of-the-art performance.
Abstract (translated)
代理型AI正在迅速从研究原型转向企业部署，其需求不仅限于可能的文字生成，还要求满足软件质量属性，如可靠性、可扩展性和可观测性。我们介绍Agentics 2.0，这是一个轻量级的Python原生框架，用于构建高质量、结构化、解释性强和类型安全的代理数据工作流。 在Agentics 2.0的核心中，逻辑变换代数将大型语言模型的推理调用形式化为一个有类型的语义转换，我们将其称为可变函数。这种可变函数强制执行模式的有效性和证据的地方性。通过基于代数的操作符，这些可变函数可以组合成更大的程序，并在异步Map-Reduce程序中以无状态的方式并行执行。 该框架提供了通过强类型提供的语义可靠性、通过输入和输出类型的槽之间证据跟踪的语义可观测性以及通过无状态并行执行的扩展性。我们实现了可重用的设计模式并在Agentics 2.0的程序上进行了评估，包括用于数据驱动发现的DiscoveryBench和用于自然语言到SQL语义解析的Archer等具有挑战性的基准测试，展示了最先进的性能。
URL
https://arxiv.org/abs/2603.04241
PDF
-
BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
2026-03-04 16:11:49Tommaso Boccato, Michal Olak, Matteo FerranteAbstract
Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13321
PDF
-
Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference
2026-03-04 15:20:55Patrick Wilhelm, Thorsten Wittkopp, Odej KaoAbstract
Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs, particularly in request-heavy scenarios. In many real-world applications, the full scale and capabilities of LLMs are often unnecessary, as Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks. When enhanced with advanced reasoning strategies, such as Chain-of-Thought (CoT) prompting or Majority Voting, SLMs can approach the performance of larger models while reducing overall computational requirements. However, these strategies can also introduce additional energy costs, creating an energy-accuracy trade-off. Our analysis examines these trade-offs in test-time compute strategies for smaller models compared to larger ones, using the MMLU benchmark. Additionally, we explore the input-output token dynamics of transformer architectures, which result in nonlinear hardware energy operation curves for LLMs. To bridge AI research with its physical impact, we propose \textit{energy efficiency metrics}, including Energy-per-Token, as complements to traditional accuracy benchmarks. Beyond model selection, we propose controlled reasoning in CoT token generation, using operating curves to regulate reasoning depth dynamically. This vision integrates a energy-aware routing mechanism, ensuring that model selection and inference strategies balance accuracy for sustainable AI deployment.
Abstract (translated)
大型语言模型（LLMs）在各类任务中展现出卓越性能，但其高昂的能耗与计算成本在请求密集场景中尤为突出。在实际应用中，LLM的完整规模与能力往往并不必要，因为小型语言模型（SLMs）已能针对简单文本生成任务提供准确响应。当SLMs结合思维链（CoT）提示或多数投票等先进推理策略时，其性能可逼近大型模型，同时降低总体计算需求。然而，这些策略也可能带来额外能耗，形成能耗-精度权衡。本研究通过MMLU基准测试，对比分析了小型与大型模型在测试时计算策略中的此类权衡。此外，我们探讨了Transformer架构的输入输出令牌动态特性，该特性导致LLMs呈现非线性硬件能耗运行曲线。为连接AI研究与物理影响，我们提出将包括每令牌能耗在内的能效指标作为传统精度基准的补充。除模型选择外，我们还提出在CoT令牌生成中实施可控推理，利用运行曲线动态调节推理深度。这一愿景整合了能感知能耗的路由机制，确保模型选择与推理策略在精度与可持续性部署间取得平衡。
URL
https://arxiv.org/abs/2603.20224
PDF
-
ScribeTokens: Fixed-Vocabulary Tokenization of Digital Ink
2026-03-03 09:45:49Douglass WangAbstract
Digital ink -- the coordinate stream captured from stylus or touch input -- lacks a unified representation. Continuous vector representations produce long sequences and suffer from training instability, while existing token representations require large vocabularies, face out-of-vocabulary issues, and underperform vectors on recognition. We propose ScribeTokens, a tokenization that decomposes pen movement into unit pixel steps. Together with two pen-state tokens, this fixed 10-token base vocabulary suffices to represent any digital ink and enables aggressive BPE compression. On handwritten text generation, ScribeTokens dramatically outperforms vectors (17.33% vs. 70.29% CER), showing tokens are far more effective for generation. On recognition, ScribeTokens is the only token representation to outperform vectors without pretraining. We further introduce next-ink-token prediction as a self-supervised pretraining strategy, which consistently improves recognition across all token-based models and accelerates convergence by up to 83x. With pretraining, ScribeTokens achieves the best recognition results across all representations on both datasets (8.27% CER on IAM, 9.83% on DeepWriting).
Abstract (translated)
数字墨水——从笔或触摸输入捕获的坐标流——缺乏统一的表现形式。连续向量表示会产生长序列并导致训练不稳定，而现有的标记表示则需要庞大的词汇表、面临词外问题，并且在识别方面不如向量表现好。我们提出了ScribeTokens，这是一种将笔迹运动分解为单像素步进的分词方法。结合两个笔状态标记，这种固定10个标记的基础词汇足以代表任何数字墨水，并能实现激进的BPE压缩。在手写文本生成任务中，ScribeTokens显著优于向量表示（CER误差率为17.33%，而向量为70.29%），表明标记对于生成过程更为有效。在识别方面，ScribeTokens是唯一一种无需预训练即能超越向量表现的标记表示形式。我们进一步引入了下一笔标记预测作为自监督预训练策略，这种方法在所有基于标记的模型中都能一致地提升识别效果，并最多可加快83倍的收敛速度。通过预训练，ScribeTokens在两个数据集的所有表示方法中均取得了最佳识别结果（IAM上的CER误差率为8.27%，DeepWriting上的为9.83%）。
URL
https://arxiv.org/abs/2603.02805
PDF
-
A Directed Graph Model and Experimental Framework for Design and Study of Time-Dependent Text Visualisation
2026-03-02 22:06:31Songhai Fan, Simon Angus, Tim Dwyer, Ying Yang, Sarah Goodwin, Helen PurchaseAbstract
Exponential growth in the quantity of digital news, social media, and other textual sources makes it difficult for humans to keep up with rapidly evolving narratives about world events. Various visualisation techniques have been touted to help people to understand such discourse by exposing relationships between texts (such as news articles) as topics and themes evolve over time. Arguably, the understandability of such visualisations hinges on the assumption that people will be able to easily interpret the relationships in such visual network structures. To test this assumption, we begin by defining an abstract model of time-dependent text visualisation based on directed graph structures. From this model we distill motifs that capture the set of possible ways that texts can be linked across changes in time. We also develop a controlled synthetic text generation methodology that leverages the power of modern LLMs to create fictional, yet structured sets of time-dependent texts that fit each of our patterns. Therefore, we create a clean user study environment (n=30) for participants to identify patterns that best represent a given set of synthetic articles. We find that it is a challenging task for the user to identify and recover the predefined motif. We analyse qualitative data to map an unexpectedly rich variety of user rationales when divergences from expected interpretation occur. A deeper analysis also points to unexpected complexities inherent in the formation of synthetic datasets with LLMs that undermine the study control in some cases. Furthermore, analysis of individual decision-making in our study hints at a future where text discourse visualisation may need to dispense with a one-size-fits-all approach and, instead, should be more adaptable to the specific user who is exploring the visualisation in front of them.
Abstract (translated)
数字新闻、社交媒体和其他文本来源的数量呈指数级增长，这使得人类难以跟上关于世界事件的快速演变叙述。各种可视化技术被提出，以帮助人们通过揭示不同时间点之间文章（如新闻报道）之间的关系来理解这样的讨论。然而，此类可视化的可理解性假设是人们能够轻松地解读这些网络结构中的关系。 为了检验这一假设，我们首先定义了一个基于有向图结构的时间依赖文本可视化的抽象模型，并从中提炼出一系列模式，这些模式捕捉了在时间变化中文章之间可能的链接方式。同时，我们开发了一种受现代大型语言模型（LLMs）启发的合成文本生成方法论，能够创建一套符合每一种模式的虚构但结构化的时间相关文本集。 因此，我们为参与者创造了一个清洁的用户研究环境（n=30），让他们识别一组合成文章中最佳代表特定模式的方式。我们的发现表明，用户很难准确地识别并恢复预定义的模式。通过分析定性数据，我们绘制了当用户的解读与预期解释出现偏差时所采用的各种丰富的用户逻辑。 此外，对生成合成数据集过程中使用LLMs的深入分析揭示了一些意外复杂性和挑战，这在某些情况下影响了研究的控制性。最后，我们在研究中关于个人决策过程的分析暗示了一个未来方向，在这个未来中，文本讨论可视化可能需要摒弃“一刀切”的方法，并且应该更加灵活地适应正在探索可视化的具体用户需求。
URL
https://arxiv.org/abs/2603.02422
PDF
-
LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval
2026-03-02 04:11:18Jiajie Jin, Yanzhao Zhang, Mingxin Li, Dingkun Long, Pengjun Xie, Yutao Zhu, Zhicheng DouAbstract
LLMs have fundamentally transformed dense retrieval, upgrading backbones from discriminative encoders to generative architectures. However, a critical disconnect remains: while LLMs possess strong reasoning capabilities, current retrievers predominantly utilize them as static encoders, leaving their potential for complex reasoning unexplored. To address this, existing approaches typically adopt rewrite-then-retrieve pipelines to generate explicit CoT rationales before retrieval. However, this incurs prohibitive latency. In this paper, we propose LaSER, a novel self-distillation framework that internalizes explicit reasoning into the latent space of dense retrievers. Operating on a shared LLM backbone, LaSER introduces a dual-view training mechanism: an Explicit view that explicitly encodes ground-truth reasoning paths, and a Latent view that performs implicit latent thinking. To bridge the gap between these views, we design a multi-grained alignment strategy. Beyond standard output alignment, we introduce a trajectory alignment mechanism that synchronizes the intermediate latent states of the latent path with the semantic progression of the explicit reasoning segments. This allows the retriever to think silently and effectively without autoregressive text generation. Extensive experiments on both in-domain and out-of-domain reasoning-intensive benchmarks demonstrate that LaSER significantly outperforms state-of-the-art baselines. Furthermore, analyses across diverse backbones and model scales validate the robustness of our approach, confirming that our unified learning framework is essential for eliciting effective latent thinking. Our method successfully combines the reasoning depth of explicit CoT pipelines with the inference efficiency of standard dense retrievers.
Abstract (translated)
大型语言模型（LLMs）从根本上改变了密集检索，将骨干从判别式编码器升级为生成性架构。然而，一个关键的脱节依然存在：尽管LLMs具备强大的推理能力，当前的检索器主要将其用作静态编码器，未充分利用其进行复杂推理的能力。为了应对这一问题，现有方法通常采用重写后再检索管道，在检索之前生成明确的原因链条（CoT）。然而，这会导致不可接受的时间延迟。在本文中，我们提出了LaSER，一种新的自我蒸馏框架，将显式的推理内部化到密集检索器的潜在空间中。基于共享的大规模语言模型骨干结构，LaSER引入了双视图训练机制：明确视图（Explicit view），该视图显式编码真实的推理路径；以及隐含视图（Latent view），该视图执行隐性的潜在思考。为了弥合这两种视角之间的差距，我们设计了一种多粒度对齐策略。除标准输出对齐外，我们还引入了轨迹对齐机制，以同步隐性路径中的中间潜在状态与显式推理片段的语义进展。这使得检索器能够在不进行自回归文本生成的情况下有效地进行无声思考。 在行业内和行业外密集推理基准上的广泛实验表明，LaSER显著优于现有的最先进基线方法。此外，在不同骨干结构和模型规模下的分析验证了我们方法的强大性能，确认统一学习框架对于提取有效的潜在思维至关重要。我们的方法成功地将显式CoT管道的推理深度与标准密集检索器的推断效率相结合。
URL
https://arxiv.org/abs/2603.01425
PDF
-
Reviewing the Reviewer: Graph-Enhanced LLMs for E-commerce Appeal Adjudication
2026-02-27 00:43:59Yuchen Du, Ashley Li, Zixi HuangAbstract
Hierarchical review workflows, where a second-tier reviewer (Checker) corrects first-tier (Maker) decisions, generate valuable correction signals that encode why initial judgments failed. However, learning from these signals is hindered by information asymmetry: corrections often depend on verification actions unavailable to Makers or automated systems. We address this challenge by introducing explicit action modeling as an inferential constraint that grounds reasoning in verifiable operations rather than unconstrained text generation. We propose the Evidence-Action-Factor-Decision (EAFD) schema, a minimal representation for adjudication reasoning that prevents hallucination through operational grounding and enables learning from correction signals via explicit conflict modeling. Building on this schema, we develop a conflict-aware graph reasoning framework that: (1) constructs EAFD graphs from historical cases capturing Maker-Checker disagreements, (2) aggregates them into a retrievable knowledge base, and (3) performs top-down deductive reasoning for new cases by projecting validated resolution paths from precedents. A distinctive capability is the Request More Information (RMI) outcome: when evidence is insufficient, the system identifies precisely which verification actions remain unexecuted and generates targeted information requests. We evaluate the framework in large-scale e-commerce seller appeal adjudication. While a standard LLM-only baseline achieves only 70.8% alignment with human experts, incorporating action modeling with RMI improves alignment to 87.5%. Augmenting this with the retrieval-based knowledge graph yields the best offline performance of 95.8%. Following online deployment, the framework maintains robust performance, achieving a 96.3% alignment rate in production, demonstrating its real-world effectiveness.
Abstract (translated)
分层审查工作流中，二级审核者（Checker）对一级决策者（Maker）的判定进行修正，会生成编码初始判断为何失败的有价值修正信号。然而，从这些信号中学习受到信息不对称的阻碍：修正往往依赖于制作者或自动化系统无法获取的验证操作。我们通过引入显式操作建模作为推理约束来解决这一挑战，将推理锚定在可验证操作而非无约束的文本生成中。我们提出了证据-行动-因素-决策（EAFD）模式，这是一种用于裁决推理的最小化表示，通过操作锚定防止幻觉，并通过显式冲突建模实现从修正信号中学习。基于该模式，我们开发了一个冲突感知图推理框架，该框架：(1) 从历史案例中构建捕获制作者-审核者分歧的EAFD图，(2) 将其聚合为可检索知识库，(3) 通过从先例投影已验证的解决方案路径，对新案例执行自上而下的演绎推理。一个独特能力是“申请补充信息”（RMI）结果：当证据不足时，系统能精确识别哪些验证操作尚未执行，并生成针对性的信息请求。我们在大规模电商卖家申诉裁决中评估该框架。标准纯LLM基线仅能达到70.8%的人类专家对齐率，而结合操作建模与RMI将对齐率提升至87.5%。再叠加基于检索的知识图谱，离线性能最优达95.8%。在线部署后，该框架保持稳健表现，生产环境中达到96.3%的对齐率，证明了其现实有效性。
URL
https://arxiv.org/abs/2603.19267
PDF
-
FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records
2026-02-26 20:14:21Michael Frew, Nishit Bheda, Bryan TrippAbstract
Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. In this work, we introduce FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. We propose a text-to-FHIRPath QA paradigm that shifts reasoning from free-text generation to FHIRPath query synthesis, significantly reducing LLM usage. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Further, we demonstrate that state-of-the-art LLMs struggle to deal with ambiguity in patient language and perform poorly in FHIRPath query synthesis. However, they benefit strongly from supervised fine-tuning. Our results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and our dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code is available at: this https URL.
Abstract (translated)
尽管患者越来越多地通过数字方式访问其电子健康记录（EHR），但现有的界面可能无法支持对特定患者的精准、可靠的问题回答。大型语言模型（LLM）在临床问答（QA）方面显示出潜力，但由于检索式方法计算效率低、容易产生错误信息并且难以部署到现实生活中的EHR上，因此存在局限性。在这项工作中，我们介绍了FHIRPath-QA，这是第一个公开的数据集和基准测试工具，用于特定患者的问答，并包括在真实世界临床数据上的开放标准FHIRPath查询。我们提出了一种文本到FHIRPath QA的范式，这种范式将推理从自由文本生成转移到了FHIRPath查询合成上，从而大大减少了LLM的使用量。该数据集基于MIMIC-IV on FHIR Demo构建，包含超过14,000个自然语言问题（以患者和临床医生的语言表述），并附有验证过的FHIRPath查询与答案配对。此外，我们还展示了最先进的LLMs在处理患者的模糊表达方面存在困难，并且在生成FHIRPath查询时表现不佳。然而，在监督微调下，它们能够显著受益。 我们的研究结果表明，文本到FHIRPath的合成有可能成为安全、高效和互操作性的消费者健康应用的实际基础，并且我们提供的数据集和基准测试为未来在此领域进行的研究提供了起点。完整的数据集和生成代码可在以下链接获取：[此处插入URL]。
URL
https://arxiv.org/abs/2602.23479
PDF
-
MAPLE: Metadata Augmented Private Language Evolution
2026-02-26 14:18:36Eli Chien, Yuzheng Hu, Ryan McKenna, Shanshan Wu, Zheng Xu, Peter KairouzAbstract
While differentially private (DP) fine-tuning of large language models (LLMs) is a powerful tool, it is often computationally prohibitive or infeasible when state-of-the-art models are only accessible via proprietary APIs. In such settings, generating DP synthetic data has emerged as a crucial alternative, offering the added benefits of arbitrary reuse across downstream tasks and transparent exploratory data analysis without the opaque constraints of a model's parameter space. Private Evolution (PE) is a promising API-based framework for this goal; however, its performance critically depends on initialization. When the private data distribution deviates substantially from the foundation model's pre-training priors--particularly in highly specialized domains--PE frequently struggles to align with the target data, resulting in degraded utility, poor convergence, and inefficient API usage. To address this initialization bottleneck, we propose Metadata Augmented Private Language Evolution (MAPLE). MAPLE leverages differentially private tabular metadata extraction and in-context learning to effectively ground the initial synthetic distribution in the target domain. Extensive experiments on challenging, domain-specific text generation tasks demonstrate that MAPLE achieves a significantly more favorable privacy-utility trade-off, converges faster, and drastically reduces API costs compared to previous PE methods.
Abstract (translated)
尽管对大语言模型进行差分隐私微调是一项强大工具，但当先进模型仅能通过专有API访问时，该技术通常面临计算成本过高或不可行的问题。在此类场景下，生成差分隐私合成数据已成为关键替代方案，其额外优势包括：可跨下游任务任意复用，且能在不触碰模型参数空间不透明限制的前提下实现透明的探索性数据分析。私有演化（PE）是实现此目标的前景广阔的API驱动框架；然而其性能严重依赖初始化。当私有数据分布与基础模型的预训练先验存在显著偏差时——尤其在高度专业化领域——PE常难以与目标数据对齐，导致效用下降、收敛困难及API使用效率低下。为突破此初始化瓶颈，我们提出元数据增强私有语言演化（MAPLE）。MAPLE利用差分隐私表格元数据提取与上下文学习，有效将初始合成分布锚定于目标领域。在具有挑战性的领域特定文本生成任务上的大量实验表明，相较于先前PE方法，MAPLE实现了更优的隐私-效用权衡、更快收敛速度，并大幅降低API成本。
URL
https://arxiv.org/abs/2603.19258
PDF
-
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
2026-02-25 07:25:01Minhao Jiang, Zhikai Li, Xuewen Liu, Jing Zhang, Mengjuan Chen, Qingyi GuAbstract
Large language models have demonstrated capabilities in text generation, while their increasing parameter scales present challenges in computational and memory efficiency. Post-training sparsity (PTS), which reduces model cost by removing weights from dense networks, is an effective approach. However, native dense matrices lack high sparsity, making existing approaches that directly remove weights disrupt model states, resulting in unsatisfactory performance recovery even with post-tuning. We propose Sparsity Induction, which promotes models toward higher sparsity at both distribution and feature levels before pruning, to push the limits of PTS. At the distribution level, we enhance distributional sparsity through mathematically equivalent scaling transformations, which are fully absorbable and incur no extra parameters or inference-time overhead. At the feature level, we introduce Spectral Norm Loss to promote feature sparsity from a low-rank perspective. Experiments across diverse model architectures and tasks demonstrate that our method further enhances sparsity-friendliness, achieving superior pruning performance over existing approaches.
Abstract (translated)
大型语言模型在文本生成方面表现出强大的能力，但随着参数规模的增加，计算和内存效率问题也日益突出。后训练稀疏化（PTS）通过从密集网络中移除权重来降低模型成本，是一种有效的方法。然而，原有的密集矩阵缺乏高斯度特性，直接移除权重会破坏模型状态，即使经过后续调优也无法达到令人满意的性能恢复。我们提出了诱导稀疏性方法，该方法在剪枝前从分布和特征两个层面推动模型向更高的稀疏性发展，从而突破PTS的限制。 在分布层面上，我们通过数学等价的缩放变换增强分布特性上的稀疏性，这些变换可以完全吸收，并且不会增加额外参数或推理时间开销。在特征层面上，我们引入了谱范数损失（Spectral Norm Loss），从低秩角度促进特征稀疏性的形成。 跨多种模型架构和任务的实验表明，我们的方法进一步提高了模型对稀疏化的友好性，在剪枝性能方面超越了现有方法。
URL
https://arxiv.org/abs/2602.21652
PDF
