Text_Generation
-
Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
2026-04-11 08:52:08Jiang Li, Tian Lan, Shanshan Wang, Dongxing Zhang, Dianqing Lin, Guanglai Gao, Derek F. Wong, Xiangdong SuAbstract
The rapid development of large language models (LLMs) has extended text generation tasks into the literary domain. However, AI-generated literary creations has raised increasingly prominent issues of creative authenticity and ethics in literary world, making the detection of LLM-generated literary texts essential and urgent. While previous works have made significant progress in detecting AI-generated text, it has yet to address classical Chinese poetry. Due to the unique linguistic features of classical Chinese poetry, such as strict metrical regularity, a shared system of poetic imagery, and flexible syntax, distinguishing whether a poem is authored by AI presents a substantial challenge. To address these issues, we introduce ChangAn, a benchmark for detecting LLM-generated classical Chinese poetry that containing total 30,664 poems, 10,276 are human-written poems and 20,388 poems are generated by four popular LLMs. Based on ChangAn, we conducted a systematic evaluation of 12 AI detectors, investigating their performance variations across different text granularities and generation strategies. Our findings highlight the limitations of current Chinese text detectors, which fail to serve as reliable tools for detecting LLM-generated classical Chinese poetry. These results validate the effectiveness and necessity of our proposed ChangAn benchmark. Our dataset and code are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.10101
PDF
-
LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention
2026-04-11 05:54:45Dongjie Xu, Hao Wu, Weijie Shi, Yue Cui, Yuanjun Liu, Jiawei Li, Haolun Ma, An Liu, Jia Zhu, Jiajie XuAbstract
Through systematic experiments on long-context generation, we observe a damaging failure mode in which decoding can collapse into persistent repetition loops. We find that this degeneration is driven by collapsed attention patterns, where a subset of heads locks onto a narrow suffix of the history, and is further stabilized by inference-time KV cache reuse. Crucially, since many existing KV cache policies rely on attention-based importance, this collapse can produce spuriously high scores for repetitive tokens, causing cache management to inadvertently amplify repetition. To study this phenomenon in a controlled and reproducible manner, we introduce LoopBench, a benchmark with explicit loop-inducing conditions and loop-oriented metrics that quantify repetition severity and generation instability beyond downstream task scores. Building on these insights, we propose LoopGuard, a lightweight, plug-in KV cache guard that detects loop onset online and disrupts the feedback cycle by pruning repetitive tail spans under a fixed cache budget. Experiments on LoopBench show that LoopGuard reduces loop incidence by over 90 percentage points, while restoring output diversity and reducing token waste.
Abstract (translated)
URL
https://arxiv.org/abs/2604.10044
PDF
-
FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer
2026-04-11 04:32:59Shenghe Zheng, Minyu Zhang, Tianhao Liu, Hongzhi WangAbstract
With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.
Abstract (translated)
URL
https://arxiv.org/abs/2604.10023
PDF
-
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
2026-04-09 11:41:58Luozheng Qin, Jia Gong, Qian Qiao, Tianjiao Li, Li Xu, Haoyu Pan, Chao Qu, Zhiyu Tan, Hao LiAbstract
Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.08121
PDF
-
iTAG: Inverse Design for Natural Text Generation with Accurate Causal Graph Annotations
2026-04-08 09:59:45Wenshuo Wang, Boyu Cao, Nan Zhuang, Wei LiAbstract
A fundamental obstacle to causal discovery from text is the lack of causally annotated text data for use as ground truth, due to high annotation costs. This motivates an important task of generating text with causal graph annotations. Early template-based generation methods sacrifice text naturalness in exchange for high causal graph annotation accuracy. Recent Large Language Model (LLM)-dependent methods directly generate natural text from target graphs through LLMs, but do not guarantee causal graph annotation accuracy. Therefore, we propose iTAG, which performs real-world concept assignment to nodes before converting causal graphs into text in existing LLM-dependent methods. iTAG frames this process as an inverse problem with the causal graph as the target, iteratively examining and refining concept selection through Chain-of-Thought (CoT) reasoning so that the induced relations between concepts are as consistent as possible with the target causal relationships described by the causal graph. iTAG demonstrates both extremely high annotation accuracy and naturalness across extensive tests, and the results of testing text-based causal discovery algorithms with the generated data show high statistical correlation with real-world data. This suggests that iTAG-generated data can serve as a practical surrogate for scalable benchmarking of text-based causal discovery algorithms.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06902
PDF
-
Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM
2026-04-08 08:50:08Chengyue Wu, Shiyi Lan, Yonggan Fu, Sensen Gao, Jin Wang, Jincheng Yu, Jose M. Alvarez, Pavlo Molchanov, Ping Luo, Song Han, Ligeng Zhu, Enze XieAbstract
Vision-language models (VLMs) predominantly rely on autoregressive decoding, which generates tokens one at a time and fundamentally limits inference throughput. This limitation is especially acute in physical AI scenarios such as robotics and autonomous driving, where VLMs are deployed on edge devices at batch size one, making AR decoding memory-bandwidth-bound and leaving hardware parallelism underutilized. While block-wise discrete diffusion has shown promise for parallel text generation, extending it to VLMs remains challenging due to the need to jointly handle continuous visual representations and discrete text tokens while preserving pretrained multimodal capabilities. We present Fast-dVLM, a block-diffusion-based VLM that enables KV-cache-compatible parallel decoding and speculative block decoding for inference acceleration. We systematically compare two AR-to-diffusion conversion strategies: a two-stage approach that first adapts the LLM backbone with text-only diffusion fine-tuning before multimodal training, and a direct approach that converts the full AR VLM in one stage. Under comparable training budgets, direct conversion proves substantially more efficient by leveraging the already multimodally aligned VLM; we therefore adopt it as our recommended recipe. We introduce a suite of multimodal diffusion adaptations, block size annealing, causal context attention, auto-truncation masking, and vision efficient concatenation, that collectively enable effective block diffusion in the VLM setting. Extensive experiments across 11 multimodal benchmarks show Fast-dVLM matches its autoregressive counterpart in generation quality. With SGLang integration and FP8 quantization, Fast-dVLM achieves over 6x end-to-end inference speedup over the AR baseline.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06832
PDF
-
AGSC: Adaptive Granularity and Semantic Clustering for Uncertainty Quantification in Long-text Generation
2026-04-08 08:25:01Guanran Luo, Wentao Qiu, Wanru Zhao, Wenhan Lv, Zhongquan Jian, Meihong Wang, Qingqiang WuAbstract
Large Language Models (LLMs) have demonstrated impressive capabilities in long-form generation, yet their application is hindered by the hallucination problem. While Uncertainty Quantification (UQ) is essential for assessing reliability, the complex structure makes reliable aggregation across heterogeneous themes difficult, in addition, existing methods often overlook the nuance of neutral information and suffer from the high computational cost of fine-grained decomposition. To address these challenges, we propose AGSC (Adaptive Granularity and GMM-based Semantic Clustering), a UQ framework tailored for long-form generation. AGSC first uses NLI neutral probabilities as triggers to distinguish irrelevance from uncertainty, reducing unnecessary computation. It then applies Gaussian Mixture Model (GMM) soft clustering to model latent semantic themes and assign topic-aware weights for downstream aggregation. Experiments on BIO and LongFact show that AGSC achieves state-of-the-art correlation with factuality while reducing inference time by about 60% compared to full atomic decomposition.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06812
PDF
-
AgentGate: A Lightweight Structured Routing Engine for the Internet of Agents
2026-04-08 05:22:16Yujun Cheng, Enfang Cui, Hao Qin, Zhiyuan Liang, Qi XuAbstract
The rapid development of AI agent systems is leading to an emerging Internet of Agents, where specialized agents operate across local devices, edge nodes, private services, and cloud platforms. Although recent efforts have improved agent naming, discovery, and interaction, efficient request dispatch remains an open systems problem under latency, privacy, and cost constraints. In this paper, we present AgentGate, a lightweight structured routing engine for candidate-aware agent dispatch. Instead of treating routing as unrestricted text generation, AgentGate formulates it as a constrained decision problem and decomposes it into two stages: action decision and structural grounding. The first stage determines whether a query should trigger single-agent invocation, multi-agent planning, direct response, or safe escalation, while the second stage instantiates the selected action into executable outputs such as target agents, structured arguments, or multi-step plans. To adapt compact models to this setting, we further develop a routing-oriented fine-tuning scheme with candidate-aware supervision and hard negative examples. Experiments on a curated routing benchmark with several 3B--7B open-weight models show that compact models can provide competitive routing performance in constrained settings, and that model differences are mainly reflected in action prediction, candidate selection, and structured grounding quality. These results indicate that structured routing is a feasible design point for efficient and privacy-aware agent systems, especially when routing decisions must be made under resource-constrained deployment conditions.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06696
PDF
-
SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future
2026-04-07 20:17:49Timothy B. Higgins, Antonios Mamalakis, Chirag AgarwalAbstract
Recent advances in visual-language models (VLMs) have led to significant improvements in a plethora of complex multimodal tasks like image captioning, report generation, and visual perception. However, generating text from meteorological data is highly challenging because the atmosphere is a chaotic system that is rapidly changing at various spatial and temporal scales. Given the complexity of atmospheric phenomena, it is critical to verifiably quantify the effectiveness of existing VLMs on weather forecasting data. In this work, we present SynopticBench, a high-quality dataset consisting of 1,367,041 text samples of Area Forecast Discussions created by the National Weather Service over the continental United States paired to images of 500mb geopotential height, 2 meter temperature, and 850mb wind velocity in weather forecasts. We also present Synoptic Phenomena Alignment and Coverage Evaluation (SPACE), a novel evaluation framework that can be used to effectively estimate the quality of text descriptions of synoptic weather phenomena. Extensive experiments on generating forecast discussions using state-of-the-art VLMs show the sensitivity of existing evaluation metrics in this domain and enable further exploration into synoptic weather and climate text generation.
Abstract (translated)
URL
https://arxiv.org/abs/2604.16451
PDF
-
A Severity-Based Curriculum Learning Strategy for Arabic Medical Text Generation
2026-04-07 18:41:53Ahmed Alansary, Molham Mohamed, Ali HamdiAbstract
Arabic medical text generation is increasingly needed to help users interpret symptoms and access general health guidance in their native language. Nevertheless, many existing methods assume uniform importance across training samples, overlooking differences in clinical severity. This simplification can hinder the model's ability to properly capture complex or high-risk cases. To overcome this issue, this work introduces a Severity-based Curriculum Learning Strategy for Arabic Medical Text Generation, where the training process is structured to move gradually from less severe to more critical medical conditions. The approach divides the dataset into ordered stages based on severity and incrementally exposes the model to more challenging cases during fine-tuning, allowing it to first learn basic medical patterns before addressing more complex scenarios. The proposed method is evaluated on a subset of the Medical Arabic Question Answering (MAQA) dataset, which includes Arabic medical questions describing symptoms alongside corresponding responses. In addition, the dataset is annotated with three severity levels (Mild, Moderate, and Critical) using a rule-based method developed in this study. The results demonstrate that incorporating severity-aware curriculum learning leads to consistent performance improvements across all tested models, with gains of around +4% to +7% over baseline models and +3% to +6% compared with conventional fine-tuning approaches.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06365
PDF
-
Severity-Aware Weighted Loss for Arabic Medical Text Generation
2026-04-07 18:26:41Ahmed Alansary, Molham Mohamed, Ali HamdiAbstract
Large language models have shown strong potential for Arabic medical text generation; however, traditional fine-tuning objectives treat all medical cases uniformly, ignoring differences in clinical severity. This limitation is particularly critical in healthcare settings, where errors in severe cases contain higher clinical risk. In this work, we propose a severity-aware weighted loss for fine-tuning Arabic language models on medical complaint-response data. The method depends on soft severity probabilities to dynamically scale token-level loss contributions during optimization, thereby prioritizing clinically critical interactions without modifying model architectures. Experiments are conducted using the MAQA dataset, which provides Arabic medical complaints and trusted human responses. Severity labels and probabilistic scores are automatically derived using a fine-tuned AraBERT-based classifier and incorporated exclusively at the loss level. The proposed approach is evaluated across ten Arabic large language models of varying architectures and parameter scales. While standard cross-entropy fine-tuning yields only modest improvements, severity-aware optimization consistently achieves larger gains. Using a balanced weighting configuration, performance improves from 54.04% to 66.14% for AraGPT2-Base, from 59.16% to 67.18% for AraGPT2-Medium, and from 57.83% to 66.86% for Qwen2.5-0.5B, with peak performance reaching 67.18%. Overall, severity-aware fine-tuning delivers improvements of up to 12.10% over non-fine-tuned baselines, demonstrating robust and architecture-consistent gains.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06346
PDF
-
PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer
2026-04-07 17:40:37David Picard, Nicolas Dufour, Lucas Degeorge, Arijit Ghosh, Davide Allegro, Tom Ravaud, Yohann Perron, Corentin Sautier, Zeynep Sonat Baltaci, Fei Meng, Syrine Kalleli, Marta L\'opez-Rauhut, Thibaut Loiseau, S\'egol\`ene Albouy, Raphael Baena, Elliot Vincent, Loic LandrieuAbstract
This paper introduces the Polynomial Mixer (PoM), a novel token mixing mechanism with linear complexity that serves as a drop-in replacement for self-attention. PoM aggregates input tokens into a compact representation through a learned polynomial function, from which each token retrieves contextual information. We prove that PoM satisfies the contextual mapping property, ensuring that transformers equipped with PoM remain universal sequence-to-sequence approximators. We replace standard self-attention with PoM across five diverse domains: text generation, handwritten text recognition, image generation, 3D modeling, and Earth observation. PoM matches the performance of attention-based models while drastically reducing computational cost when working with long sequences. The code is available at this https URL.
Abstract (translated)
本文介绍了多项式混合器（PoM），这是一种具有线性复杂度的新型令牌混合机制，可作为自注意力机制的即插式替代方案。PoM通过一个学习到的多项式函数将输入令牌聚合为紧凑表示，每个令牌再从中检索上下文信息。我们证明PoM满足上下文映射属性，确保配备PoM的变换器仍保持通用序列到序列逼近能力。我们在五个不同领域用PoM替换了标准自注意力机制：文本生成、手写文本识别、图像生成、3D建模和地球观测。PoM在长序列处理中匹配了基于注意力模型的性能，同时大幅降低了计算成本。代码已公开于该链接。
URL
https://arxiv.org/abs/2604.06129
PDF
-
UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning
2026-04-07 07:15:28Xiaolong Wei, Zerun Zhu, Simin Niu, Xingyu Zhang, Peiying Yu, Changxuan Xiao, Yuchen Li, Jicheng Yang, Zhejun Zhao, Chong Meng, Long Xia, Daiting ShiAbstract
A fundamental challenge in creative writing lies in reconciling the inherent tension between maintaining global coherence in long-form narratives and preserving local expressiveness in short-form texts. While long-context generation necessitates explicit macroscopic planning, short-form creativity often demands spontaneous, constraint-free expression. Existing alignment paradigms, however, typically employ static reward signals and rely heavily on high-quality supervised data, which is costly and difficult to scale. To address this, we propose \textbf{UniCreative}, a unified reference-free reinforcement learning framework. We first introduce \textbf{AC-GenRM}, an adaptive constraint-aware reward model that dynamically synthesizes query-specific criteria to provide fine-grained preference judgments. Leveraging these signals, we propose \textbf{ACPO}, a policy optimization algorithm that aligns models with human preferences across both content quality and structural paradigms without supervised fine-tuning and ground-truth references. Empirical results demonstrate that AC-GenRM aligns closely with expert evaluations, while ACPO significantly enhances performance across diverse writing tasks. Crucially, our analysis reveals an emergent meta-cognitive ability: the model learns to autonomously differentiate between tasks requiring rigorous planning and those favoring direct generation, validating the effectiveness of our direct alignment approach.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05517
PDF
-
Top-K Retrieval with Fixed-Size Linear-Attention Completion: Backbone- and KV-Format-Preserving Attention for KV-Cache Read Reduction
2026-04-07 05:19:54Yasuto Hoshi, Daisuke Miyashita, Jun DeguchiAbstract
Long-context generation is increasingly limited by decode-time key-value (KV) cache traffic, particularly when KV is offloaded beyond GPU memory. Query-aware retrieval (e.g., Top-K selection) reduces this traffic by loading only a subset of KV pairs, but renormalizing the softmax over the subset introduces bias when attention mass is spread over unretrieved tokens. We propose a retrieval-completion attention module that keeps backbone weights and the KV-cache format unchanged. For each query, we compute exact attention over sink/tail anchors and the query-dependent retrieved Top-K tokens, and estimate the remaining mid-region numerator and denominator using a fixed-size feature-map summary computed at prefill time. We add the exact and estimated contributions in the unnormalized domain and apply a single normalization, recovering the missing softmax mass without additional attention-side KV reads. Across long-context benchmarks, the proposed method improves over selection-only Top-K at matched token-equivalent read budgets, with the largest gains in high-entropy heads.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05438
PDF
-
BiST: A Gold Standard Bangla-English Bilingual Corpus for Sentence Structure and Tense Classification with Inter-Annotator Agreement
2026-04-06 14:22:46Abdullah Al Shafi, Swapnil Kundu Argha, M. A. Moyeen, Abdul Muntakim, Shoumik Barman PolokAbstract
High-quality bilingual resources remain a critical bottleneck for advancing multilingual NLP in low-resource settings, particularly for Bangla. To mitigate this gap, we introduce BiST, a rigorously curated Bangla-English corpus for sentence-level grammatical classification, annotated across two fundamental dimensions: syntactic structure (Simple, Complex, Compound, Complex-Compound) and tense (Present, Past, Future). The corpus is compiled from open-licensed encyclopedic sources and naturally composed conversational text, followed by systematic preprocessing and automated language identification, resulting in 30,534 sentences, including 17,465 English and 13,069 Bangla instances. Annotation quality is ensured through a multi-stage framework with three independent annotators and dimension-wise Fleiss Kappa ($\kappa$) agreement, yielding reliable and reproducible labels with $\kappa$ values of 0.82 and 0.88 for structural and temporal annotation, respectively. Statistical analyses demonstrate realistic structural and temporal distributions, while baseline evaluations show that dual-encoder architectures leveraging complementary language-specific representations consistently outperform strong multilingual encoders. Beyond benchmarking, BiST provides explicit linguistic supervision that supports grammatical modeling tasks, including controlled text generation, automated feedback generation, and cross-lingual representation learning. The corpus establishes a unified resource for bilingual grammatical modeling and facilitates linguistically grounded multilingual research.
Abstract (translated)
高质量的双语资源仍是推动低资源场景下（尤其是孟加拉语）多语言NLP发展的关键瓶颈。为缓解这一缺口，我们推出了BiST——一个经过严格筛选的孟加拉语-英语句子级语法分类语料库，在句法结构（简单句、复合句、并列句、并列复合句）与时态（现在时、过去时、将来时）两个维度进行标注。该语料库基于开放授权的百科资料与自然对话文本构建，经系统预处理与自动化语言识别后，共包含30,534句（其中英语17,465句，孟加拉语13,069句）。标注质量通过三阶段框架保障：由三名独立标注者执行，并采用分维度Fleiss Kappa（κ）一致性检验，最终结构标注κ值达0.82，时态标注κ值达0.88，确保标签可靠且可复现。统计分析显示语料库呈现真实的句法与时态分布，基线实验表明利用互补语言特定表示的双编码器架构持续优于强大多语言编码器。除基准测试外，BiST提供的显性语言监督还能支持语法建模任务，包括可控文本生成、自动反馈生成与跨语言表征学习。该语料库为双语语法建模建立了统一资源，并将推动基于语言学的多语言研究发展。
URL
https://arxiv.org/abs/2604.04708
PDF
-
FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation
2026-04-06 10:36:22Runzhe Zhang, Letian Chen, Wenpeng Zhang, Zhouhan Lin, Peilin ZhaoAbstract
We present FlowLM, a flow matching language model transformed from pre-trained diffusion language models via efficient fine-tuning. By re-aligning the curved sampling trajectories of diffusion models into straight-line flows, FlowLM enables high quality few-step generation that rivals or even outperforms the quality of 2,000-step diffusion sampling with very few training epochs. Remarkably, finetuned FlowLM reaches performance saturation with only half as many training epochs as training from scratch, both approaches greatly outperforming the original diffusion model, thereby validating our method. Furthermore, we validate a more effective training objective for flow matching: predicting clean data to consistently guide the sampling process towards the true data distribution. Empirical results demonstrate that our approach is highly effective for high-quality, few-step text generation.
Abstract (translated)
URL
https://arxiv.org/abs/2605.20199
PDF
-
A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
2026-04-06 07:27:04Tianmeng Fang, Yong Wang, Zetai Kong, Zengzhen Su, Jun Wang, Chengjin Yu, Wei WangAbstract
Multimodal large language models have become an important infrastructure for unified processing of visual and linguistic tasks. However, such models are highly susceptible to backdoor implantation during supervised fine-tuning and will steadily output the attacker's predefined harmful responses once a specific trigger pattern is activated. The core challenge of backdoor defense lies in suppressing attack success under low poisoning ratios while preserving the model's normal generation ability. These two objectives are inherently conflicting. Strong suppression often degrades benign performance, whereas weak regularization fails to mitigate backdoor behaviors. To this end, we propose a unified defense framework based on patch augmentation and cross-view regularity, which simultaneously constrains the model's anomalous behaviors in response to triggered patterns from both the feature representation and output distribution levels. Specifically, patch-level data augmentation is combined with cross-view output difference regularization to exploit the fact that backdoor responses are abnormally invariant to non-semantic perturbations and to proactively pull apart the output distributions of the original and perturbed views, thereby significantly suppressing the success rate of backdoor triggering. At the same time, we avoid over-suppression of the model during defense by imposing output entropy constraints, ensuring the quality of normal command generation. Experimental results across three models, two tasks, and six attacks show that our proposed defense method effectively reduces the attack success rate while maintaining a high level of normal text generation capability. Our work enables the secure, controlled deployment of large-scale multimodal models in realistic low-frequency poisoning and covert triggering scenarios.
Abstract (translated)
多模态大语言模型已成为统一处理视觉与语言任务的重要基础设施。然而，此类模型在监督微调过程中极易遭受后门植入，一旦特定触发模式被激活，便会持续输出攻击者预设的有害响应。后门防御的核心挑战在于：在低投毒比例下抑制攻击成功率的同时，需保持模型的正常生成能力——这两个目标本质上是冲突的。强抑制往往损害良性性能，而弱正则化则无法有效缓解后门行为。为此，我们提出一种基于补丁增强与跨视图正则化的统一防御框架，从特征表征与输出分布两个层面同步约束模型对触发模式的异常响应。具体而言，通过结合补丁级数据增强与跨视图输出差异正则化，利用后门响应对非语义扰动呈现异常不变的特性，主动拉大原始视图与扰动视图的输出分布差异，从而显著降低后门触发的成功率。同时，通过施加输出熵约束避免防御过程中对模型的过度抑制，确保正常指令的生成质量。在三个模型、两项任务与六种攻击上的实验结果表明，所提防御方法能在有效降低攻击成功率的同时，维持较高的正常文本生成能力。本研究使大规模多模态模型在现实低频率投毒与隐蔽触发场景下的安全可控部署成为可能。
URL
https://arxiv.org/abs/2604.04488
PDF
-
DP-OPD: Differentially Private On-Policy Distillation for Language Models
2026-04-06 06:18:03Fatemeh Khadem, Sajad Mousavi, Yi Fang, Yuhong LiuAbstract
Large language models (LLMs) are increasingly adapted to proprietary and domain-specific corpora that contain sensitive information, creating a tension between formal privacy guarantees and efficient deployment through model compression. Differential privacy (DP), typically enforced via DP-SGD, provides record-level protection but often incurs substantial utility loss in autoregressive generation, where optimization noise can amplify exposure bias and compounding errors along long rollouts. Existing approaches to private distillation either apply DP-SGD to both teacher and student, worsening computation and the privacy--utility tradeoff, or rely on DP synthetic text generation from a DP-trained teacher, avoiding DP on the student at the cost of DP-optimizing a large teacher and introducing an offline generation pipeline. We propose \textbf{Differentially Private On-Policy Distillation (DP-OPD)}, a synthesis-free framework that enforces privacy solely through DP-SGD on the student while leveraging a frozen teacher to provide dense token-level targets on \emph{student-generated} trajectories. DP-OPD instantiates this idea via \emph{private generalized knowledge distillation} on continuation tokens. Under a strict privacy budget ($\varepsilon=2.0$), DP-OPD improves perplexity over DP fine-tuning and off-policy DP distillation, and outperforms synthesis-based DP distillation (Yelp: 44.15$\rightarrow$41.68; BigPatent: 32.43$\rightarrow$30.63), while substantially simplifying the training pipeline. In particular, \textbf{DP-OPD collapses private compression into a single DP student-training loop} by eliminating DP teacher training and offline synthetic text generation. Code will be released upon publication at this https URL.
Abstract (translated)
大型语言模型（LLMs）正日益适配于包含敏感信息的专有和领域特定语料库，这造成了形式化隐私保证与通过模型压缩实现高效部署之间的紧张关系。通常通过DP-SGD实施的差分隐私（DP）提供记录级保护，但在自回归生成中常导致显著的效用损失，因为优化噪声可能加剧曝光偏差并沿长序列传播累积错误。现有的私有蒸馏方法要么对教师和学生均应用DP-SGD，从而恶化计算负担与隐私-效用权衡；要么依赖从DP训练教师生成DP合成文本，虽避免对学生进行DP处理，却需以DP优化大型教师为代价并引入离线生成流程。我们提出\textbf{差分隐私同策略蒸馏（DP-OPD）}，一个无合成框架，该框架仅通过学生的DP-SGD实施隐私保护，同时利用冻结的教师为学生生成轨迹上的密集词级目标提供指导。DP-OPD通过在续写词上实施\textit{私有广义知识蒸馏}来实现这一构想。在严格隐私预算（$\varepsilon=2.0$）下，DP-OPD相较于DP微调和离策略DP蒸馏提升了困惑度表现，并超越了基于合成的DP蒸馏（Yelp：44.15$\rightarrow$41.68；BigPatent：32.43$\rightarrow$30.63），同时大幅简化了训练流程。特别地，\textbf{DP-OPD通过消除DP教师训练和离线合成文本生成，将私有压缩整合至单一DP学生训练循环中}。代码将在发表后通过此https URL发布。
URL
https://arxiv.org/abs/2604.04461
PDF
-
Differences in Text Generated by Diffusion and Autoregressive Language Models
2026-04-04 17:30:35Zeyang Zhang, Chengwei Liang, Xingyan Chen, Meiqi Gu, Minrui Luo, Jingzhao Zhang, Tianxing HeAbstract
Diffusion language models (DLMs) are promising alternatives to autoregressive language models (ARMs), yet the intrinsic differences in their generated text remain underexplored. We first find empirically that off-the-shelf DLMs exhibit lower $n$-gram entropy, higher semantic coherence, and higher semantic diversity. To understand the cause, we conduct controlled experiments that decouple the effects of training objectives and decoding algorithms. Results suggest that the DLM training objective contributes to the increases in semantic coherence and semantic diversity, but has a minor influence on entropy. These differences are primarily driven by the bidirectional context; other components in the training objective, such as input masking, label masking, and the weighting function, have a much weaker influence. Further, our experiments demonstrate that the reduction in entropy stems from DLMs' decoding algorithms, particularly confidence-based remasking strategies. We provide a theoretical understanding for this entropy reduction phenomenon. Together, our work uncovers key mechanisms underlying the differences between DLMs and ARMs in text generation, and informs future design of training objectives and decoding algorithms in DLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2605.12522
PDF
-
Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation
2026-04-03 18:19:18Haziq Mohammad Khalid, Salsabeel Shapsough, Imran ZualkernanAbstract
Generating diverse, pedagogically valid stories for Arabic early-grade reading assessments requires balancing tight constraints on vocabulary, reading level, and narrative structure against the need to avoid repetitive plots that undermine assessment validity. We investigate noise steering, injecting calibrated Gaussian perturbations into the internal representations of transformer models at inference time, as a training-free diversity method evaluated across five small Arabic-centric language models (7-9B parameters). We compare four injection strategies against high-temperature sampling baselines, measuring diversity, quality, constraint adherence, and reading grade level. Residual stream noise consistently improves narrative diversity with minimal quality or constraint cost and preserves early-grade reading level across all models. Attention entropy noise injection (AENI) stabilizes the otherwise unreliable attention-logit noise while recovering quality. High-temperature sampling inflates reading grade level and causes catastrophic collapse on several models. We find internal representation-level perturbation to be a more suitable diversity strategy than output-level stochasticity for constrained educational content generation.
Abstract (translated)
为阿拉伯语早期阅读评估生成多样化且教学有效的故事，需要在词汇、阅读水平和叙事结构的严格限制与避免损害评估有效性的重复情节之间取得平衡。我们研究了噪声引导技术——在推理时向Transformer模型的内部表征注入校准过的高斯扰动——作为一种无需训练的多样性方法，并在五个以阿拉伯语为中心的小型语言模型（7-9B参数）上进行了评估。我们将四种注入策略与高温采样基线进行比较，测量多样性、质量、约束遵守度和阅读年级水平。残差流噪声始终能提升叙事多样性，且对质量和约束遵守度的负面影响极小，并在所有模型中保持了早期年级阅读水平。注意力熵噪声注入（AENI）稳定了原本不可靠的注意力-逻辑值噪声，同时恢复了生成质量。高温采样抬高了阅读年级水平，并在多个模型中导致灾难性崩溃。我们发现，对于受限的教育内容生成，内部表征层面的扰动比输出层面的随机性是一种更优的多样性策略。
URL
https://arxiv.org/abs/2604.03380
PDF
