Abstract
Generating diverse, pedagogically valid stories for Arabic early-grade reading assessments requires balancing tight constraints on vocabulary, reading level, and narrative structure against the need to avoid repetitive plots that undermine assessment validity. We investigate noise steering, injecting calibrated Gaussian perturbations into the internal representations of transformer models at inference time, as a training-free diversity method evaluated across five small Arabic-centric language models (7-9B parameters). We compare four injection strategies against high-temperature sampling baselines, measuring diversity, quality, constraint adherence, and reading grade level. Residual stream noise consistently improves narrative diversity with minimal quality or constraint cost and preserves early-grade reading level across all models. Attention entropy noise injection (AENI) stabilizes the otherwise unreliable attention-logit noise while recovering quality. High-temperature sampling inflates reading grade level and causes catastrophic collapse on several models. We find internal representation-level perturbation to be a more suitable diversity strategy than output-level stochasticity for constrained educational content generation.
Abstract (translated)
为阿拉伯语早期阅读评估生成多样化且教学有效的故事，需要在词汇、阅读水平和叙事结构的严格限制与避免损害评估有效性的重复情节之间取得平衡。我们研究了噪声引导技术——在推理时向Transformer模型的内部表征注入校准过的高斯扰动——作为一种无需训练的多样性方法，并在五个以阿拉伯语为中心的小型语言模型（7-9B参数）上进行了评估。我们将四种注入策略与高温采样基线进行比较，测量多样性、质量、约束遵守度和阅读年级水平。残差流噪声始终能提升叙事多样性，且对质量和约束遵守度的负面影响极小，并在所有模型中保持了早期年级阅读水平。注意力熵噪声注入（AENI）稳定了原本不可靠的注意力-逻辑值噪声，同时恢复了生成质量。高温采样抬高了阅读年级水平，并在多个模型中导致灾难性崩溃。我们发现，对于受限的教育内容生成，内部表征层面的扰动比输出层面的随机性是一种更优的多样性策略。
URL
https://arxiv.org/abs/2604.03380
