Abstract
The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture. It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content. The design is inspired by the multigrid (MG) paradigm, originally developed to accelerate the convergence of iterative solvers for partial differential equations and boundary value problems, and later extended to more general linear systems through algebraic multigrid methods. We refer to the method as a pseudo-projector because its linear prototype corresponds to a strictly idempotent orthogonal projector, whereas the practical formulation employs learnable restriction and prolongation operators and therefore does not, in general, satisfy the properties of an exact orthogonal projection. We evaluate the proposed approach on transformer-based text classification tasks, as well as controlled synthetic benchmarks, demonstrating its effectiveness in improving training dynamics and robustness. Experimental results, together with supporting theoretical heuristics, indicate consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise. Our next step will be to extend this approach to language models.
Abstract (translated)
伪投影器是一种轻量级的修改，可以集成到现有的语言模型和其他神经网络中而不改变其核心架构。它可以被视为一种隐藏表示纠正器，通过抑制由标签无关输入内容引起的导向来减少对噪声的敏感性。该设计受到了多重网格（MG）范式的启发，这种范式最初是为了加速偏微分方程和边值问题迭代求解器的收敛速度而开发，并且后来通过代数多重网格方法被扩展到更一般的线性系统中。我们将其称为伪投影器是因为其线性原型对应于严格的幂等正交投影器，而在实际实施方案中使用了可学习的限制操作符和延拓操作符，因此通常不满足精确正交投影所具备的性质。 我们在基于变压器的文本分类任务以及受控合成基准测试上评估了该方法，展示了其在改进训练动态性和提高鲁棒性方面的有效性。实验结果与支持性的理论依据表明，在各种设置下训练行为的一致改善，并且没有观察到其他不良影响。下一步我们将扩展此方法以应用于语言模型。
URL
https://arxiv.org/abs/2603.09815
