Abstract
Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at this https URL.
Abstract (translated)
翻译如下： 仅解码器的大规模语言模型越来越多地被用作用户表示学习的行为编码器，然而注意力屏蔽对用户嵌入质量的影响仍然缺乏深入的研究。在本研究中，我们针对大规模现实世界的支付宝数据（该数据集整合了长期异构的用户行为）进行了一项系统性研究，探讨因果、混合和双向注意掩码在统一对比学习框架中的应用效果。为了改进从因果到双向注意力过渡过程中的训练动态，我们提出了一种基于梯度引导的软屏蔽方法——Gradient-Guided Soft Masking，在线性调度器渐进式开放未来注意力优化之前进行预热。通过评估涵盖预测、偏好和营销敏感性的9个工业用户认知基准任务，我们的方法在对比因果、混合及仅使用调度器的基线模型时，展现出了更稳定的训练过程以及更高质量的双向表示，同时仍与解码器预训练保持兼容。 总体而言，我们的研究结果突显了屏蔽设计和训练转换对于适应仅解码器的大规模语言模型以实现有效的用户表示学习的重要性。相关代码可在提供的网址获取。
URL
https://arxiv.org/abs/2602.10622
