Abstract
Deep neural networks for visual perception are highly susceptible to domain shift, which poses a critical challenge for real-world deployment under conditions that differ from the training data. To address this domain generalization challenge, we propose a cross-modal framework under the learning using privileged information (LUPI) paradigm for training a robust, single-modality RGB model. We leverage event cameras as a source of privileged information, available only during training. The two modalities exhibit complementary characteristics: the RGB stream is semantically dense but domain-dependent, whereas the event stream is sparse yet more domain-invariant. Direct feature alignment between them is therefore suboptimal, as it forces the RGB encoder to mimic the sparse event representation, thereby losing semantic detail. To overcome this, we introduce Privileged Event-based Predictive Regularization (PEPR), which reframes LUPI as a predictive problem in a shared latent space. Instead of enforcing direct cross-modal alignment, we train the RGB encoder with PEPR to predict event-based latent features, distilling robustness without sacrificing semantic richness. The resulting standalone RGB model consistently improves robustness to day-to-night and other domain shifts, outperforming alignment-based baselines across object detection and semantic segmentation.
Abstract (translated)
用于视觉感知的深度神经网络在面对数据领域变化时非常脆弱，这给其在与训练数据不同的实际环境中部署带来了重大挑战。为了解决这种领域的泛化难题，我们提出了一种基于学习使用特权信息（LUPI）范式的跨模态框架来训练一种鲁棒的单模RGB模型。我们利用事件相机作为仅限于训练阶段使用的特权信息源。这两种模式表现出互补的特点：RGB流具有丰富的语义但依赖领域环境，而事件流则更为稀疏却更少受领域变化的影响。因此，在两者之间直接进行特征对齐是次优的，因为这会迫使RGB编码器模仿稀疏的事件表示，从而导致丢失了语义细节。 为了解决这一问题，我们引入了一种基于特权事件预测正则化的PEPR方法（Privileged Event-based Predictive Regularization），该方法将LUPI重新构架为共享潜在空间中的一个预测问题。与其强制执行直接跨模态对齐，我们使用PEPR训练RGB编码器来预测基于事件的潜在特征，这可以在不牺牲语义丰富性的情况下提炼出鲁棒性。最终得到的独立RGB模型在面对白天到夜晚以及其他领域的变化时，其稳健性得到了持续提高，并且无论是在目标检测还是语义分割方面都超过了基于对齐的方法基线表现。
URL
https://arxiv.org/abs/2602.04583
