Abstract
Multimodal fusion faces two robustness challenges: noisy inputs degrade representation quality, and missing modalities cause prediction failures. We propose DCER, a unified framework addressing both challenges through dual-stage compression and energy-based reconstruction. The compression stage operates at two levels: within-modality frequency transforms (wavelet for audio, DCT for video) remove noise while preserving task-relevant patterns, and cross-modality bottleneck tokens force genuine integration rather than modality-specific shortcuts. For missing modalities, energy-based reconstruction recovers representations via gradient descent on a learned energy function, with the final energy providing intrinsic uncertainty quantification (\r{ho} > 0.72 correlation with prediction error). Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate state-of-the-art performance across all benchmarks, with a U-shaped robustness pattern favoring multimodal fusion at both complete and high-missing conditions. The code will be available on Github.
Abstract (translated)
多模态融合面临两个鲁棒性挑战：噪声输入会降低表示质量，而缺失的模式会导致预测失败。我们提出了DCER框架，这是一个通过双阶段压缩和基于能量的重构同时解决这两个问题的统一方法。在压缩阶段，该框架运行于两个层面：在同一模式内的频域变换（音频使用小波变换，视频使用离散余弦变换）可以去除噪声并保留与任务相关的模式；跨模式瓶颈令牌则强制进行真实的整合而不是单一模式特有的捷径。对于缺失的模式，基于能量的重构通过在学习的能量函数上进行梯度下降来恢复表示，并且最终能量提供预测误差（\rho > 0.72 相关性）的内在不确定性量化。 实验结果表明，在CMU-MOSI、CMU-MOSEI和CH-SIMS数据集上的所有基准测试中，DCER框架表现出最先进的性能，并展示了一个U形鲁棒性模式，在完整和高缺失条件下均有利于多模态融合。代码将在Github上提供。
URL
https://arxiv.org/abs/2602.04904
