Abstract
Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles, an efficient transformer-based ensemble that prunes attention heads to create diverse members and merges them via a new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Experiments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, in zero-shot classification on ImageNet-1k, our approach surpasses state of the art methods, even without requiring additional training.
Abstract (translated)
不确定性量化（UQ）对于在安全关键环境中部署深度神经网络至关重要。尽管像Deep Ensembles这样的方法能够实现强大的不确定性量化性能，但它们的高计算和内存成本阻碍了其在大型模型中的可扩展性。我们引入了一种高效的基于Transformer的集合——Hydra Ensembles，它通过修剪注意力头来创建多样化的成员，并利用一种新的具有分组全连接层的多头注意力机制将这些成员合并起来。这种方法生成了一个紧凑的模型，其推理速度接近单个网络的速度，在不从头开始重新训练的情况下，不确定性量化的性能可以与Deep Ensembles相匹配甚至超越。 我们还提供了对修剪方法的深入分析，表明简单的修剪方式可能会损害校准效果，而Hydra Ensembles则能够保持强大的不确定性估计。在图像和文本分类任务中进行的各种架构实验显示，与Deep Ensembles相比，我们的方法具有持续的优势。尤其值得注意的是，在零样本（zero-shot）ImageNet-1k分类测试中，即使不增加额外的训练数据，我们提出的方法也超越了现有的最佳方法。
URL
https://arxiv.org/abs/2510.18358
