Abstract
This work invokes the notion of $f$-divergence to introduce a novel upper bound on the Bayes error rate of a general classification task. We show that the proposed bound can be computed by sampling from the output of a parameterized model. Using this practical interpretation, we introduce the Bayes optimal learning threshold (BOLT) loss whose minimization enforces a classification model to achieve the Bayes error rate. We validate the proposed loss for image and text classification tasks, considering MNIST, Fashion-MNIST, CIFAR-10, and IMDb datasets. Numerical experiments demonstrate that models trained with BOLT achieve performance on par with or exceeding that of cross-entropy, particularly on challenging datasets. This highlights the potential of BOLT in improving generalization.
Abstract (translated)
这项工作引入了$f$-散度的概念，提出了一种对一般分类任务的贝叶斯错误率的新上界估计。我们展示了所提出的界限可以通过从参数化模型的输出中采样来计算得出。基于这一实用解释，我们提出了贝叶斯最优学习阈值（BOLT）损失函数，其最小化能够促使分类模型达到贝叶斯错误率。我们在图像和文本分类任务中验证了该损失函数的有效性，考虑的数据集包括MNIST、Fashion-MNIST、CIFAR-10以及IMDb数据集。数值实验表明，在训练时使用BOLT的模型在性能上可以与交叉熵相媲美或超越交叉熵，尤其是在挑战性的数据集上更为明显。这突显了BOLT在提升泛化能力方面的潜力。
URL
https://arxiv.org/abs/2501.07754
