Abstract
In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model's parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness.
Abstract (translated)
在这篇论文中，我们从概率分布估计的角度出发，探讨了使用深度神经网络进行监督分类的优化机制。我们展示了当采用Fenchel-Young损失函数时，尽管相对于模型参数拟合误差是非凸的，但仍可以通过同时最小化梯度范数和结构误差来近似全局最优解。前者可通过梯度下降算法加以控制。对于后者，我们证明通过增加参数数量并确保参数独立性可以进行管理，从而为诸如过度参数化和随机初始化等机制提供了理论见解。最终，本文通过实证结果验证了所提出方法的关键结论，并展示了其实际有效性。
URL
https://arxiv.org/abs/2503.23016
