Abstract
In this paper we propose and investigate a wide class of Mirror Descent updates (MD) and associated novel Generalized Exponentiated Gradient (GEG) algorithms by exploiting various trace-form entropies and associated deformed logarithms and their inverses - deformed (generalized) exponential functions. The proposed algorithms can be considered as extension of entropic MD and generalization of multiplicative updates. In the literature, there exist nowadays over fifty mathematically well defined generalized entropies, so impossible to exploit all of them in one research paper. So we focus on a few selected most popular entropies and associated logarithms like the Tsallis, Kaniadakis and Sharma-Taneja-Mittal and some of their extension like Tempesta or Kaniadakis-Scarfone entropies. The shape and properties of the deformed logarithms and their inverses are tuned by one or more hyperparameters. By learning these hyperparameters, we can adapt to distribution of training data, which can be designed to the specific geometry of the optimization problem, leading to potentially faster convergence and better performance. The using generalized entropies and associated deformed logarithms in the Bregman divergence, used as a regularization term, provides some new insight into exponentiated gradient descent updates.
Abstract (translated)
在这篇论文中，我们提出并研究了一类广泛的镜像下降更新（MD）和相关的新型广义指数梯度（GEG）算法。通过利用各种迹形式的熵以及相应的变形对数及其逆——即变形（广义）指数函数，我们开发了这些算法。提出的算法可以被视为熵型 MD 的扩展，并且是乘法更新的一般化。 目前文献中存在超过五十种数学上严格定义的广义熵，因此不可能在一篇研究论文中全部利用它们。所以我们集中关注几种最流行的熵及其对应的对数，如 Tsallis 熵、Kaniadakis 熵和 Sharma-Taneja-Mittal 熵，还有一些它们的扩展形式，比如 Tempesta 和 Kaniadakis-Scarfone 熵。 变形对数及其逆的形式和性质可以通过一个或多个超参数进行调整。通过学习这些超参数，我们可以适应训练数据分布，并根据优化问题的具体几何特性对其进行设计，从而可能实现更快的收敛速度和更好的性能表现。在 Bregman 散度中使用广义熵及相关变形对数作为正则化项，为指数梯度下降更新提供了一些新的见解。
URL
https://arxiv.org/abs/2503.08748
