Abstract
In this paper, we explore the two-point zeroth-order gradient estimator and identify the distribution of random perturbations that minimizes the estimator's asymptotic variance as the perturbation stepsize tends to zero. We formulate it as a constrained functional optimization problem over the space of perturbation distributions. Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length. While existing research has largely focused on fixed-length perturbations, the potential advantages of directional alignment have been overlooked. To address this gap, we delve into the theoretical and empirical properties of the directionally aligned perturbation (DAP) scheme, which adaptively offers higher accuracy along critical directions. Additionally, we provide a convergence analysis for stochastic gradient descent using $\delta$-unbiased random perturbations, extending existing complexity bounds to a wider range of perturbations. Through empirical evaluations on both synthetic problems and practical tasks, we demonstrate that DAPs outperform traditional methods under specific conditions.
Abstract (translated)
在这篇论文中，我们探讨了两点零阶梯度估计器，并确定了随机扰动分布，该分布可以在扰动步长趋向于零时使估计量的渐近方差最小化。我们将这个问题表述为在扰动分布空间上的约束泛函优化问题。我们的研究结果表明，理想的扰动可以与真实梯度的方向对齐，而不仅仅是保持固定长度。虽然现有研究主要集中在固定长度的扰动上，但方向对齐的潜在优势却被忽视了。 为了填补这一空白，我们深入探讨了方向对齐扰动（DAP）方案的理论和实证特性，该方案可以自适应地在关键方向上提供更高的准确性。此外，我们还为使用$\delta$无偏随机扰动进行随机梯度下降提供了收敛性分析，并将现有的复杂度界限扩展到了更广泛的扰动范围内。 通过合成问题和实际任务上的经验评估，我们在特定条件下展示了DAP优于传统方法的性能。
URL
https://arxiv.org/abs/2510.19975
