Abstract
Training Artificial Neural Networks (ANNs) with Stochastic Gradient Descent (SGD) frequently encounters difficulties, including substantial computing expense and the risk of converging to local optima, attributable to its dependence on partial weight gradients. Therefore, this work investigates Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) - two population-based Metaheuristic Optimizers (MHOs) - as alternatives to SGD to mitigate these constraints. A hybrid PSO-SGD strategy is developed to improve local search efficiency. The findings indicate that the hybrid PSO-SGD technique decreases the median training MSE by 90 to 95 percent relative to conventional GA and PSO across various network sizes (e.g., from around 0.02 to approximately 0.001 in the Sphere function). RMHC attains substantial enhancements, reducing MSE by roughly 85 to 90 percent compared to GA. Simultaneously, RS consistently exhibits errors exceeding 0.3, signifying subpar performance. These findings underscore that hybrid and evolutionary procedures significantly improve training efficiency and accuracy compared to conventional optimization methods and imply that the Building Block Hypothesis (BBH) may still be valid, indicating that advantageous weight structures are retained during evolutionary search.
Abstract (translated)
使用随机梯度下降（SGD）训练人工神经网络（ANNs）经常遇到诸如计算成本高昂和可能陷入局部最优解等问题，这些问题归因于其对权重偏导数的依赖。因此，本研究探讨了粒子群优化（PSO）和遗传算法（GAs）——两种基于种群的元启发式优化器（MHOs），作为SGD的替代方案以缓解这些限制。开发了一种混合PSO-SGD策略来提高局部搜索效率。研究表明，与传统的GA和PSO相比，该混合PSO-SGD技术在各种网络规模下将中位训练均方误差（MSE）降低了90%到95%，例如，在Sphere函数中的误差从大约0.02降至约0.001。 RMHC（随机重启Hill-Climbing算法）也取得了显著改进，与GA相比，它使MSE减少了约85%至90%。同时，RS（随机搜索）始终表现出超过0.3的错误率，表明其性能不佳。这些发现强调了混合和进化方法在训练效率和准确性方面相较于传统优化方法有明显改善，并暗示构建块假设（BBH）可能仍然有效，这意味着有利的权重结构在进化搜索中得以保留。
URL
https://arxiv.org/abs/2506.15737
