Abstract
Recent progress in deep learning has been driven by increasingly larger models. However, their computational and energy demands have grown proportionally, creating significant barriers to their deployment and to a wider adoption of deep learning technologies. This thesis investigates principled approximation methods for improving the efficiency of deep learning systems, with a particular focus on settings that involve discrete constraints and non-differentiability. We study three main approaches toward improved efficiency: architecture design, model compression, and optimization. For model compression, we propose novel approximations for pruning and quantization that frame the underlying discrete problem as continuous and differentiable, enabling gradient-based training of compression schemes alongside the model's parameters. These approximations allow for fine-grained sparsity and precision configurations, leading to highly compact models without significant fine-tuning. In the context of architecture design, we design an algorithm for neural architecture search that leverages parameter sharing across layers to efficiently explore implicitly recurrent architectures. Finally, we study adaptive optimization, revisiting theoretical properties of widely used methods and proposing an adaptive optimizer that allows for quick hyperparameter tuning. Our contributions center on tackling computationally hard problems via scalable and principled approximations. Experimental results on image classification, language modeling, and generative modeling tasks show that the proposed methods provide significant improvements in terms of training and inference efficiency while maintaining, or even improving, the model's performance.
Abstract (translated)
最近的深度学习进展主要得益于模型规模的不断扩大。然而，这些大型模型在计算和能源需求方面也相应增加，给它们的实际部署以及更广泛地采用深度学习技术带来了显著障碍。本论文探讨了改善深度学习系统效率的原则性近似方法，特别关注包含离散约束和非可微分问题的情境。我们研究了提高效率的三个主要途径：架构设计、模型压缩和优化。 对于模型压缩，我们提出了一种新的修剪和量化近似方法，将底层的离散问题视为连续且可微的问题，使压缩方案可以与模型参数一起进行基于梯度的训练。这些近似方法允许精细调整稀疏性和精度配置，从而创建高度紧凑的模型而无需大量微调。 在架构设计方面，我们提出了一种算法用于神经网络架构搜索，通过利用层间的参数共享来高效地探索隐式递归架构。 最后，我们研究了自适应优化技术，并重新审视广泛使用的现有方法的理论特性，同时提出了一个新的自适应优化器，该优化器能够快速调整超参数。我们的贡献集中在通过可扩展且原则性的近似解决计算难题上。 实验结果表明，在图像分类、语言模型和生成式建模任务中，所提出的方法在保持或甚至提高模型性能的同时，显著提高了训练和推理的效率。
URL
https://arxiv.org/abs/2509.00174
