Abstract
In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior knowledge of the task, the sizes of individual layers are often chosen heuristically. However, this approach does not guarantee an optimal trade-off between performance and computational complexity; consequently, post hoc methods such as weight quantization or model pruning are typically employed to reduce computational cost. This occurs because stochastic gradient descent (SGD) methods can only optimize differentiable functions, while factors influencing computational complexity, such as layer sizes and floating-point operations per second (FLOP/s), are non-differentiable and require modifying the model structure during training. We propose a reparameterization technique based on feature noise injection that enables joint optimization of performance and computational complexity during training using SGD-based methods. Unlike traditional pruning methods, our approach allows the model size to be dynamically optimized for a target performance-complexity trade-off, without relying on heuristic criteria to select which weights or structures to remove. We demonstrate the effectiveness of our method through three case studies, including a synthetic example and two practical real-world applications: voice activity detection and audio anti-spoofing. The code related to our work is publicly available to encourage further research.
Abstract (translated)
在语音机器学习中，神经网络模型通常是通过选择具有固定层数和结构的架构来设计的。这些模型经过训练以最大化与任务目标一致的指标性能。虽然整体架构通常由任务的先验知识指导，但各个层的大小往往是由直觉或经验法则确定的。然而，这种方法不能保证在性能和计算复杂性之间达到最优权衡；因此，人们常常采用事后方法（如权重量化或模型剪枝）来降低计算成本。这是因为随机梯度下降(SGD)方法只能优化可微函数，而影响计算复杂性的因素（如层数和每秒浮点运算数(FLOP/s)）是不可微的，并且需要在训练过程中修改模型结构。 我们提出了一种基于特征噪声注入的重新参数化技术，该技术能够使用SGD基方法在训练期间同时优化性能和计算复杂性。与传统的剪枝方法不同，我们的方法允许动态调整模型大小以适应目标性能-复杂性的权衡，而无需依赖于选择要移除哪些权重或结构的经验准则。 我们通过三个案例研究展示了这种方法的有效性，包括一个合成示例以及两个实际应用：语音活动检测和音频防欺骗。与我们的工作相关的代码是公开的，旨在促进进一步的研究。
URL
https://arxiv.org/abs/2601.13704
