Abstract
The real-time processing of time series signals is a critical issue for many real-life applications. The idea of real-time processing is especially important in audio domain as the human perception of sound is sensitive to any kind of disturbance in perceived signals, especially the lag between auditory and visual modalities. The rise of deep learning (DL) models complicated the landscape of signal processing. Although they often have superior quality compared to standard DSP methods, this advantage is diminished by higher latency. In this work we propose novel method for minimization of inference time latency and memory consumption, called Short-Term Memory Convolution (STMC) and its transposed counterpart. The main advantage of STMC is the low latency comparable to long short-term memory (LSTM) networks. Furthermore, the training of STMC-based models is faster and more stable as the method is based solely on convolutional neural networks (CNNs). In this study we demonstrate an application of this solution to a U-Net model for a speech separation task and GhostNet model in acoustic scene classification (ASC) task. In case of speech separation we achieved a 5-fold reduction in inference time and a 2-fold reduction in latency without affecting the output quality. The inference time for ASC task was up to 4 times faster while preserving the original accuracy.
Abstract (translated)
实时处理序列信号是许多实际应用程序的关键问题。实时处理的概念在音频领域尤为重要,因为人类对声音的感知对 perceived信号中的任何扰动都非常敏感,特别是听觉和视觉模式之间的延迟。深度学习(DL)模型的崛起使信号处理领域变得复杂。尽管他们通常比标准DSP方法提供更好的质量,但这一优势随着更高的延迟而减弱。在这项工作中,我们提出了一种新的方法来最小化推断时间和内存消耗,称为短期记忆卷积(STMC)和其transposed counterpart。STMC的主要优点是低延迟,类似于长短期记忆(LSTM)网络。此外,基于STMC的模型的训练速度更快更稳定,因为方法仅基于卷积神经网络(CNNs)。在本研究中,我们演示了这种方法对U-Net模型的一个语音分离任务和一个声学场景分类(ASC)任务的应用。在语音分离的情况下,我们实现了推断时间的五fold减少和延迟的两人次减少,而输出质量没有受到影响。 ASC任务推断时间高达4倍快,同时保留了原始准确性。
URL
https://arxiv.org/abs/2302.04331
