Abstract
Speech is a natural means of conveying emotions, making it an effective method for understanding and representing human feelings. Reliable speech emotion recognition (SER) is central to applications in human-computer interaction, healthcare, education, and customer service. However, most SER methods depend on heavy backbone models or hand-crafted features that fail to balance accuracy and efficiency, particularly for low-resource languages like Bangla. In this work, we present SpectroFusion-ViT, a lightweight SER framework built utilizing EfficientViT-b0, a compact Vision Transformer architecture equipped with self-attention to capture long-range temporal and spectral patterns. The model contains only 2.04M parameters and requires 0.1 GFLOPs, enabling deployment in resource-constrained settings without compromising accuracy. Our pipeline first performs preprocessing and augmentation on raw audio, then extracts Chroma and Mel-frequency cepstral coefficient (MFCC) features. These representations are fused into a complementary time-frequency descriptor that preserves both fine-grained spectral detail and broader harmonic structure. Using transfer learning, EfficientViT-b0 is fine-tuned for multi-class emotion classification. We evaluate the system on two benchmark Bangla emotional speech datasets, SUBESCO and BanglaSER, which vary in speaker diversity, recording conditions, and acoustic characteristics. The proposed approach achieves 92.56% accuracy on SUBESCO and 82.19% on BanglaSER, surpassing existing state-of-the-art methods. These findings demonstrate that lightweight transformer architectures can deliver robust SER performance while remaining computationally efficient for real-world deployment.
Abstract (translated)
语音是传达情感的自然方式，因此成为理解并表达人类感受的有效方法。可靠的声音情感识别（SER）对于人机交互、医疗保健、教育和客户服务等应用至关重要。然而，大多数SER方法依赖于重型骨干模型或人工设计的功能，这些功能在低资源语言如孟加拉语中无法同时兼顾准确性和效率。 在此工作中，我们提出了SpectroFusion-ViT，这是一种轻量级的SER框架，利用了EfficientViT-b0这一小巧的Vision Transformer架构。该架构配备了自注意力机制，能够捕捉长时间跨度和频谱模式。模型仅有2.04M个参数，并且需要0.1GFLOPs计算力，这使其在资源有限的情况下也能部署而不影响准确性。 我们的处理流程首先对原始音频进行预处理及增强操作，随后提取Chroma（调式图）和Mel频率倒谱系数（MFCC）特征。这些表示被融合成一个互补的时间-频率描述符，同时保留了精细的频谱细节和更广泛的谐波结构。通过迁移学习，我们微调EfficientViT-b0进行多类别情感分类。 我们在两个孟加拉语情感语音基准数据集上评估了该系统：SUBESCO 和 BanglaSER。这些数据集中说话人的多样性、录音条件及声学特征各不相同。我们的方法在SUBESCO 数据集上实现了92.56%的准确率，在BanglaSER 上则为82.19%，超过了现有的最先进方法。 这些结果表明，轻量级的Transformer架构能够在实际部署中提供稳健的声音情感识别性能，并且计算效率高。
URL
https://arxiv.org/abs/2603.00746
