Abstract
Recent advancements in Speech Large Language Models have significantly enhanced multi-dimensional speech understanding. However, the majority of high-performance frameworks are predominantly optimized for GPU centric ecosystems and proprietary backbones, creating a significant gap for deployment on non-CUDA computing infrastructures. In this paper, we present OSUM-Pangu, a fully open-source speech understanding foundation model developed on a completely non-CUDA software and hardware stack. By integrating an audio encoder with the openPangu-7B LLM backbone, we successfully implement the entire training and inference pipeline on the Ascend NPU platform. To facilitate efficient task alignment under non-CUDA resource constraints, we adopt a practical training process that sequentially bridges speech perception and user intent recognition. Experimental results demonstrate that OSUM-Pangu achieves task accuracy comparable to mainstream GPU-based models while maintaining robust natural language interaction capabilities. Our work provides a reproducible, non-CUDA baseline for the open-source speech community, promoting the independent evolution of multimodal intelligence.
Abstract (translated)
最近在语音大型语言模型方面的进展显著提高了多维度的语音理解能力。然而，大多数高性能框架主要针对以GPU为中心的生态系统和专有骨干网络进行了优化，这为部署到非CUDA计算基础设施上造成了明显的差距。本文介绍了OSUM-Pangu，这是一个完全开源的语音理解基础模型，在一个完全不依赖于CUDA软件和硬件堆栈上开发而成。通过将音频编码器与开放源代码的openPangu-7B LLM骨干网络集成，我们成功地在Ascend NPU平台上实现了整个训练和推理流程。为了在非CUDA资源限制条件下促进高效的任务对齐，我们采用了一种实际的培训过程，该过程依次连接语音感知和用户意图识别。实验结果显示，OSUM-Pangu能够达到与主流GPU基础模型相当的任务准确度，并保持强大的自然语言交互能力。我们的工作为开源语音社区提供了可重复、非CUDA的标准基线，推动了多模态智能的独立发展。
URL
https://arxiv.org/abs/2603.10862
