Abstract
The deployment of large language models (LLMs) in real-world clinical applications is constrained by the fundamental trade-off between computational cost and the efficiency of linear-time models. To address this, we propose an LLM-based MambaFormer hybrid Mixture-of-Experts (MoE) framework for efficient medical question-answering (QA) and clinical assistance. The MambaFormer employs a lightweight gating mechanism that performs token-level dynamic routing to a customized Transformer expert (ET5) for short, complex queries or to a State Space Model expert (EMamba) for long, high-throughput sequences. The customized EMamba and ET5 models are tailored to accommodate input sequence dimensionality, embedding structure, sequence length, and target-specific output heads, and are fine-tuned through transfer learning on a new, custom-designed DentalQA dataset. Moreover, intelligent routing decisions are driven by the contextual complexity of token embeddings, normalized sequence length, and domain-aware features, thereby enforcing a Pareto-optimal trade-off between inference latency and prediction accuracy. Furthermore, a novel utility-guided multi-objective loss jointly optimizes decisions, router parameters, routing behavior, expert utilization, and computational cost by adaptively regulating token-level expert activation. Finally, the proposed MambaFormer is cross-validated (holdout) for medical QA on the new, custom-designed DentalQA and PubMedQA datasets and compared with state-of-the-art techniques. The proposed MambaFormer outperforms (BERTScore = 0.9180) with ultra-low latency (0.077 s), delivering a 24.4 speedup over T5-Large and establishing a scalable solution for resource-constrained clinical deployment.
Abstract (translated)
在现实世界的临床应用中部署大型语言模型（LLMs）受到计算成本和线性时间模型效率之间基本权衡的限制。为了解决这个问题，我们提出了一种基于LLM的MambaFormer混合专家组合(Mixture-of-Experts, MoE)框架，用于高效的医疗问答(QA)和临床辅助。MambaFormer采用轻量级门控机制，在处理简短复杂查询时将令牌动态路由到定制化的Transformer专家（ET5），在处理长序列或高吞吐量序列时则将其路由至状态空间模型专家（EMamba）。 定制的EMamba和ET5模型经过调整，以适应输入序列维度、嵌入结构、序列长度以及特定目标的输出头，并通过迁移学习对新的自设计牙科QA数据集进行微调。此外，智能路由决策由令牌嵌入的上下文复杂性、归一化序列长度及领域感知特征驱动，在推理延迟和预测准确性之间实现了帕累托最优权衡。同时，一种新颖的目标导向多目标损失函数通过调节令牌级专家激活来共同优化决策、路由器参数、路由行为、专家利用度以及计算成本。 最终，所提出的MambaFormer在新的自设计牙科QA数据集和PubMedQA数据集中进行交叉验证，并与最先进的技术进行了比较。该方法（BERTScore = 0.9180）在极低延迟(0.077秒)下表现出色，比T5-Large快24.4倍，在资源受限的临床部署中提供了一种可扩展的解决方案。
URL
https://arxiv.org/abs/2601.01260
