Abstract
Transfer Learning (TL) has accelerated the rapid development and availability of large language models (LLMs) for mainstream natural language processing (NLP) use cases. However, training and deploying such gigantic LLMs in resource-constrained, real-world healthcare situations remains challenging. This study addresses the limited support available to visually impaired users and speakers of low-resource languages such as Hindi who require medical assistance in rural environments. We propose PDFTEMRA (Performant Distilled Frequency Transformer Ensemble Model with Random Activations), a compact transformer-based architecture that integrates model distillation, frequency-domain modulation, ensemble learning, and randomized activation patterns to reduce computational cost while preserving language understanding performance. The model is trained and evaluated on medical question-answering and consultation datasets tailored to Hindi and accessibility scenarios, and its performance is compared against standard NLP state-of-the-art model baselines. Results demonstrate that PDFTEMRA achieves comparable performance with substantially lower computational requirements, indicating its suitability for accessible, inclusive, low-resource medical NLP applications.
Abstract (translated)
迁移学习（TL）加速了大型语言模型（LLM）在主流自然语言处理（NLP）应用场景中的快速发展和应用。然而，在资源受限的实际医疗环境中训练和部署此类大规模LLM仍然具有挑战性。本研究关注视障用户和支持低资源语言如印地语的说话者在农村环境下的医疗服务需求问题，这些人群需要医疗帮助但面临着较大的使用障碍。我们提出了PDFTEMRA（高性能蒸馏频域变换器集成模型与随机激活），这是一种基于变压器架构的小型化设计，集成了模型蒸馏、频域调制、集成学习和随机激活模式等技术，以降低计算成本并保持语言理解性能。该模型在针对印地语及可访问性场景定制的医疗问答和咨询数据集上进行训练和评估，并与标准NLP最新基准模型进行了性能对比。结果表明，PDFTEMRA实现了与现有最佳模型相当的性能水平，但计算需求显著降低，这证明了其适用于包容性强、资源有限的医学NLP应用。
URL
https://arxiv.org/abs/2512.06734
