Abstract
Alzheimer's disease and related dementias (ADRD) affect approximately five million older adults in the U.S., yet over half remain undiagnosed. Speech-based natural language processing (NLP) offers a promising, scalable approach to detect early cognitive decline through linguistic markers. To develop and evaluate a screening pipeline that (i) fuses transformer embeddings with handcrafted linguistic features, (ii) tests data augmentation using synthetic speech generated by large language models (LLMs), and (iii) benchmarks unimodal and multimodal LLM classifiers for ADRD detection. Transcripts from the DementiaBank "cookie-theft" task (n = 237) were used. Ten transformer models were evaluated under three fine-tuning strategies. A fusion model combined embeddings from the top-performing transformer with 110 lexical-derived linguistic features. Five LLMs (LLaMA-8B/70B, MedAlpaca-7B, Ministral-8B, GPT-4o) were fine-tuned to generate label-conditioned synthetic speech, which was used to augment training data. Three multimodal models (GPT-4o, Qwen-Omni, Phi-4) were tested for speech-text classification in zero-shot and fine-tuned settings. The fusion model achieved F1 = 83.3 (AUC = 89.5), outperforming linguistic or transformer-only baselines. Augmenting training data with 2x MedAlpaca-7B synthetic speech increased F1 to 85.7. Fine-tuning significantly improved unimodal LLM classifiers (e.g., MedAlpaca: F1 = 47.3 -> 78.5 F1). Current multimodal models demonstrated lower performance (GPT-4o = 70.2 F1; Qwen = 66.0). Performance gains aligned with the distributional similarity between synthetic and real speech. Integrating transformer embeddings with linguistic features enhances ADRD detection from speech. Clinically tuned LLMs effectively support both classification and data augmentation, while further advancement is needed in multimodal modeling.
Abstract (translated)
阿尔茨海默病及相关痴呆症（ADRD）在美国影响了约500万老年人，然而超过一半的病例仍未被诊断。基于语音的自然语言处理（NLP）为通过语言标志物检测早期认知衰退提供了一种有前景且可扩展的方法。为了开发和评估一种筛查流程，该流程结合了(i) 将变压器嵌入与手工制作的语言特征相融合；(ii) 使用大语言模型（LLM）生成的合成语音进行数据增强测试；以及(iii) 对ADRD检测中的单模态和多模态LLM分类器进行基准测试。使用DementiaBank“偷饼干”任务的转录文本（n = 237）。评估了十种变压器模型，采用了三种不同的微调策略。融合模型结合了表现最佳的变压器的嵌入与110个词汇衍生的语言特征。五个LLM（LLaMA-8B/70B、MedAlpaca-7B、Ministral-8B、GPT-4o）被微调以生成标签条件下的合成语音，该合成语音用于增强训练数据。测试了三种多模态模型（GPT-4o、Qwen-Omni、Phi-4），在零样本和微调设置下进行语音文本分类。融合模型实现了F1 = 83.3（AUC = 89.5）的性能，优于仅使用语言或变压器特征的基本线。用2倍MedAlpaca-7B合成语音增强训练数据后，F1提升至85.7。微调显著提升了单模态LLM分类器的效果（例如，MedAlpaca：F1 = 47.3 -> 78.5 F1）。当前的多模态模型表现出较低的性能（GPT-4o = 70.2 F1；Qwen = 66.0）。性能提升与合成语音和真实语音之间的分布相似性相关。将变压器嵌入与语言特征相结合增强了从语音中检测ADRD的能力。经过临床调优的LLM有效地支持了分类和数据增强，但仍需在多模态建模方面进一步改进。
URL
https://arxiv.org/abs/2508.10027
