Abstract
Large language models (LLMs) achieve impressive results in terms of fluency in text generation, yet the nature of their linguistic knowledge - in particular the human-likeness of their internal lexicon - remains uncertain. This study compares human and LLM-generated word associations to evaluate how accurately models capture human lexical patterns. Using English cue-response pairs from the SWOW dataset and newly generated associations from three LLMs (Mistral-7B, Llama-3.1-8B, and Qwen-2.5-32B) across multiple temperature settings, we examine (i) the influence of lexical factors such as word frequency and concreteness on cue-response pairs, and (ii) the variability and typicality of LLM responses compared to human responses. Results show that all models mirror human trends for frequency and concreteness but differ in response variability and typicality. Larger models such as Qwen tend to emulate a single "prototypical" human participant, generating highly typical but minimally variable responses, while smaller models such as Mistral and Llama produce more variable yet less typical responses. Temperature settings further influence this trade-off, with higher values increasing variability but decreasing typicality. These findings highlight both the similarities and differences between human and LLM lexicons, emphasizing the need to account for model size and temperature when probing LLM lexical representations.
Abstract (translated)
大语言模型（LLMs）在文本生成的流畅性方面取得了令人瞩目的成果，然而其语言知识的本质——尤其是其内部词汇的人类相似性——仍不明确。本研究通过比较人类与LLM生成的词汇联想，评估模型在多大程度上能捕捉人类的词汇模式。利用SWOW数据集的英语线索-反应词对，以及三个LLM（Mistral-7B、Llama-3.1-8B和Qwen-2.5-32B）在多种温度设置下新生成的联想，我们考察了（i）词汇频率和具体性等词汇因素对线索-反应词对的影响，以及（ii）LLM反应与人类反应在变异性和典型性上的差异。结果表明，所有模型在频率和具体性上均反映了人类趋势，但在反应变异性和典型性上存在差异。Qwen等较大模型倾向于模仿单一“原型”人类参与者，生成高度典型但变异极小的反应；而Mistral和Llama等较小模型则产生更多变异但典型性较低的反应。温度设置进一步影响这种权衡，较高的温度值会增加变异但降低典型性。这些发现揭示了人类与LLM词汇的异同，强调在探究LLM词汇表征时需考虑模型规模和温度设置。
URL
https://arxiv.org/abs/2603.18171
