Abstract
Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms including conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMs understand \textit{social} language. Here, we introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge which we group into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. In tests on the benchmark, we demonstrate that current models attain only moderate performance but reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero-shot evaluations, we show that pretrained models already possess some innate but limited capabilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides a systematic way to analyze model performance on an important dimension of language and points to clear room for improvement to build more socially-aware LLMs. The associated resources are released at this https URL.
Abstract (translated)
大型语言模型(LLMs)在语法、言语和推理任务中表现出色。尽管LLMs正在越来越应用于各种形式，包括与人类的交互对话机器人，但我们缺乏一个基准来测量LLMs如何理解社交语言。在这里，我们介绍了一个新的基于理论的基准，SocKET，它包含58个NLP任务测试社交知识，我们将它们分为五个类别：幽默和讽刺、攻击性、情感和可靠性。在基准测试中，我们表明，当前模型只能达到中等表现，但揭示了从理论预测的Task转移的重要潜力。通过零样本评估，我们表明预训练模型已经具备一些固有的但有限的社交语言理解能力，而训练在一个类别任务中可以改进其他类别任务的零样本测试。我们的基准提供了一种系统的方式来分析模型在语言一个重要的维度上的表现，并指出了建立更社交意识的LLMs的巨大改进空间。相关的资源在这里发布。
URL
https://arxiv.org/abs/2305.14938
