Abstract
Traditionally, natural language processing (NLP) models often use a rich set of features created by linguistic expertise, such as semantic representations. However, in the era of large language models (LLMs), more and more tasks are turned into generic, end-to-end sequence generation problems. In this paper, we investigate the question: what is the role of semantic representations in the era of LLMs? Specifically, we investigate the effect of Abstract Meaning Representation (AMR) across five diverse NLP tasks. We propose an AMR-driven chain-of-thought prompting method, which we call AMRCoT, and find that it generally hurts performance more than it helps. To investigate what AMR may have to offer on these tasks, we conduct a series of analysis experiments. We find that it is difficult to predict which input examples AMR may help or hurt on, but errors tend to arise with multi-word expressions, named entities, and in the final inference step where the LLM must connect its reasoning over the AMR to its prediction. We recommend focusing on these areas for future work in semantic representations for LLMs. Our code: this https URL.
Abstract (translated)
传统上，自然语言处理（NLP）模型通常使用由语言专业知识创建的丰富特征集，例如语义表示。然而，在大型语言模型（LLMs）的时代，越来越多的任务被转化为通用序列生成问题。在本文中，我们研究了在LLM时代语义表示的作用：具体来说，我们研究了抽象意义表示（AMR）在五个不同NLP任务上的效果。我们提出了一个基于AMR的思绪提示方法，我们称之为AMRCoT，并发现它通常会损害性能，而不是帮助。为了研究AMR在这些任务上可能提供的优势，我们进行了一系列分析实验。我们发现很难预测AMR可能会帮助或损害哪些输入示例，但错误往往会在多词表达、命名实体和最后推理步骤中出现，LLM必须将推理跨越AMR与预测相结合。我们建议将未来LLM语义表示工作集中在这些领域上。我们的代码：<https://this URL>。
URL
https://arxiv.org/abs/2405.01502
