Abstract
Two studies tested the hypothesis that a Large Language Model (LLM) can be used to model psychological change following exposure to influential input. The first study tested a generic mode of influence - the Illusory Truth Effect (ITE) - where earlier exposure to a statement (through, for example, rating its interest) boosts a later truthfulness test rating. Data was collected from 1000 human participants using an online experiment, and 1000 simulated participants using engineered prompts and LLM completion. 64 ratings per participant were collected, using all exposure-test combinations of the attributes: truth, interest, sentiment and importance. The results for human participants reconfirmed the ITE, and demonstrated an absence of effect for attributes other than truth, and when the same attribute is used for exposure and test. The same pattern of effects was found for LLM-simulated participants. The second study concerns a specific mode of influence - populist framing of news to increase its persuasion and political mobilization. Data from LLM-simulated participants was collected and compared to previously published data from a 15-country experiment on 7286 human participants. Several effects previously demonstrated from the human study were replicated by the simulated study, including effects that surprised the authors of the human study by contradicting their theoretical expectations (anti-immigrant framing of news decreases its persuasion and mobilization); but some significant relationships found in human data (modulation of the effectiveness of populist framing according to relative deprivation of the participant) were not present in the LLM data. Together the two studies support the view that LLMs have potential to act as models of the effect of influence.
Abstract (translated)
两项研究测试了假设,即大型语言模型(LLM)可以用来模拟受到有影响力的输入影响后的心理变化。第一项研究测试了一种影响的模式,即幻觉的真实性效应(ITE),该效应是指在早期接触一个陈述(例如评估它的 interest 程度)会增强后来的真实性测试评级。数据是通过在线实验从1000名人类参与者中收集的,以及通过工程刺激和 LLM 完成从1000名模拟参与者中收集的。每个参与者都收到了64个评级,使用了所有暴露和测试的属性组合:真相、兴趣、情感和重要性。人类参与者的结果再次证实了 ITE,并证明了除了真相的其他属性没有影响,而且当使用相同的属性进行暴露和测试时也没有。相同的影响模式也存在于 LLM 模拟的参与者中。第二项研究关注一种特定的影响模式,即民粹主义 framing 新闻来提高其说服和政治动员效果。从 LLM 模拟参与者中收集的数据与之前在15国实验中公开发表的数据进行了比较。之前从人类研究中证明的一些效果被模拟研究重复了,包括令人惊讶地震惊人类研究作者的理论期望的效果(反移民 framing 新闻会降低其说服和动员效果)。但是,一些在人类数据中发现的重要关系(根据参与者相对剥夺的 Modifiable 民粹主义 framing 效果的变化)在 LLM 数据中不存在。两项研究共同支持观点,即LLM 有潜力作为影响效应的模型。
URL
https://arxiv.org/abs/2303.06074
