Abstract
Recent breakthroughs in natural language processing (NLP) have permitted the synthesis and comprehension of coherent text in an open-ended way, therefore translating the theoretical algorithms into practical applications. The large language-model (LLM) has significantly impacted businesses such as report summarization softwares and copywriters. Observations indicate, however, that LLMs may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. Large-scale benchmarks for accountable LLMs should consequently be developed. Although several empirical investigations reveal the existence of a few ethical difficulties in advanced LLMs, there is no systematic examination and user study of the ethics of current LLMs use. To further educate future efforts on constructing ethical LLMs responsibly, we perform a qualitative research method on OpenAI's ChatGPT to better understand the practical features of ethical dangers in recent LLMs. We analyze ChatGPT comprehensively from four perspectives: 1) \textit{Bias} 2) \textit{Reliability} 3) \textit{Robustness} 4) \textit{Toxicity}. In accordance with our stated viewpoints, we empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks, and hence illustrate them via additional case studies. In addition, we examine the implications of our findings on the AI ethics of ChatGPT, as well as future problems and practical design considerations for LLMs. We believe that our findings may give light on future efforts to determine and mitigate the ethical hazards posed by machines in LLM applications.
Abstract (translated)
近年来在自然语言处理(NLP)方面的进展使得我们可以以开放的方式将连贯的文本合成和理解,从而将理论算法转化为实际应用。大型语言模型(LLM)已经对诸如报告摘要软件和写作人员等企业产生了重大影响。然而,观察表明,LLM可能表现出社会偏见和毒性,导致不负责任行为所带来的伦理和社会危险。因此,应该建立大规模的责任性LLM基准。虽然多个实证研究揭示了高级LLM中存在的一些伦理困难,但目前还没有系统检查和用户研究当前LLM使用的伦理问题。为了进一步教育未来努力在建设负责任的LLM方面的责任,我们对OpenAI的ChatGPT进行了定性研究,更好地理解最近LLM中的伦理危险的实际特征。我们从四个方面对ChatGPT进行了全面分析:1) 偏见2) 可靠性3) 鲁棒性4) 毒性。按照我们提出的论点,我们在不同的样本数据集上 empirical 基准ChatGPT。我们发现,现有基准无法解决大量伦理风险,因此通过额外的案例研究来阐明这些风险。此外,我们检查了我们 findings 对ChatGPT AI伦理的影响,以及LLM 未来的问题和实用设计考虑。我们相信,我们的发现可能会启示未来努力确定和减轻机器在LM应用中带来的伦理风险。
URL
https://arxiv.org/abs/2301.12867
