Abstract
Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user's personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.
Abstract (translated)
大语言模型（LLMs）在学术界和公众中常被讨论为支持文本生成几乎所有应用场景的工具，包括软件工程。目前有很多关于LLM工具的辩论，但很少有实证证据，关于这些工具在工业界工程师中的应用效果。我们对24名职业软件工程师在工作的一个星期内使用ChatGPT进行观察研究，并对其与聊天机器的对话以及整体经验（通过退出调查进行捕捉）进行定性分析。我们发现，实践者更倾向于使用ChatGPT获得有关任务解决方案的指导，而不是期望该工具生成可用的软件输出（例如，代码）。我们也提出了一个理论框架，即（i）交互的目的，（ii）内部因素（例如用户的个性）和（iii）外部因素（例如公司政策）共同塑造了体验（以感知有用性和信任为基础）。我们展望，我们的框架可以为未来的研究提供一个进一步探讨LLM在软件工程师中的应用、为该领域未来实证LLM研究的指导，以及作为未来研究的一个参考点的框架。
URL
https://arxiv.org/abs/2404.14901
