Title: Evaluation of ChatGPT Usability as A Code Generation Tool Show Chinese title

Title: ChatGPT作为代码生成工具的可用性评估

Abstract: With the rapid advance of machine learning (ML) technology, large language models (LLMs) are increasingly explored as an intelligent tool to generate program code from natural language specifications. However, existing evaluations of LLMs have focused on their capabilities in comparison with humans. It is desirable to evaluate their usability when deciding on whether to use a LLM in software production. This paper proposes a user centric method. It includes metadata in the test cases of a benchmark to describe their usages, conducts testing in a multi-attempt process that mimic the uses of LLMs, measures LLM generated solutions on a set of quality attributes that reflect usability, and evaluates the performance based on user experiences in the uses of LLMs as a tool. The paper reports an application of the method in the evaluation of ChatGPT usability as a code generation tool for the R programming language. Our experiments demonstrated that ChatGPT is highly useful for generating R program code although it may fail on hard programming tasks. The user experiences are good with overall average number of attempts being 1.61 and the average time of completion being 47.02 seconds. Our experiments also found that the weakest aspect of usability is conciseness, which has a score of 3.80 out of 5. Our experiment also shows that it is hard for human developers to learn from experiences to improve the skill of using ChatGPT to generate code. Show Chinese abstract

Abstract: 随着机器学习（ML）技术的快速发展，大型语言模型（LLMs）越来越被探索作为一种智能工具，从自然语言规范生成程序代码。 然而，现有的LLM评估主要集中在与人类能力的比较上。在决定是否在软件生产中使用LLM时，评估其可用性是很有必要的。 本文提出了一种以用户为中心的方法。它包括在基准测试用例中加入元数据，以描述其用途，在多尝试过程中进行测试，模拟LLM的使用，根据反映可用性的质量属性来衡量LLM生成的解决方案，并基于用户在使用LLM作为工具时的体验来评估性能。 本文报告了该方法在评估ChatGPT作为R编程语言代码生成工具的可用性中的应用。 我们的实验表明，尽管ChatGPT可能在困难的编程任务上失败，但它对于生成R程序代码非常有用。用户的体验良好，总体平均尝试次数为1.61次，完成平均时间为47.02秒。 我们的实验还发现，可用性的最薄弱方面是简洁性，得分为3.80分（满分5分）。 我们的实验还表明，对于人类开发人员来说，很难通过经验学习来提高使用ChatGPT生成代码的技能。