Effectiveness of ChatGPT in Coding: A Comparative Analysis of Popular Large Language Models-Reference-Cited by-同舟云学术

Effectiveness of ChatGPT in Coding: A Comparative Analysis of Popular Large Language Models

Published:2024-01-08 Issue:1 Volume:4 Page:114-125
ISSN:2673-6470
Container-title:Digital
language:en
Short-container-title:Digital

Author:

Coello Carlos Eduardo Andino¹,Alimam Mohammed Nazeh¹,Kouatly Rand¹^ORCID

Affiliation:

1. Faculty of Tech and Software Engineering, University of Europe of Applied Sciences, 14469 Potsdam, Germany

Abstract

This study explores the effectiveness and efficiency of the popular OpenAI model ChatGPT, powered by GPT-3.5 and GPT-4, in programming tasks to understand its impact on programming and potentially software development. To measure the performance of these models, a quantitative approach was employed using the Mostly Basic Python Problems (MBPP) dataset. In addition to the direct assessment of GPT-3.5 and GPT-4, a comparative analysis involving other popular large language models in the AI landscape, notably Google’s Bard and Anthropic’s Claude, was conducted to measure and compare their proficiency in the same tasks. The results highlight the strengths of ChatGPT models in programming tasks, offering valuable insights for the AI community, specifically for developers and researchers. As the popularity of artificial intelligence increases, this study serves as an early look into the field of AI-assisted programming.

Publisher

MDPI AG

Link

https://www.mdpi.com/2673-6470/4/1/5/pdf

Reference31 articles.

1. Language models are few-shot learners;Brown;Adv. Neural Inf. Process. Syst.,2020

2. Fan, L., Li, L., Ma, Z., Lee, S., Yu, H., and Hemphill, L. (2023). A bibliometric review of large language models research from 2017 to 2023. arXiv.

3. Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.T., Wang, S., and Lin, X.V. (2023, January 23–29). Lever: Learning to verify language-to-code generation with execution. Proceedings of the International Conference on Machine Learning 2023, Honolulu, HI, USA.

4. OpenAI, and Pilipiszyn, A. (2023, November 26). GPT-3 Powers the Next Generation of Apps. Available online: https://openai.com/blog/gpt-3-apps/.

5. Hardesty, L. (2023, July 25). Explained: Neural Networks, MIT News. Massachusetts Institute of Technology. Available online: https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414.

Cited by 6 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Testing the Feasibility of Schema.org Metadata Refinement Through the Use of a Large Language Model;Journal of Library Metadata;2024-08-27

2. Comparison of the Performance of ChatGPT, Claude and Bard in Support of Myopia Prevention and Control;Journal of Multidisciplinary Healthcare;2024-08

3. Transforming Children's Python Turtle Graphics Learning with LLM Technology: A Design Proposal;2024 9th International STEM Education Conference (iSTEM-Ed);2024-07-31

4. Dear ChatGPT – can you teach me how to program an app for laboratory medicine?;Journal of Laboratory Medicine;2024-05-15

5. Applications of Large Language Models in Pathology;Bioengineering;2024-03-31