ICSE 2025
Sat 26 April - Sun 4 May 2025 Ottawa, Ontario, Canada
Tue 29 Apr 2025 09:50 - 10:10 at 210 - APR Session 1 Chair(s): Tegawendé F. Bissyandé, Chao Peng

LLMs have long demonstrated remarkable effectiveness in automatic program repair (APR), with OpenAI’s ChatGPT being one of the most widely used models in this domain. Through continuous iterations and upgrades of GPT-family models, their performance in fixing bugs has already reached state-of-the-art levels. However, there are few works comparing the effectiveness and variations of different versions of GPT-family models on APR. In this work, inspired by the recent public release of the GPT-o1 models, we conduct the first study to compare the effectiveness of different versions of the GPT-family models in APR. We evaluate the performance of the latest version of the GPT-family models (i.e., O1-preview and O1-mini), GPT-4o, and the historical version of ChatGPT on APR. We conduct an empirical study of the four GPT-family models against other LLMs and APR techniques on the QuixBugs benchmark from multiple evaluation perspectives, including repair success rate, repair cost, response length, and behavior patterns. The results demonstrate that O1’s repair capability exceeds that of prior GPT-family models, successfully fixing all 40 bugs in the benchmark. Our work can serve as a foundation for further in-depth exploration of the applications of GPT-family models in APR.

Tue 29 Apr

Displayed time zone: Eastern Time (US & Canada) change

09:00 - 10:30
APR Session 1APR at 210
Chair(s): Tegawendé F. Bissyandé University of Luxembourg, Chao Peng ByteDance
09:00
50m
Keynote
Baishakhi Ray's Keynote
APR
Baishakhi Ray Columbia University
09:50
20m
Talk
Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs
APR
Haichuan Hu Alibaba Cloud, Tongke Zhang Nanjing University, Guolin Xu Chongqing University, Congqing He University Sains Malaysia, Quanjun Zhang Nanjing University
10:10
20m
Talk
Memorization in LLM-Based Program Repair
APR
Jiaolong Kong Singapore Management University, Mingfei Cheng Singapore Management University, Xiaofei Xie Singapore Management University