Title: Transfer of Large Language Model Generated Feedback Beyond Assisted Programming Tasks A Secondary Crossover Analysis of ProgFeed
Authors: Mushtaq Talib Tally
Volume: 10
Issue: 9
Pages: 115-120
Publication Date: 2026/09/28
Abstract:
Large language models can improve students' performance during programming tasks, but whether such gains transfer to later work completed without artificial intelligence remains uncertain. This study presents a secondary crossover analysis of ProgFeed, an open classroom dataset containing 6693 submissions from 215 consenting students in an introductory Python course. The analysis focused on three counterbalanced intervention periods followed by artificial intelligence free prelabs designed as posttests of the preceding lab. These periods yielded 607 student lab assignments from 211 students, while 89 students completed 190 subsequent posttest observations. Baseline first submission performance was comparable across assigned conditions, and prior feedback assignment did not significantly predict posttest completion. Fixed effects analyses controlling for student and period found no detectable advantage of either test case feedback or natural language feedback in first attempt posttest performance compared with no artificial intelligence feedback. First attempt and eventual full correctness were also not significantly different. The findings suggest a performance transfer gap in which feedback can support debugging within an assisted task without necessarily producing detectable gains on a subsequent independent task. Because the posttests were optional and completion was limited, the estimates should be interpreted cautiously.