Title: Characterizing Large Language Model Failure Modes in Code Refactoring and Multi-Turn Reasoning: A Controlled Study Design
Authors: Stanley Nwakamma, Serif Oyindamola Oyesiji, Kingsley Chinazaekpere Ndupu, Emmanuel Eniola Ajala.
Volume: 10
Issue: 9
Pages: 226-237
Publication Date: 2026/09/28
Abstract:
Large language models are increasingly used not to write code from scratch but to change code that already works: renaming and restructuring, extracting functions and modules, migrating interfaces, and tightening designs across successive conversational turns. Refactoring is a demanding target for such assistance because its defining contract, behavior preservation under structural change, is exactly the property that generative models do not natively guarantee, and because realistic refactoring is multi-turn, with requirements, corrections, and constraints arriving incrementally in a dialogue whose earlier content the model must keep honoring. Existing code generation benchmarks measure single-shot synthesis against input-output tests and therefore see neither the preservation contract nor the conversational dynamics. This paper develops a research concept for a systematic empirical study of failure modes in LLM-assisted code refactoring under multi-turn interaction. The concept contributes a failure taxonomy spanning behavior-preservation violations, scope violations, constraint forgetting across turns, instruction-following decay, regression reintroduction, sycophantic acceptance of incorrect user assertions, and self-repair pathologies; a task corpus of refactoring scenarios over real codebases with layered verification, regression suites, differential and metamorphic checks, and refactoring-type-specific invariants; a controlled multi-turn protocol with scripted personas that inject corrections, contradictions, distractions, and erroneous claims at defined points; and an annotation and metrics framework that localizes each failure to taxonomy category, turn, and trigger. Prespecified hypotheses target constraint decay with conversation length, the divergence between edit locality instructed and edit locality performed, and the conditions under which iterative repair converges rather than oscillates. Applications to model evaluation, tooling design, and review practice are analyzed alongside limitations.