Educational Psychology Review Education
How Generative AI Influences Learning Outcomes in Programming Education: A Three-level Bayesian Meta-analysis
Abstract
Generative AI (GenAI) has shown promise in programming education, yet empirical evidence remains inconsistent due to varied pedagogical and technical contexts. This research addressed this inconsistency through a three-level Bayesian meta-analysis of 35 empirical studies with 131 effect sizes published between 2022 and 2025. Learning outcomes were classified into AI-assisted programming outcomes (AIPO), independent programming outcomes (IPO), higher-order skills (HOS), and motivational-emotional outcomes (MEO). Results demonstrated credible small-to-medium positive effects of GenAI on all four outcome categories, and no credible difference among categories was detected. The moderator analyses did not detect credible moderation by educational contexts, instructional designs, or GenAI system designs in any outcome category. A few exploratory pairwise differences were credible for MEO under specific educational contexts and strategy-training conditions. Within the current evidence, GenAI shows consistent positive average effects, whereas the conditions that strengthen or weaken these effects remain undetermined. Because the corpus is small and unevenly distributed across moderator levels, the absence of detected moderation may reflect limited statistical information rather than equivalence across conditions. Larger and more balanced studies with more complete reporting of implementation details are needed to identify these conditions.