When students use AI, do they learn more, or do they only produce more?
Seventeen recent studies that tried to measure what students could do once the AI was taken away.
What this is
This is a biweekly reading of new research on generative AI in university teaching. Each issue takes one question and gathers recent studies that bear on it, from as many angles as the evidence allows. There is one featured article per issue, read in depth, and a set of shorter reviews written so you can decide in a minute whether the original is worth your time.
It is meant to start conversations, not settle them. Nothing here is institutional guidance or policy. It is one reader's account of the field, shared with colleagues at MRU and anywhere else it is useful. Disagreement is welcome, and the listening groups are where it goes.
This issue's question
When students use AI, do they learn more, or do they only produce more?
The two are easy to confuse. A student with ChatGPT open writes a better essay, solves more problems and finishes sooner. Whether that student can do any of it the following week, without the tool, is a separate question, and it is the one a degree certifies. Every study below tries to measure the second thing: what students could do once the AI was taken away, or whether AI raised their performance at all.
A note on how these were chosen. We read 109 full papers for this issue, and most did not hold up: statistics that did not match their own tables, one class per condition, effects too large to be believed. These ten are 2026 studies that survived a close reading of the full text, from seven countries and nine fields, plus an international review. Each review says how much weight it can bear. The long edition has seven more, including two widely cited 2025 studies.
Featured article
A semester of access to a course chatbot made no difference to exam scores
Thoeni & Fryer (2026). AI chatbots in higher education: Comparing expectations to evidence. Computers in Human Behavior Reports, 22, 101061. Read the paper
Randomized trial over a full semester · 454 students · Peer reviewed · April 2026
If any study was set up to show that an AI study assistant helps, it was this one. In an introductory marketing course at a public university in the southeastern United States, about 450 students were randomly assigned, within their sections, to get access to a course chatbot or not. This was not a generic ChatGPT link. The chatbot was built on the course materials, and in testing it could answer 199 of 200 exam questions correctly. Students in the comparison group got credit for using Quizlet instead.
At the end of the semester, on proctored exams, the chatbot made no difference. The effect was 0.015 of a standard deviation, with a margin of error running from about minus 0.08 to plus 0.10. There was also no effect on students' interest in the subject, their confidence, or the time they spent with the e-textbook. Students liked the chatbot, rating their enjoyment of it at about 80 out of 100. The authors put the gap plainly: "students' satisfaction with an AI chatbot does not equal students learning from the AI chatbot."
The obvious objection is that students barely used it: on average less than one login a week, and about 68 words per session. That is a fair reading, and it cuts both ways. A tool that is available, accurate and well liked, and that students mostly ignore, is a realistic picture of what many institutional AI licences will look like in practice.
How much weight it bears. This is the most robust study in the issue: large, randomized, a full semester, measured on proctored exams with a precise null result, and its arithmetic checks out. The weaknesses are real but modest. Usage was low. Students in the chatbot group gave up the Quizlet points the other group kept. Nobody measured whether the comparison group used public AI tools on their own. The exams came at the end of term, not weeks later.
For discussion. If a well-built, accurate course chatbot changes nothing when students can use it as they please, is the problem the tool, or the assumption that access alone does anything? And what should we read into the high satisfaction ratings that accompany so many AI pilots?
When students used AI freely, learning did not follow
Randomized ChatGPT access raised unaided test scores a little, and only for students who used it to understand
Contractor & Reyes (2026). Experimental evidence on the learning impact of generative AI. arXiv preprint 2607.08849. Read the paper
Randomized experiment · 211 undergraduates · USA · Not yet peer reviewed · July 2026, revised October 2026
At Middlebury College, students spent 35 minutes learning an unfamiliar topic and writing an essay, with ChatGPT either allowed or forbidden. They were then tested without AI, right away and again a week later without warning. AI access raised scores by 0.28 standard deviations immediately and 0.21 a week later, though the one-week result is only marginally significant. Students who used the AI to explain concepts kept their gains. The seven who mostly had it write for them produced the strongest essays and lost the advantage a week later.
Two details are worth the full read. Students who were not given AI predicted it would raise their score by 25.5 points, more than six times the real effect. And the authors warn against the obvious reading: time on task was fixed in the lab, so "AI helped learning here" does not mean "allowing AI in your course will help learning." Weight: strong design, small effects, one selective college, a short task, a preprint.
On a national math platform, students stopped spending time on problems they could paste into a chatbot
Rismanchian et al. (2026). Faster completion, less learning: Generative AI reduced study time on math problems and the knowledge they build. arXiv preprint 2605.21629. Read the paper
Trace data from millions of interactions · USA · Not yet peer reviewed · Four of five authors work for the platform's owner
This paper uses ALEKS, a math platform used at hundreds of colleges, to compare word problems, which can be pasted into ChatGPT, with graph problems, which cannot easily be. Over eleven quarters after ChatGPT's release, college students' time on word problems fell about 27% relative to graph problems. Under proctoring, the gap disappeared. On proctored retention items assigned at random, the odds of a correct answer on word problems fell about 25%.
If it holds, unsupervised online homework in gateway math courses may now partly measure the chatbot rather than the student. Weight: large and directly relevant to college courses. The retention result is fragile, the pre-ChatGPT trends were not flat, and the company that sells the platform wrote the paper.
Reading with Copilot or ChatGPT open did not improve comprehension of a difficult text
Marquès-Donoso, Benguría-Aguirreche, Neira-Calama & Sánchez-Huete (2026). Generative AI use and reading comprehension: Associations with metacognitive experience in higher education. Youth, 6, 144. Read the paper
Quasi-experiment, six conditions · 531 students · Spain · Peer reviewed · 1 October 2026
Students at two Madrid institutions, mostly in teacher education with some in law and business, read a dense regulatory text. Some read it alone. Others used Copilot or ChatGPT, with or without instructions, and one group had a 35-minute session on using AI well beforehand. They were then tested with the text closed and no AI. No AI condition beat plain reading. On literal comprehension, the no-AI group scored highest. The group that had received AI training scored lowest on interpretation and reported the least confidence.
The authors' summary applies well beyond this study: "The confidence reported by students cannot be taken as a direct indicator of their comprehension." Weight: large, the statistics reproduce, and the authors are candid. But groups differed by semester and institution as well as condition, and the test was close to its ceiling.
ChatGPT users learned more vocabulary at first, then forgot faster than dictionary users
Albalawi, Alzeer, Alharbi, Alsharidi, Alqahtani & Asiri (2026). Comparing the effectiveness of ChatGPT and dictionaries in second language vocabulary learning. International Journal of Lexicography, 39(1), 1–16. Read the paper
Quasi-experiment, three conditions · 93 students · Saudi Arabia · Peer reviewed · June 2026
English learners at five Saudi universities looked up unfamiliar words with ChatGPT, with an online learner dictionary, or not at all. Right afterwards, the ChatGPT group recalled the most: 12.3 words out of 20, against 10.0 for dictionary users. A week later the ChatGPT group had dropped to 7.3, while dictionary users held at 9.1. The authors: "while ChatGPT may facilitate task completion or production efficiency in some contexts, such advantages do not necessarily imply stronger memory traces."
Weight: a delayed outcome with a fair non-AI comparison, in a solid journal. But it is small, classes rather than students were assigned, and the tests were taken online without supervision.
Students allowed to use GPT-4o during a real-patient exam scored no higher
Saloojee, Gramanie, Mwale, Bassett, Madhi & Kalla (2026). AI at the bedside: Randomised controlled trial of ChatGPT's impact on student performance in real-patient clinical exams. Medical Teacher. Read the paper
Randomized trial · 73 final-year medical students · South Africa · Peer reviewed · April 2026
This one tests the other half of the question: does AI even raise performance? Final-year students at four Johannesburg teaching hospitals were randomly allowed or not allowed to use GPT-4o on their phones while examining a real patient for a scored assessment. Scores were 66.7 with ChatGPT and 68.2 without. How much students used it made no difference. Most found it helpful, yet 37% said it distracted them and 45% said it slowed the encounter. The authors' line: "AI should be a scaffold, not a shortcut."
Weight: a registered trial in a genuinely authentic setting with open data. It was too small to detect anything but a large effect, and it measures performance with AI rather than learning.
When the design kept students thinking, it helped
Students who judged AI's work learned more than students whose work AI judged
Fan & Chang (2026). Telling students to evaluate does not make it happen: Task stage, offloading tendency, and error detection in AI-assisted student writing. Behavioral Sciences, 16(9), 1671. Read the paper
Randomized experiment · 744 students randomized · China · Peer reviewed · September 2026
Over six weeks, students writing graded reports were randomly assigned a division of labour with the AI. In one arm the AI drafted and the student critiqued and revised. In another the student drafted and the AI gave feedback. In a third the AI revised and the student accepted or rejected each change with a reason. A fourth arm used no AI. On a new report written without AI, the two arms where students judged AI-produced text did best. AI feedback on the student's own draft was statistically identical to no AI at all.
The catch is who benefited. Students who were already inclined to hand off thinking quietly asked the AI to do the critiquing too, and gained nothing. In the authors' words: "Prescribing a division of labour governs the assignment, not the cognition." Weight: large, randomized and blind-scored, with numbers that reproduce. Students in different arms shared class sections, the test was immediate only, and the study's timeline should be confirmed with the authors.
A scaffolded AI revision tool beat plain ChatGPT on essays written a week later without AI
Huang (2026). Designing a scaffolded GenAI revision environment for EMI academic writing: A randomized active-control study of no-AI transfer and feedback use. Interactive Learning Environments. Read the paper
Randomized trial · 428 students · Taiwan · Peer reviewed · August 2026
Students in a university-wide English-medium writing module revised a draft either with standard ChatGPT or with a version that asked them to think first and explain which suggestions they accepted and why. A week later, they wrote a new essay in class, with no AI and no internet. The scaffolded group scored 75.9 against 73.0, about a third of a standard deviation. The paper's opening line states this issue's question better than we can: "Generative artificial intelligence (GenAI) can improve assisted drafts without demonstrating students' ability to write independently."
Weight: randomized, large, blind-rated, delayed and unaided, with a plausible effect size. It was a single activity, the paper does not report attrition by group or an ethics approval, and the author notes that some secondary data could not be re-verified.
Engineering students who tried first and checked with Copilot after did better months later
Syromiatnykov et al. (2026). Independent engineering transfer after traceable generative-AI-assisted learning: A six-university controlled trial. Applied Sciences, 16, 8949. Read the paper
Controlled trial, 28 classes at six universities · 656 students · Uzbekistan · Peer reviewed · September 2026
Agricultural engineering students worked through problems in the same routine: attempt the problem alone, then check. One group checked with Copilot, limited to four turns. The other checked against a curated textbook solution. On a test with no AI and no internet, the Copilot classes scored 2.8 points higher out of 32 a week later, and 2.3 points higher about sixteen weeks later. The authors stress that what they measured was "tool-free engineering transfer rather than performance while AI assistance remained available."
Weight: large, multi-site, with an active comparison and a genuinely delayed unaided test. But classes were not randomly allocated, the trial was not registered, the authors wrote the tests, and the effect sizes are reported generously.
Context
A meta-analysis of AI in STEM learning finds the large average effect is mostly publication bias
Boolzen, Kuhn, Flegr, Rott, Stausberg & Küchemann (2026). Evidence of impact and interpretational limits of generative AI in STEM education: A systematic review and meta-analysis on cognitive learning outcomes. Artificial Intelligence Review, 59, 223. Read the paper
Preregistered review of 85 studies · Peer reviewed · August 2026
Pooled the usual way, studies of generative AI in STEM learning show a large positive effect, g = 0.84. Corrected for the tendency of positive results to get published and null ones not to, it falls to 0.08, which the authors call effectively no overall effect. What remains is a design lesson that matches Part 2 above: AI comparisons look good mainly when the AI group was given a more active task than the comparison group. In other words, much of this literature compares good teaching with AI against weaker teaching without it.
Weight: large, preregistered and bias-corrected, and the headline reproduces. It pools school and university studies, includes some pre-LLM chatbots, and has reporting inconsistencies in its subgroup tables, so use the headline and not the details.
What the ten add up to
Access alone did not produce learning. A course chatbot for a whole semester, AI while reading, AI for vocabulary lookup, AI during a clinical exam: none of these left students measurably better off. The vocabulary study and the math platform suggest free access can leave them worse off.
Arrangements that kept students thinking did better. Judging AI's drafts, explaining which AI suggestions to accept, and attempting a problem before checking with AI all improved what students could do alone. The STEM meta-analysis points the same way: the positive effects in the literature come mostly from more active designs.
Students' feelings were not a guide. Students enjoyed the chatbot that did nothing, and readers felt more confident than their scores justified. Satisfaction with an AI tool is not evidence of learning.
How this issue was made
Candidate papers came from searches of OpenAlex and of a larger review database covering January 2025 to early October 2026, about 1,170 abstracts in all. An AI model screened them against written inclusion rules and rated how rigorous each looked. We retrieved full texts through legal open-access routes, and a further 30 promising papers through the library. AI readers produced a fact sheet for each of the 109 full papers, quoting numbers with page references and recomputing statistics where possible. Most papers did not survive that step. I chose these ten from those that did, for fit to the question, quality, and range of disciplines and settings, and checked the key claims against the papers before writing. The AI drafted parts of the reviews, and I rewrote them.
Preprints are labelled, as is the study whose authors work for the platform it examines. This is one reader's view of a fast-moving literature, not institutional guidance, and corrections are welcome.
Reading list
| # | Study | Field, country | Evidence | Status |
|---|---|---|---|---|
| F | Thoeni & Fryer (2026), Computers in Human Behavior Reports | Marketing, USA | Randomized trial, 454 | Peer reviewed |
| 1 | Contractor & Reyes (2026), arXiv | Mixed majors, USA | Randomized experiment, 211 | Preprint |
| 2 | Rismanchian et al. (2026), arXiv | Math, USA | Platform trace data | Preprint, industry authors |
| 3 | Marquès-Donoso et al. (2026), Youth | Education, Spain | Quasi-experiment, 531 | Peer reviewed |
| 4 | Albalawi et al. (2026), International Journal of Lexicography | English vocabulary, Saudi Arabia | Quasi-experiment, 93 | Peer reviewed |
| 5 | Saloojee et al. (2026), Medical Teacher | Medicine, South Africa | Randomized trial, 73 | Peer reviewed |
| 6 | Fan & Chang (2026), Behavioral Sciences | Writing, China | Randomized experiment, 744 | Peer reviewed |
| 7 | Huang (2026), Interactive Learning Environments | Academic writing, Taiwan | Randomized trial, 428 | Peer reviewed |
| 8 | Syromiatnykov et al. (2026), Applied Sciences | Agricultural engineering, Uzbekistan | Controlled trial, 656 | Peer reviewed |
| 9 | Boolzen et al. (2026), Artificial Intelligence Review | STEM, international | Review of 85 studies | Peer reviewed |
What this is
This is a biweekly reading of new research on generative AI in university teaching. Each issue takes one question and gathers recent studies that bear on it, from as many angles as the evidence allows. There is one featured article per issue, read in depth, and a set of shorter reviews written so you can decide in a minute whether the original is worth your time.
It is meant to start conversations, not settle them. Nothing here is institutional guidance or policy. It is one reader's account of the field, shared with colleagues at MRU and anywhere else it is useful. Disagreement is welcome, and the listening groups are where it goes.
This issue's question
When students use AI, do they learn more, or do they only produce more?
The two are easy to confuse. A student with ChatGPT open writes a better essay, solves more problems and finishes sooner. Whether that student can do any of it the following week, without the tool, is a separate question, and it is the one a degree certifies. Every study below tries to measure the second thing: what students could do once the AI was taken away, or whether AI raised their performance at all.
A note on how these were chosen. We read 109 full papers to find these 17. Most did not hold up: statistics that did not match their own tables, one class per condition, effects too large to be believed. Every study here survived a close reading of the full text. They come from eleven countries and more than a dozen disciplines, and only four are from medicine, which is otherwise the field that publishes the most trials on this topic. Each review says how much weight the study can bear.
Featured article
A semester of access to a course chatbot made no difference to exam scores
Thoeni & Fryer (2026). AI chatbots in higher education: Comparing expectations to evidence. Computers in Human Behavior Reports, 22, 101061. Read the paper
Randomized trial over a full semester · 454 students · Peer reviewed · April 2026
If any study was set up to show that an AI study assistant helps, it was this one. In an introductory marketing course at a public university in the southeastern United States, about 450 students were randomly assigned, within their sections, to get access to a course chatbot or not. This was not a generic ChatGPT link. The chatbot was built on the course materials, and in testing it could answer 199 of 200 exam questions correctly. Students in the comparison group got credit for using Quizlet instead.
At the end of the semester, on proctored exams, the chatbot made no difference. The effect was 0.015 of a standard deviation, with a margin of error running from about minus 0.08 to plus 0.10. There was also no effect on students' interest in the subject, their confidence, or the time they spent with the e-textbook. Students liked the chatbot, rating their enjoyment of it at about 80 out of 100. The authors put the gap plainly: "students' satisfaction with an AI chatbot does not equal students learning from the AI chatbot."
The obvious objection is that students barely used it: on average less than one login a week, and about 68 words per session. That is a fair reading, and it cuts both ways. A tool that is available, accurate and well liked, and that students mostly ignore, is a realistic picture of what many institutional AI licences will look like in practice.
How much weight it bears. This is the most robust study in the issue: large, randomized, a full semester, measured on proctored exams with a precise null result, and its arithmetic checks out. The weaknesses are real but modest. Usage was low. Students in the chatbot group gave up the Quizlet points the other group kept. Nobody measured whether the comparison group used public AI tools on their own. The exams came at the end of term, not weeks later.
For discussion. If a well-built, accurate course chatbot changes nothing when students can use it as they please, is the problem the tool, or the assumption that access alone does anything? And what should we read into the high satisfaction ratings that accompany so many AI pilots?
Part 1 · When students used AI freely, learning did not follow
Randomized ChatGPT access raised unaided test scores a little, and only for students who used it to understand
Contractor & Reyes (2026). Experimental evidence on the learning impact of generative AI. arXiv preprint 2607.08849. Read the paper
Randomized experiment · 211 undergraduates · USA · Not yet peer reviewed · July 2026, revised October 2026
At Middlebury College, students spent 35 minutes learning an unfamiliar topic and writing an essay, with ChatGPT either allowed or forbidden. They were then tested without AI, right away and again a week later without warning. AI access raised scores by 0.28 standard deviations immediately and 0.21 a week later, though the one-week result is only marginally significant. Students who used the AI to explain concepts kept their gains. The seven who mostly had it write for them produced the strongest essays and lost the advantage a week later.
Two details are worth the full read. Students who were not given AI predicted it would raise their score by 25.5 points, more than six times the real effect. And the authors warn against the obvious reading: time on task was fixed in the lab, so "AI helped learning here" does not mean "allowing AI in your course will help learning." Weight: strong design, small effects, one selective college, a short task, a preprint.
On a national math platform, students stopped spending time on problems they could paste into a chatbot
Rismanchian et al. (2026). Faster completion, less learning: Generative AI reduced study time on math problems and the knowledge they build. arXiv preprint 2605.21629. Read the paper
Trace data from millions of interactions · USA · Not yet peer reviewed · Four of five authors work for the platform's owner
This paper uses ALEKS, a math platform used at hundreds of colleges, to compare word problems, which can be pasted into ChatGPT, with graph problems, which cannot easily be. Over eleven quarters after ChatGPT's release, college students' time on word problems fell about 27% relative to graph problems. Under proctoring, the gap disappeared. On proctored retention items assigned at random, the odds of a correct answer on word problems fell about 25%.
If it holds, unsupervised online homework in gateway math courses may now partly measure the chatbot rather than the student. Weight: large and directly relevant to college courses. The retention result is fragile, the pre-ChatGPT trends were not flat, and the company that sells the platform wrote the paper.
Reading with Copilot or ChatGPT open did not improve comprehension of a difficult text
Marquès-Donoso, Benguría-Aguirreche, Neira-Calama & Sánchez-Huete (2026). Generative AI use and reading comprehension: Associations with metacognitive experience in higher education. Youth, 6, 144. Read the paper
Quasi-experiment, six conditions · 531 students · Spain · Peer reviewed · 1 October 2026
Students at two Madrid institutions, mostly in teacher education with some in law and business, read a dense regulatory text. Some read it alone. Others used Copilot or ChatGPT, with or without instructions, and one group had a 35-minute session on using AI well beforehand. They were then tested with the text closed and no AI. No AI condition beat plain reading. On literal comprehension, the no-AI group scored highest. The group that had received AI training scored lowest on interpretation and reported the least confidence.
The authors' summary applies well beyond this study: "The confidence reported by students cannot be taken as a direct indicator of their comprehension." Weight: large, the statistics reproduce, and the authors are candid. But groups differed by semester and institution as well as condition, and the test was close to its ceiling.
ChatGPT users learned more vocabulary at first, then forgot faster than dictionary users
Albalawi, Alzeer, Alharbi, Alsharidi, Alqahtani & Asiri (2026). Comparing the effectiveness of ChatGPT and dictionaries in second language vocabulary learning. International Journal of Lexicography, 39(1), 1–16. Read the paper
Quasi-experiment, three conditions · 93 students · Saudi Arabia · Peer reviewed · June 2026
English learners at five Saudi universities looked up unfamiliar words with ChatGPT, with an online learner dictionary, or not at all. Right afterwards, the ChatGPT group recalled the most: 12.3 words out of 20, against 10.0 for dictionary users. A week later the ChatGPT group had dropped to 7.3, while dictionary users held at 9.1. The authors: "while ChatGPT may facilitate task completion or production efficiency in some contexts, such advantages do not necessarily imply stronger memory traces."
Weight: a delayed outcome with a fair non-AI comparison, in a solid journal. But it is small, classes rather than students were assigned, and the tests were taken online without supervision.
ChatGPT improved essays more than a human expert did, and students learned no more
Fan et al. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology. Read the paper
Randomized lab experiment, four conditions · 117 students · Peer reviewed · 2025
Students revised an essay with help from ChatGPT, a human writing expert, checklist tools, or nothing. The ChatGPT group's essays improved most: 3.6 points out of 25, about twice the others. On knowledge gain and transfer there was no difference at all: the ChatGPT group's transfer score was 0.782, against 0.785 for students with no help. Trace data showed their revision looping around the chatbot rather than back to the readings.
This is older than our usual window, but it is cited in nearly every talk on the subject, so we read it in full. "Metacognitive laziness" is a catchy label for something the study did not measure directly. Weight: a strong design with a human-expert comparison, a single one-hour task, and the version we read is the preprint.
Design students with ChatGPT produced as many ideas, but more of them already existed
Baltà-Salvador, Brasó-Vives & Peña (2026). Evaluating AI-assisted creative ideation: A crossover study in higher education. Thinking Skills and Creativity, 59, 101958. Read the paper
Randomized crossover · 51 students · Spain · Peer reviewed
Design-engineering students in Barcelona brainstormed product ideas twice, once with ChatGPT and once without, in random order. ChatGPT did not change how many ideas they produced, how varied they were, or how original. It did change their character: ideas that resembled existing products nearly doubled, and descriptions got shorter. Nearly half of all prompts were generic requests along the lines of "give me innovative ideas," with a median length of eight words.
Weight: a randomized design in a real course with useful prompt logs. It is small, and the authors' claim that using AI first makes students more creative later is not supported by the design.
In a statistics course, practising with ChatGPT made no overall difference to quiz scores
Kabba & Ejbari (2026). ChatGPT as scaffold: Quiz performance across session complexity. International Journal of Evaluation and Research in Education, 15(4), 3172–3181. Read the paper
Quasi-experiment, intact classes · 193 students · Morocco · Peer reviewed · August 2026
Tourism students in Tangier practised statistics with or without ChatGPT across seven sessions, then took quizzes with AI banned. Overall averages were 7.30 versus 7.06, no real difference. Six of seven sessions showed essentially nothing. One session, on harmonic means and grouped-data modes, showed a large gain for the AI classes.
The authors suggest AI starts to help when formulas get complex. A skeptic would note that the "complex" label was applied after the result. Weight: a useful null from a real course with unaided quizzes. Classes were not randomized, and the one positive session could be chance.
Students allowed to use GPT-4o during a real-patient exam scored no higher
Saloojee, Gramanie, Mwale, Bassett, Madhi & Kalla (2026). AI at the bedside: Randomised controlled trial of ChatGPT's impact on student performance in real-patient clinical exams. Medical Teacher. Read the paper
Randomized trial · 73 final-year medical students · South Africa · Peer reviewed · April 2026
This one tests the other half of the question: does AI even raise performance? Final-year students at four Johannesburg teaching hospitals were randomly allowed or not allowed to use GPT-4o on their phones while examining a real patient for a scored assessment. Scores were 66.7 with ChatGPT and 68.2 without. How much students used it made no difference. Most found it helpful, yet 37% said it distracted them and 45% said it slowed the encounter. The authors' line: "AI should be a scaffold, not a shortcut."
Weight: a registered trial in a genuinely authentic setting with open data. It was too small to detect anything but a large effect, and it measures performance with AI rather than learning.
Part 2 · When the design kept students thinking, it helped
Students who judged AI's work learned more than students whose work AI judged
Fan & Chang (2026). Telling students to evaluate does not make it happen: Task stage, offloading tendency, and error detection in AI-assisted student writing. Behavioral Sciences, 16(9), 1671. Read the paper
Randomized experiment · 744 students randomized · China · Peer reviewed · September 2026
Over six weeks, students writing graded reports were randomly assigned a division of labour with the AI. In one arm the AI drafted and the student critiqued and revised. In another the student drafted and the AI gave feedback. In a third the AI revised and the student accepted or rejected each change with a reason. A fourth arm used no AI. On a new report written without AI, the two arms where students judged AI-produced text did best. AI feedback on the student's own draft was statistically identical to no AI at all.
The catch is who benefited. Students who were already inclined to hand off thinking quietly asked the AI to do the critiquing too, and gained nothing. In the authors' words: "Prescribing a division of labour governs the assignment, not the cognition." Weight: large, randomized and blind-scored, with numbers that reproduce. Students in different arms shared class sections, the test was immediate only, and the study's timeline should be confirmed with the authors.
A scaffolded AI revision tool beat plain ChatGPT on essays written a week later without AI
Huang (2026). Designing a scaffolded GenAI revision environment for EMI academic writing: A randomized active-control study of no-AI transfer and feedback use. Interactive Learning Environments. Read the paper
Randomized trial · 428 students · Taiwan · Peer reviewed · August 2026
Students in a university-wide English-medium writing module revised a draft either with standard ChatGPT or with a version that asked them to think first and explain which suggestions they accepted and why. A week later, they wrote a new essay in class, with no AI and no internet. The scaffolded group scored 75.9 against 73.0, about a third of a standard deviation. The paper's opening line states this issue's question better than we can: "Generative artificial intelligence (GenAI) can improve assisted drafts without demonstrating students' ability to write independently."
Weight: randomized, large, blind-rated, delayed and unaided, with a plausible effect size. It was a single activity, the paper does not report attrition by group or an ethics approval, and the author notes that some secondary data could not be re-verified.
Engineering students who tried first and checked with Copilot after did better months later
Syromiatnykov et al. (2026). Independent engineering transfer after traceable generative-AI-assisted learning: A six-university controlled trial. Applied Sciences, 16, 8949. Read the paper
Controlled trial, 28 classes at six universities · 656 students · Uzbekistan · Peer reviewed · September 2026
Agricultural engineering students worked through problems in the same routine: attempt the problem alone, then check. One group checked with Copilot, limited to four turns. The other checked against a curated textbook solution. On a test with no AI and no internet, the Copilot classes scored 2.8 points higher out of 32 a week later, and 2.3 points higher about sixteen weeks later. The authors stress that what they measured was "tool-free engineering transfer rather than performance while AI assistance remained available."
Weight: large, multi-site, with an active comparison and a genuinely delayed unaided test. But classes were not randomly allocated, the trial was not registered, the authors wrote the tests, and the effect sizes are reported generously.
Novices who studied with ChatGPT alone did worst, until a lecture followed
Digiacomo et al. (2025). ChatGPT vs traditional pedagogy: A comparative study in urological learning. World Journal of Urology, 43, 286. Read the paper
Randomized trial, three conditions · 121 medical students · Italy · Peer reviewed · 2025
Italian medical students had 90 minutes to learn a new topic with ChatGPT alone, a lecture alone, or ChatGPT followed by a lecture. ChatGPT plus lecture beat ChatGPT alone by about 2.6 points out of 30. Lecture alone was not significantly better than ChatGPT alone. The authors' takeaway: "ChatGPT loses its potential if used without a previous background." Their headline claim, that the combination beats a lecture, is never actually tested.
Weight: a real randomized comparison, a single session, and an unvalidated test.
Part 3 · Where AI appeared to help, and why to read carefully
Clerkship students with an AI assistant did better on knowledge and clinical skills, and kept it for three months
Yu et al. (2026). Effectiveness of artificial intelligence–assisted peer teaching in orthopedic clinical education: Historical cohort study. JMIR Medical Education. Read the paper
Historical cohort comparison · 190 students · China · Peer reviewed · 2026
Students in a 2025 orthopedic clerkship who could consult DeepSeek during structured peer teaching scored about half a standard deviation higher than the 2024 cohort on a knowledge exam and a clinical skills exam. The knowledge advantage held at three months, and both cohorts forgot at the same rate, which the authors read as evidence that AI did not create dependency.
Weight: well reported and the statistics reproduce, but it compares two different years rather than randomized groups. Both cohorts sat identical exam items, and the 2025 peer teachers were likely 2024 examinees.
A free chatbot alongside video lessons helped dental students who struggled with spatial reasoning
Huang, Wen, Bai et al. (2025). Exploring the application capability of ChatGPT as an instructor in skills education for dental medical students: Randomized controlled trial. Journal of Medical Internet Research, 27, e68538. Read the paper
Randomized trial · 187 dental students · China · Peer reviewed · 2025
Students learning a dental procedure from videos either had ChatGPT-3.5 alongside for a week or did not. On an automatically scored virtual-reality skill test, the ChatGPT group scored 73.1 versus 65.5. Nearly all the gain came from students with low spatial ability, 70.2 versus 55.4. High-spatial students barely differed.
Weight: a decent-sized registered trial. But "videos plus a responsive helper" beating "videos alone" is a low bar, and some of the paper's numbers do not reconcile.
Part 4 · Context
A meta-analysis of AI in STEM learning finds the large average effect is mostly publication bias
Boolzen, Kuhn, Flegr, Rott, Stausberg & Küchemann (2026). Evidence of impact and interpretational limits of generative AI in STEM education: A systematic review and meta-analysis on cognitive learning outcomes. Artificial Intelligence Review, 59, 223. Read the paper
Preregistered review of 85 studies · Peer reviewed · August 2026
Pooled the usual way, studies of generative AI in STEM learning show a large positive effect, g = 0.84. Corrected for the tendency of positive results to get published and null ones not to, it falls to 0.08, which the authors call effectively no overall effect. What remains is a design lesson that matches Part 2 above: AI comparisons look good mainly when the AI group was given a more active task than the comparison group. In other words, much of this literature compares good teaching with AI against weaker teaching without it.
Weight: large, preregistered and bias-corrected, and the headline reproduces. It pools school and university studies, includes some pre-LLM chatbots, and has reporting inconsistencies in its subgroup tables, so use the headline and not the details.
Outside the classroom: AI closed three-quarters of an education gap, and most of it came back without the tool
Cruces, Fernández Meijide, Galiani, Gálvez & Lombardi (2026). Does generative AI narrow education-based productivity gaps? Evidence from a randomized experiment. arXiv preprint. Read the paper
Preregistered randomized experiment · 1,174 working adults · Argentina · Not yet peer reviewed · Not students
Adults with and without postsecondary education did a workplace writing-and-analysis task, with or without a GPT assistant. AI raised everyone's scores and narrowed the education gap from 0.55 to 0.14 standard deviations. Tested minutes later without the tool, the gap was back to 0.20. About 64% of AI users simply pasted the AI's text. These are not students, so treat it as context, but it is the clearest picture in this issue of the distance between what someone produces with AI and what they can then do.
What the 17 add up to
Access alone did not produce learning. A course chatbot for a whole semester, AI while reading, AI while practising statistics, AI for vocabulary lookup, AI for essay revision: in none of these did free access leave students measurably better off without the tool. Sometimes it left them worse off, as with the faster forgetting in the vocabulary study and the shift away from effort on the math platform.
Arrangements that kept students thinking did better. Judging AI's drafts, explaining which AI suggestions to accept, attempting a problem before checking with AI, and having a lecture to make sense of what AI said all improved what students could do on their own. The STEM meta-analysis points the same way from the other end: the positive effects in the literature come mostly from more active designs.
Students' feelings were not a guide. Students enjoyed the chatbot that did nothing, felt more confident after reading with AI than their scores justified, and vastly overestimated what AI would do for them. Course evaluations of AI tools measure something, but not learning.
The evidence is thin and uneven. We read 109 full papers to find these 17. Very few studies test learning weeks later, and fewer still are randomized at scale. The most reliable results here are null or modest.
Held for later issues
These studies also passed a full-text reading but answer a neighbouring question. They will appear in the issues where they belong.
| Study | Why it is held | Issue |
|---|---|---|
| Modirrousta-Galian et al. (2026), AI-generated practice quizzes | AI-made assessment items | AI as assessor, 14 Dec |
| Xiao & Zhang (2026), hints versus full solutions | Type of AI feedback | AI as assessor, 14 Dec |
| Bauer et al. (2025), adaptive AI feedback in statistics | Type of AI feedback | AI as assessor, 14 Dec |
| Kestin et al. (2025), AI physics tutor at Harvard | Purpose-built tutor | Built for learning, 25 Jan |
| Nathaniel et al. (2026), structured ChatGPT use in programming | Training students to use AI | AI literacy, 8 Mar |
| Connell Pensky et al. (2026), AI-literacy modules across 65 sections | AI-literacy instruction | AI literacy, 8 Mar |
| Zuo et al. (2026), AI-written clinical cases | Instructor-made AI materials | The instructor's side, 22 Mar |
| Gavoille et al. (2026), AI clone of the lecturer's voice | Instructor-made AI materials | The instructor's side, 22 Mar |
How this issue was made
Candidate papers came from searches of OpenAlex and of a larger review database covering January 2025 to early October 2026, about 1,170 abstracts in all. An AI model screened them against written inclusion rules and rated how rigorous each looked. We retrieved full texts through legal open-access routes, and a further 30 promising papers through the library. AI readers produced a fact sheet for each of the 109 full papers, quoting numbers with page references and recomputing statistics where possible. Most papers did not survive that step. I chose these 17 from those that did, for fit to the question, quality, and range of disciplines and settings, and checked the key claims against the papers before writing. The AI drafted parts of the reviews, and I rewrote them.
Preprints are labelled, as is the study whose authors work for the platform it examines. This is one reader's view of a fast-moving literature, not institutional guidance, and corrections are welcome.
Reading list
| # | Study | Field, country | Evidence | Status |
|---|---|---|---|---|
| F | Thoeni & Fryer (2026), Computers in Human Behavior Reports | Marketing, USA | Randomized trial, 454 | Peer reviewed |
| 1 | Contractor & Reyes (2026), arXiv | Mixed majors, USA | Randomized experiment, 211 | Preprint |
| 2 | Rismanchian et al. (2026), arXiv | Math, USA | Platform trace data | Preprint, industry authors |
| 3 | Marquès-Donoso et al. (2026), Youth | Education, Spain | Quasi-experiment, 531 | Peer reviewed |
| 4 | Albalawi et al. (2026), International Journal of Lexicography | English vocabulary, Saudi Arabia | Quasi-experiment, 93 | Peer reviewed |
| 5 | Fan et al. (2025), British Journal of Educational Technology | Writing, China/Australia | Randomized experiment, 117 | Peer reviewed |
| 6 | Baltà-Salvador et al. (2026), Thinking Skills and Creativity | Design engineering, Spain | Randomized crossover, 51 | Peer reviewed |
| 7 | Kabba & Ejbari (2026), IJERE | Statistics, Morocco | Quasi-experiment, 193 | Peer reviewed |
| 8 | Saloojee et al. (2026), Medical Teacher | Medicine, South Africa | Randomized trial, 73 | Peer reviewed |
| 9 | Fan & Chang (2026), Behavioral Sciences | Writing, China | Randomized experiment, 744 | Peer reviewed |
| 10 | Huang (2026), Interactive Learning Environments | Academic writing, Taiwan | Randomized trial, 428 | Peer reviewed |
| 11 | Syromiatnykov et al. (2026), Applied Sciences | Agricultural engineering, Uzbekistan | Controlled trial, 656 | Peer reviewed |
| 12 | Digiacomo et al. (2025), World Journal of Urology | Medicine, Italy | Randomized trial, 121 | Peer reviewed |
| 13 | Yu et al. (2026), JMIR Medical Education | Medicine, China | Cohort comparison, 190 | Peer reviewed |
| 14 | Huang, Wen, Bai et al. (2025), JMIR | Dentistry, China | Randomized trial, 187 | Peer reviewed |
| 15 | Boolzen et al. (2026), Artificial Intelligence Review | STEM, international | Review of 85 studies | Peer reviewed |
| 16 | Cruces et al. (2026), arXiv | Workplace task, Argentina | Randomized experiment, 1,174 adults | Preprint, not students |
Faculty dialogue
Join the conversation
These studies open questions rather than settle them. Where do they match your teaching, and where don't they? What did they miss? Challenge the reading, or the choice of studies.
- A well-built course chatbot made no difference when students could use it as they liked. What would have to be true of an AI tool, or of the assignment around it, for access to matter?
- The arrangements that worked kept students judging, explaining or attempting first. Which of your assignments would change if you designed for that?
- Students repeatedly felt they learned more with AI than tests showed. How should that change the way we read student feedback on AI tools?
- The South African trial found AI did not even raise performance in a real clinical exam. Should assessments allow AI and grade how well students use it, as those authors propose?
Comments open soon
Named comments, written or recorded, are on their way. Until then, bring your thoughts to the conversation in person.
Next gathering: Lunch and learn on this theme: date to be announced