If you are a student, you have probably asked a chatbot to walk you through a problem and gotten the finished answer instead. If you are the parent of one, you have probably made a kinder assumption: that help on tap at two in the morning can only add to what your kid learns. More help, more learning. What could be wrong with that?
Wharton's Ethan Mollick describes the first one precisely: you may want guidance "on how to approach your homework, but it will often just give you the answer instead." The University of Maryland tested the second assumption, running a randomized trial of its own AI tutor in fall 2025 before considering whether to deploy it campus-wide.
Its working paper, dated September 30, 2026, reports that in sections of the same course, final grades came in 4.12 points lower out of 100 where students were offered the tutor, and that they took part far less on the course site. Yet only about 15 percent of the students offered it ever used it. Figuring out how both can be true changes what you should ask before your student leans on one of these tools.
Thirty-four sections, instructors assigned at random, and a chatbot inside the course page
How do you test a tutor fairly? You take the choice away from the students. Randomization "occurred at the instructor level": within blocks of matched courses, each instructor was assigned either to switch the tutor on for the whole class or to change nothing. The study ended with 30 instructors, 2,379 students and 34 course sections, half of them offered the tutor. Nine blocks were sections of the very same course, and that comparison (22 instructors, 1,353 students) is the cleanest, because it "holds course content fixed." Three instructors assigned to offer the tutor never switched it on and were dropped afterward (purists will wince: removing people after randomization can tilt a comparison, and the paper doesn't show results with them left in).
The tutor, the university's Virtual Study Assistant (VSA), was a GPT-4o chatbot built into each course's page. The university's IT division recruited instructors with a promise to "Reduce the amount of repetitive questions during your office hours" and a switch between "answer mode (providing direct answers)" and "tutor mode (guiding students without providing direct answers)." The default gave answers, and only two instructors finished the semester in the guided mode. (Hold on to that detail.)
Maryland's tutor is a different animal from the one I set against an English schools trial in September — Harvard's physics tutor, whose students learned significantly more in less time than in an active-learning class, and which I described then as "a tutor purpose-built on pedagogical principles, used by consenting undergraduates."
So what did it find? In the same-course comparison, being offered the tutor came with final grades 4.12 points lower (0.37 standard deviations). Across the full sample (the paper's primary analysis, mixing different courses), the authors report "not significant evidence of adverse treatment effects," likely, they say, because of "the lack of alignment between courses and grading."
What held everywhere was engagement. In the same-course comparison, students offered the tutor logged about 26 fewer recorded activities on the course site (discussion replies, quizzes and the like) and about seven fewer active days; the authors call the engagement effects "large and statistically significant in both samples." They add that this does not show the drop caused the lower grades, only that the two "tended to co-occur among treated students."
Who is telling us this? The lead author, Jing Liu, is the founding director of the university's Center for Educational Data Science and Innovation; his co-authors include the university's chief information officer, Jeffrey Hollingsworth, and its provost, Jennifer King Rice. That cuts both ways (an institution grading its own product), but I credit them for publishing a result this unflattering. It is also a working paper in EdWorkingPapers, a series whose papers "have not gone through a peer review process."
The tutor most students never touched
How does a tool that about 85 percent of the students offered it never touched move anything? First, what was measured: every headline estimate is the effect of being offered the tutor, not of using it. Of the roughly 15 percent who used it at least once, about 40 percent had a single session. And in the raw same-course averages, "lower final grades were not confined to students who used the VSA but were similar to those who did not use the tool." This study does not show that students who used the tutor did worse than classmates who didn't.
Second, weigh the authors' explanation, offered as an interpretation, not a finding. Students in both groups already had other university-supported AI tools, and the authors suspect that introducing the course tutor changed AI use more broadly: the trial "likely evaluates an impact of broader AI use induced by introducing the VSA and not just the isolated effect of individual VSA use." In a survey (answered by only 18 percent of students, so a clue rather than a count), treated students were more likely to agree that "I asked the instructor fewer questions about course content," roughly 57 percent against 44. The findings, they write, "suggest that introducing an AI tutor may displace engagement with existing learning activities without improving academic performance."
And what did users ask for? Of 3,460 requests, 73.8 percent were classified as "direct-answer requests," a broad bin covering facts, definitions and conceptual explanations as well as solutions; requests for feedback on students' own work were "particularly rare." Such requests do "not necessarily indicate cognitive offloading" (handing the thinking itself to the machine), the authors note, but the results "suggest that students see the VSA as a source of immediate information gathering rather than as a partner in sustained, interactive learning."
One flag, not a verdict: the estimated grade losses among first-generation college students were "more than twice those among continuing-generation students in both samples," which the authors call "the clearest recurring pattern" among student groups, though "statistical precision varies across outcomes and samples."
What would make the four points disappear?
The strongest objections come from the paper itself. First, the outcome is the grade each instructor gave, not an independent test; the authors concede that this "makes interpreting a learning effect challenging," and they couldn't see which exams were proctored. The same-course comparison rests on 22 instructors, the treated ones older on average (about 55 against 40) and assigning fewer homework points, and "sections may still differ in grading." The authors ran an extra check suited to so few groups and report, "Our conclusions are robust to this approach."
Second, the tool was barely introduced. Of ten treated instructors who answered a survey, all mentioned it, 40 percent demonstrated it, and "none incorporated it into tasks or assignments"; the authors say more training might have changed the results.
Third, fewer page views "could reflect more efficient information retrieval, which is one goal of the tool's design," the authors grant. But homework submission held steady while broader participation fell, a sign, they suggest, that students "may have lowered their broader engagement with course activities while continuing to meet assignment requirements."
So do AI tutors lower grades? Liu himself warns against that reading: "These findings do not establish that purpose-built AI tutoring tools are not beneficial," he wrote in a post that EdTech Innovation Hub quoted on October 2, 2026.
Here is where things get interesting. The obvious fix, a tutor that guides instead of answering, hits a different wall. Over two years in 18 Tennessee middle schools, when Khan Academy's Khanmigo refused and instead asked questions or offered hints, students "largely stopped using it" (Khan Academy's founder says its current version pops up on its own after a wrong answer). The researchers, Philip Oreopoulos and Nina Low, concluded in a National Bureau of Economic Research working paper, also not peer-reviewed, that "realizing the promise of AI tutoring will require getting students to use it, not just giving them access." And in a field experiment with nearly a thousand high school math students in Turkey, published in PNAS, those given unrestricted GPT-4 did worse once it was taken away than peers who never had it; a guardrailed tutor version "essentially eradicated" that harm, "though we still do not observe a positive effect."
In short, a tutor that hands out answers may crowd out the work that teaches — and one that withholds them may go unused. Either way, the question is what the tool replaces.
South Korea ran the experiment in the opposite order
What does scaling first and checking later look like? South Korea gave AI textbooks official status, then legislated it away.
In November 2024, Korea's Ministry of Education approved 76 AI digital textbooks in English, math and coding for 2025, pressing ahead, the Korea Times reported, "despite concerns about the effectiveness of digital learning." The plan was AI textbooks in every school. It never got there. On January 21, 2025, Acting President Choi Sang-mok vetoed a proposed revision of the Elementary and Secondary Education Act that would have made them optional educational resources, warning of "a significant blow to the national competitive edge," but limited 2025 to schools that volunteered. That March, 3,870 of 11,932 schools (32.4 percent) had chosen them for at least one subject; "Use of the AI textbooks is optional," the Korea Herald noted.
Then the government changed. On August 4, 2025, the National Assembly voted to reclassify the AI textbooks as "education materials" after resistance from teachers and concerns about screen time, literacy and budgets, Yonhap reported; lawmakers of the opposition People Power Party denounced it as a setback for classroom innovation that deprives underprivileged students of educational opportunities. The ruling Democratic Party passed it "unilaterally," effectively removing "the legal and financial foundations for using AI digital textbooks in schools." The law, in force since August 14, 2025, says AI learning software cannot be approved as a textbook, and as of October 3, 2026, it still does.
The audit came last. In December 2025, Korea's Board of Audit and Inspection found that the ministry had promised a 2024 pilot and "skipped the pilot entirely due to tight development timelines." In schools that chose the textbooks, an average of 60 percent of students never logged in even once, the Kyunghyang Shinmun reported, and the auditor told the ministry to pilot any new kind of textbook and verify its effectiveness first. By the first semester of 2026, Education Ministry figures in the same paper showed 622 schools (5.2 percent) still using them, down from 4,095 (34.2 percent) a year earlier; a ministry official cited shifting perceptions after the change in legal status, among other things.
Kim Cha-myung, an elementary school teacher who found the textbooks convenient, put it more simply: "The program failed because everything was rushed; it should have been rolled out gradually after testing its effectiveness."
To be clear, this compares approaches, not results: nothing here measures Korean students' learning against a control group, and the ministry skipped the very pilot that might have. Korea pushed access from the top and then reversed it by law; Maryland measured first.
Imagine the syllabus of 2029
Maryland ran its trial "prior to considering campus-wide deployment." As of October 3, 2026, no campus-wide rollout had been announced, and the university still lists the assistant among its AI services with a button that reads "Request a deployment." So what follows is my extrapolation, not a forecast.
Imagine every course page shipping with an assistant switched on in answer mode, because defaults stick (few of Maryland's instructors changed it). Imagine the budget meeting where "Reduce the amount of repetitive questions during your office hours" quietly becomes an argument for fewer teaching-assistant hours. Imagine your questions becoming data: instructors can already view students' anonymized prompts in an administrative console, and the authors note that being watched "may also have influenced what students were willing to ask." Scoring those prompts as participation is a short step.
And imagine the whiplash: a state that makes AI tutors official after one election and strips that status after the next, much as Korea did with its textbooks, with no one measuring what students learned in between. As I wrote in August about ten AI-in-schools laws passed by states in 2026, "only one wrote down what the software may not decide."
There is a better version, and an optimist has sketched it. AEI's John Bailey imagines "a coordinated team of AI agents," one of them "deciding when to offer a hint versus when to let the student struggle." That is nearly the opposite of Maryland's default, and which one students get is a configuration choice.
The left, the right and an AI optimist end up in the same place
Who else sees it this way?
On the center-left, Brookings concluded in January 2026 that at this point in AI's trajectory "the risks of utilizing AI in education overshadow its benefits," though tools "explicitly designed to support the learning of children and youth" can "foster enriched learning experiences." (Its subject is school-age students, not undergraduates.)
At MIT, an ad hoc committee on AI in teaching and learning reported in August 2026 that AI has driven "decreased attendance at office hours, reduced participation in online discussions," and warned of "cognitive surrender," when students "fall back on AI at the first hint of struggle."
On the center-right, Bailey argues that "AI works in education when it's used not to give answers but to orchestrate the conditions for productive struggle." On the right, the Manhattan Institute's Brian J.A. Boyd warns, in a report for public university boards, that "The greatest risk of AI to higher education is simply that it will be a substitute for, rather than a complement to, human thought," and says AI "should be banned from the core curriculum." The Heritage Foundation's Annie Chestnut Tutor and Jonathan Butcher, warning that "AI tools designed to supplant cognitive effort can undermine rather than support student growth," want chatbot tutors studied "before deploying them at scale in schools."
Mollick names the mechanism: "the default mode of AI is to do the work for you, not with you."
Set Boyd's sentence beside the Maryland authors' call to study whether design and teaching "can enable AI to be a complement to, rather than a substitute for, existing opportunities for practice, feedback, and human interaction." A conservative think tank, an AI optimist and the tool's own evaluators start from three places and land in the same one: AI as a complement, not a substitute.
What does this mean for you?
Ask which mode the course tool is in. Maryland's default gave answers. If your instructor can flip the switch, ask for the one that guides.
If you study with a general chatbot, try its learning mode, and know its limits. ChatGPT's Study Mode, Claude's Learning Mode and Gemini's Guided Learning each launched in 2025 as something you choose to turn on. That is the catch: when the Tennessee tutor withheld answers, students largely stopped using it. A guided mode works only if you want it to.
Try first, then ask for feedback on your attempt. It was the request Maryland's students almost never made. In the Turkish trial, the guardrailed version avoided the harm (though it produced no gain).
Keep asking people. Office hours and questions to instructors "may feel inefficient, but they are essential building blocks of learning," Liu wrote. The help a bot replaces may be the help that was doing the teaching.
Do some of the work cold, on purpose. The line from my piece on endoscopists fits homework too: "schedule the unassisted rep."
If you are the first in your family at college, or raising someone who is, know that the estimated losses were larger for first-generation students: a reason to guard office hours and study groups, not for alarm.
Parents and campus decision-makers: ask for the evidence before the rollout, and ask what the tool replaces. Liu's test: "when evaluating an AI tool's impact on learning, we need to consider what it might replace." If no one can name it, ask for a pilot first.
Read the headline twice. Offered is not used. A story that says students who used the Maryland tutor got lower grades has misread the study.
The lesson, as I see it
Access is not learning. It sounds obvious written down, yet I suspect it is the unspoken premise behind a great many AI rollouts, and behind the comforting thought that help at two in the morning must be good help.
Maryland deserves credit for doing what Korea skipped: it tested before scaling and published an unflattering result with its own provost's name on it. My September piece praised Estonia for spending public money to find out whether its bet paid off, in a line that fits Maryland too: "That is the posture worth copying. Not the confidence. The checking."
What should you watch? If the university moves toward campus-wide use, ask which mode ships as the default. And Liu has said that a second paper, from a separate study, would follow, EdTech Innovation Hub reported on October 2, 2026. Read it — and whatever evaluation comes next — with the two questions this one taught: what did the tool replace, and did it answer, or teach?
Offer the tutor, by all means. Then count what it replaced.
The HAIA Foundation argues for AI that leaves people more capable, and for testing that promise before anyone scales it. Subscribe if you would rather read the results than the brochure.