Try a small test before you read on. (It's painless, and you grade it yourself.)
Think of the last meeting you sat through and then read an AI recap of, or the last long email thread you caught up on through its summary. Pick one small detail: who raised the objection, which deadline was agreed, whether the figure was twelve or twenty. Got it? Now the harder question. Are you remembering the meeting, or the recap?
If you can't say for sure, you've just met the problem this piece is about. You have probably assumed that if the summary had gotten that detail wrong, you would have caught it. You were there, after all.
That assumption carries a lot of weight now. Six in ten U.S. adults say they read AI search engine summaries, the Pew Research Center reported in June, and summaries have spread, in Georgetown University's words, "from news article previews to workplace meeting transcriptions" to police body-camera footage. Wherever they go, the deal is the same — the machine drafts, a person checks. In September, writing about California's law on AI-drafted police reports, I argued that when even veteran supervisors can't spot the machine, "the honor system was never a system."
A new study from Georgetown and the University of Washington puts the deal itself to the test. Humans in the loop "are often expected to correct for AI's mistakes," the authors write, yet "our work suggests human memory can instead be distorted by these mistakes."
A 25-second crash, two road signs and one swapped sentence
So what did the researchers do? (Bear with the details; they decide what the result can mean.)
Mattea Sim and Tadayoshi Kohno of Georgetown and Yael Eiger of the University of Washington recruited U.S. adults online. Each watched a 25-second animated clip built in Grand Theft Auto V: a red car approaches an intersection with either a stop sign or a yield sign and hits a pedestrian. Then came a 24–48 hour gap — chosen because misleading information does more damage once the original memory has had time to fade.
Back for part two, they read a 21-sentence summary of the clip, one sentence per page, credited either to AI software or to "a person who works as a professional video transcriber." Everyone got the same ChatGPT-written text except one sentence the researchers controlled: the sign either matched what they had seen or was swapped, in ChatGPT's own wording. Then, told to think back to the video rather than the summary, they answered twelve two-choice questions. The one that counted: at which sign had the car stopped? Stop, or yield?
Among the 328 people analyzed, 83.6 percent of those who had read the accurate summary got the sign right, "compared to only 44.8%" of those who had read the misleading one.
Two cautions. These are two groups compared side by side, not one group getting worse. And with two choices, a blind guess is right half the time, so the misled group did no better on that detail than a coin toss would have.
It was only that detail, though. On average, participants got 91.4 percent of the other eleven questions right. They remembered the video; the detail that gave way was the one the summary changed.
Here is where things get interesting: did being told it was AI help? Not measurably. Among those misled, 46.3 percent of people told the summary came from AI got the sign right, against 43.4 percent of those told a person wrote it — a gap the authors found was not significant. Trust in AI and how much people used it didn't change their susceptibility either; the authors treat that "with caution," but suggest that "increasing AI literacy alone may not be enough."
None of this is new psychology. The design echoes a 1978 experiment by Elizabeth Loftus and colleagues, and people in such studies have long been "misled into remembering a yield sign as a stop sign." What's new is a machine as the source.
And how often does AI get a summary wrong on its own? In a separate check, two popular consumer models, ChatGPT-5.5 and Gemini 2.5 Flash-Lite, summarized each video five times in spring 2026, told to "Make sure everything important is included in the summary." All twenty summaries included mistakes, at least seven apiece. Most often they left things out — 51.6 percent of central details on average — and nineteen of twenty omitted "the single most central detail of the event: the car colliding with the pedestrian." (Ten summaries per model is a small sample, as the authors acknowledge, and no basis for ranking the two.)
The full paper is a preprint; a shortened version appears in the proceedings of a peer-reviewed AI ethics conference meeting in Malmö, Sweden, October 12–14.
Doesn't the officer have the footage?
It's the strongest objection. The participants weren't proofreading; they read the summary one sentence at a time and only later answered from memory. An officer using Axon's Draft One gets the opposite instruction, in the product page's words: "Review your draft alongside the original footage and transcript." And drafts arrive within five minutes of an incident ending, by Axon's account, not a day or two later.
The authors are candid about other limits. The crash was an animation ("We did not use actual police body cam footage," Kohno said). The summaries came from consumer chatbots working from video, while Draft One drafts, Axon told GeekWire in 2024, "strictly from the audio transcript" of the body-camera recording (the authors' untested hypothesis: audio-only summaries "could be less accurate"). The memory-test summary was re-prompted until it included the accident, then picked from among the better outputs. And human reports were never a gold standard; the Council on Criminal Justice's nonpartisan AI task force notes that ordinary practice already includes "an under-measured baseline error rate."
So where does the objection run out?
First, review on paper isn't always review in practice: in records Mother Jones obtained in 2025, six of the seven departments that responded had "turned off features that required officers to review AI-generated reports."
Second, rewatching the footage catches a wrong sentence only if something prompts you to look. The commonest error in the study was omission, and a missing sentence doesn't look wrong.
Third, noticing isn't immunity. Spotting a discrepancy can reduce the effect, the authors note, but Loftus has written that "false memories can still occur even if a discrepancy is noticed," and sketched what the rememberer sometimes thinks: "Gee, I thought I saw a stop sign, but the new information mentions a yield sign, I guess I must be wrong and it was a yield sign."
And the report outlives the review. "When cases take months or years to reach trial, officers often depend on their reports to refresh their memory about key details," notes Fair and Just Prosecution, a network of reform-minded prosecutors.
In King County, prosecutors refused the draft itself
Is a signature enough? One prosecutor's office said no, two years before this study.
In September 2024, the King County Prosecuting Attorney's Office in Seattle told local police chiefs that it "will not accept any police report narratives that have been produced with the assistance of AI. All reports must be produced entirely by the authoring officer." (Departments had asked about Axon's Draft One and other tools, ChatGPT among them.) The memo's reasoning reads like this study's premise in a practitioner's words: "While an officer is required to edit the narrative and assert under penalty of perjury that it is accurate, some of the errors are so small that they will be missed in review." Its example was an otherwise excellent report that mentioned "an officer who was not even at the scene."
The memo insisted, "We do not fear advances in technology," and Axon replied at the time that "critical safeguards require every report to be edited, reviewed and approved by a human officer" — the very safeguard the memo doubted. Fair and Just Prosecution calls King County "the first prosecutorial office in the country to publicly announce" such a refusal. It's an office policy, not a law, and could change without a vote; on September 17, 2026, a Scripps News report still listed King County among the places where "prosecutors have hit the brakes."
The states known to have legislated took the other road. At least two states, California and Utah, have passed laws on AI-written police reports, and both chose a label and a signature. California requires a statement, on each page or within the body of the text, that AI helped write the report, plus the officer's signature verifying that "the facts contained in the official report are true and correct." Utah requires a disclaimer and, in the Electronic Frontier Foundation's summary, requires officers to certify that they checked the report for accuracy. R Street's Logan Seacrest, on the free-market right, gave both states credit: "One thing Utah and California both get right is keeping a human in the loop."
Even Washington's own AI Task Force recommended "officer attestation of completed review" for AI-written reports, though the Legislature enacted that recommendation only "in part" — and the part was about license-plate readers. A Justice Department community-policing newsletter summed up the standard arrangement in January 2025: a disclosure, and "the officer's signature testifies to the accuracy of the document." Truleo's chief executive describes the same sign-off for its own tool.
In 2026, the refusal side grew. In April, Connecticut's chief state's attorney, Patrick Griffin, backed by the state's police chiefs and State Police, imposed a statewide moratorium on AI programs "to draft, author and/or narrate criminal reports" while the software is tested and rules are written. Under present technology, he said, officers would be "essentially swearing under oath to what a computer believes it heard." The moratorium mooted a public defenders' proposal for a label and a signature. And in mid-September, Wayne County, Michigan's prosecutor, Kym Worthy, stopped short of refusal but urged departments "to be cautious and restrained" while she writes a countywide policy.
So which side does the study favor? To be clear, it backs King County's worry without proving King County's remedy. It shows that a reader's memory can absorb an error in a summary; it doesn't show how often police software makes such errors, or that refusing the draft beats labeling it. What it changes is what a signature can mean. A California officer signs that the facts are "true and correct." After a fluent draft, some of what feels true may have come from the draft.
Imagine the summary as the only version anyone reads
What follows is my extrapolation, not the study's. Every piece of it is already for sale, on someone's agenda or in a lab.
Imagine the review screen of 2029. The draft arrives with a planted mistake to prove someone read it (Draft One can already insert a suspect "absolved by presenting a convincing alibi involving a time loop"). The officer deletes the time loop, the log records a careful review, and a swapped road sign sails through — because nothing about it looks wrong. Outlandish errors "may be more easily corrected," the authors note: a Utah draft that said an officer had been turned into a frog, after the software picked up The Princess and the Frog playing in the background, was corrected before the report was filed. Utah lawmakers tried this year to make agencies publish whether officers may switch such safeguards off; the bill died on March 6.
Imagine the summary becoming the record. Axon told investors in May that its Brief One works by "compressing hours of video review into minutes through AI-powered summarization"; Mark43 sells a BriefAI that summarizes case information. The study tested neither. But picture the prosecutor, the defense lawyer and the judge each reading the summary instead of the footage, and the officer, a year later on the stand, refreshing a memory from a report the machine drafted.
Imagine the interview going the same way. In an MIT Media Lab experiment (a 2024 preprint), a chatbot interviewer whose questions included five misleading ones induced "over 3 times more immediate false memories than the control" in a simulated witness interview.
And imagine it leaving policing. Axon says "Draft One, for example, is expanding into healthcare," where clinicians already sign AI-drafted notes; in the study I wrote about in August, one in seven notes was signed with no edits at all. Sim's own next question is whether this can "manipulate people's memories of police-civilian interactions." The answer isn't in yet.
What a signature proves depends on whom you ask
Who else is worried, and who isn't?
On the civil-liberties side, the ACLU's Jay Stanley warned in late 2024 that an AI rehash of the footage "can reshape and contaminate the officer's memory," and concluded that "we do not think police departments should use this technology." (This study comes close to measuring that warning, if not in officers.) The Electronic Frontier Foundation calls the label laws "an important first step" and predicted in December 2025 that more states would join in "regulating, or perhaps even banning" AI-written reports. EPIC's Calli Schroeder put the stakes in a line: an AI transcript or summary "may be 98% accurate, but that 2% could completely change who is at fault."
Even Seacrest warns that "automation bias — the tendency to defer to computers — can lull officers into a false sense of security." Ben Green, a policy scholar who surveyed 41 policies requiring human oversight of government algorithms, concluded that such policies "provide a false sense of security in adopting algorithms."
The study's authors see their work as part of "growing evidence that the human-in-the-loop strategy is not enough to account for AI's mistakes." Sim calls AI "a new method of delivering misinformation." Their deeper point is a question rather than a verdict: "whether AI should be used in these contexts at all."
In the middle sits the Council on Criminal Justice's task force (its funders include Microsoft). "Automation bias is acute in this context," it says, and its fix is the most practical on the table: where feasible, an independent statement from the officer, written before reviewing the AI draft, "to preserve independent recall."
Every voice here keeps a human in the picture. The quarrel is over what that human's signature proves: a safeguard, or a false sense of security.
What does this mean for you?
Write down what you remember before you read the summary. It's the task force's advice for officers, and you can borrow it for any meeting. In one study, recall within 24 hours helped preserve eyewitness memory for at least a month; whether it also shields you from later misinformation is contested, so treat it as a habit that may help.
Treat the summary as a claim. As I wrote in August about AI search answers, "Treat the summary as a claim, not a conclusion." For anything that matters, go back to the recording, the thread or the source.
Look for what's missing, not just what's wrong. Nineteen of twenty summaries in the study left out the collision itself.
Don't count on the label, or on your own skepticism. In the study, neither helped measurably (a finding the authors read cautiously, and so should you).
If you sign it, you're vouching for your memory, not the machine's. California's signature line says the facts are "true and correct." Before you sign anything a model drafted (a report, a chart note, the minutes), check the details that matter against the source, not the draft.
If you or someone close to you faces a criminal case, know what to look for. In California, the report must say AI helped write it and the first draft must be kept; ask the lawyer for the retained first draft and the audit trail, as my September piece advised. In Utah, look for the disclaimer and the certification. In King County, prosecutors say they won't accept AI-assisted narratives. Elsewhere, ask the department in writing whether it uses AI drafting.
The lesson, as I see it
The human in the loop was meant to be the check on the machine. This study suggests the loop runs both ways: the machine's draft can edit the human, too. That doesn't make a signature worthless, but it makes it a different promise from the one the law assumes, closer to I read this and it seemed right than to this matches what I saw.
I've written about automation bias, "a failure in the moment, with the tool present," and about what's left of a skill once the tool is taken away. This is a third failure, and a quieter one — the tool's mistake becomes your memory, and you sign it in good faith.
What to watch? Los Angeles police commissioners approved a $235 million Axon contract in September, and its text says Draft One's phased rollout to officers begins January 2028, with a pilot when the department is ready after the contract is executed. Wayne County's policy is still being written, and Connecticut's pause, as last reported, holds.
King County's prosecutors didn't want officers unwittingly "certifying false police reports," so they refused the draft. The legislatures that acted kept the draft and trusted the signature. Whichever way your own jurisdiction goes, the cheapest protection is yours to take: write down what you saw before a machine tells you what you saw.
Whoever signs off on machine-drafted summaries near you (a sergeant, a nurse, the person who keeps the minutes) deserves the test at the top of this piece. Send it their way.