For a long time I argued about facial recognition the way almost everyone argues about it — which is to say, wrongly.
I went straight for the accuracy numbers. Error rates. False positives. Whether the software performed worse on some faces than others. It felt rigorous — but I was reading a specification sheet when I should have been reading a booking log.
Here is what finally rearranged my thinking, and it has almost nothing to do with how good the software is. A face recognition result is not an identification. It is a lead — a ranked list of faces that resemble a face, produced by a machine that has no idea who anyone is. Everybody agrees on this. The vendors print it in the report. The trade association that sells the systems says so in public. Departments write it into their own manuals.
And then, in August 2024, deputies drove to a house in Fort Myers, Florida and arrested a man over an incident at a fast-food restaurant at the other end of the state. Robert Dillon spent the night in a cell. To get out, according to the lawsuit he filed this June, he had to borrow money and pledge the title to his truck to post bond.
The restaurant was more than 300 miles away from his home.
That is the gap this piece is about: the distance between what a face match is and what it gets treated as. Not a technical gap. A human one.
First, what the machine actually hands you
Let me define the thing plainly, because the argument collapses without it.
Fingerprints and DNA are confirmatory: you run the sample, and the science tells you how unlikely it is that the match belongs to anyone else. Face recognition does not do that. The industry's own advocates say so in the law-enforcement trade press — and note the messenger: Don Erickson, chief executive of the Security Industry Association. The software "does not confirm a match," it "generates multiple images from a database that are the most similar," and results are never to be considered the sole basis for probable cause.
So the rule is not contested — it is printed on the product. In one Ohio prosecution, the vendor's own disclaimer warned police that Clearview AI's findings were not "intended or permitted" to be used as admissible evidence, and the search result still ended up as the reason a man was charged.
Nor is the weakness a secret. The government's own testing program at the National Institute of Standards and Technology reported in July 2022 that false positive differentials "are widespread, occuring even in pristine images." The failure mode that survives good photography is the machine pointing confidently at the wrong person. Georgetown Law's Center on Privacy and Technology has a name for wiring a tool like that into a criminal case anyway — a forensic without the science.
I have written before about how London says scan every face and Brussels says ban it. That was an argument about whether to switch the cameras on. This is the harder one: what happens after they produce a name.
What the complaint says happened next
Here I have to be careful, and I want you to hold me to it. Everything that follows about the Florida officers is an allegation from Dillon's complaint. The case is pending, no court has found any of it, and both Jacksonville Beach police and the Jacksonville Sheriff's Office declined to comment to CBS News. But the allegations are specific, and that is what makes this case matter.
The suit was filed in the Middle District of Florida on June 10, 2026 by the ACLU and the ACLU of Florida, and it asks for two things: damages, and a rule — policy changes to stop this happening to the next person. Its opening line is the whole argument in a sentence: this is a case about what happens when police let an error-prone AI system stand in for an investigation.
What the lawsuit says police did not tell the judge should make you sit up. A restaurant employee described the man they were looking for as a regular at that location. Dillon lived five hours away and had told police he had never been to Jacksonville Beach in his life. An automatic license plate reader search turned up no sign of his car near the restaurant in the days around the incident. And investigators, the complaint says, never requested or obtained mobile ordering records, payment data, or online account information from the restaurant itself.
Read that list again. Not one item on it requires a warrant, a subpoena, a laboratory or a specialist. It requires a phone call and an afternoon.
This is a pattern, not an outlier. In Maryland, an Oklahoma woman named Kimberlee Williams spent months in jail on charges brought after a face search. The Washington Post found that across the incident reports and charging documents, no officers reported investigating her whereabouts or whether she had any connection to the state at all — and that police never told the court she had been identified by facial recognition. By the ACLU's April tally she was the fourteenth person publicly known to have been arrested this way; two months later, once Dillon's case was filed, fifteen. Read those as at least — they are the cases that surfaced, not a census.
In New York, Trevis Williams was jailed two days without matching the description the victim gave — the man police wanted, as Williams put it, was eight inches shorter and seventy pounds lighter — and his own phone data placed him miles away. In a second Florida case, Jalil Richardson was flagged as an 85 percent match in a Jacksonville stolen-car case while living in Charlotte. Time cards showed he was at work four hundred miles away. He was held thirty-three days in North Carolina, extradited, and held another fifty-three.
Eighty-six days. The exonerating evidence existed the entire time. It was on a time card.
"But the technology works now" — and that is exactly the problem
Let me take the other side seriously, because the strongest argument for police facial recognition is not the one critics usually knock down.
It goes like this: you are fighting the last war. The bias findings that made headlines came from an earlier generation of algorithms, and the systems have improved enormously since. The data broadly supports that. Two researchers at the Federation of American Scientists reported this May that from 2020 to 2025 the error rate fell by a factor of three. The Bipartisan Policy Center is blunter still: the most accurate identification algorithms are now highly accurate overall and across demographic groups, and roughly 97 percent of end-to-end system errors trace back to the camera rather than the matching algorithm.
So if your case rests on "it is inaccurate," you are standing on a floor being replaced under your feet.
Here is the trouble. The arrests did not stop.
Why not? Because the component that fails is not the ranking algorithm. It is the person reading the screen. Those same FAS researchers say it directly: the failures were not just the product of a bad model but of human failure to follow effective procedures. And the Bipartisan Policy Center — hardly a radical outfit — names the hole in our testing regime precisely: we measure the algorithm exhaustively, and there is no standardized third-party testing that measures the impact of automation bias, the technical term for the very human habit of believing the screen.
We have built an elaborate science for auditing the machine and no science at all for auditing the trust we place in it.
A forensics writer at Forbes put it best: the technology did what it was designed to do, it found a face that looked similar, and everything that went wrong after that was entirely human. A lead is supposed to trigger an investigation. It is not supposed to replace one.
And the standard fix — a warning label on the output — has demonstrably failed. The ACLU's answer, backed by case after case, is that officers treat the face recognition result as a positive identification regardless. Detroit's own police chief conceded something more corrosive still: dropping the software's top candidate into a photo lineup can taint the lineup. The "independent" witness identification that supposedly corroborates the match has already been steered by it.
None of this requires bad faith — only a tired detective at the end of a long shift, holding a name the computer produced with a confidence score attached.
The rule already exists. It is just not the law where you live.
Put two things side by side, because the contrast is the story.
On one side, written policy. Departments that deploy this technology typically have a rule requiring officers to corroborate a match before acting on it. On paper, American policing already agrees with everything above.
On the other side, what happened. When The Washington Post pulled the records, it found that of twenty-three departments where detailed facial recognition records are available, fifteen departments across twelve states had arrested suspects identified by AI matches with no independent evidence connecting them to the crime — in most cases contradicting their own internal policies. The Electronic Frontier Foundation says it in fewer words: officers routinely ignore protocol and arrest whoever the computer put at the top of the list.
A rule broken by roughly two-thirds of the departments you can actually audit is not a policy. It is a press release.
What does an enforceable version look like? It exists in one American city, by accident of litigation. After Robert Williams was arrested in Detroit in 2020 and subjected to some thirty hours in a cell, the case settled in June 2024 on terms that converted guidance into obligation: Detroit police must now back up face recognition results with independent and reliable evidence linking a suspect to a crime before making any arrest. That is the fix. It is written down. It binds exactly one department.
Zoom out and the map should bother you regardless of your politics. By one policy tracker's count, a face match cannot be the sole basis for an arrest in seven states — Alabama, Colorado, Maine, Maryland, Montana, Virginia and Washington. In the other forty-three, whether the computer's guess is enough to put you in handcuffs is a matter of departmental preference. Federal legislation has been introduced. Congress has not acted.
The courts have barely begun. In late June 2026 New Jersey's Supreme Court held that prosecutors must hand over discovery identifying the tools used, including the name and manufacturer of the software and publicly available information about its error rates — though the ruling is narrower than the headlines suggest, making disclosure case-by-case rather than a blanket entitlement. In Ohio, where a trial judge threw out evidence in January 2025 because undisclosed reliance on Clearview meant police lacked probable cause, an appeals court sent the case back without answering it that September. Do not file that as a win for anyone. File it as the state of American law: unresolved.
The ACLU of Ohio named the mechanism in its amicus brief: when police hide their use of the technology from judges, it undermines the courts' ability to protect constitutional rights. The magistrate is not weighing probable cause at all. He is initialling a search result he cannot see, described to him as though a person had done the work.
Now run the tape forward five years
Everything above describes a system where a human still types the query and clicks the button. That constraint is dissolving, so let me get imaginative.
Picture the near version, neither science fiction nor far off. The cameras are already on the poles; the matching happens live. A patrol terminal does not wait to be asked — it pushes. A name arrives before the officer does, with a photograph, a similarity score to two decimal places, and a prior-contacts summary assembled from half a dozen databases. The officer steps out of the car already holding a hypothesis with a number attached, and every question from that point is confirmatory. Not because anyone is corrupt, but because that is what a mind does with a hypothesis it did not have to work for.
Then add the layer I find genuinely unsettling: automated corroboration. Some earnest vendor will notice that departments keep getting sued for failing to corroborate, and will sell them a module that does it automatically — cross-referencing the candidate against plate readers, location data, social accounts, utility records — returning a tidy verdict: CORROBORATED. It will be right most of the time. When it is wrong, the affidavit will say corroboration was performed, which will be technically true and entirely hollow.
Nobody wrote down that they had checked in the Maryland case. In the automated version, everybody will have written down that they checked.
Which is why I keep returning to a principle I have argued here before: handing a task to an agent does not hand off your responsibility along with it. A detective who delegates an identification to software has still made the identification. The signature at the bottom of that affidavit is not a formality — it is the architecture of accountability, and we are automating our way out from under it.
What the smart people are saying — and how little they disagree
The spread of serious opinion here is unusually narrow, and that should carry weight with you. The civil-liberties left, as you have seen, is not asking for a ban — the ACLU's Florida suit seeks damages and a policy change.
From the libertarian right, Reason — a magazine that agrees with the ACLU about very little — reached the same conclusion covering the same case: no such investigative tool should form the sole basis for an arrest warrant.
From the center-right, the R Street Institute warns that coupling cameras with real-time recognition and connectivity risks "creating a biometric surveillance dragnet that treats every citizen interaction as a criminal investigation."
From the defense bar, the National Association of Criminal Defense Lawyers calls what departments are deploying "unregulated, untested, and flawed" — a criticism of deployment, not of physics.
And from the industry: never the sole basis for probable cause — the vendors' own trade association stating the thing the ACLU is now suing to enforce.
Everyone agrees on the rule. Agreeing on a rule and being bound by one are entirely different conditions.
What does this mean for you?
You are unlikely to be arrested on a bad face match. But your jurisdiction has made a decision about this on your behalf, probably without telling you.
Ask for the corroboration rate, not the accuracy rate. When someone tells you the system is 99-point-something percent accurate, agree — then ask what the department does with the output. How many arrests last year followed a match and nothing else? That number, not the match rate, is the story.
Check whether your state is one of the seven. If it is, you have a floor. If it is not, your protection is a departmental memo that fifteen of twenty-three audited departments were found to be departing from.
If you are ever named, ask how. "How was I identified?" is a plain-English question you or your lawyer should ask early and get answered on the record. If software is in the answer, the follow-ups are: which product, what error rate, and was the judge told any of this?
Do not comfort yourself with an alibi. This is the bleak lesson of Richardson's time cards and Trevis Williams's phone data: the exonerating evidence usually exists, and nobody goes looking. Receipts do not protect you if the procedure never reaches them.
If you write to one official this year, write one sentence. No arrest on a face match alone. That is already Detroit's binding obligation, seven states' law and the vendors' own stated position. Not a radical ask — a request that everyone's existing promise be made enforceable.
The lesson, as I see it
The machine did not arrest Robert Dillon. A person did — after reading a name off a screen and, according to the complaint, declining to spend an afternoon on the phone calls that would have ended it.
That is why "is it accurate?" turned out to be the wrong question, and why I regret the years I spent asking it. Accuracy is a property of a component. Justice is a property of a procedure. You can improve the component forever — and researchers genuinely are — but if the procedure lets a similarity score walk into a courtroom wearing the costume of an identification, all you have built is a faster route to the same night in a cell.
What I find almost hopeful is how small the remedy is. The fix fits on an index card: a face match is a lead, corroborate it before you arrest, and tell the judge you used it. Detroit signed that. Seven states legislated it. The industry advertises it. The only missing ingredient is the unglamorous step of making it binding — and then checking.
My vote? Keep the software, if it earns its place. Keep the afternoon of phone calls too. The moment we treat those as alternatives rather than partners, the machine stops being a tool and becomes a verdict — and a verdict nobody signed is the one thing a free country cannot afford to automate.
The HAIA Foundation exists for exactly this question: not whether a technology works, but who stays accountable when it does. If that is the kind of thinking you want more of, subscribe — it arrives quietly, and it costs nothing.
