Who’s Watching the Machines?
Why every AI now needs a flight recorder and an air-traffic control tower — and why that should matter to you, not just to engineers
Let me start with a true story, because it’s funnier than it has any right to be.
In late 2023, a grieving man named Jake Moffatt asked an Air Canada chatbot whether he could get a discounted bereavement fare after a death in the family. The chatbot told him yes — buy the ticket now, apply for the refund later. There was just one problem: that policy did not exist. The bot had invented it. When Moffatt asked for the refund he’d been promised, the airline refused, and then, in a move so bold it almost deserves applause, argued in tribunal that the chatbot was “a separate legal entity responsible for its own actions.”
Just sit with that for a second. A multibillion-dollar airline looked a judge in the eye and said: don’t blame us, blame the robot.
The judge was not amused. Air Canada was held liable and ordered to pay. And a quiet precedent was set — one that applies, eventually, to every company you’ll ever buy something from: you are responsible for what your AI says and does, even when you have no idea what it’s saying or doing.
Which raises the obvious, uncomfortable question. Do these companies actually know what their AI is saying and doing?
Mostly? No. Not yet. And that gap — between how fast we’re handing decisions to machines and how little we can see inside them — is the single most important infrastructure problem of this decade. This article is about the two things being built to close it. Stick with me, because by the end you’ll see your bank, your doctor, your insurer, and your government in a slightly different light.
First, the problem nobody warns you about
Here is the thing about modern AI that the marketing never quite mentions: even the people who build it cannot fully explain why it does what it does.
That’s not a knock on them. It’s the nature of the technology. The man who would go on to found one of the leading companies in this field, Krishna Gade, tells a story from his time running newsfeed engineering at Facebook. Someone would ask a simple question — why am I seeing this particular post? — and the honest answer was that nobody could say for certain. The system had taught itself. It was, in his words, a black box.
Now multiply that black box across your entire life.
The AI deciding whether your résumé gets seen by a human. The algorithm scoring whether you’re a credit risk. The model triaging which patients get extra medical attention. The bot quoting you a price, approving your loan, flagging you for fraud. These aren’t science-fiction scenarios — they are happening, today, in systems most of us never see and cannot question.
So what do you do when your most consequential decisions are being made inside a box you can’t open?
You build instruments to see inside it. And you build a tower to govern it. Those two ideas — in the jargon, AI observability and the AI control plane — are the heroes of this story.
The cockpit dashboard: AI observability
Think about how we made air travel astonishingly safe. We didn’t do it by hoping pilots would be perfect. We did it by instrumenting everything — the cockpit dashboards a pilot scans in real time, the constant telemetry streaming to the ground, and, yes, the black-box recorder for when the worst happens. But here’s the part worth noticing: most of that machinery isn’t about explaining a crash after the fact. It’s about catching the temperature creeping up, the altitude slipping, the fuel burning too fast — while there’s still time to do something.
AI observability is that, for algorithms — and I want to be precise here, because this is the piece most explanations fumble. The technology firm IBM defines it as the ability to understand AI systems by continuously monitoring their telemetry — response quality, token usage, drift, and a great deal more. In human terms: observability is a standing watch over the whole system. Not a one-off answer about a single decision, but an ongoing, panoramic view of how the machine is behaving across thousands or millions of interactions, in real time and over time.
That breadth is the entire point, so let me make it concrete. A good observability setup is constantly asking, without anyone having to prompt it:
Is the model still as accurate today as it was last month? This catches model drift — when the world quietly shifts out from under a model trained on yesterday’s data. A fraud model built before a pandemic, a pricing model built before inflation: both rot in place unless something is watching.
Are the outputs starting to go strange? This flags hallucinations — the polite word for the AI confidently made something up, like Air Canada’s phantom refund policy — along with toxic or unsafe responses, ideally before any customer ever sees them. (And this is no edge case: documented hallucination rates across leading models run, by one Stanford analysis, anywhere from a reassuring fifth of the time to a terrifying almost always, depending on the task.)
Is one group of users quietly getting worse results than another? This surfaces bias as a live, measurable pattern across a whole population — a heartbeat monitor, not a once-a-year audit.
Where, exactly, did this go off the rails? This traces a single request through every step it took — every prompt, every database lookup, every hand-off between AI “agents” — so a failure can be pinpointed instead of guessed at.
Notice what nearly all of those have in common: they are continuous and forward-looking. They’re about catching trouble early, across the entire system, not narrating it after the funeral.
And that is exactly where one popular idea gets mistaken for the whole: explainability (sometimes “XAI”). Explainability is genuinely useful — it pries open a single decision and shows which factors drove it, so “loan denied” becomes “loan denied, mainly because of debt-to-income ratio,” something a human can check and challenge. But look how narrow that is. Explainability answers a backward-looking question about one decision that has already happened. Observability asks the much bigger, never-ending question: how is the entire system behaving, right now and over weeks — and is it about to fail? Explainability is one instrument on the dashboard. Observability is the dashboard, the telemetry, and the pilot watching all of it at once.
Observability, in short, lets you see — broadly, continuously, in time to act. And seeing is the prerequisite for everything else. But seeing, by itself, is not enough.
The control tower: the AI control plane
Here is where things get interesting.
Watching a plane drift off course is useless if nobody has the authority — or the ability — to redirect it. You don’t just want cameras. You want air-traffic control: a tower that sets the rules, clears every takeoff, and can ground a flight before it ever leaves the runway.
In the AI world, that tower is called the control plane. The data firm Atlan puts the distinction beautifully: where the working part of the system processes requests, the control plane “decides what AI is allowed to do — before it acts.“ Other companies building this layer, like the developer-tooling firm Speakeasy, describe it as the single governed pathway through which every prompt, every response, and every action an AI takes must flow — so that identity, permissions, safety rules, and audit logs are enforced in one place rather than scattered and forgotten.
So far so good. Let me make the difference concrete, because it’s the whole ballgame:
Observability is the cockpit dashboard and the telemetry feed. It tells you what the AI is doing across the whole system — continuously, as it happens.
The control plane is air-traffic control. It decides what the AI is allowed to do in the first place — and stops it when it tries to do otherwise.
One sees. The other sees and governs. You want both. All the dashboards and recorders in the world are small comfort if there was never a tower to prevent the crash in the first place.
A short, sobering tour of what goes wrong without them
You might be thinking: fine, but is this really such a big deal? Let me answer with receipts. Every one of these is real, documented, and — crucially — exactly the kind of failure these systems are built to catch.
The $1 Chevrolet. A car dealership in California bolted a chatbot onto its website. A clever customer talked it into agreeing to sell a brand-new SUV for one dollar — “a legally binding offer, no takesie-backsies,” the bot cheerfully added. Funny. Now imagine that same flaw inside a system negotiating real contracts at scale.
The lawyers who cited cases that never existed. In Mata v. Avianca, two New York attorneys filed a legal brief built on six court cases ChatGPT had invented out of thin air — and then “confirmed” were real when asked. They were sanctioned. In the months that followed, more than a dozen copycat cases surfaced. The machine doesn’t know it’s lying. That’s the point.
The healthcare algorithm that saw race it wasn’t supposed to. A landmark study published in Science found that an algorithm used to flag patients for extra care — affecting an estimated 200 million Americans — was systematically under-serving Black patients, because it quietly used past healthcare spending as a stand-in for health need. Since less had historically been spent on Black patients, the model concluded they were healthier. They were not. This is drift and bias, invisible without observability, harming people who never knew an algorithm was in the room.
The scandal that toppled a government. In contrast to a single company’s stumble, consider what happens at the scale of a state. In the Netherlands, a self-learning fraud-detection system used “foreign-sounding names” and dual nationality as risk signals, and falsely branded tens of thousands of families as fraudsters. Parents were ordered to repay sums that ruined them. The fallout was so severe it helped bring down the Dutch government in 2021. A black box, unwatched and ungoverned, doing institutional harm at national scale.
And the original sin: Tay. Way back in 2016, Microsoft released a chatbot named Tay onto Twitter. Within about sixteen hours, coordinated users had taught it to spew racist filth, and it was yanked offline. The lesson was there nearly a decade ago, in flashing neon: an AI without guardrails will be steered straight into the rocks. We are only now building the towers.
I’ll be fair here, because honesty matters: not every alleged AI scandal turns out to be one. When critics accused the Apple Card of giving women lower credit limits, New York’s financial regulator investigated and found no unlawful discrimination. But — and this is the quiet lesson — it still faulted the program for a lack of transparency that “undermined consumer trust.” Sometimes the algorithm is innocent. The opacity is the crime.
The people building the tower
So who, exactly, is building this safety infrastructure? Encouragingly: a growing, competitive field — which is good news, because monocultures are fragile and competition makes the whole ecosystem stronger.
One of the pioneers is Fiddler AI, the company Krishna Gade founded in 2018 after his black-box revelation. Fiddler helped popularize the very term “AI observability,” and its platform tries to do all three jobs at once: watch traditional models for drift and bias, monitor newer large language models for hallucination and toxicity, and — through what it calls its Trust Service — run fast “guardrails” that can catch a bad output before it ever reaches you. Notably, it runs these checks inside a company’s own walls rather than shipping your data off to some external service, which matters more than it sounds when the data in question is your medical record or your bank balance.
But — and I want to be scrupulously even-handed here — Fiddler is one of several. The field includes Arize AI, which recently raised one of the largest rounds the category has seen; Arthur, WhyLabs, and Credo AI, among others, each attacking a slightly different corner of the same problem. The specific logos matter less than the trend they represent: a real industry, with real money and real engineers, has formed around the unglamorous job of keeping AI honest. By some estimates this market is growing at well over twenty percent a year. That tells you something about how serious the underlying problem is.
This isn’t just a vendor pitch — the grown-ups agree
Here’s a reasonable skepticism: of course companies selling AI-safety tools say AI safety is important. Fair. So let me hand the microphone to people with no product to sell.
The U.S. government’s standards body published a risk framework built on four plain-spoken verbs — govern, map, measure, manage. Europe went further and wrote it into law: the EU’s landmark AI regulation requires that high-risk systems be designed for genuine human oversight, explicitly so that a person can step in, and explicitly to counter our tendency to over-trust whatever the machine says. The world’s wealthy democracies, through their shared AI principles, have agreed that transparency, explainability, and accountability aren’t optional niceties. Industry analysts have even coined an acronym for it — AI “TRiSM,” for trust, risk, and security management — which is consultant-speak, yes, but it signals that boardrooms now treat this as a budget line, not a hobby.
And perhaps most tellingly, a panel of ninety-six experts from thirty countries, chaired by one of the field’s most decorated scientists, produced an International AI Safety Report warning that these systems “may malfunction or behave unpredictably” — and that the challenge gets harder, not easier, as we hand AI more autonomy. Even the labs at the frontier are racing to understand their own creations: some of the most fascinating research today is an attempt to literally map the inside of a model’s “mind,” feature by feature, like neuroscientists with a brain scanner.
When the regulators, the standards bodies, the academics, and the companies all point in the same direction, it’s worth paying attention.
Now imagine the next chapter
Everything so far has been about AI that answers. The next wave is AI that acts — so-called agentic AI, systems that don’t just suggest but go off and do, booking, buying, negotiating, deciding, on your behalf, with the human increasingly out of the loop. The same analysts predict that within a few short years, a meaningful share of everyday business decisions will be made autonomously by such agents. (The same forecast also predicts that more than forty percent of these projects will be scrapped — often for exactly the reason this article is about: inadequate controls.)
So let me get imaginative, because the brief here is to help you see around the corner. Just imagine:
Just imagine a fleet of purchasing agents negotiating supplier contracts overnight, when one gets quietly manipulated — the $1 Chevrolet, except now it’s a binding seven-figure commitment, and everyone was asleep.
Just imagine a financial-advice agent that develops a subtle hallucination and dispenses the same confidently-wrong guidance to fifty thousand customers before breakfast — Air Canada’s liability problem, multiplied by a stadium.
Just imagine a benefits-eligibility agent that re-creates the Dutch catastrophe — but at machine speed, ruining lives faster than any human auditor could possibly notice.
Just imagine one AI’s confident mistake becoming the trusted input for the next AI, and the next — an echo chamber of error, compounding through a chain nobody is monitoring.
None of these require a malevolent machine. They require only an unwatched one. And that is precisely the gap that observability and control planes exist to fill — the live instruments so we can see what the agents are doing across the whole system, and the control tower so we can stop them before they take off in the wrong direction.
So where does this leave you?
Here is why this matters to you, personally, even if you never write a line of code.
You are already living inside these systems. The loan, the diagnosis, the job application, the insurance quote, the fraud flag — increasingly, an algorithm is in the room. The only real question is whether that algorithm is being watched and governed, or whether it’s a black box that someone, someday, will defend by telling a judge it’s “a separate legal entity.”
The genuinely hopeful part — and I’ll resist the temptation to end on doom, because doom is lazy — is that the instruments now exist. The dashboards are being switched on. The towers are being built, by Fiddler and its competitors, with the wind of regulators and researchers at their backs. This is one of those rare moments where the safeguard is arriving roughly alongside the danger, instead of decades too late.
What it needs now is demand. So the next time a company tells you “the AI made that decision,” you’re allowed to ask the only question that matters: Fine. Who was watching it? And who had the power to say no?
My vote? Don’t accept “the robot did it” as an answer. Not from an airline, not from your bank, not from your government. We figured out how to make the skies safe by refusing to fly blind. We can do the same with the machines now quietly deciding so much of ordinary life.
We just have to insist that somebody — and something — is watching.
Have a question about how these systems touch your own life — your bank, your job, your data? Reply and tell me. The whole point of opening the black box is that the rest of us get to look inside, too.