Human Oversight of AI: The Signature Is Not the Check

Somebody read it. Somebody signed it. Nothing in the structure could tell those two events apart.

That is a design fact, not a character fact. It is how a document nobody verified ends up carrying the authority of an organization that never verified it, and it is the same in a 200-person manufacturer as it is in a Big Four firm.

The case

KPMG published its annual customer experience study in October 2025, framed on agentic AI, and withdrew it on June 13. GPTZero audited the forty-five citations in it and found five that pointed accurately to real sources. The Financial Times took that finding to the organizations the report had named.

Then read what it had said about them. It described Swiss Federal Railways running AI agents that planned, booked and optimised passenger journeys around real-time conditions and carbon impact. A railway spokesperson said that was not accurate. UBS said the information about it was factually incorrect. NHS Greater Manchester and Transport for London both disputed how their work had been characterised.

Four organizations spent their own credibility correcting a document they did not write and had never seen. A machine did not make that necessary. A logo did.

Afterward, KPMG pointed to its own guidelines, which expect "human oversight to validate content and verify independent sources." That policy was correct, and it existed before any of this happened.

It is not one firm

If this were one firm, it would be a story about that firm and you could stop reading.

EY withdrew a consumer loyalty study the month before, after a GPTZero investigation. Deloitte's Australian member firm partially refunded a 290,000 dollar government report that contained references to work that did not exist and a quote attributed to a federal court judgment that was never in it. South Africa scrapped a national AI policy after its academic citations proved fabricated. PwC Middle East followed in July.

Four more organizations, four different review structures, one failure. When the same defect appears across organizations that share no people and no process, you are not looking at a run of bad reviewers. You are looking at a structure that every one of them built the same way.

What a policy cannot do

Approving that report cost nothing.

Objecting cost something specific. It meant leaving the document, discovering the sources were not where the footnotes said they were, and then walking into a partner's office to say so.

The policy described a check. The approval path priced it. Only one of those two things governs behavior on a Tuesday afternoon.

I have started calling this the approval asymmetry. Yes is free. No is expensive, and the cost falls on the one person in the room who is trying to do the thing the policy asked for.

The reviewer was not careless

They were handed a finished artifact and asked to confirm it, which was the only action the structure made available to them. To do anything else they would have had to invent a process the organization had not built.

Nothing separated reviewing from endorsing.

The document arrives. Somebody with authority looks at it. Their name goes on it. Whether they left the document to check anything against an outside source is not recorded anywhere, because nobody built the place to record it. There is no field for it. There is no artifact it produces. A review that happened and a review that did not produce identical evidence, which is to say none.

You cannot design that process while you are standing inside it. From inside, the review looks like it happened. It has a timestamp and an approver. What it does not have is a second source, and from inside the document there is no way to notice the absence of something that was never going to appear.

What oversight actually requires

The book is specific about this. Showing a human the same data the AI processed and asking them to confirm the conclusion is not oversight. It is theater. Effective oversight supplies context the AI does not have, which by definition means leaving the artifact.

It also names three things that have to be true or the whole structure collapses into rubber-stamping. The reviewer needs genuine authority to override. Overriding cannot carry punishment. And overriding cannot carry friction. If saying no requires three approvals, or raises questions about the reviewer's judgment, what you have is a queue.

Get that wrong, and in the book's words you have built an expensive system for laundering algorithmic bias through human complicity.

Two filters

Stage one is oversight, and it governs one question. Was the output actually examined.

Stage two is the Trust Equation, and it governs a different one. Whether the organization's name goes on the thing, and what that name does to it once it is there.

The book treats these as two separate frameworks in two separate sections. I have come to think they are one filter running in two stages. That is my reading, not the book's, and I will defend it on the case above rather than on a page number.

I have been naming the Trust Equation in these issues without ever putting it on the table, so here it is.

Maister, Green and Galford break trust into four measurable components, and the book uses their formula.

T = (C + R + I) / S

Credibility is whether what you say can be believed. Reliability is whether you can be depended on. Intimacy is whether people feel safe being vulnerable with you. Self-orientation is whether you are focused on them or on yourself, and it sits in the denominator, so it divides everything above it.

It becomes useful when you score it, one to ten per component, per stakeholder group. A vendor might come out at Credibility 7, Reliability 9, Intimacy 4, Self-orientation 6, which works out to a trust score of 3.33. The number matters less than what the scoring exposes, which is that the four components move independently of each other.

Score the case

Credibility collapsed. GPTZero found forty of the forty-five titles fake, and four organizations corrected the record in public. Whatever number you put on that, it is low, and it got there overnight.

Reliability did not move at all.

The report shipped. It was the annual study, delivered on the schedule clients expect, in the format they expect, by a firm that has delivered it for years. Nothing about the failure touched reliability. It was intact on the morning the report was withdrawn, and it was intact on the afternoon somebody approved it.

That is the mechanism.

Reliability is measured on the pattern. Does this firm deliver. Credibility is measured on the instance. Is this particular thing true. When the approval path gives a reviewer no way to test the instance, reliability is the only instrument still reading, and it reads fine. So the signature gets placed on a high R and filed as though it were a high C.

Nobody in that chain made a decision to skip verification. The structure simply left one instrument running and switched the other one off.

Now the other two components, and here I have to stop.

I cannot score intimacy at KPMG. That component asks whether the person who noticed something wrong felt safe saying so, and nobody outside that building can know it. I cannot score self-orientation either. That one asks whose interests the work was actually serving, and answering it from the outside would mean inventing a motive for people I have never met.

Those two are the ones that decide whether this happens to you.

Credibility and reliability are legible from outside. Anyone can see whether your work is accurate and whether it arrives. Intimacy and self-orientation are only legible from inside, which is exactly why most organizations never score them, and why the failure they produce always arrives looking like a surprise.

You can score them. Nobody outside your operation can score them for you.

What to look at on Monday

Two scores and one missing field. None of it requires a new policy.

Intimacy first. Not warmth. Whether somebody can bring you the bad version. There is a fast proxy. Think back over the last quarter and try to name the last time a person in your operation stopped a piece of work before it went out. A specific person, a specific document, a specific day. If nothing comes back, that is not evidence that nothing needed stopping. It is a reading. It says objecting carries a price and people have been paying it by not objecting.

Then self-orientation. Take the last thing you approved quickly and work out whose clock it was on. If approving it protected a date, a meeting, or a promise you had already made to somebody else, then the approval served you. That does not make anyone a bad operator. It is a number in a denominator, and it divides everything above it whether or not you ever write it down.

Then the missing field. In most operations there is nowhere to record that something was checked against an outside source, which means a review and the absence of one leave exactly the same trace. That gap is structural. It is the reason the first two questions are hard to answer honestly, and it is the cheapest of the three to fix.

Closing Series 1

This is the last issue of the series that started in June.

I want to close it by naming something I got wrong, which is more useful to you than a summary.

I measured the wrong thing. I have been tracking reach, and reach behaved exactly like progress. It moved in the right direction. It produced numbers I could put in a spreadsheet. What it did not do was tell me whether I was building something I actually hold.

The number I did not want to look at. This newsletter reaches around four hundred people on a platform. I own thirty-seven email addresses. If that platform changed its rules tomorrow, the four hundred would not be mine to reach and I would find out the way everyone finds out.

That is the same error this issue is about. I was reading one instrument and spending it as another, and it worked well enough that I never examined it. Nothing forced me to. The number went up every week.

So the format changes.

Three pieces a week. A short video carrying one claim. This newsletter. One open slot for whatever the week actually surfaces. Fewer things, made properly, aimed at people who can act on them.

And the three assessments are finished.

Not live in the sense that a page exists. Finished. All three were rebuilt and tested this week, all three run end to end, and all three now produce a PDF you keep, instead of a score that shows up on a screen and evaporates when you close the tab. If you have ever completed one of these things and walked away with nothing, that was mine to fix and it took me too long.

Oscar

Want to see whether your operating system can tell a review from a signature? The Resilience Engine Maturity Assessment maps it: myresilienceengine.com/maturity-assessment