What an AI Detector Actually Measures

Run the US Constitution through a popular AI detector and it scores as almost entirely AI written. Same with passages from the Bible, the lyrics to Bohemian Rhapsody, and chunks of Harry Potter.

None of those were written by a machine. Some of them predate the machine by centuries.

That is not a glitch. It is the clearest window we have into what these tools actually measure, and it matters now because the score has quietly started deciding real things.

A few years ago, AI detectors were a school thing. Now they are everywhere. Surveys in 2025 found close to one in five recruiters would reject a candidate for an AI written resume or cover letter, and some employers in law and finance screen for it automatically. Freelance writers lose clients over a single score. A self published novelist watched a major publisher pull her hit book after readers decided the prose had the hallmarks of AI. And a friend who coaches college essays told me a student got flagged at 85% AI, which is where this started for me.

So I went and figured out what the number is. Here is what it measures, and what that means if one ever gets pointed at you.

Graphic showing a visual overview of what an AI detector truly measures

What The Number Measures

Every detector has a language model inside it. To score your text, it goes word by word and asks one question: how predictable is the next word? Predictable text gets labeled AI.

Predictability has a technical name, perplexity. Low perplexity means the next word is easy to guess. AI scores low because guessing the most likely next word is literally the thing it does. Smooth, fluent, unsurprising.

There is a second signal too, how much that predictability bounces around. Researchers call it burstiness. People spike. A long winding sentence, then a short one. An odd word out of nowhere. Machines hold a flatter, steadier line.

Smooth and steady gets called AI. That is the whole engine, under almost every detector on the market.

Why The Constitution Gets Flagged

Here is the catch, and it is the most important thing to understand about these tools.

The Constitution does not score as AI because it looks machine made. It scores as AI because the machine has read it. It sits in the training data, quoted across millions of pages the model learned from. The model has it nearly memorized. Of course the next word feels obvious. It has seen this document thousands of times.

That is a real flag for a fake reason. The detector is not catching a machine. It is recognizing something it already knows.

This splits into three completely different situations the detector cannot tell apart:

1. A machine writes the text fresh. This is the only thing we actually want to catch.
2. A person writes simple, steady prose in their own words. Flagged, but innocent.
3. A person uses language the model already knows cold, like a famous passage, a common phrase, or the plain formal English of a cover letter. Flagged, but not AI at all.

All three look identical to the detector, because all it measures is predictability. It takes “a machine wrote this,” “a careful human wrote this,” and “the model has seen this before,” and crushes them into one number. Then it calls that number AI.

That is the whole flaw, in one paragraph.

The Proof

This is not a hunch. It is measured, on fresh human writing, which is the case that actually matters.

Stanford researchers ran 91 essays through seven detectors. Every essay was written by a real person, in their second language. The detectors flagged 61% of them as AI. Almost 98% got flagged by at least one tool. The same kind of essay written by a native speaker passed clean. The tool was not detecting machines. It was detecting a smaller vocabulary.

It is not even consistent with itself. Writers have found that the same text, run through the same detector on different days, comes back with different scores. When researchers tested 14 detection tools, not one reached 80% accuracy.

And the accuracy numbers the companies advertise are marketing. Tested on an independent benchmark instead of their own demo, one popular detector’s false alarm rate came in around 4.79%, roughly ten times what it claims. On Wikipedia text it falsely flagged 13%.

It fails the actual cheaters too, which tells you how shaky the signal is. Run AI text through a basic paraphrasing tool once and detection on some tools drops from 70% to under 5%. So the motivated person with a rewriting app walks free, and the honest person who happens to write cleanly gets caught.

The Math Nobody Prints

Say a detector is 99% accurate. Sounds airtight. Now point it at a big pile of writing where actual AI text is rare, maybe one in two hundred.

Out of 200 documents, one is really AI. The detector probably catches it. But it also falsely flags about 1% of the other 199, which is two clean ones. Now there are three flags, and two of the three are innocent.

Run the real numbers at realistic rates and the odds that a flagged document is actually AI fall to around 1%. A flag is far more likely to be noise than a catch. You cannot make a real decision about someone on a tool that fires on 99 innocent people for every guilty one.

Not All Of Them Are Junk

Here is the honest part. “All detectors are fake” is wrong. One tool, Pangram, genuinely holds a near zero false alarm rate in independent tests, even against the apps built to fool it. Some people built theirs carefully.

But even a good score is not a verdict, and the reason has nothing to do with quality. The number is not a real probability, and the companies do not even agree on what it means. One tool’s “85%” is the share of your text it thinks is AI. Another tool’s “85%” is its confidence that the whole document is AI. Two different things wearing the same costume. They are not comparable, they are not probabilities, and they are not proof of anything.

So when a piece of writing comes back “85% AI,” the honest read is this: a tool that cannot define its own number is fairly sure about something it cannot measure.

Even The Makers Backed Away

OpenAI, the company that makes the AI everyone is worried about, built a detector for it. It caught 26% of AI text while falsely accusing 9% of human writing. They shut it down in six months.

Vanderbilt turned off Turnitin’s AI detector and said why. At the volume they process, even a 1% false rate means hundreds of students wrongly accused every year. Northwestern, Michigan State, and more than a dozen other universities did the same. The people closest to these tools trust them the least.

If You Ever Get Flagged

A score is not evidence. If you write cleanly, or you write in a second language, you can get flagged for work that is entirely your own. Protect yourself before it happens.

1. Write in a document with version history. Google Docs, Word, anything that saves your edits over time. A revision trail that shows the piece grow, word by word, is the single best answer to a flag.
2. Keep your notes, outlines, and rough drafts. Messy is good. Messy is human.
3. If you are accused, ask one question back: what is this score actually measuring? Make them explain the number. They usually cannot.
4. Do not try to “fix” it by running it through a humanizer tool. That ties you to the cheating ecosystem and proves nothing. Authentic and documented beats clean and untraceable, every time.

If you are the one holding the detector, hiring or teaching or judging, the rule is shorter. Treat a flag as the start of a conversation, never the end of one. The math says it is usually wrong.

And for the students this started with: there is no documented case of anyone being rejected from college because a detector flagged their essay. None with a name, a school, and a date. In December 2025, Common App confirmed it does not run essays through AI detectors at all. The real risk was never getting caught. It is that AI written work comes out generic and forgettable, and a tired reader feels that long before any tool does.

Which points at the useful question. The detector measures predictability. The cure for predictable writing is not avoiding AI. It is using AI to think instead of to type.

That is the next piece.