I Found A Reliable Way To Tell If Someone’s Writing With ChatGPT

It has nothing to do with AI detectors.

A man writing on his laptop. Photo by Tima Miroshnichenko from Pexels.

I recently came across an article from someone claiming they’d written ten articles in a single day. Out of curiosity, I clicked through to read a few more from their profile, partly because I wanted to see how one person had impressively become an expert on so many different subjects overnight.

It turned out they covered just about everything. One article was about building better habits. Another was about data science. Then there were articles on climate change, upcoming elections, and leaving a corporate job. For somebody who’d apparently mastered that many subjects, though, and also had a ton of incredible life experience, the writing sounded bland.

A couple of years ago, I probably would have pasted the article into an AI detector to see what score came back. But these days, I don’t spend much time doing any of that because AI detectors are notoriously unreliable and can come back with wildly different scores. Case in point, I pasted the Declaration of Independence into two of the most “accurate ones.” And sure enough, ZeroGPT estimated 92.5% AI. And GPTZero gave the exact same text a score of 2% AI.

You get my point.

AI detectors just look for patterns in speech and common habits that ChatGPT and other LLMs use to generate writing. And if you believe the marketing hype that some AI detectors are 100% foolproof, then boy have I got a bridge to sell you. But since ChatGPT is trained on good human writing, you don’t have to be a genius to realize that actual writers using metaphors and polished grammar are falsely being accused.

That’s why I now try to look at each writer’s portfolio of work from a different angle.

Does it give me a reason to believe the writer actually knows what they’re talking about? And are there stories only they could tell, or details that would’ve been impossible to invent by generative AI?

Basically, if I finish reading several pieces and still don’t feel like I know anything about the person behind the byline, I find it plausible to believe the articles were written (at least in part) with AI.

I’ve been writing full time for about six years now, which means I’ve published enough articles to notice that my writing today isn’t exactly the same as it was when I started.

That’s hardly surprising.

Over the years, I’ve read hundreds of biographies, psychology books, memoirs, and way more novels than I can count. Every now and then, I’ll read an old article and catch myself thinking, “This was definitely written after I finished Bill Bryson,” or, “I was clearly on a Ryan Holiday kick when I wrote that one.”

Research published by The University of Waterloo shows I’m not alone. After all, what’s on someone’s nightstand can have a big impact on their writing style:

“When we read, we unconsciously pick up on the vocabulary, phrasing, structure, and flow in what’s being read. When we read lots, we’re essentially learning a wide repertoire of writing techniques which we internalize and later use in our own writing.”

Reading books has influenced my writing, for sure. Always has. But one thing I’ve never really tried to change is my voice because I’ve always written the way I speak. Or at least, tried to.

If you came to Scotland and we grabbed a coffee, we’d probably spend an hour talking about writing, travel, books, etc. I’d interrupt myself halfway through a story because I’d remembered another detail, disappear down an unnecessary tangent about airplanes, and almost certainly end up recommending a book before we’d finished our drinks. But between you and me, the published article would simply be the version of the same story where I’d had a chance to fix the waffling and typos during the editing process.

A friend of mine went to an author event for a history book he’d just finished reading. The author spent most of the evening talking about the research, the years it took to write, and some of the stories that made it into the final version.

Apparently, everything was going well until the Q&A started.

A member of the audience asked about an event that happened just before the period covered in the book. Another wanted to know why one historian had been cited but another more well-known expert had been left out. Somebody else asked whether any discoveries published after the book was written would’ve changed the author’s opinion. And if so, how?

None of the questions sounded especially difficult, yet the answers became increasingly vague, and a couple of dates the author mentioned during the Q&A of various battles had to be corrected by people sitting in the audience. Before long, it was clear several readers seemed to know the subject better than the person who’d written the book.

So here’s one of the most reliable tests I actively use to determine if someone is secretly using AI to write…

Build a strong rapport with an author and eventually invite them for a cup of coffee whenever they’re in town. But offer to pay because many authors like myself who run on caffeine won’t turn down a free one if our calendar is empty. Or, if the author lives on the other side of the world, jump on a Zoom call instead. Before long, you’ll end up discussing another book they’d read on the subject, a mistake they’d made while researching a piece, the rabbit hole that kept them awake until two in the morning, or the opinion they’d completely changed after finding one extra source at the last minute. And it’s also a good way to make friends with other writers… which in this current age of loneliness is important than ever.

Authors who’ve genuinely spent years learning about something almost never run out of things to say whenever I speak to them. But if the conversation goes dry right away, I’m far more likely to believe AI plays a major role in the writing process than the author publicly admits.

Did you find this article interesting? Consider reposting this story so more people can read it.