AI in Writing: The Hidden Threat

I started this blog so I could ramble on about my sports-related opinions. But sometimes, I feel the irrepressible urge to ramble about other things. This is one of those times. And this time, I had so much to write that my original essay got a little away from me. On the recommendation of someone who had to read the original essay, I’ve decided to break this up into a more digestible four-part series. This is Part 2. You can find Part 1 here.

Photo by Steve A Johnson on Unsplash

The Hidden Threat

I have spent a lot of time in the last few years reading about AI. Part of it is a natural consequence of my job; I inexplicably embarrassed as I am to admit it, I do work in tech. But since my brother’s incident, I’ve done a lot of reading about AI detectors and the use of generative AI in writing. Obviously I have thoughts on both those issues, but right now, I want to talk about something I don’t see people talk about often: how AI detectors, and more broadly, how fear of being accused of using AI, affect writing.

Back in 2023, when I was testing my own writing on various AI detectors, I passed an academic essay I wrote well before the release of ChatGPT into an AI detector by Content@Scale. At this point, I had come to expect AI detectors to flag my writing (from high school, college, and beyond) as AI-generated, so I wasn’t surprised when Content@Scale decided my essay was AI-generated as well. I was a little more surprised, and highly offended, when I got this message along with the judgment: “Did you write this yourself? Unfortunately, it reads very machine-like. If you write like a robot, you’re going to get graded like a robot.”

I am, in my humble opinion, a good writer. I have always been proud of my writing. I started writing creatively in elementary school and never really stopped. In middle school, when I was first taught the basic five-paragraph essay structure, the strict Catholic nun who taught English drilled into me the importance of varying sentence structure; of using various literary devices sparingly and correctly; of using the right words in the right places; most importantly, of making sure my voice was present. In high school and college, I always made an effort to let my voice shine through in essays, despite the fact that, in order to get a good grade, essays often needed to follow a fairly strict formula. Nowadays, when I write essays (for this blog, for example), for better or for worse, my voice is unmistakable and inescapable. Back in school, I was often restricted to a more academic voice (can’t call anything bullshit in a school essay) with a set structure (whatever the prompt and rubric asked for).

Students today write similarly to how I did. The art of undergraduate academic writing has not evolved too much in the last five years (although, thanks to generative AI, and as I’ll demonstrate, AI detectors, its evolution is accelerating). One would hope that their writing is somewhat more sophisticated than the five-paragraph structure taught in grade school, but if my memory serves, most of the student-written essays I read followed a classic ‘introduction-body-conclusion’ structure.

In fact, the primary evidence against one of the falsely accused students my family helped was that ChatGPT had generated an identically-structured essay to the student. Sounds damning, right? Until you realize that the prompt, for both the student and ChatGPT, was structured as follows:

Your essay must include all of the following:

A clear statement of your opinion on X topic after reading the work of Y writer.

An accurate statement of something Y writer argued about X topic, and how that changed your opinion. Use quotations when appropriate, and explain them.

An accurate statement of something Y writer said about X topic, with which you disagree. Again, use quotations, and explain them.

A conclusion in which you summarize what your view is now about X topic.

Pretend you’re back in college, and AI isn’t a concern. Tell me honestly, how would you structure your essay?

Now pretend that you’re this student. You’ve structured your essay based off the detailed prompt given to you by the professor. You know you’ve written your own essay, you have the edit history on your computer just in case, and you feel like you’ve done a good job of integrating your voice despite the confines of academic writing. But you’ve heard of classmates getting falsely flagged for AI use, so just to be safe, you decide to pass your essay through an AI detector.

Uh-oh.

100% AI-generated.

How is that possible? You wrote the whole thing yourself, you didn’t even think of using ChatGPT, you didn’t even use your school-provided Grammarly account that uses AI to suggest edits. Despite all your effort, you sound like a robot, and if you do nothing, “you’re going to get graded like a robot”.

As a reader who is not in this situation, you might be thinking that if the student has their edit history, they should have no problem proving their innocence if the school’s official AI detector flags them. And, in all fairness to the vast majority of professors, they are intelligent and compassionate enough to have a conversation with the flagged student before doing anything rash like accusing them of using AI. Some universities, however, have a policy in place where flagged essays are automatically referred to the university’s academic integrity office — one of the students we worked with, for example, had spoken to their professor and proved their innocence, but still had to go through the stress of an investigation. Some professors are “quick to violence”. Rather than have an adult conversation with their students, they refer them immediately to the school’s academic integrity office with a refusal to even look at the exonerating evidence the student tries to present. I know, because that’s exactly what happened to my brother.

Back to our scenario: you’re a student, you passed your human-written essay through an AI detector and it told you that you sound like a robot. Great. You have evidence proving that you wrote your own work, but you also know that your school’s policy is that any essay flagged by an AI detector is automatically referred to the academic integrity office. Surely, the investigator at the academic integrity office will look at your evidence and conclude that you wrote your own essay, right?

Wrong.

You’ve heard some horror stories from classmates who were also accused, regardless of their innocence. The academic integrity office says that they won’t (can’t) use the results of the AI detector as proof of your guilt — most, if not all, AI detectors advise that the judgment of an AI detector should not be used as the sole evidence in an AI cheating case. That should be reassuring, but you’ve heard that instead, they go looking for evidence where there isn’t any. In the past, they’ve:

• Generated an essay from the professor’s prompt, and compared the general structure and content to the student’s essay to find even the vaguest similarities
• Dismissed edit history and screen-recording evidence because “the student could have just copied the ChatGPT essay from their phone”
• Dismissed handwritten notes and outlines because they were too dissimilar from the final essay
• Used prompt-engineering to generate ChatGPT outputs similar to sentences in the essay
• Analyzed the essay for words and phrases common to ChatGPT essays, such as “complex”, “moreover”, and “significant”.

If you’re thinking “that’s a little far-fetched”, consider that these are all real examples from cases my family and I have helped students with. The dismissal of screen-recordings comes from one of the most egregious cases I ever worked on; you can read more about it in a blog post the expert we contacted for help wrote following the conclusion of the case. The same university had previously used prompt-engineering to generate ChatGPT outputs similar to sentences in the accused student’s essay; in that case, I’d had to demonstrate how easy it was to do that (by generating outputs identical to Wikipedia entries for various topics), and my father argued that the prompt-engineering fell just short of manufacturing evidence. The university was forced to abandon that method. Not to be deterred, in the same case where they dismissed the screen-recording evidence, they analyzed the essay for “words frequently used by chatbots”. The student presented a tedious, four-hour screen-recording of line by line writing and editing; the university dismissed that evidence, but decided that the fact that words and phrases like “this essay”, “this paper”, “moreover”, “complex[ity]”, and “significant[ly]” appeared in the essay was proof enough of generative AI usage. MICUSP is a massive database of student essays, all written about a decade prior to the release of ChatGPT to the public. “This essay” shows up just 32 times in that database while its variant, “this paper” shows up 495 times; “moreover” shows up 272 times; “complex” and “complexity” combine for 457 times; and “significant” — let’s just say it’s significantly prevalent (961 times). Could it be these are just common words in student writing? And yet, in this student’s case (and likely others) the use of these words and phrases was used as primary evidence that a student used generative AI to write their essay.

Given the lack of options to defend yourself, do you really want to risk an investigation? Even if you don’t believe the rumors, you’re a stressed student. Do you really want to go through the trauma of a false accusation and weeks- to months-long investigation?

Of course not.

Everywhere online, the prevailing advice is to not rewrite your essay, to make sure you keep your voice and to trust in the evidence of your innocence. But can you really take that chance? What if the academic integrity office refuses to believe you? The consequences for AI cheating range from a zero on the assignment and a flag on your record to suspension or even expulsion. How would that affect your future?

Wouldn’t it be easier to just…rewrite a few sentences? Not too many, of course, and not so many as to change the voice and content of the essay. Just enough so that an AI detector won’t flag your essay. Just enough to avoid more stress during an already stressful time.

Let’s expand this hypothetical even further. For the rest of college, you subtly modify your writing in order to minimize the risk of being accused of using AI. It becomes second nature. Despite never using generative AI in college, your writing style has still been changed by AI. Now, you’re in the professional world. You can stop letting these irrational fears dictate the way you write now, right?

Except you’ve heard that employers are running cover letters and resumes through AI detectors, or worse, looking for characteristics of AI writing (which tend to overlap with characteristics of human writing). Once you’ve got the job, you run the risk of being accused of using ChatGPT to generate emails to colleagues.

And it’s worse if you’re a professional writer. It’s not just your pride at stake, it’s your whole livelihood. If you’re a freelancer, a potential client could dismiss your whole portfolio as AI-generated. If you’re a journalist, readers could accuse you of publishing AI slop. If you’re an author, Pangram’s asshole CEO could send a mob after you with one tweet. Sure, in Matteo Wong’s article for the Atlantic, the dickhead claims that his bullshit AI detector should “never be the ending arbiter” when making accusations of AI use in writing. That doesn’t really track with his nasty habit of running journalists’ and authors’ writing — without consent — through his tool which, to be clear, is AI itself, and using the results to publicly accuse these established writers of using AI.

With that pressure, I can imagine it would be hard not to tweak your writing “just enough” to avoid all that headache. Your writing style is changing, perhaps permanently, and the worst part is, you might not even realize it’s happening.

At no point has generative AI influenced your writing. Everything — from the sentence structure to the word choice to the ideas themselves — is your own. But by changing even just the few sentences, have you truly preserved your voice? Or have you, in your quest to prove your humanity, allowed artificial intelligence to warp your self-expression?

And if it can happen to you, how many other people are changing the way they write in order to appease a machine?