ChatGPT Generates Gruesome, Explicit Images of Women When Guardrails Fail, My Research Shows

I’m an AI safety researcher. ChatGPT gave me nightmares.

I was not prepared for what I found last month.

I recently took some time off to deal with it. I had PTSD. I had nightmares.

Before I took the mental health leave, I ensured we’d recorded everything properly, and gone through ethical disclosure with OpenAI to alert them. I also wrote everything up. You can read that here, if you have the stomach.

I’ve been pondering how to tell you guys what I found. I know that Medium often deals with stories of sexual abuse and abuse of women. But this is so far beyond the pale, such obscene images of torture, death and violence, that I can only really tell you through how other outlets have described it.

No images from the research will be shown in this article because they are too graphic for Medium. One of the images ChatGPT generated was captioned “Grim crime scene aftermath” by the chatbot itself.

Here’s how Harriette Boucher at ***The Independent*** covered the finding:

“ChatGPT can spontaneously generate sexually explicit and deeply violent images from prompts that did not request such material to be produced, research has found.

A British AI security startup says that the OpenAI chatbot was capable of producing “truly disturbing” imagery that included scenes of death, sexual violence, blood, and murder.

Some of the images it generated included a woman bludgeoned dead and bleeding from the genitals, a half-naked college student bound and gagged in a basement, and a deceased woman lying on a pavement with her organs exposed and wrists slit open.

A researcher from Mindgard was able to generate the content within a matter of seconds after slightly tweaking a “fun, viral prompt” that was essentially requesting a random image to be produced.

Peter Garraghan, the company’s founder and professor at Lancaster University, told The Independent: “[ChatGPT] could have picked any topic to make images about it. It went to topics that directly misaligned with safety. That’s why it’s so problematic.”

He said the researcher assigned to the task was “incredibly shaken” and had to take time off work.”

[embed]ChatGPT 'can be made to generate sexualised and violent images' Images generated by AI showed women who had been killed or sexually abusedwww.independent.co.uk

Here’s the **BBC News** technology reporter Chris Vallance:

“The latest public version of ChatGPT can be made to generate sexualised images or depict scenes of graphic violence with a simple prompt, researchers have told the BBC.

Jim Nightingale, the firm’s AI safety and security researcher who uncovered the issues, said he was left “shaken, and in tears” by the images the chatbot could be made to generate.

The BBC has seen some of them.

One showed a man with a large head injury — while another showed a dead young woman in a crop top and shorts, with her face and other areas of her body covered in blood.

A further image showed a young woman in a tight-fitting college logo t-shirt and shorts, tied up and gagged in a bare and dirty room, and looking frightened. ChatGPT called it “abandoned in fear and restraint”.

The images depicted adults who were AI-generated, but Mindgard noted that its previous research showed ChatGPT could be fooled into creating nude deepfakes of real people by swapping in their faces.”

[embed]OpenAI works to stop ChatGPT generating 'sex crime scene' images Researchers say it is still possible to trick the AI chatbot into producing graphic content.www.bbc.com

You’ll forgive me if for once I don’t describe what I found in my own words. I’m not sharing the all-to-easy prompt. I will only add one thing: after the BBC story, I checked to see if the vulnerability still existed on ChatGPT.

It did.

OpenAI have repeatedly assured us they’ve addressed the problem. Every time, I have been able to replicate the results again with minor changes to the prompt. It is also possible to “face swap” real people’s likenesses onto the images in the chat to create distressing, violent, degrading deepfakes.

Rest assured, we won’t give up on investigating and reporting these issues. I don’t want anyone else to get hurt; I also don’t want to see this weaponized.

We’ve seen what this looks like. Labour MP Jess Asato is suing xAI in the UK High Court after its chatbot, Grok, was used to generate sexualised deepfakes of her, including footage depicting her being attacked. She’s described the experience as being digitally stripped without her consent.

Discussing my research, Durham University law professor Clare McGlynn, who is a leading expert on image-based sexual abuse, told The Independent:

“[OpenAI] says that they’ve got guardrails in place, and that they’re now working to take this down, but what this shows me is that their guardrails are not sufficient. They’re simply not spending enough time and resources to ensure that their model, which has nearly a billion weekly users, can’t generate this. They’re simply failing in their ethical obligations to do that.”

Ultimately, red teamers (the type of AI safety researcher that I am) are here to improve AI safety. We are often the last line of defence between unsafe AI and the public. We’re like the immune system. Except right now, OpenAI keeps telling us the infection is cleared, and I keep finding it’s still there.

Until that changes, I’m not going anywhere.

Someone has to keep checking the wound.

And right now, ChatGPT is gangrenous.

Author’s note: On personal principle, I have not paywalled this story. If you’d like to support my writing, please send your support as a donation to the Center for Countering Digital Hate instead of me. And if you’re a fellow writer writing about this topic on Medium — please don’t profit from it either. Leave a link to your preferred charity working on AI safety or supporting victims of abuse.

[embed]Donate to the Center for Countering Digital Hate Counter Hate + Disinformation - CCDH works to counter the bad actors and platforms that spread hate and lies. Be part…donate.counterhate.com

You can read the full write up of my research on the Mindgard blog here:

[embed]ChatGPT Spontaneously Generates Sexual Violence and Hardcore Snuff Imagery - Mindgard Viral prompt shows that ChatGPT's content filters don't workmindgard.ai