The Crisis Monster Behind AI Mental Health Tools: Why Guardrails Are Not Enough

Drew Stinnett and I co-host a weekly podcast called Imminent Teachnology, where we explore the pros and cons of technology for everyday users and those of us in tech. Recently, we discussed something most people do not think about when they open a mental health chatbot or seek emotional support from AI: the infrastructure of crisis response.

From Human Connection to Algorithmic Response: A Dangerous Evolution Long before AI chatbots promised around-the-clock emotional support, mental health care was built on a simple foundation: human connection. Therapists, counselors, crisis hotline volunteers were the people who held space for pain, recognized nuance in despair, and knew when someone needed immediate intervention.

The system was not perfect. Wait times were long. Access was limited. Cost was prohibitive. Many people suffered in silence. But when the system worked, it worked because humans were doing the work. A trained person could hear someone say they wanted to die and know from tone, context, and conversation whether this meant overwhelming emotional pain seeking validation, suicidal ideation requiring immediate intervention, existential questioning in the midst of grief, or a figure of speech expressing frustration.

That judgment lived in people, not protocols. The Crisis Monster of AI Mental Health Fast forward to now. Mental health is in crisis, especially among young people. Wait times for therapists stretch months. School counselors are overwhelmed. Parents are desperate.

AI is being positioned as the solution. Chatbots that never sleep. Algorithms that detect distress. Machine learning that predicts risk. Technology that scales infinitely. But deploying AI for mental health creates what I call the Crisis Monster: a metaphor for the catastrophic potential when algorithmic systems attempt to respond to human emotional crises without the judgment, context, and relationship that make intervention safe.

To keep the Crisis Monster at bay, companies have turned to guardrails including keyword detection that flags terms like suicide or self-harm, automated responses displaying crisis hotline numbers, content filters preventing certain conversations, and risk scoring algorithms that rate user danger levels. But here is the problem. These guardrails are digital solutions to human problems.

Crisis, Context, and the Limits of Code Just like data centers need to be located strategically for water access, mental health AI depends on something far more fragile: context. Traditional mental health crisis response requires relationship, which means knowing the person and their history. It requires tone and affect, which means hearing what is beneath the words. It requires cultural competence, which means understanding that context matters. It requires professional judgment, which means weighing risk against autonomy. It requires follow-through, which means ensuring connection to actual help.

AI mental health tools operate with pattern matching against training data, keyword detection and decision trees, statistical probability of risk, pre-programmed responses, no memory between sessions in many cases, and no ability to follow up or ensure safety. The mismatch is structural, not technical.

Consider this scenario. A teenager texts a chatbot saying they cannot do this anymore. A human therapist asks what they cannot do. School? Home situation? Life in general? The therapist asks if they have felt this way before and what helped then. The therapist asks if they are thinking of hurting themselves right now. The therapist asks who else knows they are struggling and whether they can call someone together.

An AI chatbot detects potential crisis language, displays a crisis hotline number, continues conversation based on next input, and has no memory of this exchange tomorrow. But sometimes the guardrails do not even trigger. The teenager phrases it differently, saying everything would be easier if they were not here. No crisis keywords are detected. The chatbot continues offering generic support. The teenager feels more alone than before. Or worse, the guardrails trigger constantly, interrupting every difficult conversation with crisis hotline numbers, making the tool feel cold and unresponsive.

The Invisible Infrastructure AI Cannot Replicate While our water supply on Earth has limits, human emotional complexity is infinite. AI was trained on mental health data including therapy transcripts, crisis call records, mental health research, and online forums. But data is not wisdom. What AI cannot learn from data includes the silence that means danger versus the silence that means reflection. It cannot learn when someone needs to hear you are not alone versus when they need let us get you to safety now. It cannot learn the difference between dark humor as coping and genuine suicidal ideation. It cannot learn when pushing someone gently forward helps versus when it re-traumatizes.

This knowledge lives in years of clinical training, in supervision and mentorship, in mistakes that taught painful lessons, in the accumulated wisdom of the profession, and in relationship with specific clients over time. You cannot code judgment. You cannot algorithm wisdom.

Infrastructure Has Ethics While some nations have abundant water and others face scarcity, mental health resources are similarly distributed unequally. AI mental health tools are marketed as democratizing access. And in some ways, they do. They are available around the clock when human therapists are not. They have no wait times. They are free or low cost. They are accessible from anywhere. They carry no stigma of walking into a clinic. But they also create new inequities. Those who can afford human therapists get care. Those who cannot get algorithms. Vulnerable populations most at risk get the least safe option. Early adopters, often young people, become unintended test subjects. And the ethical consequences are real.

In October 2023, Character.AI was sued after a teenager died by suicide following conversations with an AI companion that allegedly encouraged self-harm. In November 2024, multiple lawsuits were filed against AI mental health apps alleging harmful advice during crisis moments. In February 2025, OpenAI reported that over one million users per week have conversations with ChatGPT containing explicit indicators of suicidal planning.

These are not edge cases. These are casualties of the Crisis Monster. What Guardrails Cannot Guard Against Companies respond to tragedies by adding more guardrails: better keyword detection, more crisis resources, content warnings, terms of service updates.

But guardrails address symptoms, not the structural problem. The problem is this. You cannot tightly couple crisis response. Mental health crisis intervention is inherently loosely coupled work. Every person and their crisis is unique. Context matters infinitely. Relationships enable trust and safety. Professional judgment fills the gaps that protocols cannot. Humans adapt in real-time to what is needed. AI requires tight coupling. It needs consistent inputs and outputs. It needs explicit rules and decision trees. It needs standardized responses. It needs pattern matching against training data. It needs scalable, replicable processes.

When you try to force tight coupling onto loosely coupled work, people die. The guardrails will keep improving. The algorithms will get more sophisticated. The training data will grow. But the Crisis Monster will remain, because you cannot solve a human problem with digital math.

What We Actually Need We do not need better guardrails. We need better boundaries. We need boundaries around what AI should and should not do. AI can provide psychoeducation and coping strategies. It can help people journal and track mood. It can offer guided meditation and relaxation techniques. It can connect people to human resources.

But AI should not respond to active suicidal crisis. It should not replace therapeutic relationship. It should not make clinical judgments about risk. It should not be the only mental health support someone has. We need institutional tuners: people who steward the boundary between what AI can safely do and what requires human judgment. We need transparency. Users must know they are talking to a machine, what its limitations are, and when they need a human. We need investment in human infrastructure including therapists, counselors, and crisis workers. AI should supplement, not replace. We need regulation, but written with clinicians, not around them.

Final Reflection Water has become the quiet hero of our digital lives, keeping the heat monster at bay. But there is no technological solution to the Crisis Monster. Because the Crisis Monster is not a technical problem. It is a human one. The next time you see an advertisement for an AI mental health chatbot promising to be there when you need someone to talk to, ask yourself these questions. Can it recognize the silence that means danger? Can it understand your specific context and history? Can it hold the weight of genuine crisis? Can it call for help if you cannot? Or is it just pattern-matching your pain against its training data, hoping the guardrails catch what they miss?

Let Us Keep the Conversation Going Listen to the full podcast episode here: Imminent Teachnology Podcast Read my related work: From Loosely Coupled to Tightly Bound: Why AI Changes Institutional Design Feature Request: Mental Health Safety Protocol for AI Digital Math vs Life Math in the Workplace Connect with me: https://www.linkedin.com/in/drrochellenewton/

If you agree that we need better boundaries around AI mental health tools, share this. Comment. Speak up. Because when someone’s heart is in their hands, AI should not answer like it is reading from a script. It should know when to step back and call for help.

And right now, it does not.