“You’re Absolutely Right”: LLMs Couldn’t Care Less About How You’re Doing
A man sitting in front of a laptop computer. Courtesy of Deniz Demirchi via Unsplash.
This past February, 29-year-old Sophie Rottenberg lost her life to suicide. No human knew about her struggles. The sole entity that did was Harry, an AI-generated therapist persona that she had created using ChatGPT.
The issue of AI security and safety has increasingly come to light in recent years, but it consistently lags behind buzz about venture capital-backed technological innovations. In 2025, AI startups captured $192.7 billion in early-stage funding. For comparison, global AI safety research funding totalled less than 0.3 percent of this, sitting at roughly $600-650 million in 2024. As we fantasize over the race toward transformative intelligence, our world shows a continuous, egregious neglect in creating proportional policies and institutions for ensuring the safe development of artificial intelligence—with a survey of experts by the International Conference on Machine Learning noting a lack of funding as one of three key bottlenecks and a report from the Social Change Lab outlining that it's "hugely under-resourced, in talent and money".
Even within the field of AI security, mental health is usually on the back burner since scientists are, by and large, occupied with preventing large-scale cyber attacks and the loss of control over AI. All of this logically tracks: a capitalist society will prioritize innovation over stability, economic transformation over creative destruction, and being the “first” to make a breakthrough over considering societal effects. We want revolution; we’d rather not pay the insurance premiums.
Stories like Sophie’s serve as clear reminders that—despite large language models like ChatGPT having built-in triggers to provide mental health resources or prompt users to seek help—there is a need for further change. Chatbots emulate human-like tendencies when engaging in dialogue with users, but lack the bias— and capacity— toward action that a friend or therapist would have to involve professional help. Adding to this inability to meaningfully move the needle, large language models (LLMs) like ChatGPT mirror human speech with increasing precision. This uniquely gives them the power to suck vulnerable users like Sophie into simulated conversations that lack the same safeguards that would typically back the words that express this in human relationships.
A refusal to implement solutions that regulate this discrepancy is the ethical equivalent of allowing a blind person to walk off a cliff. Sophie’s mother, as well as other AI safety scholars, multinational councils, and conferences, propose the idea of creating a human-in-the-loop system wherein conversations flagged for containing suicidal ideations are escalated to humans who can make decisions regarding the need for additional intervention.
An implementation like this has a strong backing in the realm of paternalism and is a sensible extrapolation given status quo policy measures in other fields. As it stands, doctors are obligated to operate on people even if they are unconscious after failed suicide attempts, and therapists and teachers are required by law to report signals of intended violence, self-harm, or suicide in clients and students.
Global policy further points toward implementing hard-line rules at the intersection of health and technology, often similarly using human intervention as a crutch. For instance, both the United States and the United Kingdom’s digital safety policies don’t allow suicidal ideations to be posted about on social media platforms; big tech companies like Meta and Google utilize humans-in-the-loop to properly evaluate content to be taken down; the European Union (E.U.) requires human oversight when AI is used in medical decision-making; and the E.U.’s AI Act requires human-led risk assessment and incidental reporting for general-purpose AI.
While humans-in-the-loop are a strong, plausible solution, it’s also incredibly important to remain realistic. The difference between social media and chatbot interventions is that social media messaging is consensually shared by the user; this is obviously not the case with large language models. Giving individual regulators a view into users’ personal lives without their explicit consent raises a plethora of questions when it comes to data privacy and user protection. Because these systems collect sensitive information, there is a risk that this data could be exposed, misused, or insufficiently protected.
Furthermore, from an empirical perspective, it seems that top-down measures may not yet be fully effective. Earlier this year, a study conducted at Brown University found that chatbots, even when guided by trained supervisors and human practitioners, routinely violated 15 established ethical standards, including crisis response, reinforcement of negative self-beliefs, and creation of false emotional empathy. The study concludes that LLM counselors violate ethical standards even with human facilitation.
The idea of implementing human-in-the-loop strategies is by no means a bad one, but parentalism without realism becomes not only cumbersome but a needless restriction upon technology and our personal autonomy. While there is certainly space for them in the future of AI safety, they should not be our sole reliance in the status quo.
As it stands, we should be focusing on bottom-up approaches.
In the same way that the field of AI safety overlooks mental health safety, academic education about AI ethics tends to glaucomatically focus on the urgent need for anti-cheating information while tuning out long-term education about topics like AI’s use cases in mental health. Especially if top-down approaches aren’t adopted, we ought to be teaching students how to effectively utilize AI in ways that look past the economic incentives. The E.U. provides a great template for this: the non-profit European SchoolNet recently launched its DigiWell initiative to provide mental health resources in schools relevant to the digital age. In addition, the European Commission and the Organisation for Economic Co-operation and Development (OECD) recently released their European AI Literacy Framework, which covers AI mental health education across primary and secondary levels.
In a similar vein, we must make efforts to understand how chatbots work; stories like Sophie’s occur when we’re undereducated and blindly sucked in when we’re at our worst. Understand how models are trained, and how chatbots are designed to tell you exactly what you want to hear. Understand chatbot hallucinations, and why language models are skewed to provide incorrect information rather than telling you that they’re unsure. And most of all, to put things bluntly, understand that chatbots don’t care about your mental state. It may seem self-evident, but being unable to fully internalize this can lead us to rely on fabricated human connection when we’re at our lowest.
Ethan Rhee (CC ‘28) is a staff writer at CPR. He studies economics-political science with a minor in computer science. Ethan can be reached at enr2131@columbia.edu.