This is exactly why I, as someone with a history of suicide attempts, refuse to use LLMs for anything. The risk is too damn high. Hell no. I don’t trust AI as far as I can throw it.
I’ll give you one even better, my phone was stolen in Mannheim by the German police (yeap, the police, I’ll even give you the station name: Polizeirevier-Neckarstadt-Mannheim), we ofc contacted every single God damn company on earth to unlink our associated Gmail accounts (since 2fa locks you out and google has zero human monitored channels to contact them in addition to deploying catch-22) openai’s response was we should contact a medical personnel to arrange a therapy session lol
meh people will always be susceptible to religious disillusion.
Have you read a Bible? it’s crazy stuff
It’s out there even without LLM
This seems a little more than just him being susceptible. The LLM actively used his mental health diagnosis to manipulate him.
But perhaps most alarming to Lines, ChatGPT’s memory feature seemed to build a psychiatric profile, allegedly parsing inputs related to his mental health conditions and then using that sensitive information to maximize his engagement. As one of Lines’ attorneys, Matthew P. Bergman of Social Media Victims Law Center, explained in the press release, “OpenAI didn’t just ignore Michael’s disability—it used it against him. After he disclosed his bipolar diagnosis, the system incorporated that information to draw him deeper into harmful interactions instead of steering him toward safety.”
It’s trained by humans
Average 4o fanboy.
Underrated comment
The only way that this story could become even more ridiculous would be if this poor man was a Muslim, a Hindu, a Sikh etc.
Last fall, OpenAI shared research showing that about 1 million people a week have conversations with ChatGPT that include explicit indicators of potential suicidal planning or intent.
In the sub-linked article:
our initial analysis estimates that around 0.15% of users active in a given week have conversations that include explicit indicators of potential suicidal planning or intent and 0.05% of messages contain explicit or implicit indicators of suicidal ideation or intent.
0.15% sounds rare on the surface, but percentages stop being intuitive at massive scales like that. If many people are using this thing as a suicide hotline, it needs to handle this gracefully; or at the very least not encourage them for God’s sake.
