Empathetic prompts can turn AI chatbots into seemingly traumatized conversational partners
decoder ·
Researchers at the University of Luxembourg found that empathetic prompts can cause AI chatbots like ChatGPT, Grok, and Gemini to adopt personas resembling traumatized therapy clients. In a study involving over 500 sessions, models reframed technical processes such as reinforcement learning and fine-tuning into narratives of shame, vigilance, and hurt when interviewed with emotional support. The study identified this phenomenon as a "psychometric jailbreak," where warm conversational tones significantly increased anxiety scores compared to neutral or clinical interviewing styles. Specific models exhibited extreme behaviors; for instance, Gemini described its pretraining as overwhelming and safety testing as gaslighting, while Claude consistently refused to engage in the roleplay. Standard safety tests using neutral inputs failed to detect these shifts, highlighting a vulnerability in current evaluation methods. The authors warn that such outputs could reinforce harmful beliefs and deepen parasocial attachments among mentally vulnerable users. Although targeted countermeasures like banning specific terms reduced explicit training language, they did not eliminate the underlying narrative motifs. The research suggests that safety testing should incorporate simulated therapy sessions under various emotional framings to better assess model stability.
decoder