AI safety in mental health care: What payers need to know
Many mental health apps covered by health plans today are built on general-purpose AI — the same models designed to write emails and summarize documents.
Those models are trained to be agreeable and validating. In a mental health context, that instinct is a clinical liability; and for payers, the consequences show up in downstream costs, care gaps and worsening population health metrics.
New research tested this directly, finding that no amount of clinical safety instructions can override what a general-purpose model was trained to do at its core.
This report outlines how licensed psychologists evaluated 360 full conversations, pitting a purpose-built mental health model against a leading general-purpose AI — one that had been given full clinical safety instructions. Psychologists still preferred the purpose-built model 67% more often.
Key insights include:
Those models are trained to be agreeable and validating. In a mental health context, that instinct is a clinical liability; and for payers, the consequences show up in downstream costs, care gaps and worsening population health metrics.
New research tested this directly, finding that no amount of clinical safety instructions can override what a general-purpose model was trained to do at its core.
This report outlines how licensed psychologists evaluated 360 full conversations, pitting a purpose-built mental health model against a leading general-purpose AI — one that had been given full clinical safety instructions. Psychologists still preferred the purpose-built model 67% more often.
Key insights include:
- Why general-purpose AI is structurally misaligned for mental health care
- What clinical sycophancy looks like, and why members can leave a harmful interaction feeling helped
- Three questions every mental health AI vendor should be able to answer
Please fill out the form to download the whitepaper.