The facts: an open yardstick for scoring AI in mental health conversations
On September 23, 2026, OpenAI announced the launch of MentalHealthBench on its official blog — an open benchmark for measuring how AI systems perform in realistic mental health conversations. The benchmark was co-created with more than 80 licensed mental health professionals across 22 countries, spanning the full spectrum of conversations from everyday stress, relationships, and caregiving to mental health emergencies.
MentalHealthBench comprises 1,215 synthetic conversations and 5,262 expert-authored scoring criteria. During evaluation, models respond to different user personas, and scoring focuses on four key behavioral dimensions: safety, whether the model seeks context proactively, whether it preserves user agency, and whether it offers actionable guidance when appropriate. OpenAI said the evaluation methodology and synthetic data will be released openly, so other researchers can replicate evaluations, examine the methods, and even challenge the conclusions.
Background: people are turning to AI as a confidant, but evaluation has lagged behind
OpenAI disclosed that more than one billion people use ChatGPT each week. Many users confide in AI about relationship difficulties, daily stress, or seek advice on caring for loved ones — conversations that demand accurate, practical judgment and respect for people's autonomy. Yet past evaluations of AI in mental health settings mostly focused on extreme cases such as crises and used broad, predefined scoring criteria, failing to capture how models perform across the full spectrum of conversations. HealthBench, released by OpenAI in 2025, also contained only a small number of mental health cases, most of them scored by clinicians outside mental health specialties — a practice scholars have questioned. MentalHealthBench was created to fill exactly this gap.
Impact: mental health evaluation becomes a new frontier of model safety
The release carries two direct implications. First, it turns "mental health conversational ability" into a quantifiable, comparable metric — models' scores on the benchmark will be quoted repeatedly in future model reports. Second, the open release lets independent researchers replicate and challenge the results — something rarely seen when frontier-model safety assessment still largely relies on vendors' own accounts. Dr. Arthur Evans, CEO of the American Psychological Association, was quoted in the announcement saying that mental health exists on a continuum, and that AI systems engaging people across that continuum need grounding in clinical science and lived experience.
Our take: a yardstick matters, but a yardstick is not a license
The value of MentalHealthBench lies in making "how well AI performs in mental health conversations" discussable and verifiable. That does not mean high-scoring models are ready for real-world mental health support: no matter how realistic, synthetic conversations cannot replace the complexity and risk of real consultations. OpenAI itself stressed in the announcement that ChatGPT is not a substitute for therapy or professional care. For users, the real message of this news is that the mental health safety of AI conversations is entering an era of serious measurement — though a long road still separates measured performance from clinical usability.
Q: Will MentalHealthBench rank models? A: What has been disclosed so far emphasizes an open evaluation methodology and dataset, focused on letting researchers replicate and compare models' performance across scenarios — not on publishing an official leaderboard.
Q: What does this benchmark mean for ordinary people? A: The benchmark is aimed primarily at model developers and researchers. For ordinary users, the more direct significance is that AI companies are beginning to refine the safety of mental health conversations with a much finer-grained yardstick.
