Google published its full framework for evaluating health-related AI on November 4, with a direct core claim: a health answer can sound right without being safe or useful, so accuracy, safety, and usefulness must be graded separately. The whitepaper was led by Sarah Montgomery-Taylor, clinical evaluation lead at Google Health, together with outside health experts Elizabeth McGlynn and Dana Gelb Safran. Google says the framework is already used to assess its own models, and publishing it is meant to make the method transparent and help developers, researchers, and clinicians build more trustworthy health AI tools.
Three dimensions, three different ways of testing
The framework first breaks a health AI use case apart: who it serves, how severe the consequences of failure are, and who reviews it. Evaluation covers three areas: accuracy looks at medical consensus, evidentiary support, factuality, clinical reasoning, and contextual appropriateness, with source grounding added for summarization tasks; safety looks at the harm that could follow when someone acts — or fails to act — on an answer, weighing both severity and likelihood; usefulness looks at health literacy, relevance, comprehensiveness, and whether the answer can be acted on. The testing methods also come in three layers that check one another: closed-question benchmarks test knowledge, AI autoraters grade against written rubrics, and expert human review examines high-risk scenarios and red-teams the system. The three methods run at three stages: while the base model is trained, while a product is built on it, and after launch. The layering means a single multiple-choice score says little; a genuinely high-risk answer must have been seen and checked by people to count as passed.
What the framework does not answer are precisely the two most important points
First, the whitepaper publishes a method, not results: Google's blog gives no scores for any model and draws no pass line, so readers cannot tell from this release how well Google's models actually perform on health questions. Second, this is Google's method for evaluating its own models — not an independent audit, not a certification, and not regulatory approval; when other organizations adopt it, scenario coverage, data quality, and rating standards will shape the results. The more grounded reminder comes from the framework itself: when two physicians rate the same answer, their judgments commonly diverge, which means “expert review” is not an error-free yardstick either. For ordinary users, the framework's direct significance is limited — it will not immediately change the quality of any one health answer; its value is on the supply side: OpenAI previously worked with more than 80 mental-health experts to publish the open benchmark MentalHealthBench, which specifically measures AI conversations about mental health; together, the two approaches are turning “is health AI good?” from a vague reputation into checkable, reviewable items. What is worth watching next is whether third parties or regulators adopt a similar framework and publish measured results — only then will ordinary users be able to use health AI that has genuinely been checked.
