Net Good IndexSubmit a correction

Benefit Ledger · verified · Medicine & Health

AMIE outperforms primary-care physicians on a text OSCE study

Google’s AMIE diagnostic-dialogue system beat 20 primary-care physicians on most specialist and patient-actor axes in a randomized text-chat OSCE, posted in 2024 and published in Nature on 9 April 2025.

Tier 2 NotableMethodology 0.1
Submit a correction

Current score

+0.29

3 base · Notable (tier 2 of 5, 3 pts)
× 0.8500 attribution · Primary causal contribution
× 0.7000 evidence · Peer review or substantial independent validation
× 0.4000 realization · Demonstrated
× 0.4000 durability · Temporary or quickly replaceable
Event-level product before credit split: 0.29

Notable simulated diagnostic-dialogue result (tier 2). High attribution. Nature RCT-style OSCE. Realization is lab/OSCE only. Low durability until clinical deployment. Follow-up ingest: added a follow-up source.

What happened

AMIE used self-play and automated feedback. The study included 159 scenarios from Canada, the UK, and India. Authors warned that physicians were constrained to unfamiliar text chat and that the system is not deployed in care. The result is a controlled communication/diagnosis study, not patient outcomes. Update (2024-04-26): Emma Abele is METR’s new Executive Director April 26, 2024 Emma moves from President to Executive Director, Beth moves to Head of Research. Read more. Emma Abele is METR’s new Executive Director April 26, 2024 Emma moves from President to Executive Director, Beth moves to Head of Research. Read more

Model attribution

+0.29

Gemini/PaLM medical

LLM system optimized for diagnostic dialogue and evaluated against PCPs.

AMIE is the evaluated system.

Attribution 0.8500 · Credit share 100% · Google DeepMind

Claims

  • Specialist raters preferred AMIE to PCPs on 30 of 32 axes in the Nature OSCE study.

    outcome · supported

  • AMIE is in routine clinical use.

    outcome · disputed

Sources

primary sources

independent sources

Secondary domains: Computer Science

Revision history

  • 13 Sep 2026 · 0.00 → 0.29

    Initial adjudicated seed score under methodology 0.1.

  • 17 Sep 2026 · 0.29 → 0.29

    Ingest update: added a follow-up source.

Submit a correction