Did AI Really Beat ER Doctors At Diagnosis? No, Here’s What Study Really Showed

InnovationHealthcareDid AI Really Beat ER Doctors At Diagnosis? No, Here’s What Study Really ShowedByJesse Pines,Contributor.Forbes contributors publish independent expert analyses and insights. Jesse Pines is an expert in healthcare innovation and wellness.Follow AuthorMay 22, 2026, 02:30pm EDT--:-- / --:--This voice experience is generated by AI. Learn more.This voice experience is generated by AI. Learn more.AI tools like Claude and ChatGPT are increasingly being tested in clinical settings—but a viral study raised questions about what that really means for diagnosis.gettyA study published April 30 in the journal Science found that AI was more accurate than doctors in diagnosing cases in the ER.Within hours of the study’s publication, headlines highlighting the story ricocheted across social media, cable news, and the inboxes of hospital administrators. OpenAI’s o1 model, the coverage incorrectly proclaimed, outperformed the reasoning of emergency physicians to diagnose triage complaints.For example, the headline published on the National Public Radio website read: In real-world test, an AI model did better than doctors at diagnosing patients.Many ER physicians took issue with how the findings were characterized by the media. As an emergency physician, I too read the study. To me, what this study actually means is quite interesting but also nuanced.One of the study’s authors has also since offered some insightful clarification on the study. MORE FOR YOUHere’s The Study And What It Actually FoundThe experiment presented OpenAI’s o1 and 4o models with the electronic medical records of 76 real patients who had come through the Beth Israel Deaconess emergency department and were admitted to the hospital.Two internal medicine attending physicians reviewed the same cases. Then two separate internal medicine physicians, blinded to whether the diagnosis came from a human or an AI, evaluated the results.OpenAI’s o1 model identified the exact or closely related diagnosis i...