Every model in the study got the final diagnosis right more than 90 percent of the time, once it had been handed all the relevant patient information up front. That is the number that ends up in a press release. But the researchers also scored an earlier stage, the part where a clinician has to build a differential diagnosis from incomplete information, and every single model failed that stage more than 80 percent of the time.
I have spent the better part of a year writing about this exact split, usually from the direction of arguable systems and evidence synthesis. This study puts hard numbers next to something I had mostly been arguing on philosophical grounds, and it is worth sitting with properly.
Why Two Scores, One Model
A final diagnosis is, in a narrow sense, a lookup problem once the labs, imaging, and history are already sitting in front of the model. Match the case to the closest thing seen in training and the answer usually lands close to correct. A differential diagnosis asks for something else entirely: a ranked list of plausible explanations built before all the evidence is in, a decision about which test would actually distinguish between them, and a willingness to revise that list as new information comes in.
The researchers, led by Arya Rao and Marc Succi, built a new scoring tool called PrIME-LLM that grades models across the separate stages of a clinical workflow rather than just checking whether the last line matches an answer key. Final diagnosis accuracy came in above 90 percent across the twenty one models. Differential diagnosis failure came in above 80 percent across the same set. Same case, same model, two very different scores depending on which part of the reasoning chain gets checked.
Oversight Only Works on Something Visible
The corresponding author, Marc Succi, said plainly that these models are not ready for unsupervised clinical grade deployment and that they need a human in the loop with close oversight. I agree with the conclusion, but I want to sit with the word oversight for a moment, because it is carrying a lot of quiet weight in that sentence.
If a model hands back a correct final answer and nothing else, there is nothing for a clinician to actually supervise. The reasoning that got skipped, the alternative diagnoses that were never seriously weighed, none of that shows up in a single confident output. A clinician looking at a right answer with no visible reasoning trail cannot tell a model that reasoned well and got lucky apart from one that reasoned badly and still landed close by proximity to its training data.
Not by pushing the final answer accuracy higher, that number is already high, but by forcing the intermediate reasoning into a form a clinician can actually contest. A visible, ranked differential diagnosis is not an explanation bolted on after the fact. It is the clinical work itself. Skipping straight to a final answer is the same as a radiology report with no impression section, a diagnosis with no shown path for how anyone got there.
The Part That Worries Me More
Mass General Brigham has the infrastructure to run a real human in the loop workflow, with the staffing to actually catch a bad differential before it reaches a patient. Most of the clinics I have worked with in Bangladesh do not have that luxury. A rural health worker using a triage tool cannot always call in a second opinion, and often does not have the time or training to interrogate a confident sounding output even when the tool offers one to interrogate.
In that setting, a model scoring 90 percent on final diagnosis sounds like a success story right up until the missed 10 percent lands on a case where the differential reasoning was never sound to begin with, and there was no one positioned to catch it. The failure rate on differential diagnosis is not a rounding error to patch later. It is exactly where low resource deployment is most exposed.
What Actually Needs to Change
Three Takeaways for Researchers and Clinics
- Score the reasoning stages separately. Any benchmark that only checks the final answer will keep this problem hidden.
- Treat "human in the loop" as a design requirement, not a disclaimer. It only works when the system exposes something for a person to review, question, and overrule.
- Prioritize low resource deployment scrutiny first, not last. The settings with the least capacity to catch a reasoning failure are the ones most likely to be handed a tool that hides its reasoning entirely.