Microsoft’s AI Doctor Claim Starts With A Particular Test

🔍 Read the full analysis: Microsoft’s AI Doctor Claim Starts With A Particular Test on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored 83.2% on 304 difficult medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed paper published in Science on October 22. The results describe performance on historical, text-based cases in a simulator—not diagnoses made for patients receiving care.

The paper, “Sequential diagnosis with language models,” also reports results on 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. Microsoft said its system was designed to request information and tests in stages, then weigh their expected diagnostic value against their estimated cost. The benchmark assessed both the final diagnosis and the cost of information gathering.

In the difficult-case set, GPT-5 alone scored 78.9%, close to MAI-DxO’s 83.2%. Microsoft’s chart also put the model’s estimated testing cost at $8,409 per case. That suggests the underlying language model accounted for much of the measured diagnostic score; the study reports an additional gain from the MAI-DxO orchestration method. These are estimates from the benchmark, not observed costs in a hospital.

The physician comparison was with generalists working individually, including primary-care doctors and hospital generalists. The test did not compare MAI-DxO with specialists or physician teams. Microsoft’s results are its own study findings; publication in a peer-reviewed journal does not by itself mean independent researchers have reproduced them.

A Benchmark, Not a Clinical Trial

The results matter because they offer a test of a different part of medical reasoning than a conventional multiple-choice exam. In this benchmark, participants began with a short case summary and had to decide what to ask or test next. That resembles one element of diagnostic work, where information is incomplete and each next step has a cost.

But a strong benchmark score does not establish that an AI system is safe or useful in routine care. No patients were examined, and the cases were presented as text. The study therefore does not show how MAI-DxO performs with physical examinations, imaging in clinical settings, incomplete records, communication with patients, or the range of ordinary cases seen in hospitals and clinics.

The comparison also affects how to read the headline. The AI system’s panel of agents was measured against individual generalists, rather than specialists or care teams. A system that performs well on difficult cases could eventually help clinicians, but this study does not establish that it improves decisions, reduces costs, or changes patient outcomes in practice.

How the Test Was Built

Microsoft’s Sequential Diagnosis Benchmark, or SDBench, gives each participant a short initial case summary. Further details are held by a simulated gatekeeper and must be requested. The MAI-DxO system uses a language model in a panel-like process: agents propose possible diagnoses, choose questions or tests, and challenge one another over multiple rounds. The benchmark then scores the diagnosis and the estimated cost of the requests.

The source account describes several methodological details from an earlier, non-peer-reviewed 2025 preprint; it cautions that some details may differ in the final Science paper, whose full text is behind a paywall. In that earlier version, a model judged answers against the case’s published final diagnosis, and researchers reported good but imperfect agreement when physicians re-graded a sample. If a requested test result was absent from a case file, the gatekeeper generated a synthetic result. Test costs were estimates based on billing codes and a 2023 U.S. hospital price list, with a fixed $300 physician-visit charge.

The earlier preprint described 21 U.S. and UK generalists, including 17 primary-care physicians and four hospital generalists. It reported that doctors were tested on 56 held-out cases, while the AI was tested on all 304. The Science paper reports a physician result of 36.1% with textbooks and internet access, as well as 22.3% without those resources. Microsoft’s June 2025 preprint used a different model version and reported a larger performance gap; those earlier figures should not be treated as the final paper’s results.

“Building Medical Superintelligence”

— Microsoft, in its blog post about the system

Limits Beyond the Case Files

Real-world diagnostic performance remains untested in the reported study. The cases were historical and text-only, with no real patients, physical exams, or clinical imaging. The source account also says the NEJM set contained no healthy patients or harmless explanations, so the benchmark could not assess false alarms across that kind of case mix.

Other open questions include whether the system performs as well on common, less difficult presentations; how its decisions compare with specialists or teams; and whether the estimated test costs match actual care. The study reports results produced by Microsoft’s researchers. Independent replication and patient-outcome evidence are not established by the information available here. The source account notes that model judging, synthetic missing test results, and estimated prices also affect how the benchmark should be interpreted.

Testing Alongside Patient Care

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how the tool operates in a clinical workflow, but the source material does not give a start date, study size, protocol, or results.

Until such evidence is available, the published percentages should be read as performance on defined case benchmarks, not as proof that MAI-DxO can replace clinicians or improve care. Details to watch include how researchers measure diagnostic accuracy and safety, whether they track false alarms and costs, and how the system’s suggestions are reviewed by medical staff.

Key Questions

What did Microsoft report about MAI-DxO?

Microsoft reported that MAI-DxO scored 83.2% on 304 difficult historical cases and 80.3% on a separate set of 325 emergency department cases. The results appeared in a Science paper published October 22, 2026.

Was the system tested on patients?

No. The reported benchmark used historical text-based cases in a simulator. The source account says a separate prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their usual care.

Who were the doctors in the comparison?

The comparison was with generalist physicians working individually. It did not include specialist doctors or physician teams, according to the source account.

Does the study show that MAI-DxO is better than doctors in hospitals?

No. It reports higher scores on specific case benchmarks, but does not establish performance in routine clinical care or improved patient outcomes. Real-world effectiveness remains unconfirmed.

How much did GPT-5 contribute to the result?

Microsoft’s chart reported that GPT-5 alone scored 78.9% on the difficult cases, compared with 83.2% for MAI-DxO. That indicates the underlying model scored near the full system, though benchmark performance does not show how either would perform in clinical practice.

Source: ThorstenMeyerAI.com