Large frontier models such as GPT-5 and Gemini have demonstrated remarkable performance in a wide range of health application benchmarks. However, underneath the seemingly promising results lie salient growth areas, especially in cutting-edge frontiers such as multimodal reasoning. Here we systematically apply and integrate a series of adversarial stress tests to assess the robustness of flagship models and health benchmarks. Our study reveals prevalent brittleness in the presence of simple adversarial transformations: leading systems can guess the correct answer even with key inputs removed yet may get confused by the slightest prompt alterations while fabricating convincing but flawed reasoning traces. Using clinician-guided rubrics, we demonstrate that popular health benchmarks vary widely in what they truly measure. Our study reveals considerable gaps between benchmark performance and the robustness evidence needed to support claims about multimodal medical reasoning in health applications.<\/p>\n","protected":false},"excerpt":{"rendered":"
Large frontier models such as GPT-5 and Gemini have demonstrated remarkable performance in a wide range of health application benchmarks. However, underneath the seemingly promising results lie salient growth areas, especially in cutting-edge frontiers such as multimodal reasoning. Here we systematically apply and integrate a series of adversarial stress tests to assess the robustness of […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"text","value":"Yuxian Gu","user_id":0},{"type":"user_nicename","value":"Jingjing Fu","user_id":"32287"},{"type":"user_nicename","value":"Xiaodong Liu","user_id":"34877"},{"type":"user_nicename","value":"Jeya Maria Jose Valanarasu","user_id":"43491"},{"type":"user_nicename","value":"Noel Codella","user_id":"41635"},{"type":"user_nicename","value":"Reuben Tan","user_id":"43827"},{"type":"text","value":"Qianchu Liu","user_id":0},{"type":"text","value":"Ying Jin","user_id":0},{"type":"user_nicename","value":"Sheng Zhang","user_id":"39087"},{"type":"user_nicename","value":"Jinyu Wang","user_id":"42963"},{"type":"user_nicename","value":"Rui Wang","user_id":"40192"},{"type":"user_nicename","value":"Lei Song","user_id":"41078"},{"type":"user_nicename","value":"Guanghui Qin","user_id":"43527"},{"type":"user_nicename","value":"Naoto Usuyama","user_id":"38670"},{"type":"user_nicename","value":"Cliff Wong","user_id":"38508"},{"type":"user_nicename","value":"Hao Cheng","user_id":"39922"},{"type":"text","value":"HoHin Lee","user_id":0},{"type":"text","value":"Praneeth Sanapathi","user_id":0},{"type":"text","value":"Sarah Hilado","user_id":0},{"type":"user_nicename","value":"Tristan Naumann","user_id":"37929"},{"type":"user_nicename","value":"Javier Alvarez-Valle","user_id":"32137"},{"type":"user_nicename","value":"Jiang Bian","user_id":"38481"},{"type":"text","value":"Mu Wei","user_id":0},{"type":"text","value":"Khalil Malik","user_id":0},{"type":"user_nicename","value":"Lidong Zhou","user_id":"32673"},{"type":"user_nicename","value":"Jianfeng Gao","user_id":"32246"},{"type":"user_nicename","value":"Eric Horvitz","user_id":"32033"},{"type":"user_nicename","value":"Matthew Lungren","user_id":"42792"},{"type":"user_nicename","value":"Doug Burger","user_id":"31582"},{"type":"text","value":"E. Topol","user_id":0},{"type":"user_nicename","value":"Hoifung Poon","user_id":"32016"},{"type":"text","value":"Paul Vozila","user_id":0}],"msr_publishername":"","msr_publisher_other":"","msr_booktitle":"","msr_chapter":"","msr_edition":"Nature Medicine","msr_editors":"","msr_how_published":"","msr_isbn":"","msr_issue":"","msr_journal":"Nature Medicine","msr_number":"","msr_organization":"","msr_pages_string":"","msr_page_range_start":"","msr_page_range_end":"","msr_series":"","msr_volume":"","msr_copyright":"","msr_conference_name":"","msr_doi":"10.1038\/s41591-026-04501-8","msr_arxiv_id":"","msr_mag_id":"","msr_other_authors":"","msr_other_contributors":"","msr_speaker":"","msr_award":"","msr_affiliation":"","msr_institution":"","msr_host":"","msr_version":"","msr_duration":"","msr_release_tracker_id":"","msr_highlight_type":"","msr_date_display_format":"","msr_main_download_label":"","msr_external_link_label":"","msr_doi_label":"","msr_published_date":"2026-06-26","msr_startdate":"","msr_presentation_date":"","msr_highlight_text":"","msr_notes":"","msr_longbiography":"","msr_publicationurl":"","msr_external_url":"","msr_secondary_video_url":"","msr_conference_url":"","msr_journal_url":"","msr_year":2026,"msr_month":6,"msr_day":26,"msr_microsoftintellectualproperty":false,"msr_pub_id":"8ae1b067222068f1a998a6a1c9bc21034d7d47c8","msr_publication_uploader":[{"type":"url","title":"https:\/\/pubmed.ncbi.nlm.nih.gov\/42362863","label_id":243109,"id":false,"viewUrl":false}],"msr_related_uploader":[],"msr_original_fields_of_study":[],"msr_s2_paper_id":"","msr_s2_pdf_url":"","msr_citation_count_updated":"","msr_citation_count":0,"msr_influential_citations":0,"msr_reference_count":0,"msr_s2_open_access":false,"msr_s2_author_ids":[],"msr_pub_ids":[{"provider":"s2","id":"8ae1b067222068f1a998a6a1c9bc21034d7d47c8"},{"provider":"doi","id":"10.1038\/s41591-026-04501-8"},{"provider":"pubmed","id":"42362863"},{"provider":"corpusid","id":"289611831"}],"msr_hide_image_in_river":0,"footnotes":""},"msr-research-highlight":[],"research-area":[13556,13553],"msr-publication-type":[193715],"msr-publisher":[],"msr-publication-cta":[],"msr-focus-area":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[246985],"msr-conference":[],"msr-journal":[265818],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1179051","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-medical-health-genomics","msr-locale-en_us","msr-field-of-study-medicine"],"msr_publishername":"","msr_edition":"Nature Medicine","msr_affiliation":"","msr_published_date":"2026-06-26","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"Nature Medicine","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":0,"msr_main_download":"","msr_publicationurl":"","msr_doi":"10.1038\/s41591-026-04501-8","msr_publication_uploader":[{"type":"url","title":"https:\/\/pubmed.ncbi.nlm.nih.gov\/42362863","label_id":243109,"id":false,"viewUrl":false}],"msr_related_uploader":[],"msr_citation_count":0,"msr_citation_count_updated":"","msr_s2_paper_id":"","msr_influential_citations":0,"msr_reference_count":0,"msr_arxiv_id":"","msr_s2_author_ids":[],"msr_s2_open_access":false,"msr_s2_pdf_url":null,"msr_attachments":[],"msr-author-ordering":[{"type":"text","value":"Yuxian Gu","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Jingjing Fu","user_id":32287,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jingjing Fu"},{"type":"user_nicename","value":"Xiaodong Liu","user_id":34877,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Xiaodong Liu"},{"type":"user_nicename","value":"Jeya Maria Jose Valanarasu","user_id":43491,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jeya Maria Jose Valanarasu"},{"type":"user_nicename","value":"Noel Codella","user_id":41635,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Noel Codella"},{"type":"user_nicename","value":"Reuben Tan","user_id":43827,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Reuben Tan"},{"type":"text","value":"Qianchu Liu","user_id":0,"rest_url":false},{"type":"text","value":"Ying Jin","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Sheng Zhang","user_id":39087,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Sheng Zhang"},{"type":"user_nicename","value":"Jinyu Wang","user_id":42963,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jinyu Wang"},{"type":"user_nicename","value":"Rui Wang","user_id":40192,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Rui Wang"},{"type":"user_nicename","value":"Lei Song","user_id":41078,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Lei Song"},{"type":"user_nicename","value":"Guanghui Qin","user_id":43527,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Guanghui Qin"},{"type":"user_nicename","value":"Naoto Usuyama","user_id":38670,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Naoto Usuyama"},{"type":"user_nicename","value":"Cliff Wong","user_id":38508,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Cliff Wong"},{"type":"user_nicename","value":"Hao Cheng","user_id":39922,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Hao Cheng"},{"type":"text","value":"HoHin Lee","user_id":0,"rest_url":false},{"type":"text","value":"Praneeth Sanapathi","user_id":0,"rest_url":false},{"type":"text","value":"Sarah Hilado","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Tristan Naumann","user_id":37929,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Tristan Naumann"},{"type":"user_nicename","value":"Javier Alvarez-Valle","user_id":32137,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Javier Alvarez-Valle"},{"type":"user_nicename","value":"Jiang Bian","user_id":38481,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jiang Bian"},{"type":"text","value":"Mu Wei","user_id":0,"rest_url":false},{"type":"text","value":"Khalil Malik","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Lidong Zhou","user_id":32673,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Lidong Zhou"},{"type":"user_nicename","value":"Jianfeng Gao","user_id":32246,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Jianfeng Gao"},{"type":"user_nicename","value":"Eric Horvitz","user_id":32033,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Eric Horvitz"},{"type":"user_nicename","value":"Matthew Lungren","user_id":42792,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Matthew Lungren"},{"type":"user_nicename","value":"Doug Burger","user_id":31582,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Doug Burger"},{"type":"text","value":"E. Topol","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Hoifung Poon","user_id":32016,"rest_url":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Hoifung Poon"},{"type":"text","value":"Paul Vozila","user_id":0,"rest_url":false}],"msr_impact_theme":[],"msr_research_lab":[199565,849856],"msr_event":[],"msr_group":[],"msr_project":[],"publication":[],"video":[],"msr-tool":[],"msr_publication_type":"article","related_content":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1179051","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":2,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1179051\/revisions"}],"predecessor-version":[{"id":1179053,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1179051\/revisions\/1179053"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1179051"}],"wp:term":[{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=1179051"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1179051"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=1179051"},{"taxonomy":"msr-publisher","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publisher?post=1179051"},{"taxonomy":"msr-publication-cta","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-cta?post=1179051"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=1179051"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1179051"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1179051"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=1179051"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=1179051"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=1179051"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1179051"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1179051"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}