Benchmarking vision capabilities of large language models in surgical examination questions

• Verfasst von: Bereuter, Jean-Paul [VerfasserIn]
• Verfasst von: Geißler, Mark Enrik [VerfasserIn]
• Verfasst von: Klimova, Anna [VerfasserIn]
• Verfasst von: Steiner, Robert-Patrick [VerfasserIn]
• Verfasst von: Pfeiffer, Kevin [VerfasserIn]
• Verfasst von: Kolbinger, Fiona [VerfasserIn]
• Verfasst von: Wiest, Isabella [VerfasserIn]
• Verfasst von: Muti, Hannah Sophie [VerfasserIn]
• Verfasst von: Kather, Jakob Nikolas [VerfasserIn]
• Titel: Benchmarking vision capabilities of large language models in surgical examination questions
• Verf.angabe: Jean-Paul Bereuter, MD, Mark Enrik Geissler, MS, Anna Klimova, PhD, Robert-Patrick Steiner, MD, Kevin Pfeiffer, Fiona R. Kolbinger, MD, Isabella C. Wiest, MD, Hannah Sophie Muti, MD, and Jakob Nikolas Kather, MD
• E-Jahr: 2025
• Jahr: April 2025
• Umfang: 11 S.
• Illustrationen: Diagramme
• Fussnoten: Online verfügbar: 9. Februar 2025, Artikelversion: 9. Februar 2025 ; Gesehen am 03.06.2025
• Titel Quelle: Enthalten in: Journal of surgical education
• Ort Quelle: New York, NY : Elsevier, 2007
• Jahr Quelle: 2025
• Band/Heft Quelle: 82(2025), 4 vom: Apr., Artikel-ID 103442, Seite 1-11
• ISSN Quelle: 1878-7452
• DOI: doi:10.1016/j.jsurg.2025.103442
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: exam questions
• Sach-SW: large language models
• Sach-SW: vision capabilities
• Sach-SW: vision language models
• K10plus-PPN: 1927319676

Abstract

Objective - Recent studies investigated the potential of large language models (LLMs) for clinical decision making and answering exam questions based on text input. Recent developments of LLMs have extended these models with vision capabilities. These image processing LLMs are called vision-language models (VLMs). However, there is limited investigation on the applicability of VLMs and their capabilities of answering exam questions with image content. Therefore, the aim of this study was to examine the performance of publicly accessible LLMs in 2 different surgical question sets consisting of text and image questions. - Design - Original text and image exam questions from 2 different surgical question subsets from the German Medical Licensing Examination (GMLE) and United States Medical Licensing Examination (USMLE) were collected and answered by publicly available LLMs (GPT-4, Claude-3 Sonnet, Gemini-1.5). LLM outputs were benchmarked for their accuracy in answering text and image questions. Additionally, the LLMs’ performance was compared to students’ performance based on their average historical performance (AHP) in these exams. Moreover, variations of LLM performance were analyzed in relation to question difficulty and respective image type. - Results - Overall, all LLMs achieved scores equivalent to passing grades (≥60%) on surgical text questions across both datasets. On image-based questions, only GPT-4 exceeded the score required to pass, significantly outperforming Claude-3 and Gemini-1.5 (GPT: 78% vs. Claude-3: 58% vs. Gemini-1.5: 57.3%; p < 0.001). Additionally, GPT-4 outperformed students on both text (GPT: 83.7% vs. AHP students: 67.8%; p < 0.001) and image questions (GPT: 78% vs. AHP students: 67.4%; p < 0.001). - Conclusion - GPT-4 demonstrated substantial capabilities in answering surgical text and image exam questions. Therefore, it holds considerable potential for the use in surgical decision making and education of students and trainee surgeons.