Navigation überspringen
Universitätsbibliothek Heidelberg
Status: Bibliographieeintrag

Verfügbarkeit
Standort: ---
Exemplare: ---
heiBIB
 Online-Ressource
Verfasst von:Bereuter, Jean-Paul [VerfasserIn]   i
 Geißler, Mark Enrik [VerfasserIn]   i
 Klimova, Anna [VerfasserIn]   i
 Steiner, Robert-Patrick [VerfasserIn]   i
 Pfeiffer, Kevin [VerfasserIn]   i
 Kolbinger, Fiona [VerfasserIn]   i
 Wiest, Isabella [VerfasserIn]   i
 Muti, Hannah Sophie [VerfasserIn]   i
 Kather, Jakob Nikolas [VerfasserIn]   i
Titel:Benchmarking vision capabilities of large language models in surgical examination questions
Verf.angabe:Jean-Paul Bereuter, MD, Mark Enrik Geissler, MS, Anna Klimova, PhD, Robert-Patrick Steiner, MD, Kevin Pfeiffer, Fiona R. Kolbinger, MD, Isabella C. Wiest, MD, Hannah Sophie Muti, MD, and Jakob Nikolas Kather, MD
E-Jahr:2025
Jahr:April 2025
Umfang:11 S.
Illustrationen:Diagramme
Fussnoten:Online verfügbar: 9. Februar 2025, Artikelversion: 9. Februar 2025 ; Gesehen am 03.06.2025
Titel Quelle:Enthalten in: Journal of surgical education
Ort Quelle:New York, NY : Elsevier, 2007
Jahr Quelle:2025
Band/Heft Quelle:82(2025), 4 vom: Apr., Artikel-ID 103442, Seite 1-11
ISSN Quelle:1878-7452
Abstract:Objective - Recent studies investigated the potential of large language models (LLMs) for clinical decision making and answering exam questions based on text input. Recent developments of LLMs have extended these models with vision capabilities. These image processing LLMs are called vision-language models (VLMs). However, there is limited investigation on the applicability of VLMs and their capabilities of answering exam questions with image content. Therefore, the aim of this study was to examine the performance of publicly accessible LLMs in 2 different surgical question sets consisting of text and image questions. - Design - Original text and image exam questions from 2 different surgical question subsets from the German Medical Licensing Examination (GMLE) and United States Medical Licensing Examination (USMLE) were collected and answered by publicly available LLMs (GPT-4, Claude-3 Sonnet, Gemini-1.5). LLM outputs were benchmarked for their accuracy in answering text and image questions. Additionally, the LLMs’ performance was compared to students’ performance based on their average historical performance (AHP) in these exams. Moreover, variations of LLM performance were analyzed in relation to question difficulty and respective image type. - Results - Overall, all LLMs achieved scores equivalent to passing grades (≥60%) on surgical text questions across both datasets. On image-based questions, only GPT-4 exceeded the score required to pass, significantly outperforming Claude-3 and Gemini-1.5 (GPT: 78% vs. Claude-3: 58% vs. Gemini-1.5: 57.3%; p < 0.001). Additionally, GPT-4 outperformed students on both text (GPT: 83.7% vs. AHP students: 67.8%; p < 0.001) and image questions (GPT: 78% vs. AHP students: 67.4%; p < 0.001). - Conclusion - GPT-4 demonstrated substantial capabilities in answering surgical text and image exam questions. Therefore, it holds considerable potential for the use in surgical decision making and education of students and trainee surgeons.
DOI:doi:10.1016/j.jsurg.2025.103442
URL:Bitte beachten Sie: Dies ist ein Bibliographieeintrag. Ein Volltextzugriff für Mitglieder der Universität besteht hier nur, falls für die entsprechende Zeitschrift/den entsprechenden Sammelband ein Abonnement besteht oder es sich um einen OpenAccess-Titel handelt.

kostenfrei: Volltext: https://doi.org/10.1016/j.jsurg.2025.103442
 kostenfrei: Volltext: https://www.sciencedirect.com/science/article/pii/S1931720425000236
 DOI: https://doi.org/10.1016/j.jsurg.2025.103442
Datenträger:Online-Ressource
Sprache:eng
Sach-SW:exam questions
 large language models
 vision capabilities
 vision language models
K10plus-PPN:1927319676
Verknüpfungen:→ Zeitschrift

Permanenter Link auf diesen Titel (bookmarkfähig):  https://katalog.ub.uni-heidelberg.de/titel/69351828   QR-Code
zum Seitenanfang