Evaluation of four chatbots in autoimmune liver disease

• Verfasst von: Daza Barragán, Jimmy Andres [VerfasserIn]
• Verfasst von: Bezerra, Lucas Soares [VerfasserIn]
• Verfasst von: Santamaría, Laura [VerfasserIn]
• Verfasst von: Rueda-Esteban, Roberto [VerfasserIn]
• Verfasst von: Bantel, Heike [VerfasserIn]
• Verfasst von: Girala, Marcos [VerfasserIn]
• Verfasst von: Ebert, Matthias [VerfasserIn]
• Verfasst von: Van Bömmel, Florian [VerfasserIn]
• Verfasst von: Geier, Andreas [VerfasserIn]
• Verfasst von: Aldana, Andres Gomez [VerfasserIn]
• Verfasst von: Yau, Kevin [VerfasserIn]
• Verfasst von: Alvares-da-Silva, Mario [VerfasserIn]
• Verfasst von: Peck-Radosavljevic, Markus [VerfasserIn]
• Verfasst von: Ridruejo, Ezequiel [VerfasserIn]
• Verfasst von: Weinmann, Arndt [VerfasserIn]
• Verfasst von: Teufel, Andreas [VerfasserIn]
• Titel: Evaluation of four chatbots in autoimmune liver disease
• Titelzusatz: a comparative analysis
• Verf.angabe: Jimmy Daza, Lucas Soares Bezerra, Laura Santamaría, Roberto Rueda-Esteban, Heike Bantel, Marcos Girala, Matthias Ebert, Florian Van Bömmel, Andreas Geier, Andres Gomez Aldana, Kevin Yau, Mario Alvares-da-Silva, Markus Peck-Radosavljevic, Ezequiel Ridruejo, Arndt Weinmann, Andreas Teufel
• E-Jahr: 2025
• Jahr: January/February 2025
• Umfang: 7 S.
• Illustrationen: Illustrationen
• Fussnoten: Online verfügbar: 13. August 2024, Artikelversion: 5. September 2024 ; Gesehen am 24.02.2025
• Titel Quelle: Enthalten in: Annals of hepatology
• Ort Quelle: México : Assoc., 2002
• Jahr Quelle: 2025
• Band/Heft Quelle: 30(2025), 1 vom: Jan./Feb., Artikel-ID 101537, Seite 1-7
• DOI: doi:10.1016/j.aohep.2024.101537
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: Artificial intelligence
• Sach-SW: Autoimmune liver disease
• Sach-SW: Chatbots
• Sach-SW: Clinical decision support tools
• K10plus-PPN: 1917883641

Abstract

Introduction and Objectives - Autoimmune liver diseases (AILDs) are rare and require precise evaluation, which is often challenging for medical providers. Chatbots are innovative solutions to assist healthcare professionals in clinical management. In our study, ten liver specialists systematically evaluated four chatbots to determine their utility as clinical decision support tools in the field of AILDs. - Materials and Methods - We constructed a 56-question questionnaire focusing on AILD evaluation, diagnosis, and management of Autoimmune Hepatitis (AIH), Primary Biliary Cholangitis (PBC), and Primary Sclerosing Cholangitis (PSC). Four chatbots -ChatGPT 3.5, Claude, Microsoft Copilot, and Google Bard- were presented with the questions in their free tiers in December 2023. Responses underwent critical evaluation by ten liver specialists using a standardized 1 to 10 Likert scale. The analysis included mean scores, the number of highest-rated replies, and the identification of common shortcomings in chatbots performance. - Results - Among the assessed chatbots, specialists rated Claude highest with a mean score of 7.37 (SD = 1.91), followed by ChatGPT (7.17, SD = 1.89), Microsoft Copilot (6.63, SD = 2.10), and Google Bard (6.52, SD = 2.27). Claude also excelled with 27 best-rated replies, outperforming ChatGPT (20), while Microsoft Copilot and Google Bard lagged with only 6 and 9, respectively. Common deficiencies included listing details over specific advice, limited dosing options, inaccuracies for pregnant patients, insufficient recent data, over-reliance on CT and MRI imaging, and inadequate discussion regarding off-label use and fibrates in PBC treatment. Notably, internet access for Microsoft Copilot and Google Bard did not enhance precision compared to pre-trained models. - Conclusions - Chatbots hold promise in AILD support, but our study underscores key areas for improvement. Refinement is needed in providing specific advice, accuracy, and focused up-to-date information. Addressing these shortcomings is essential for enhancing the utility of chatbots in AILD management, guiding future development, and ensuring their effectiveness as clinical decision-support tools.