Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer

• Verfasst von: Fink, Matthias A. [VerfasserIn]
• Verfasst von: Bischoff, Arved [VerfasserIn]
• Verfasst von: Fink, Christoph Andreas [VerfasserIn]
• Verfasst von: Moll, Martin [VerfasserIn]
• Verfasst von: Kroschke, Jonas [VerfasserIn]
• Verfasst von: Dulz, Luca [VerfasserIn]
• Verfasst von: Heußel, Claus Peter [VerfasserIn]
• Verfasst von: Kauczor, Hans-Ulrich [VerfasserIn]
• Verfasst von: Weber, Tim [VerfasserIn]
• Titel: Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer
• Verf.angabe: Matthias A. Fink, MD, Arved Bischoff, MD, Christoph A. Fink, MD, Martin Moll, MD, Jonas Kroschke, MD, Luca Dulz, MSc, Claus Peter Heußel, MD, Hans-Ulrich Kauczor, MD, Tim F. Weber, MD
• E-Jahr: 2023
• Jahr: September 2023
• Umfang: 9 S.
• Illustrationen: Illustrationen
• Fussnoten: Online veröffentlicht: 19. September 2023 ; Gesehen am 06.06.2024
• Titel Quelle: Enthalten in: Radiology
• Ort Quelle: Oak Brook, Ill. : Soc., 1923
• Jahr Quelle: 2023
• Band/Heft Quelle: 308(2023), 3 vom: Sept., Artikel-ID e231362, Seite 1-9
• ISSN Quelle: 1527-1315
• DOI: doi:10.1148/radiol.231362
• Datenträger: Online-Ressource
• Sprache: eng
• K10plus-PPN: 1890800619

Abstract

Background - - The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. - - Purpose - - To compare the performance of the LLMs ChatGPT and GPT-4 in data mining and labeling oncologic phenotypes from free-text CT reports on lung cancer by using user-defined prompts. - - Materials and Methods - - This retrospective study included patients who underwent lung cancer follow-up CT between September 2021 and March 2023. A subset of 25 reports was reserved for prompt engineering to instruct the LLMs in extracting lesion diameters, labeling metastatic disease, and assessing oncologic progression. This output was fed into a rule-based natural language processing pipeline to match ground truth annotations from four radiologists and derive performance metrics. The oncologic reasoning of LLMs was rated on a five-point Likert scale for factual correctness and accuracy. The occurrence of confabulations was recorded. Statistical analyses included Wilcoxon signed rank and McNemar tests. - - Results - - On 424 CT reports from 424 patients (mean age, 65 years ± 11 [SD]; 265 male), GPT-4 outperformed ChatGPT in extracting lesion parameters (98.6% vs 84.0%, P < .001), resulting in 96% correctly mined reports (vs 67% for ChatGPT, P < .001). GPT-4 achieved higher accuracy in identification of metastatic disease (98.1% [95% CI: 97.7, 98.5] vs 90.3% [95% CI: 89.4, 91.0]) and higher performance in generating correct labels for oncologic progression (F1 score, 0.96 [95% CI: 0.94, 0.98] vs 0.91 [95% CI: 0.89, 0.94]) (both P < .001). In oncologic reasoning, GPT-4 had higher Likert scale scores for factual correctness (4.3 vs 3.9) and accuracy (4.4 vs 3.3), with a lower rate of confabulation (1.7% vs 13.7%) than ChatGPT (all P < .001). - - Conclusion - - When using user-defined prompts, GPT-4 outperformed ChatGPT in extracting oncologic phenotypes from free-text CT reports on lung cancer and demonstrated better oncologic reasoning with fewer confabulations.