Semantic hyperspectral image synthesis for cross-modality knowledge transfer in surgical data science

• Verfasst von: Tran Ba, Viet [VerfasserIn]
• Verfasst von: Hübner, Marco [VerfasserIn]
• Verfasst von: Qasim, Ahmad Bin [VerfasserIn]
• Verfasst von: Rees, Maike [VerfasserIn]
• Verfasst von: Sellner, Jan [VerfasserIn]
• Verfasst von: Seidlitz, Silvia [VerfasserIn]
• Verfasst von: Christodoulou, Evangelia [VerfasserIn]
• Verfasst von: Özdemir, Berkin [VerfasserIn]
• Verfasst von: Studier-Fischer, Alexander [VerfasserIn]
• Verfasst von: Nickel, Felix [VerfasserIn]
• Verfasst von: Ayala, Leonardo [VerfasserIn]
• Verfasst von: Maier-Hein, Lena [VerfasserIn]
• Titel: Semantic hyperspectral image synthesis for cross-modality knowledge transfer in surgical data science
• Verf.angabe: Viet Tran Ba, Marco Hübner, Ahmad Bin Qasim, Maike Rees, Jan Sellner, Silvia Seidlitz, Evangelia Christodoulou, Berkin Özdemir, Alexander Studier-Fischer, Felix Nickel, Leonardo Ayala, Lena Maier-Hein
• E-Jahr: 2025
• Jahr: 24 April 2025
• Umfang: 9 S.
• Illustrationen: Illustrationen, Diagramme
• Fussnoten: Gesehen am 03.06.2025
• Titel Quelle: Enthalten in: International journal of computer assisted radiology and surgery
• Ort Quelle: Berlin : Springer, 2006
• Jahr Quelle: 2025
• Band/Heft Quelle: (2025), Seite 1-9
• ISSN Quelle: 1861-6429
• DOI: doi:10.1007/s11548-025-03364-7
• Datenträger: Online-Ressource
• Sprache: eng
• K10plus-PPN: 1927313112

Abstract

Purpose Hyperspectral imaging (HSI) is a promising intraoperative imaging modality, with potential applications ranging from tissue classification and discrimination to perfusion monitoring and cancer detection. However, surgical HSI datasets are scarce, hindering the development of robust data-driven algorithms. The purpose of this work was to address this critical bottleneck with a novel approach to knowledge transfer across modalities. Methods We propose the use of generative modeling to leverage imaging data across optical imaging modalities. The core of the method is a latent diffusion model (LDM) capable of converting a semantic segmentation mask obtained from any modality into a realistic hyperspectral image, such that geometry information can be learned across modalities. The value of the approach was assessed both qualitatively and quantitatively using surgical scene segmentation as a downstream task. Results Our study with more than 13,000 hyperspectral images, partially annotated with a total of 37 tissue and object classes, suggests that LDMs are well-suited for the synthesis of realistic high-resolution hyperspectral images even when trained on few samples or applied to annotations from different modalities and geometric out-of-distribution annotations. Using our approach for generative augmentation yielded a performance boost of up to 35% in the Dice similarity coefficient for the task of semantic hyperspectral image segmentation. Conclusion As our method is capable of augmenting HSI datasets in a manner agnostic to the modality of the leveraged data, it could serve as a blueprint for addressing the data bottleneck encountered for novel imaging modalities.