Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation

• Verfasst von: Sharan, Lalith [VerfasserIn]
• Verfasst von: Romano, Gabriele [VerfasserIn]
• Verfasst von: Köhler, Sven [VerfasserIn]
• Verfasst von: Kelm, Halvar [VerfasserIn]
• Verfasst von: Karck, Matthias [VerfasserIn]
• Verfasst von: De Simone, Raffaele [VerfasserIn]
• Verfasst von: Engelhardt, Sandy [VerfasserIn]
• Titel: Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation
• Verf.angabe: Lalith Sharan, Gabriele Romano, Sven Koehler, Halvar Kelm, Matthias Karck, Raffaele De Simone and Sandy Engelhardt
• E-Jahr: 2022
• Jahr: January 2022
• Umfang: 11 S.
• Fussnoten: Gesehen am 07.04.2022
• Titel Quelle: Enthalten in: Institute of Electrical and Electronics EngineersIEEE journal of biomedical and health informatics
• Ort Quelle: New York, NY : IEEE, 2013
• Jahr Quelle: 2022
• Band/Heft Quelle: 26(2022), 1, Seite 127-138
• ISSN Quelle: 2168-2208
• DOI: doi:10.1109/JBHI.2021.3099858
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: CycleGAN
• Sach-SW: Generative adversarial networks
• Sach-SW: landmark detection
• Sach-SW: landmark localization
• Sach-SW: Maintenance engineering
• Sach-SW: mitral valve repair
• Sach-SW: Semantics
• Sach-SW: Surgery
• Sach-SW: surgical simulation
• Sach-SW: surgical training
• Sach-SW: Task analysis
• Sach-SW: Training
• Sach-SW: Valves
• K10plus-PPN: 1798141248

Abstract

The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance of the same surgical target structure. This can be viewed as a novel augmented reality approach, which we coined Hyperrealism in previous work. In this use case, it is of paramount importance to display objects like needles, sutures or instruments consistent in both domains while altering the style to a more tissue-like appearance. Segmentation of these objects would allow for a direct transfer, however, contouring of these, partly tiny and thin foreground objects is cumbersome and perhaps inaccurate. Instead, we propose to use landmark detection on the points when sutures pass into the tissue. This objective is directly incorporated into a CycleGAN framework by treating the performance of pre-trained detector models as an additional optimization goal. We show that a task defined on these sparse landmark labels improves consistency of synthesis by the generator network in both domains. Comparing a baseline CycleGAN architecture to our proposed extension (DetCycleGAN), mean precision (PPV) improved by $+61.32$, mean sensitivity (TPR) by $+37.91$, and mean $F_1$ score by $+0.4743$. Furthermore, it could be shown that by dataset fusion, generated intra-operative images can be leveraged as additional training data for the detection network itself.