3D-guided face manipulation of 2D images for the prediction of post-operative outcome after cranio-maxillofacial surgery

• Verfasst von: Andlauer, Robin [VerfasserIn]
• Verfasst von: Wachter, Andreas [VerfasserIn]
• Verfasst von: Schaufelberger, Matthias [VerfasserIn]
• Verfasst von: Bouffleur, Frederic [VerfasserIn]
• Verfasst von: Kühle, Reinald [VerfasserIn]
• Verfasst von: Freudlsperger, Christian [VerfasserIn]
• Verfasst von: Nahm, Werner [VerfasserIn]
• Titel: 3D-guided face manipulation of 2D images for the prediction of post-operative outcome after cranio-maxillofacial surgery
• Verf.angabe: Robin Andlauer, Andreas Wachter, Matthias Schaufelberger, Frederic Weichel, Reinald Kühle, Christian Freudlsperger, and Werner Nahm
• E-Jahr: 2021
• Jahr: July 15, 2021
• Umfang: 15 S.
• Fussnoten: Gesehen am 07.10.2021
• Titel Quelle: Enthalten in: Institute of Electrical and Electronics EngineersIEEE transactions on image processing
• Ort Quelle: New York, NY : IEEE, 1992
• Jahr Quelle: 2021
• Band/Heft Quelle: 30(2021), Seite 7349-7363
• ISSN Quelle: 1941-0042
• DOI: doi:10.1109/TIP.2021.3096081
• Datenträger: Online-Ressource
• Sprache: eng
• Sach-SW: 3D morphable model
• Sach-SW: Cranio-maxillofacial surgery
• Sach-SW: cycle loss
• Sach-SW: CycleGAN
• Sach-SW: face editing
• Sach-SW: face manipulation
• Sach-SW: Faces
• Sach-SW: gan
• Sach-SW: generative adversarial network
• Sach-SW: Planning
• Sach-SW: post-operative face
• Sach-SW: Predictive models
• Sach-SW: Solid modeling
• Sach-SW: Surgery
• Sach-SW: surgery planning
• Sach-SW: Three-dimensional displays
• Sach-SW: Tools
• Sach-SW: unsupervised learning
• K10plus-PPN: 1772749699

Abstract

Cranio-maxillofacial surgery often alters the aesthetics of the face which can be a heavy burden for patients to decide whether or not to undergo surgery. Today, physicians can predict the post-operative face using surgery planning tools to support the patient's decision-making. While these planning tools allow a simulation of the post-operative face, the facial texture must usually be captured by another 3D texture scan and subsequently mapped on the simulated face. This approach often results in face predictions that do not appear realistic or lively looking and are therefore ill-suited to guide the patient's decision-making. Instead, we propose a method using a generative adversarial network to modify a facial image according to a 3D soft-tissue estimation of the post-operative face. To circumvent the lack of available data pairs between pre- and post-operative measurements we propose a semi-supervised training strategy using cycle losses that only requires paired open-source data of images and 3D surfaces of the face's shape. After training on "in-the-wild" images we show that our model can realistically manipulate local regions of a face in a 2D image based on a modified 3D shape. We then test our model on four clinical examples where we predict the post-operative face according to a 3D soft-tissue prediction of surgery outcome, which was simulated by a surgery planning tool. As a result, we aim to demonstrate the potential of our approach to predict realistic post-operative images of faces without the need of paired clinical data, physical models, or 3D texture scans.