(J-369) Landmark-Guided Graph Neural Network for Predicting Facial Appearance following Orthognathic Surgery

Introduction:: Craniomaxillofacial (CMF) deformities refer to various congenital and acquired disfigurements that affect the structures of the skull, face, and jaw [1,2]. Orthognathic surgery is a common option sought by individuals with facial deformities to enhance their facial appearance [3-5]. Currently, surgical planning is a bone-driven approach, focusing on restoring a normal bone structure through computer-aided surgical simulation (CASS) [6,7]. However, bone-driven approach requires time-consuming plan revision to achieve a desired ideal facial outcome by iteratively simulating facial changes [8]. This makes the approach practically impossible to be performed in daily clinical practice [9]. Soft-tissue-driven planning can present a solution to overcome the limitations of bone-driven approaches by directly estimating required bony movements to achieve the ideal facial outcome. In the soft-tissue-driven approach, accurate estimation of the ideal facial outcome is critically important. Ideal facial outcome can be efficiently estimated by positioning the preoperative facial landmarks into ideal positions. However, only limited number of facial landmarks exist and estimating the ideal facial outcome based on the sparse ideal facial landmarks is a substantially challenging task.

This study introduces a new approach called the Facial Landmark Guided Appearance Prediction network (FLGAP) that can predict the change of the entire face based on given sparse set of facial landmarks. We assume that a small number of clinically significant facial landmarks can provide sufficient information to reconstruct the entire face. The proposed method is a significant step towards the soft-tissue-driven approach that can overcome the limitations of the bone-driven surgical planning approaches.

Materials and Methods:: FLGAP (Figure 1) takes two inputs, facial landmark displacements and preoperative facial mesh, to predict facial change based on the displacements of the facial landmarks.

First, the points of facial mesh are divided into 23 clusters using K-means clustering with 23 facial landmarks as centroids. Clustered facial mesh is then processed through a GCN [10] block and downsampled. Random point sampling based on a uniform distribution is used for downsampling to maintain the topological information of the mesh. The facial mesh passes through two GCN blocks and downsampling, followed by average pooling applied to each cluster, to generate 23 feature vectors. Second, features of the landmark displacements between preoperative and postoperative are extracted using PointNet++ [11]. Landmark displacement features are concatenated with facial mesh feature vectors. Finally, upsampling is performed to estimate the point-wise displacements of the facial mesh and generate a predicted facial mesh. We employ the Farthest Point Sampling (FPS) method for upsampling. As a result, point-wise displacements are estimated and applied to the preoperative facial mesh to generate the predicted facial mesh.

The experiments are conducted on a dataset of paired preoperative and postoperative facial surface meshes of 56 patients who underwent orthognathic surgery. To avoid overfitting, we applied 5-fold cross-validation on the training set to train five models. The prediction accuracy compared with the postoperative facial mesh was quantitatively evaluated using two measurements: 1. the average distance error of clinically important six landmarks (Figure 3) and 2. surface deviation errors.


Results, Conclusions, and Discussions:: Results
Table 1 summarizes the prediction accuracy of three different methods. DeepGCN and GraphUNet showed similar landmark-based errors, while FLGAP demonstrated significantly improved accuracy, particularly at the Pog where the largest facial deformity occurs. Figure 4 and Figure 5 show average and worst examples of the estimated facial mesh with color-coded surface deviation error, respectively. FLGAP demonstrated a considerably smaller maximum error and improvements in clinically important regions (i.e., the lips and chin). Especially for the worst examples (Fig. 5), the accuracy of our method in the chin region was far superior to other methods.


Discussion
Our FLGAP accurately predicted facial change based on the displacement of the sparse facial landmarks using the information of both facial mesh and facial landmarks. The accuracy evaluation results indicate that FLGAP successfully utilized the information of a small number of the landmarks to predict the change of entire facial mesh, outperforming other GCN-based methods. FLGAP was superior to DeepGCN which does not utilize downsampling-upsampling process, suggesting face encoder-decoder structure of FLGAP well preserved the topological information of the non-regular facial mesh structure. It is particularly noteworthy that FLGAP outperformed GraphUNet that shares the same encoder-decoder structure, highlighting the importance of incorporating landmark information to predict facial change.

Our proposed FLGAP offers promising avenues for the future of orthognathic surgery planning. FLGAP can be directly utilized to predict the ideal facial outcome by providing the ideal position of facial landmarks, which is very efficient because information of only a small number of facial landmarks is necessary to estimate the ideal facial outcome. More importantly, the ideal position of such facial landmarks can be easily derived by leveraging the cephalometric measurements of normal subjects. Ultimately, the patient-specific ideal face estimation using FLGAP can enable soft-driven planning, remarkably accelerating surgical planning procedure.

Conclusion
We proposed a landmark-guided facial change prediction method FLGAP that successfully predicted facial change, utilizing information of facial mesh and small number of sparse facial landmarks. Accuracy evaluation proved superior performance of our method compared to other GCN-based methods.


Acknowledgements (Optional): :

References (Optional): : 1. Robinson, Randolph C., and Rebecca L. Holm. "Orthognathic surgery for patients with maxillofacial deformities." AORN journal 92.1 (2010): 28-52.
2. David, David J., David C. Hemmy, and Rodney D. Cooter. Craniofacial deformities: atlas of three-dimensional reconstruction from computed tomography. Springer Science & Business Media, 2012.
3. Shafi, M. I., et al. "The accuracy of three-dimensional prediction planning for the surgical correction of facial deformities using Maxilim." International journal of oral and maxillofacial surgery 42.7 (2013): 801-806.
4. Proffit, W. R., T. A. Turvey, and C. Phillips. "Orthognathic surgery: a hierarchy of stability." The International journal of adult orthodontics and orthognathic surgery 11.3 (1996): 191-204.
5. Khechoyan, David Y. "Orthognathic surgery: general considerations." Seminars in plastic surgery. Vol. 27. No. 03. Thieme Medical Publishers, 2013.
6. Xia, James J., Jaime Gateno, and John F. Teichgraeber. "Three-dimensional computer-aided surgical simulation for maxillofacial surgery." Atlas of the oral and maxillofacial surgery clinics of North America 13.1 (2005): 25-39.
7. Xia, J. J., et al. "Algorithm for planning a double-jaw orthognathic surgery using a computer-aided surgical simulation (CASS) protocol. Part 2: three-dimensional cephalometry." International journal of oral and maxillofacial surgery 44.12 (2015): 1441-1450.
8. Fang, Xi, et al. "Deep Learning-Based Facial Appearance Simulation Driven by Surgically Planned Craniomaxillofacial Bony Movement." Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VII. Cham: Springer Nature Switzerland, 2022.
9. Ma, Lei, et al. "Deep simulation of facial appearance changes following craniomaxillofacial bony movements in orthognathic surgical planning." Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24. Springer International Publishing, 2021.
10. Kipf, Thomas N., and Max Welling. "Semi-supervised classification with graph convolutional networks." arXiv preprint arXiv:1609.02907 (2016).
11. Qi, Charles Ruizhongtai, et al. "Pointnet++: Deep hierarchical feature learning on point sets in a metric space." Advances in neural information processing systems 30 (2017).
12. Li, Guohao, et al. "Deepgcns: Can gcns go as deep as cnns?." Proceedings of the IEEE/CVF international conference on computer vision. 2019.
13. Gao, Hongyang, and Shuiwang Ji. "Graph u-nets." international conference on machine learning. PMLR, 2019.