Manager Department of Cardiovascular Surgery, Suita Tokushukai Hospital Suita, Osaka, Japan
Introduction: Cardiac surgeons must understand the anatomy and dysfunction of the heart and must be well-trained for the repairs, concomitant procedures, and bailouts during surgeries. CT scans have contributed to accurate diagnosis, treatment, and understanding of heart structures. In addition, 3D models from CT images have been used and are helpful, especially for young surgeons. However, 3D heart models have two time-consuming steps: image segmentation to construct 3D polygon data and 3D printing of complicated heart structures. Because exact shapes of small details such as valvular leaflets and muscular protrusions are essential for surgical training, manual segmentation is used but causes a prolonged duration time. We trained a convolutional neural network (CNN) with a large CT dataset of human hearts for automatic segmentation. We created augmented reality (AR) heart models from the segmentation output of the trained CNN. This approach enables the rapid creation of AR heart models for surgical assessments.
Materials and Methods: First, we trained our original CNN with CT images of 339 human hearts (20 to 44 images in each case). A cardiac surgeon created grand-truth images of 24 cases (732 images) for supervised training. We used the other 315 cases (11,170 images) for complementary training with virtual adversarial regularization of the Neural Structured Learning package in TensorFlow (Google LLC). We applied data augmentation to prepare 22,045 images for supervised training, 5,265 for validation, and 223,400 for virtual adversarial loss and saved a CNN with the best Intersection over Union (IoU) as a result of deep learning. Second, we used the trained CNN for automatic segmentation of CT images of surgical cases and converted the results to the 3D polygon data in STL format using a CT viewer OsiriX MD (Pixmeo SARL). We created AR models using Blender (Blender Foundation) and Reality Converter (Apple Inc.) and checked whether they were available for surgical assessments and training.
Results: The mean IoU attained 0.82 in the best-trained CNN. The time required to complete the image segmentation and 3D polygon conversion was several minutes without manual correction. The duration for AR model creation depended on the quality of 3D polygon data, up to ten hours.
Conclusion: The CNN-based segmentation aided the construction of heart shapes from CT images. Although the performance, reaching the mean IoU of 0.82, was insufficient for fully automatic segmentation, the trained CNN could replace most of the manual labor for detailed structural segmentation. The methodology can shorten the long-duration works and enhance the usefulness of 3D shapes for anatomical assessments of the heart.
Discussions: 3D shapes are helpful for anatomical assessment. Still, the time and cost are limited in clinical settings, and those created 3D models or AR data will be unnecessary after the treatment or surgery completes. Generally, around 150 images with a 1-mm slice interval can cover a heart in a CT scan. Thus, manual segmentation for each image is a tough job with a long duration. In addition, 3D printing of complicated shapes is still laboratory or manufacturing work with chemical materials. In our experience, manual segmentation of a heart needs more than twenty hours, and 3D printing takes several days, including post-processing such as brushing. A rapid and laborless method with good quality is required.
Acknowledgements: The authors thank the expert radiographers at Suita Tokushukai Hospital for performing the precise CT scans. This work was partly achieved through the use of SQUID at the Cybermedia Center at Osaka University.
