(B-50) Application of Cell Vision Transformer Model for the Precise Segmentation and Classification of Mitochondria

Introduction:: The identification and segmentation of the mitochondria from tissue images are critical for the development of various treatments with widespread clinical applications. As the identification and segmentation of mitochondria with the utilization of classical techniques of pathology have proven to be arduous and time consuming at times, digital pathology is increasingly becoming a more popular and necessary field of study to improve clinical efficiency. However, the identification and segmentation task may prove to be arduous due to the variability in the sizes and locations of the mitochondria between different cells. To combat this issue, image segmentation techniques based off machine learning and deep learning such as the convolutional neural network has been applied. However, there has been minimal research on application of the vision transformer model, a relatively novel development in deep learning, for the identification and segmentation of mitochondria from electron microscopy images. In this study, the accuracy of mitochondria segmentation through a vision transformer model is investigated.

Materials and Methods:: For this study, the methodology is the modification of a deep learning algorithm that utilizes a vision transformer model to classify and segment nuclei from hematoxylin and eosin-stained histology images called CellViT. The modification completed by fine-tuning the CellViT model with MitoEM, a large-scale three-dimensional dataset with annotated mitochondria in electron microscopy images of rat and human cortices, to teach develop a novel vision transformer model that identifies and segments mitochondria from images. After fine-tuning, the network architecture includes a vision transformer encoder and a decoder connected to two output branches through skip connections: the Mitochondria Prediction (MP) Branch and Horizontal and Vertical (HV) Branch, both of which require post-processing to create the final segmented image. A validated postprocessing pipeline is used to create the final output of an image with segmented mitochondria.

Results, Conclusions, and Discussions:: The fine-tuned vision transformer model onto electron microscopy images for the identification and segmentation of mitochondria produced a DICE coefficient of 86.7% and a F1 score of 89.2%. Further modifications of the model are necessary to increase the accuracy of the identification and segmentation of mitochondria on tissue images. Research into the most effective and accurate segmentation decoders that can be connected to the vision transformer model could improve the accuracy of segmentation of not only mitochondria from electron microscopy images but also for a variety of tasks.

Acknowledgements (Optional): :

References (Optional): Hörst, F., Rempe, M., Heine, L., Seibold, C., Keyl, J., Baldini, G., ... & Kleesiek, J. (2023). CellViT: Vision Transformers for Precise Cell Segmentation and Classification. arXiv preprint arXiv:2306.15350. :