(A-13) The Chimera Algorithm, a 3-Dimensional Nuclear Segmentation Model, Accurately Segments Highly Overlapping Nuclei from Confocal Images

Introduction:: Nuclear segmentation—the process of identifying the contour of each cell—is invaluable to the analysis of cell imaging data. Recent methods have developed segmentation pipelines on 2-dimensional (2D) images based on conventional image processing methods (e.g., marker-controlled watershed [1] and Otus thresholding [2]) or machine learning neural networks (e.g., U-net Resnet [3], R-CNN [4]) to improve the accuracy of nuclear segmentations. However, these methods often suffer from over-segmentation (false positives that identify nuclei clusters as a single nucleus) and under-segmentation (false negatives that miss nuclei) when applied to 3D images. These over and under-segmentation errors are often due to overlapping nuclei and irregular nuclear shapes (Fig. 1). Labelling cells inaccurately can cause false interpretations on cell movement, changes in cell morphology, and cell-cell interaction. To overcome this challenge, we developed the Chimera Algorithm; this algorithm accurately segments nuclei across the z-plane of 3D live cell microscopy images using a sliding window. It serves as a chassis for any state-of-the-art segmentation model and optimizes segmentation accuracy by labelling and replacing nuclear clusters with isolated segmentations. Furthermore, unlike conventional machine learning methods, the chimera algorithm identifies hyperparameters using knowledge of cell behaviors.




Materials and Methods:: Label segmentations from the same nucleus and create neighbor list.
The input to the algorithm is the segmentation output obtained from any state-of-the-art models. To flag falsely labeled nuclear clusters (Fig. 1b and 1d), a window slides from the first confocal slice in the z-plane to the last slice, marking each segmentation center. As nuclei positions are generally consistent across slices, segmentations with similar cell morphologies are identified as the same nucleus by customized k-mean clustering. To reduce the computational cost, the entire image is partitioned into equally sized grids, and each segmentation is assigned to the nearest grid. Subsequently, each segmentation is only compared with those in the adjacent eight grids.

Mark and replace nuclear clusters.
In the event where segmentation centers are enclosed by segmentations from other groups, the algorithm identifies the segmentation with the largest area as a nuclear cluster. The cluster is subsequently replaced by smaller nuclear segmentations whose centers are located within the cluster.

Pad missing nuclei.
Apart from identifying nuclear clusters, the algorithm also detects missing nuclei resulting from irregular shapes and dim intensities by recording the confocal slices where a nucleus first appears and disappears in (si and sj). Knowledge that a nucleus exists between stacks s_i to sj enables the algorithm to propose segmentations in confocal slices where a nucleus disappears within s_i and sj. Thus, the missing nucleus locations are filled with segmentation results from the same nucleus at the nearest confocal slice.

Results, Conclusions, and Discussions:: Results and Discussions

Segmentations from the Chimera Algorithm are compared against alternate methods using images with manually curated ground-truth labels. To conduct a qualitative evaluation, Mesmer-only and those from the Chimera Algorithm are visually inspected. Figure 2 (a) and (b) display the raw human-induced pluripotent stem cell (hi-PSC) and the bilayer epithelial tissue of zebrafish images, along with their corresponding segmentations. Additionally, we qualitatively assess the performance of the Chimera Algorithm by measuring the change in false positive and negative rates. To obtain a false positive (FP) and negative (FN) rate for Mesmer, we sum the number of FP and FN and divide by the total number of segmentations. The resulting values is 200/901 = 22.19%. However, the Chimera Algorithm successfully detects and corrects all these errors.


Furthermore, the Chimera Algorithm detects a missing nucleus in the Mesmer-only result on the hi-PSC images at the 5^th and 6^th stack. The missing nucleus is padded by the same nucleus at the nearest stack (indicated by a red arrow). Besides, the segmentations from identical nuclei at different z stacks have the same contour colors in the Chimera Algorithm results. Therefore, the Chimera Algorithm results can track cells in 3D, a feature that does not exist in Mesmer.


Conclusion

The accuracy of nuclear segmentations plays a crucial role on the downstream analysis. While recent studies have shown that pipelines combining conventional image processing methods (e.g., marker-controlled watershed) and machine learning techniques (e.g., U-net) improve the accuracy of nuclear segmentation, they suffered from over- and under-segmentation problems. These errors are due to overlapping nuclei, noises in the background, and low contrast between nuclei and other impurities. To minimize the number of false segmentations (e.g., nuclear clusters and missing nuclei) I establish a 3D nuclear segmentation pipeline, the Chimera Algorithm, that replaces all failures with isolated segmentations by moving the “sliding window” across entire z-planes of 3D images. The Chimera Algorithm are evaluated qualitatively through visual inspections. The evaluations demonstrate that Chimera Algorithm solves most over- and under-segmentation challenges and can track cells in 3D.




Acknowledgements (Optional): :

References (Optional): :

1. Miao H, Xiao C. 2018. Simultaneous segmentation of leukocyte and erythrocyte in microscopic images using a marker-controlled watershed algorithm. Computational and Mathematical Methods in Medicine 2018(9):1-9


2. Tareef, Afaf, et al. Multi-pass fast watershed for accurate segmentation of overlapping cervical cells. IEEE transactions on medical imaging 37.9 (2018): 2044-2059.


3. U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, 2015, Volume 9351


4. He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision. 2961–2969.


5. Greenwald, N.F., Miller, G., Moen, E. et al. Whole-cell segmentation of tissue images with humanlevel performance using large-scale data annotation and deep learning. Nat Biotechnol 40, 555–565 (2022). https://doi.org/10.1038/s41587-021-01094-0