(G-253) Efficient Annotation of Large ICU Patient Mobility Data Using a YOLOv5-based Active Learning Pipeline

Introduction:: Every year more than 5 million patients are admitted to Intensive Care Units (ICU) in the US. Monitoring patients’ mobility in ICU is necessary to understand their recovery status, which is challenging due to large manpower costs. An autonomous, accurate, and privacy-preserving monitoring system using deep learning (DL) can address this challenge. For training state-of-the-art DL models, a curated and annotated dataset is required. We have collected over 17 million depth images recordings of patients' mobility status. However, manually annotating the entire dataset is impractical. Therefore, we proposed an Active Learning inspired model-in-loop pipeline using state-of-art object detection model You Only Look Once version 5 (YOLOv5) to improve the efficiency of the annotation process. Our method reduced the number of samples that needed manual annotation to 5.39% of the original dataset i.e., 583,343 images.


Materials and Methods:: The data used in this study were obtained from ICUs at the University of Florida (UF) and UF Health Shands Hospital (N=10,813,519). 16,461 images from 71 patients were annotated at the first stage, where we encouraged the annotators to label only one person even when multiple people were present in a frame to reduce the annotation workload. This consisted our initial dataset. A pretrained YOLOv5 large model was fine-tuned on this initial dataset to detect the patient's mobility. Mobility was represented by 3 classes: Assisted, Sitting, and Standing. We reported the performance of the model using mean average precision (mAP) at 0.5 Intersection over Union (IoU) and mAP @ 0.95 IoU thresholds, F1-Score, Sensitivity, and Specificity. This model was then used as a people detector to select samples which required further manual annotation. Additionally, a similarity metric called structural similarity index (SSIM) was applied to select only one image from a sequence of samples which were similar to each other. This reduced the annotation workload significantly. In the future, AL will be incorporated to further minimize the workload. In the AL setup, a model trained on our initial dataset will perform inference on the large unlabeled dataset. The samples with low confidence will be sent to annotators. Then the model will be iteratively fine-tuned with the newly annotated data.


Results, Conclusions, and Discussions:: We achieved 0.98 F1 score on the task of people detection using the model trained on our initial dataset (N=10,942), with Sensitivity=0.97 and Specificity=0.99 on the test set (N=5,519). For the mobility detection task, we achieved mAP @ 0.5 IoU=0.84 and mAP @ 0.95 IoU=0.59 on the same test set. With the help of the model-in-loop annotation pipeline, we extracted 2,081,634 out of 10,813,519 samples only by the YOLOv5 model. 583,343 samples were further selected out of 2,081,634 samples by the SSIM metric. The overall ratio of selection was 5.39%.

In this study, we deployed a deep learning algorithm to automatically select samples that need to be annotated for mobility detection in the ICU. The pipeline we designed can not only largely reduce the workload and costs associated with manual annotation but also perform the task of mobility detection in the ICU with a small number of annotated examples.

We proposed a model-in-loop annotation pipeline to make an image annotation process for mobility detection in the ICU more efficient. There are some limitations in our work: 1) the number of mobility classes was limited, 2) the initial dataset used only had one bounding box per sample. In the future, we will include more mobility classes to improve the granularity of our dataset and the clinical relevance of our model. Additionally, we will develop the AL pipeline to iteratively fine-tune the model with newly annotated samples.


Acknowledgements (Optional): :

References (Optional): : Redmon, Joseph, et al. "You only look once: Unified, real-time object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Monarch, Robert Munro. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster, 2021.
Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." IEEE transactions on image processing 13.4 (2004): 600-612.