(D-121) Video Augmentation Techniques for Better Pose Estimation During Human Falls

Introduction:: Human Pose Estimation (HPE) is a computer vision task that involves identifying the position and orientation of individuals within an image or video. This technology has potential applications in various fields, including healthcare, sports analysis, and action recognition [1, 2]. One potential area of focus for action recognition could be the detection of falls in the older adult community. However, current HPE software often struggles to accurately track such events due to a lack of adequate training data [3], which can lead to inaccurate pose estimations or tracking failures. Video augmentation techniques, which modify videos to better align with the training data used by HPE software, may offer a solution to this problem [4].

We hypothesized that rotating the video in the direction of the fall could improve tracking accuracy. This is based on the assumption that most HPE software is not trained to recognize humans in horizontal positions, which is common during fall events. By rotating the video, we aim to keep the subject in a vertical orientation, which is more typical in the training data. In addition to rotation, we also hypothesized that the use of grayscale and zoom video augmentations will improve fall tracking, both of which are commonly used in data science [5,6]. Grayscale conversions can potentially reduce the computational load on the HPE software, potentially improving the efficiency and accuracy of pose estimation, while zooming in on the subject could potentially make it easier for the model to identify and track a falling subject.

Materials and Methods:: 68 videos from an open-source data set containing falls at long-term care facilities [7] featuring sagittal views of falling subjects were analyzed. The videos were subjected to various augmentations: 45° rotations in the direction of the fall, grayscale, grayscale with a 45° rotation in the direction of the fall, 2x zoom, and 2x zoom with grayscale.  Using the pose estimation software, OpenPose Version 1.7.0 [8], each non-augmented and augmented version of the video was processed (Figure 1).

A Python script analyzed each keypoint file, counting frames where the nose keypoint, as per the COCO 25 keypoint model [9, 8], was detected. This keypoint was chosen due to its relevance in tracking fall events, as the head is usually the last to hit the ground. The count of nose keypoints was compared to the total frame count to calculate the percentage of successful nose keypoint tracking, which was used to assess the effectiveness of each augmentation compared to the raw video. Paired t-tests (alpha < 0.05) with Bonferroni corrections were used to compare augmentation techniques.

Results, Conclusions, and Discussions:: Out of 68 videos, 2 failed to track any pose estimation regardless of the applied augmentation, leading to their removal from the dataset. Additionally, 3 videos were removed for over-tracking (detecting more than 1 individual), leaving 63 videos for our analysis.

Not all augmentations proved equally effective (Figure 2). The raw videos captured an average of 18.5 % of nose keypoints. The most successful augmentations were the rotations in the direction of the fall (p < 0.001), and the grayscale (p < 0.001). Not as effective but still notable was the grayscale and rotation in the direction of the fall, which tracked an average of 29.4% of nose keypoints: a 10.9% increase from raw (p < 0.001).  The zoom augmentations did not prove as helpful, capturing fewer keypoints compared to the raw videos as indicated by the positive t-statistics in Table 1.

While acknowledging the limitations of our study, we must also consider the implications of our findings within the context of the dataset used. This dataset presented several challenges, including distant subjects, low video resolution, and poor lighting conditions, all of which independently posed difficulties for OpenPose. Despite these obstacles, our video augmentations resulted in significant improvements in fall tracking. When OpenPose could only identify less than a fifth of possible nose keypoints, simply rotating the video in the direction of the fall led to an average improvement of 1.7x more keypoints. Given these promising results, it's plausible that even better outcomes could be achieved with datasets that don't share these limitations.

In conclusion, our results demonstrate that video augmentations can improve the performance of HPE software in tracking keypoints in videos containing fall events. Specifically, applying a grayscale or rotating the video in the direction of the fall relative to the camera proved successful. These findings suggest that until HPE software is specifically trained on fall events, applying video augmentations can effectively compensate for the gaps in pose estimation software.


Acknowledgements (Optional): :

References (Optional): : 1.       J. Hong, M. Fisher, M. Gharbi, and K. Fatahalian, "Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition," in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, pp. 9234-9243. [Online]. Available: https://doi.org/10.1109/ICCV48922.2021.00912

2.       D. C. Luvizon, D. Picard, and H. Tabia, "Multi-Task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 8, pp. 2752-2764, Aug. 2021. [Online]. Available: https://doi.org/10.1109/TPAMI.2020.2976014

3.       M. Salimi, J. J. M. Machado, and J. M. R. S. Tavares, "Using Deep Neural Networks for Human Fall Detection Based on Pose Estimation," in Sensors, vol. 22, no. 12, 4544, 2022. [Online]. Available: https://doi.org/10.3390/s22124544

4.       C. Khosla and B. S. Saini, "Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey," in Proc. International Conference on Intelligent Engineering and Management (ICIEM), London, UK, 2020, pp. 79-85. [Online]. Available: https://doi.org/10.1109/ICIEM48762.2020.9160048

5.       J. Wang and S. Lee, "Data Augmentation Methods Applying Grayscale Images for Convolutional Neural Networks in Machine Vision," in Applied Sciences, vol. 11, no. 15, 6721, 2021. [Online]. Available: https://doi.org/10.3390/app11156721

6.       M. Z. Islam, M. S. Hossain, R. ul Islam, and K. Andersson, "Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation," in Proc. Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR), Spokane, WA, USA, 2019, pp. 324-329. [Online]. Available: https://doi.org/10.1109/ICIEV.2019.8858563

7.       S. Robinovitch, "Falls experienced by older adult residents in long-term care homes," Databrary, 2018. [Online]. Available: https://doi.org/10.17910/b7.739

8.       Z. Cao, G. Hidalgo, T. Simon, S. -E. Wei, and Y. Sheikh, "OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 172-186, Jan. 2021. [Online]. Available: https://doi.org/10.1109/TPAMI.2019.2929257

9.       T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft COCO: Common Objects in Context," in Proc. Computer Vision–ECCV 2014, Zurich, Switzerland, 6–12 September 2014; D. Fleet, T. Pajdla, B. Schiele, T. Tuytelaars, Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 740–755. [Online]. Available: https://doi.org/10.1007/978-3-319-10602-1_48