Intelligent upper-limb exoskeleton using deep learning to predict human intention for sensory-feedback augmentation

Introduction:: The decline in musculoskeletal strength and control due to age or stroke affects a significant portion of the global population, impacting the functional independence of individuals with neuromotor disorders. Current exoskeleton robots struggle to provide viable solutions due to limitations like the lack of human intention prediction, sensory feedback, and support for multiple upper limb joint movements. This study proposes an intent-driven robotic exoskeleton that integrates a cloud-based human intention predicting deep-learning algorithm and skin-like bioelectronics for sensory feedback. Soft lightweight pneumatics provide force and displacement to assist upper-limb joint movements. Wireless bioelectronics collect real-time EMG signals, which are cloud-computed to determine the user's intended movement with an average accuracy of 96.2%. The exoskeleton can augment human strength by 5.15 times, offering a promising solution for those with neuromotor disorders and transforming their lifestyles in the future.

Materials and Methods:: PAM&Exoskeleton Fabrication:
The actuator consists of silicone tubing, polyester mesh, and aluminum fittings. Quasistatic testing evaluates performance at various pressures. The exoskeleton uses carbon fiber, aluminum connectors, and steel fasteners. It features telescoping collets and 3D-printed arm mounts for adjustability.

Skin-like EMG Sensor Fabrication and Characterization:
Fabrication process involves fabric substrate, circuit development, nanomembrane electrodes, and assembly. Stretchable sensors are made with Silbione, gold/chromium electrodes, and PDMS. Mechanical and electrical tests measure sensor properties, including cyclic stretching, circuit bending, and electrode-skin impedance, and finite element analysis.

Data Processing and Acquisition:
Data is processed using Python, bandpass filters, notch filters and Strandard Scaler method. Instrumentation amplifiers and negative feedback circuits capture biopotentials, followed by filtering and amplification.

Cloud Computing Interface and Deep-learning Motion Classifications
A microcontroller and Bluetooth system enable wireless EMG recording. Data is transmitted to Google Cloud for real-time applications. Cloud software hosts the machine learning algorithm for motion classifications. For Deep-Learning algorithm, CNN+LSTM model is used for motion classification, with optimized hyperparameters selected via random search. Performance is evaluated using accuracy metrics.

Strength Augmentation Demonstration:
Exoskeleton effectiveness is assessed by analyzing EMG signals, comparing average signal amplitudes with and without assistance. The impact of added weight on performance is also examined.

Results, Conclusions, and Discussions:: This study delivers the total integration of a motion-predicting cloud computing platform, a robotic exoskeleton for strength augmentation, and soft bioelectronics-enabled sensory feedback to develop the fully-intent driven robotic exoskeleton that assists the human upper-extremity joint movements. The PAM module can provide a maximum power source of 895 N of force and 87 mm of displacement that is translated to the exoskeleton frame such that the multiple upper-extremity joint movements can be augmented while retaining a lightweight structure. The skin-like bioelectronics enables sensory feedback by monitoring the EMG signals, which are processed through cloud computing to predict the intended movement of the user. The cloud-based deep-learning algorithm can classify four upper-extremity movements in total with a high test accuracy of 95.4% and 97.0% for biceps/triceps and medial deltoid/latissimus dorsi movements respectively. The overall response time for the movement assistance, which includes the EMG onset signal detection, cloud-based motion prediction, and actuation, consumes approximately 300-350 ms. Moreover, owing to the high force and displacement generation capability of PAM units, the intent-driven exoskeleton augments the upper-extremity strength substantially as it reduced the EMG activities by 6.9 and 3.4 times during elbow and shoulder flexion respectively. In addition, the exoskeleton also assisted the loaded movements by 1.7 and 2.1 times for elbow and shoulder flexion respectively, demonstrating that it can be used to carry or move moderately heavy objects in daily life. Until now, there has been no work that seamlessly integrated all the cutting-edge technologies into an intent-driven, fully working robotic exoskeleton that can be directly employed in real-life situations. In this context, we expect that the intent-driven exoskeleton in this study will not only make a significant contribution to next-generation robotics technology but it will also have a transformative impact on the lifestyle of individuals with neuromotor disorders.

Acknowledgements (Optional): : The authors acknowledge the support from the SEMI-FlexTech and the IEN Center for Human-Centric Interfaces and Engineering at Georgia Tech. This material is based on research sponsored by Army Research Laboratory under Agreement Number W911NF- 19-2-0345. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Army Research Laboratory (ARL) or the U.S. Government.

References (Optional): : 1. in Why stroke matters, Vol. 2022 Facts and Figures about Stroke (World Stroke Organization, 2023).
2. Li, S. Spasticity, motor recovery, and neural plasticity after stroke. Frontiers in neurology 8, 120 (2017).
3. Demaerschalk, B.M., Hwang, H.-M. & Leung, G. US cost burden of ischemic stroke: a systematic literature review. The American journal of managed care 16, 525-533 (2010).
4. Irshaidat, M., Soufian, M., Al-Ibadi, A. & Nefti-Meziani, S. in 2019 2nd IEEE international conference on soft robotics (RoboSoft) 630-635 (IEEE, 2019).
5. Ang, B.W. & Yeow, C.-H. Design and modeling of a high force soft actuator for assisted elbow flexion. IEEE Robotics and Automation Letters 5, 3731-3736 (2020).
6. Das, S. & Kurita, Y. ForceArm: a wearable pneumatic gel muscle (PGM)-based assistive suit for the upper limb. IEEE Transactions on Medical Robotics and Bionics 2, 269-281 (2020).
7. Lotti, N. et al. Adaptive model-based myoelectric control for a soft wearable arm exosuit: A new generation of wearable robot control. IEEE Robotics & Automation Magazine 27, 43-53 (2020).
8. Nassour, J., Zhao, G. & Grimmer, M. Soft pneumatic elbow exoskeleton reduces the muscle activity, metabolic cost and fatigue during holding and carrying of loads. Scientific Reports 11, 1-14 (2021).
9. Samper-Escudero, J.L., Coloma, S., Olivares-Mendez, M.A., González, M.Á.S.-U. & Ferre, M. A compact and portable exoskeleton for shoulder and elbow assistance for workers and prospective use in space. IEEE Transactions on Human-Machine Systems (2022).
10. Babič, J., Petrič, T., Peternel, L. & Morimoto, J. in 2017 IEEE International Conference on Robotics and Automation (ICRA) 5929-5934 (IEEE, 2017).
11. Samper-Escudero, J.L., Giménez-Fernandez, A., Sánchez-Urán, M.Á. & Ferre, M. A cable-driven exosuit for upper limb flexion based on fibres compliance. IEEE Access 8, 153297-153310 (2020).
12. Kim, Y.G. et al. A voice activated bi-articular exosuit for upper limb assistance during lifting tasks. Robotics and Computer-Integrated Manufacturing 66, 101995 (2020).
13. Proietti, T. et al. Sensing and control of a multi-joint soft wearable robot for upper-limb assistance and rehabilitation. IEEE Robotics and Automation Letters 6, 2381-2388 (2021).
14. Missiroli, F. et al. Rigid, soft, passive, and active: A hybrid occupational exoskeleton for bimanual multijoint assistance. IEEE Robotics and Automation Letters 7, 2557-2564 (2022).
15. Khan, A.M. et al. Estimation of desired motion intention and compliance control for upper limb assist exoskeleton. International Journal of Control, Automation, and Systems 15, 802 (2017).
16. Liu, H., Tao, J., Lyu, P. & Tian, F. Human-robot cooperative control based on sEMG for the upper limb exoskeleton robot. Robotics and Autonomous Systems 125, 103350 (2020).
17. McDonald, C.G., Dennis, T.A. & O'Malley, M.K. in 2017 international conference on rehabilitation robotics (ICORR) 164-169 (IEEE, 2017).
18. Zhou, Y. et al. in 2021 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP) 498-503 (IEEE, 2021).
19. Georgarakis, A.-M., Xiloyannis, M., Wolf, P. & Riener, R. A textile exomuscle that assists the shoulder during functional movements for everyday life. Nature Machine Intelligence 4, 574-582 (2022).
20. Shin, J.H. et al. Wearable EEG electronics for a Brain–AI Closed-Loop System to enhance autonomous machine decision-making. npj Flexible Electronics 6, 32 (2022).
21. Mahmood, M. et al. Fully portable and wireless universal brain–machine interfaces enabled by flexible scalp electronics and deep learning algorithm. Nature Machine Intelligence 1, 412-422 (2019).
22. Edelman, B.J. et al. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Science robotics 4, eaaw6844 (2019).
23. Dittmar, D., Dahlin, J., Persson, C. & Schuttelaar, M.-L. Allergic contact dermatitis caused by acrylic acid used in transcutaneous electrical nervous stimulation. Contact Dermatitis 77, 409-412 (2017).
24. Hansel, K. et al. Contact sensitivity to electrocardiogram electrodes due to acrylic acid: A rare cause of medical device allergy. Contact Dermatitis 82, 118-121 (2020).
25. Wu, H. et al. Materials, devices, and systems of on‐skin electrodes for electrophysiological monitoring and human–machine interfaces. Advanced Science 8, 2001938 (2021).
26. Kalita, B., Leonessa, A. & Dwivedy, S.K. in Actuators, Vol. 11 288 (MDPI, 2022).
27. Su, H. et al. in Actuators, Vol. 11 92 (MDPI, 2022).
28. Tiziani, L.O. & Hammond, F.L. Optical sensor-embedded pneumatic artificial muscle for position and force estimation. Soft robotics 7, 462-477 (2020).
29. Riemann, B.L. & Lephart, S.M. The sensorimotor system, part I: the physiologic basis of functional joint stability. Journal of athletic training 37, 71 (2002).
30. Ludewig, P.M. & Cook, T.M. Alterations in shoulder kinematics and associated muscle activity in people with symptoms of shoulder impingement. Physical therapy 80, 276-291 (2000).
31. Chow, D.H. et al. The effect of backpack load on the gait of normal adolescent girls. Ergonomics 48, 642-656 (2005).
32. Kim, H., Kim, E., Choi, C. & Yeo, W.-H. Advances in soft and dry electrodes for wearable health monitoring devices. Micromachines 13, 629 (2022).
33. Zhang, Y. et al. Mechanics of ultra-stretchable self-similar serpentine interconnects. Acta Materialia 61, 7816-7827 (2013).
34. Kwon, Y.-T. et al. All-printed nanomembrane wireless bioelectronics using a biocompatible solderable graphene for multimodal human-machine interfaces. Nature communications 11, 3450 (2020).
35. Kwon, Y.-T. et al. Printed, wireless, soft bioelectronics and deep learning algorithm for smart human–machine interfaces. ACS Applied Materials & Interfaces 12, 49398-49406 (2020).
36. Sabir, S.U.B., Ahmed, K., Sabir, U. & Naseer, N. in 2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS) 1-6 (IEEE, 2021).
37. Wu, Q., Chen, B. & Wu, H. Neural-network-enhanced torque estimation control of a soft wearable exoskeleton for elbow assistance. Mechatronics 63, 102279 (2019).
38. Dindorf, R. & Wos, P. Using the bioelectric signals to control of wearable orthosis of the elbow joint with bi-muscular pneumatic servo-drive. Robotica 38, 804-818 (2020).