(K-409) Surface Electromyography based Game Controller with Machine Learning Assistant

Surface electromyography (sEMG) has been incorporated as a neuro-rehabilitation technique to train motor functionalities in children with cerebral palsy (CP) [1]. It has been reported that computer games would be able to extend attention and training period for physical therapy, which is beneficial for children with CP. However, although the previous projects demonstrated applicability of integrating a sEMG-based game controller, operators had a hard time obtaining the control-reference levels of the EMG signal as multiple buttons of a game controller [2]. Series of sEMG data sets were difficult to compare to one another because there was high variability between individual measurements. Data measurements taken from the same participants, and even same target muscles, could differ drastically when taken even just minutes apart. Data collection is affected by factors such as skin texture, muscle fatigue, and shifting electrodes [3]. This exploration project incorporated the current trend of Machine Learning (ML) technologies to overcome this unique biological signal dilemma. The aim was to enhance sEMG analysis techniques to produce a novel therapy for muscle training by connecting muscle activation to a video game.

For the ML training, as well to minimize the subject variabilities, training reference data was generated using the LabVIEW (National Instruments, Austin, TX) Biomedical Toolkit. This data was fed though a ML model created using TensorFlow (Google Brain Team) libraries. The model consisted of three fully connected layers. The developed model was able to detect a strong, weak, or no contraction conditions when simulated sEMG signals were inputted into the model. The ML model will then be uploaded to an Arduino Nano 33 BLE Sense using the TensorFlow Lite Micro library to allow the game controller to operate as an independent embedded system.

Furthermore, another validation program was developed to translate muscle activation to button-press operations. sEMG data was collected using the MyoWare sEMG sensors (Advancer Technologies). An Arduino Leonardo was used to translate a positive muscle contraction signal to a game control input, an example being a right arrow key press on a keyboard. The Arduino Nano 33 BLE and Leonard were integrated into a unified system, so that when prompted by the Arduino Nano 33 BLE running the machine learning model, the Arduino Leonardo prompted a computer running a video game to hit the right arrow key. The process flow of the system can be seen in Figure 1. 

Results: LabVIEW was used to generated a total of 3,000 sEMG reference signals for each condition. The conditions were strong, weak, or no muscle contractions. These simulated training data sets were fed into the ML model to train it to classify the different types of muscle contraction. At the present time, the ML model classified the simulated signals with 62.5% accuracy between the strong, weak, and no contraction conditions. Additionally, the analog input capability of Arduino Leonard and MyoWare sEMG sensors were connected, and sample sEMG data was accurately obtained from a living subject. The sEMG translation into key board operation on Arduino Leonardo was developed with a human interface device (HID) library. It was verified that Leonard was able to control a video game running on a computer based on muscle contraction. This was evaluated by setting a threshold value for what constitutes muscle activation. The ML model was developed in a PC environment, and it will be uploaded to the Arduino Nano 33 BLE in the next phase of development using TensorFlow Lite Micro to accommodate the memory limitation of the Arduino Nano 33 BLE.

Conclusion: This exploration project identified applicability of ML models in a sEMG-based game controller.  ML has a huge potential to distinguish unique biological signals. The next phase of development will include uploading the model to the Arduino Nano 33 BLE and editing the parameters of the model to improve its detection accuracy.

Discussions: This project was expected to support people suffering from muscular disorders whose sEMG signals may be weaker due to their disorder. This study validated the usefulness of ML in a biological signal-based game controller. This project will have the potential to support many users, though it involved numerous processes to make completely functional. Additionally, this study incorporated several cost-effective components, which promise to reduce health care costs. 

[1] Yoo JW, Lee DR, Cha YJ, You SH. Augmented effects of EMG biofeedback interfaced with virtual reality on neuromuscular control and movement coordination during reaching in children with cerebral palsy. NeuroRehabilitation. 2017;40(2):175-185. doi: 10.3233/NRE-161402. PMID: 28222541.

[2] Legato B, Arico M, Asaki T. Development of Universal sEMG-based Human Interface System. 2020 BMES Annual Meeting.

[3] Raez MB, Hussain MS, Mohd-Yasin F. Techniques of EMG signal analysis: detection, processing, classification and applications. Biol Proced Online. 2006;8:11-35. doi: 10.1251/bpo115. Epub 2006 Mar 23. Erratum in: Biol Proced Online. 2006;8:163. PMID: 16799694; PMCID: PMC1455479.