Icon Legend

This session is not in your schedule.

This session is in your schedule. Click again to remove it.

Back

Device Technologies and Biomedical Robotics

Integrating Vibrational Haptic Feedback on Sonomyography Hand Configuration Estimation

(K-410) Integrating Vibrational Haptic Feedback on Sonomyography Hand Configuration Estimation

Saturday, October 14, 2023

9:30 AM – 10:30 AM PDT

Location: Exhibit Hall - Row K - Poster # 410

Presenting Author(s)

Joseph Garza (he/him/his)

Biomedical Engineering Student
University of Texas at San Antonio
Converse, Texas, United States

Co-Author(s)

KB

Keshav Bimbraw

Research Assistant
Worcester Polytechnic Institute, United States

Last Author(s)

YT

Yichuan Tang

Research Assistant
Worcester Polytechnic Institute, United States

Primary Investigator(s)

hZ

haichong K. Zhang

Assistant Professor
Worcester Polytechnic Institute, United States

Introduction:: The absence of live one-way feedback in an HCI (Human Computer Interface) utilizing a Linear SVC (Support Vector Classifier) causes users to make preventable mistakes. Addressing this limitation represents a significant advancement in controlling prehensile triple-appendage robotic hands. This research explores using hand configuration estimation with a SVC HCI to relay adaptive haptic feedback through basic classification comparisons; offering a potential solution to aid preventing user error when using HCIs.

Materials and Methods:: The Python based HCI, a mode of physical hardware communication between a device and a human, consists of a Linear SVC, [1] a machine that recognizes and then separates images based on visual characteristics, model generator program created by Keshav Bimbraw [1-2] and a SPC (Serial Port Communicator) that enables data exchange with an Arduino, a programmable hardware controller. The Arduino consists of a byte variable receiver and a haptic module relay task handler. The Linear SVC recognizes images from an onscreen SMG (sonomyography) live-feed of an SMG image dataset collected from a forearm-mounted ultrasound probe into one of three classes: ‘thumb’, ‘index’, and ‘middle’ digits of the hand. These classifications are passed to the SPC program as a byte to the haptic module; dependent on the forearm musculature image classification related to finger actuation. If the user actuates the incorrect finger a vibrational pulse will be sent to the finger to reverse actuation.

Results, Conclusions, and Discussions:: The SVC was trained on an image dataset at a 20% test to 80% train split. The images were collected at a resolution of 627 x 500 pixels and the forearm was kept still with slight variation in forearm orientation. The latencies were detected using the built-in time.time() function within the time library of Python for each significant task executed by the SPC program amounting to a global average latency of 8.633 seconds. The best performing Linear SVC model predicted SMG image classifications correctly at an average of 98% accuracy on 3000 images per class when the model was tested in three series of one hundred trials.

Acknowledgements (Optional): : Thank you, Shang Gao and Xihan Ma for providing essential insight on my personal professional development from current industry knowledge during this REU. This work was supported by NSF REU grant EEC2150076.

References (Optional): : [1] Bimbraw, K., et al. 2020. Towards Sonomyography-Based Real-Time Control of Powered Prosthesis Grasp Synergies. IEEE EMBC,4753-4757. [2] Simultaneous Estimation of Hand Configurations and Finger Joint Angles Using Forearm Ultrasound. IEEE TMRB, 5(1), 120-132. [3] Gaba, Jacob A. 2016. Air Keyboard: Mid-Air Text Input Using Wearable EMG Sensors and a Predictive Text Model. DCU Theses. 114. [4] Tripathi, A., et al. 2023. SurfMyoAiR: A Surface Electromyography-Based Framework for Airwriting Recognition IEEE TIM, 72, 1-12.