(D-149) First Experiences Integrating Functional Near-infrared Spectroscopy (fNIRS) and Virtual Reality

Integrating Virtual Reality (VR) with brain imaging provides opportunities to collect data of cognitive tasks that more accurately represent the ‘real world’ than what can traditionally be collected in the lab. While neuroimaging studies testing simple cognitive tasks provide important data in advancing the field, they are limited in their scope of providing information about cognition in real-world events. Data collected in a lab benefit from the controlled environment but lacks the context of real-world elements that factor into cognitive tasks. Integrating VR into neuroimaging studies allows researchers to collect data from subjects immersed in environments that more accurately mimic real world cognitive decision-making whilst still being able to provide reliable data from a controlled environment. VR has had an increasing role in clinical and cognitive research. Expanding research done with VR into neuroimaging allows researchers to better understand brain activity during tasks proven to have clinical significance.  

Functional Near-Infrared Spectroscopy (fNIRS) is a non-invasive method of collecting brain activity based on changes in the local concentrations of oxygenated hemoglobin (oxy-Hb) and deoxygenated hemoglobin (deoxy-Hb) in the brain during a task. Oxy-Hb and deoxy-Hb scatter and absorb near-infrared light at different wavelengths. fNIRS uses light sources and detectors to measure these optical changes and convert them to changes in brain activity (hemodynamic response) via modified Beer-Lambert law. fNIRS is portable and not as sensitive to movement as other methods of brain imaging which makes it a desirable method for integrating with VR.  

fNIRS 

To collect brain activity during the task, we used the NIRX NIRSPORT2. 42 channels were distributed across bilateral frontal and sensorimotor brain regions. Fig. 1 shows the sensitivity of the probes overlying Brodmann areas. During brain activity, there is a change in concentrations hemoglobin optically measured by the system. In this experiment, we used the wearable fNIRS system. The participant wore the Oculus Meta Quest 2 to run the VR program (Fig. 2). The headset was designed to integrate with the Oculus headset to limit mechanical and optical interference.  

Box Task 

We built a program in Unreal Engine to perform a spatial working memory box task where the user was placed in a room with 16 boxes (Fig. 3). The user initially started in a survey period where they clicked on every box to find the 3 reward boxes. When pressed, a box would either turn green to indicate it was a reward or turn red to indicate it was not a reward. The user would then go through a 30 second baseline period where the boxes became invisible followed by a retrieval period where they are spawned at a random location in the room, all the box's colors are reset to the default, and they are given 30 seconds to find the 3 reward boxes. In a level, the participant goes through a survey period and then alternating rest and retrieval periods for five trials. We recorded the participant’s brain activity for two levels. 

Results 

We recorded fNIRS data during the VR task with n=1 to find preliminary results. Brain activity changes were estimated between the Survey and the Retrieval periods compared to the baseline. The fNIRS results are shown in Fig. 4. This figure showing the t-value (p-value [corrected] < 0.05) for oxy- and deoxyhemoglobin changes.  

Discussion 

The goal of this project was to investigate the feasibility of recording brain activity during a VR task. Combining fNIRS with VR required balancing several competing design factors related to both the physical hardware of the devices and the experimental design requirements needed to collect and analyze fNIRS data.  

We found that users experienced motion sickness or dizziness. This was likely made worse by discomfort of the fNIRS head cap. This led us to modify the VR game to reduce character speed and motion controls. However, this limited the number of trial and level repetitions that could be comfortably performed by a participant. Frequent, shorter duration trials are generally used to estimate reliable brain activity, but we found fewer, longer duration tasks were better tolerated.  

Another challenge was the familiarity of the participant with the Oculus’ user interface. The participant often had to be talked through how to set up the Oculus’ physical scene boundaries and reestablish the connection allowing the experimenter to co-view the VR. We found that blocking the light sensor used by the device to detect when it is “off subject” helped but was still imperfect.  

Finally, synchronizing the timing of the VR and fNIRS had to be worked out. Since the brain’s hemodynamic response takes about 8-12seconds after the onset of a task to peak, fNIRS is less sensitive to timing errors compared to other methods such as EEG. We manually marked event times by watching the screen cast along with logged response information. In the future, a UDP/IP protocol (Lab Streaming Layer) can be used to reduce timing errors.  

Conclusions 

While integrating VR with neuroimaging presents challenges with experimental design, we have found ways to overcome such challenges. Integrating fNIRS and Virtual Reality shows promise for advancing the field of neuroimaging.