(I-343) Can Deep Convolutional Neural Network Associate Emotions with Gabor Patches?

Introduction: Deep convolutional neural networks (DCNNs) provide reasonably good models of the human ventral visual system. Efforts are currently underway to extend these models to include additional modules so that more advanced cognitive functions such as learning, memory, and emotion can be studied computationally. In this work, we first developed a deep convolutional neural network (DCNN) model that can recognize the motivational significance of natural images containing affective scenes. Then, we extended the model to the domain of associative learning, where we paired neutral stimuli (Gabor patches with different orientations) with both positive- and negative-valenced natural images and tested whether the model can be trained to assign emotional significance to the neutral stimuli through appetitive and aversive conditioning.


Materials and Methods: The VGG-16 network, a popular CNN model originally designed for object classification tasks, was augmented by adding (1) an emotion recognition module that simulates the function of the amygdala and other emotion-processing prefrontal structures such as the orbitofrontal cortex and (2) a rapid information transfer pathway that links the early visual areas (i.e., early layers of VGG-16) to the emotion recognition module. The resulting DCNN model was trained on 2 different datasets of emotional images to recognize the valence of the images. We then paired two Gabor patches (conditioned stimuli or CS) with appetitive and aversive images from the International Affective Picture System (IAPS), respectively, which served as unconditional stimuli or US. The learning objective of the DCNN model was to minimize the difference between the valence of the IAPS images and the DCNN-derived valence from the CS-US compound stimuli.

Results, Conclusions, and Discussions:

We found that (1) the conditioned DCNN can assign emotional significance to the CS which were initially neutral prior to learning, (2) the learning effects generalize to neighboring Gabor patches which the DCNN has not seen during training, and (3) associative learning takes place in the deeper layers of DCNN, consistent with the idea that associative learning process is a high-order cognitive process.



These results show that DCNN is capable of associative learning, which hitherto has not been demonstrated in deep neural networks, and suggests that DCNN can be extended to model more advanced cognitive functions such as associative learning.


Acknowledgements (Optional): :

References (Optional): :