Prediction of Visual Memorability with EEG Signals using Deep Neural Networks

Visual memorability is a method to measure how easily media contents can be memorized after glancing them. To predict the visual memorability of media contents such as images and videos has recently become more important because it can affect the design principles for visualization, advertisement, and education using multimedia contents. Previous studies on prediction of the visual memorability have generally exploited visual features (e.g., color intensity, contrast) or semantic information (e.g., class labels) that can be obtained from multimedia contents. Compared to previous work, we propose a novel approach to predict the visual memorability of multimedia contents by exploiting human biological feedback. In this study, electroencephalography (EEG) signals are recorded from subjects as human biological feedback during a visual memory task. The collected EEG signals are then used to train a deep convolutional neural network (CNN) for prediction of the visual memorability. In our experiment, 21 subjects were recruited to conduct a visual memory task where each subject is asked to answer if he/she correctly remember a particular image 30 minutes after glancing at a set of images sampled from LaMem dataset. Finally, the performance of deep CNNs trained with EEG signals were compared with classical methods such as SVM, LDA, and k-NN algorithms. From the experimental results, we found that 1) a deep CNN-based framework outperforms classical machine learning methods and 2) the performance of the EEG-based framework is comparable to the previous studies which exploit visual features.