Describing Environmental Information in Videos Using Machine Learning

The previous researches of video captioning task have focused on human actions or objects in videos, however, environmental information such as place, time, weather among others is also important information to understand videos. Therefore, in this paper, we create a new dataset which adds environmental information labels to MSVD dataset and train the machine learning model to analyze environmental information from videos. We apply R(2+1)D which is a 3D CNN model to extract video features and S2VT which is a RNN model to encode the video features and to decode the environmental information. The reason why we define the problem as a sequence to sequence problem, not multilabel classification, is that the input is a video, which is a sequence of frames, and the output is also related with each other. For example, if the place label is outside, then next label would be weather. We analyze the experimental results based on BLEU, METEOR, ROUGE-L, and CIDEr and it shows the competitive results compared to the state-of-the-art video captioning model.