MULTIMODAL ANALYSIS OF VIDEO INTERACTIONS

Abstract

Emotion recognition is an important part of both human-to-human interactions and humancomputer interactions. Recent advancements in this field aim to make the technology emotionally intelligent, much like humans. A key element of this process is the automatic recognition of emotions. Automatic emotion recognition can be approached in two ways: individual and group emotions. While there is a lot of research on the individual aspect, there is limited research on recognizing emotions across groups. This study focuses on understanding how emotions are perceived within a group context. To address the problem, we have proposed a multimodal decision fusion technique. Multimodal features include audio signals, scene context, pose and video. We have experimented with various feature combinations to enhance sentiment classification accuracy. Advanced tools have been used for feature extraction, like YOLOv8 for pose and object and TimeSformer for video features. Various fusion techniques in ensemble learning were explored to combine features and understand group emotion contexts. Our experiments, conducted on the VGAF dataset, revealed several key findings. The Multimodal approach, which combines different data modalities, outperforms the unimodal approach. Our proposed method showed better results than existing methods on the VGAF dataset. Modalities and their corresponding emotions exhibit a significant correlation.