AUDIO-VISUAL ANOMALY DETECTION AND DEEPFAKE DETECTION

Abstract

Considering rapidly increasing threats in digital multimedia, such as face change, audio manipulation or audio generation, and fake video generation, digital forensics has an important role in identifying any manipulation in digital data using forensics techniques. More specific anomalies include manipulating video data by altering the duration of the video, swapping faces in the video, changing the audio of identity in the video, and so on. Various detection techniques and algorithms based on image processing and audio processing were implemented separately to identify the anomaly. A lot of ideas were implemented using various deep learning algorithms, which involve the use of Convolution Neural Networks (CNNs), Artificial Neural Networks (ANNs), Residual Neural Networks (RNNs), etc. This thesis adopts the technique used to detect zero-day anomalies on video using unsupervised learning and deepfake detection using supervised learning using off-the-shelf encoders. In anomaly detection, the synchronization model is trained on real data for synchronization scores between video frames and audio segments. Using these features from the synchronization model and then using the decoder-based autoregression model, the feature of the next frame and audio is predicted. In this thesis, experiments are performed on the dataset to check the robustness of the model. Further in the thesis, analysis of deepfake detection is done using supervised learning by feeding video and audio inputs to an off-the-shelf encoder (e.g. BERT). The similarity between each video frame and the corresponding audio is found using a similarity function. The frames with a low similarity score are flagged as fake. Here, the performance of both the anomaly detection and deepfake detection models is compared with other supervised and self-supervised models in terms of accuracy. The comparison metrics used are Average Precision (AP) and Area Under the Curve (AUC). The anomaly detection model has high accuracy for unseen data of around AP of 90% and AUC of 85% for fake data and can detect zero-day anomalies.