AUTONOMOUS VIDEO SKIMMING: TAXONOMY, FRAMEWORKS AND QUANTITATIVE EVALUATION

Abstract

Video skimming is the process of generating an abridged version (video summary) of a given user video by including only the significant contents of the video. Such a video summary helps in conveying the intended message of the video in a significantly shorter time. This thesis contributes in the development of various components of an autonomous video skimming system, focusing on the taxonomy to organize the existing literature on video skimming; different novel frameworks to perform video skimming through importance ranking; and new quantitative measures to evaluate and carry out comparative analysis. Firstly, a comprehensive literature survey is performed using a novel taxonomy for categorizing the existing literature on user video skimming. User videos are unstructured videos that capture interesting events in day-to-day activities. The review elicits the research area’s evolution demonstrating the key developments, and gives insight into possible future directions. Further, it brings out the typical video skimming framework generally preferred by various video skimming algorithms, processing paradigms, the variety of evaluation strategies, and the datasets in vogue. Secondly, a video segment ranking algorithm is proposed for user video skimming that works by considering the relationship between a representative video segment and all other video segments, where the concept of R-ordering of vectors is employed to find the representative video segment. The video segments are grouped based on the number of components in their feature vector greater /smaller than the representative feature vector. This feature component based comparison and grouping preserves their individual contributions during the ranking process. This ranking process is further enhanced by designing the provision to allocate different priorities to the feature components using linear and Gaussian process regression. The priorities are learned using a set of frames and their corresponding importance scores as indicated in the ground truth summaries. Thirdly, a fusion framework is provided for generating user video summaries utilizing different modalities like audio and video. Our ranking algorithm is used to order the video segments according to their importance derived from the chosen audio and visual features separately. A strategy is then proposed to combine these rankings corresponding to all the modalities. Finally, a novel measure to evaluate a dynamic video summary against multiple reference summaries is proposed. To this end, concepts of granular computing are leveraged to theoretically deduce the measure, which captures the inherent (dis)agreement between the multiple references and the resulting clustering tendency. Along with a few attributes of the proposed evaluation measure, it is theoretically shown that the popularly used average F-measure is a special case of the novel measure, which is referred to as the granule-aware F-measure.