MODALITY HALLUCINATION FOR MULTI MODAL APPROACHES TO VISUAL INFERENCE

Abstract

The aim is to address the problem of exploiting multiple sources of information for object classi cation tasks when additional modali- ties that are present in the labeled training set are not available for inference. Considering the practicality of RGB-D object classi er, a modality hallucination architecture using multi-modal ConvNets has been proposed to incorporate depth information at training time. The modality hallucination network is trained to mimic mid-level features of depth images and learns a new RGB image representa- tion . The single modality RGB test image is jointly processed using hallucination and RGB network and it outperforms the RGB model. As the deep networks based object classi ers require prohibitive run- times to process images for real world applications, knowledge distil- lation framework has been proposed for the modality hallucination architecture with improved accuracy