Using Information Theoretic Concepts to Understand Autoencoders

Abstract

Deep neural networks currently require theory - building approaches for analysis, despite their extensive usage in practical uses. We present a new information-theoretic model to understand learning dynamics and building autoencoders, a type of deep learning architecture that resembles a communication channel. " By generalizing the information plane to any cost function and analyzing the roles and dynamics of various levels using layer-wise information quantity, we emphasize the importance of mutual information in measuring knowledge from data. " Using the data processing inequality, we also suggest and scientifically illustrate three essential hypotheses relating to the layer-wise flow of information and intrinsic dimensionality of the bottleneck layer for mean square error training and the recognition of a data-controlled bifurcation point in the information plane. Our findings have a direct influence on the best design of autoencoders, alternative feedforward training approaches, and even the generalization problem.