A FRAMEWORK FOR DISTRIBUTED LEARNING OF CONVOLUTIONAL NEURAL NETWORKS

Abstract

The task of object identification for large scale Image classification has seen rapid advances in the last few years due to advances in architecture of convolutional networks for automated -' feature learning. There is increased interest since these networks have been proven to outperform hand engineered features for the task of Image classification when the number of classes is very large. As the complexity of models increases along with the sizes of datasets, significant development efforts and resources are required to train networks of the scale of billions of parameters. With increasing efforts being expended on multi-GPU training, spreading out the computation across multiple OPUs in cluster requires considerable effort in division of work across the available resources. In this dissertation, we have designed and developed a framework for distributed computation of convolutional neural networks over multiple OPUs, which aims to separate the network definition and the details of distribution from the code that is to be run on the system. The framework is designed to be extensible, allowing for future additions of computation layers- that are the building blocks of neural networks, and the update rules- that determine the network convergence behavior. Further, the support of distributed training allows one to leverage more resources for faster training or for training larger networks. The framework supports this distribution by providing a distributed parameter store for parameter sharing between models, and shared memory abstraction for data sharing between processes, using a combination of which allows for both model and data parallelism as we demonstrate. We discuss the architecture of the framework we built for this purpose, along with its components and the network definition format, demonstrating both data parallel and model parallel execution of computation for a sample network.