SIGN LANGUAGE RECOGNITION

Abstract

Sign Language Recognition using machine learning is an image classification project that has an important social motive behind it. The problem statement of this project can be summarized as a problem of image and video classification using deep learning algorithms. The things that make this problem challenging is to use easily accessible hardware such as simple web cams or smartphone cameras instead of advanced devices such as kinect. This eliminates the element of depth from the data, and compromises the accuracy of the classification task, but this trade-off is necessary in order to make this project usable and scalable. Other challenges include getting good quality labeled data. The data of ISL gestures consists of videos of dynamic hand signs which involves use of one or both hands depending upon gesture complexity. For the CNN-LSTM and CNN-MLP models features were extracted from the video frames using transfer learning on the Inception V3 model with preloaded ImageNet weights. The models trained displayed good accuracy as well as the demo developed is capable of running in a real time with reasonably good prediction accuracy. The technologies behind this project is computer vision and deep learning. We used library OpenCV with python that gives the capability to work with images and Keras with Tensorflow for building models. This project has lot of social and marketable potential. This project was extended using recurrent convolutional network models to identify phrases and words. With the help of existing APIs provided by google or other companies for Natural Language Processing, this project can be developed into full-fledged translator, a product which will have a very low cost of scalability and can affect lives of many people.