SIGN LANGUAGE RECOGNITION USING MULTI-MODEL ENSEMBLE

Abstract

Communication is very crucial to human beings, as it enables us to express ourselves. We communicate through speech, gesturing, body movement language, read, write or visuals, speech being one of the most commonly used among them. But for the hearing impaired people this is not an option. Sign language is done by doing hand and facial expressions to express various information which the person wants to communicate with the other person. Majority of the common people who are not verbally or hearing impaired will not be able to get what those signs mean because no one does really learn it.There is a massive gap between the community because of this and no one really gives it much attention. With the advancement of deep learning and computer vision we are able to built an approach that can help in the interpretation of these gesture into simple human understandable text. It help normal communication between the normal and disabled people. In this report we present a method for word-level American sign language recognition using Multi model ensemble on various modalities using different models. Multi model ensemble has recently seen alot of utilization in vision community since it utilises properties of various different mortalities and strength of different models. We have also improved a 3DCNN network with attention.