WORD EMBEDDING IMPROVEMENTS IN NATURAL LANGUAGE PROCESSING

Abstract

The growing intricacy of problems related to natural language processing has brought to light the shortcomings of static word embeddings in terms of encapsulating sentence context. Transformer-based models that use self-attention processes have been used to solve this, however they have problems because of BERT's tokenizers word piece algorithm, which divide words into sub-words and may lead to problems with contextual meaning understanding. By improving the tokenizer, this research aims to improve vector embeddings of words. The tokenizer was altered with domain-specific terms or tokens, and the model was trained on a dataset that contained these tokens. According to our research, the prepared words' cosine similarity has improved, suggesting improved contextual representation. These stretching of word embeddings imply that certain words have better embedding quality than previously existing models, with some words having more accurate vector space representations. Furthermore, our methodology emphasizes the significance of tailoring tokenization techniques to particular domains, which may result in more accurate language models dynamic embeddings. This research shows the problem of existing tokenizer which are word piece based algorithm. By adding the special tokens into the tokenizers vocabulary and it shows the significant difference of dynamic embeddings generated by the BERT model.