Sign language translator using LSTM and Generative Adverserial Networks: A Review

Abstract

Sign language is an essential mode of communication for the deaf and hard-of-hearing community. However, the communication gap between sign language users and non-signers persists, hindering ef- fective interaction and inclusivity. In recent years, advancements in deep learning techniques, particularly Long Short-Term Memory (LSTM) networks and generative artificial intelligence (AI), have shown promise in bridging this gap by enabling the development of sign language translation systems.

Generative Adversarial Networks (GANs) with LSTM-based archi- tectures have shown promise in this domain. The GAN model con- sists of a generator and discriminator network working in an ad- versarial manner. The generator generates synthetic sign language videos, while the discriminator determines the authenticity of the generated videos. The GAN is trained using a dataset of real sign language videos paired with text annotations. The generator learns to generate realistic sign language videos by capturing the under- lying patterns and characteristics of the training data. The discrim- inator learns to classify the generated videos as real or fake. Once trained, the generator’s output videos can

be further processed us- ing computer vision techniques to extract visual features. These features are then used as input to an LSTM translational model that maps them to corresponding textual translations. The LSTM model learns the relationship between the visual features and the associ- ated text annotations. The GAN with LSTM generator and discrim- inator provides a framework for sign language translation, combin-ing the benefits of generative modeling and sequential processing. Further research and refinement of these models are needed to im- prove the accuracy and effectiveness of sign language translation systems.

To test the effectiveness of our method, we ran experiments us- ing a variety of datasets of sign language videos and compared the outcomes with those from conventional LSTM-based methods. The outcomes of our study show that the GAN-based system out- performs traditional LSTM-based methods. The generated text has better fluency and semantic coherence, closely matching the sign language gestures. These outcomes demonstrate how our system can aid in effective communication between sign language users and non-sign language users. The system can support a wide range of sign language expressions thanks to the GAN-based approach’s flexibility in handling variations in signing styles and gestures. In a number of areas, including education, communication platforms, and accessibility tools, this technology has the potential to improve communication accessibility.