Open Access Open Access  Restricted Access Subscription Access

The Language of Faces: From Gestures to Text

K. Sabitha, R. Pradeeswari, G. Rathimalar, A. Thenmozhi, J.Varshini Varshini

Abstract


With the rapid growth of artificial intelligence and computer vision, technology is now playing a key role in bridging communication barriers for the deaf and hard-of-hearing community. Indian Sign Language (ISL) relies on a combination of hand gestures and subtle non-manual cues such as facial expressions, head orientation, and body posture to convey meaning and emotion. However, most existing translation systems focus only on hand movements, leading to partial or unclear interpretations. To address this limitation, the proposed system integrates Mediapipe Face Mesh for detecting facial landmarks along with Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models for visual feature extraction and sequential pattern recognition. By combining these techniques, the system efficiently converts both manual and non-manual features of ISL into accurate and context-aware text. This work enhances the reliability and naturalness of sign language translation, promoting better digital accessibility and inclusivity for individuals with hearing impairments. 

Full Text:

PDF

References


Al Abdullah, Bashaer A., Ghada A. Amoudi, and Hanan S. Alghamdi. "Advancements in sign language recognition: A comprehensive review and future prospects." IEEE Access 12 (2024): 128871-128895.

Baihan, Abdullah, et al. "Sign language recognition using modified deep learning network and hybrid optimization: a hybrid optimizer (HO) based optimized CNNSa-LSTM approach." Scientific

Reports 14.1 (2024): 26111.

Joshi, Abhinav, et al. "isign: A benchmark for Indian sign language processing." arXiv preprint arXiv: 2407.05404 (2024).

Khartheesvar, G., et al. "Automatic Indian sign language recognition using MediaPipe holistic and LSTM network." Multimedia Tools and Applications 83.20 (2024): 58329-58348.

Kumar, Malay, et al. "Enhanced Sign Language Translation between American Sign Language (ASL) and Indian Sign Language (ISL) Using LLMs." arXiv preprint arXiv:2411.12685 (2024).

Lahari, V. Rama, et al. "Sign Language Classification Using Deep Learning Convolution Neural Networks Algorithm." Journal of the Institution of Engineers (India): Series B 105.5 (2024): 1347-1355. Easa S. Journal of Surveying Engineering. 1989, 115(3), 356359p.

Miah, Abu Saleh Musa, et al. "Sign language recognition using graph and general deep neural network based on large scale dataset." IEEE Access 12 (2024): 34553-34569.

Patra, Suvajit, et al. "Hierarchical windowed graph attention network and a large scale dataset for isolated indian sign language recognition." arXiv preprint arXiv:2407-14224 (2024).

Radhika, K. "Long Short-Term Memory Models and Mediapipe Based Framework for Indian Sign Language Translator." Bio-Inspired Computing: Proceedings of the 14th International Conference on Innovations in Bio-Inspired Computing and Applications (IBICA 2023) held during December 14-15, 2023, Volume 3: Information Retrieval and Applications. Vol. 1230. Springer Nature, 2025.

Ramkarthik, D., and A. Benita. "Integrating quantum networking with explainable AI and ensemble learning approaches for enhanced sign language recognition: Indian Sign language (ISL), convolutional neural networks (CNNs)." Multidisciplinary Applications of AI and Quantum Networking. IGI Global, 2025.

-226.


Refbacks

  • There are currently no refbacks.