EduSign: A Multi-Modal Deep Learning Model for Educational Sign Language Recognition

Abstract

Sign language recognition (SLR) systems aim to improve accessibility to deaf and hard-of-hearing people. The existing SLR systems fail to meet educational needs because they only support common vocabulary. The paper proposes EduSign as a solution to existing problems through its multi-modal deep learning system which uses spatial landmark data and temporal motion data to recognize educational sign language. The system uses MediaPipe to extract hand and upper body and facial landmarks which combine with dense optical flow to create a complete educational sign language system that shows both structural and dynamic features. The system processes fused features through a lightweight hybrid architecture which includes multi-scale temporal convolutions and squeeze-and-excitation attention and hierarchical residual learning and multi-head temporal attention. The assessment of the framework proceeds through testing on a new educational Kannada Sign Language (KSL) dataset which contains ten academic signs used in classroom settings. The experimental results show that the training achieved an accuracy of 96.04% while the validation accuracy reached 94.57% with only a 1.47% generalization gap. Additional comparative and ablation studies illustrate the efficiency of multimodal fusion and attention mechanism. The results suggest that joint use of spatial representation based on landmark and temporal modeling based on motion presents a good framework for educational SLR within the intelligent learning system.