Search

Isolated dynamic sign language recognition (IDSLR) has the potential to change accessibility and inclusion by enabling speech and/or hearing-impaired people to engage more completely in a variety of spheres of life, including social interactions, work, and more. IDSLR is a challenging task due to considering a sequence of image frame analysis with multiple linguistic features for a single gesture in cluttered backgrounds and an illumination variation environment. We have proposed a Hybrid Efficient Convolution (HEC) model that ensembles EfficientNet-B3 and a few modified layers as an alternative to traditional machine learning techniques with improved performances in cluttered backgrounds with illumination variation environments. The architecture of the HCE integrates pre-trained layers of EfficientNet-B3 loaded with customized weights and a new custom dense layer featuring 256 units, followed by batch normalization, dropout, and the final output layer. To enhance the robustness of the system, we employed the augmentation technique during pre-processing. Then, the system executes channel-wise feature transformation through point-wise convolution that reduces the computational complexity and increases the accuracy. The updated dense layer with 256 units processes the output from the standard EfficientNet-B3, shaping the model into a hybrid form to achieve better performance. We have created our own gesture dataset, called "BdSL_OPA_23_GESTURES," which consists of 6000 video clips of 100 isolated dynamic Bangla Sign Language words, with 60 videos for each word from 20 different people in the cluttered background with illumination variation environments to train and evaluate the performances of the proposed model. We have considered 80% of the total dataset for training purpose, while the remaining 20% is dedicated to testing and validation. In a small number of epochs, our proposed HEC model achieves a superior accuracy of 93.17% on our created "BdSL_OPA_23_GESTURES" dataset. All the information of the proposed model with the dataset has been shared along with the scientific community to provide access publicly at: https://github.com/Prothoma2001/Bangla-Continuous-Sign-Language-Recognition.git. [ABSTRACT FROM AUTHOR]

Sign Language Recognition (SLR) is crucial for enabling communication between the deaf-mute and hearing communities. Nevertheless, the development of a comprehensive sign language dataset is a challenging task due to the complexity and variations in hand gestures. This challenge is particularly evident in the case of Bangla Sign Language (BdSL), where the limited availability of depth datasets impedes accurate recognition. To address this issue, we propose BdSL47, an open-access depth dataset for 47 one-handed static signs (10 digits, from ০ to ৯; and 37 letters, from অ to ँ) of BdSL. The dataset was created using the MediaPipe framework for extracting depth information. To classify the signs, we developed an Artificial Neural Network (ANN) model with a 63-node input layer, a 47-node output layer, and 4 hidden layers that included dropout in the last two hidden layers, an Adam optimizer, and a ReLU activation function. Based on the selected hyperparameters, the proposed ANN model effectively learns the spatial relationships and patterns from the depth-based gestural input features and gives an F1 score of 97.84 %, indicating the effectiveness of the approach compared to the baselines provided. The availability of BdSL47 as a comprehensive dataset can have an impact on improving the accuracy of SLR for BdSL using more advanced deep-learning models.

Sign language, often referred to as silent conversation, serves as a visual gesture-based primary communication medium for hearing-impaired individuals. People unable to speak or hear communicate with themselves and general persons with static and dynamic sign languages. To ensure that people who are deaf and unable to talk do not feel excluded from this fast-paced world, it is even more crucial that sign language is incorporated into our culture and that a cost-effective and straightforward method of detecting it be developed. With this goal in mind, an automatic Bangla sign language (BSL) detection system has been developed in this work using deep learning approaches and a Jetson Nano edge device. The deep learning models have been trained with Okkhornama, an open-source database and a custom author-curated dataset of 49 categories and 3,760 images have been used to validate these techniques. These images are preprocessed by auto-orienting and resizing into 416 × 416 pixels in the Roboflow framework. Next, we implemented Detectron2, EfficientDet-D0 with TensorFlow and the PyTorch-built YOLOv7 models. Jetson Nano, a portable and sufficiently dynamic NVIDIA microprocessor, is also utilized in this project to generate an object detection model and infer it from test images in real-time. The Dectectron2 model performed best concerning detection accuracy, with a mAP@.5 of 94.915 and an AP of 54.814. The YOLOv7 model achieved mAP@.5 accuracies in detecting various classes from 85 percent to 97 percent and mAP@.5-.95 from 41 percent to 53 percent. Finally, the YOLOv7 Tiny approach has been used in the Jetson Nano edge device for real-time Bangla sign language detection because of its lowest total training time and frame rate. The proposed system is expected to provide an effective solution for Bangladeshi people suffering from hearing disabilities and the inability to speak in a simple and affordable approach.

Sign Language provides the means of conveying messages for hearing impaired people. Effective communication with the masses is a great challenge for the hearing impaired community, as Sign Language is not commonly understood. Many researchers have done numerous works in foreign language datasets like English, French, Japanese, etc. However, for Bangla, one of the most widely spoken languages, much significant work has not been done yet. Most of the works on Bangla Sign Language are executed on small datasets and report satisfactory performance. However, when small datasets are evaluated from the perspective of generalizability, particularly when using deep learning based solutions, these models fail to reproduce to the same performance. Therefore, this paper poses inter-dataset evaluation as the main evaluation criteria and evaluates several deep learning based models. This evaluation is done for Bangla by leveraging two popular datasets of Bangla Sign Language. Unsurprisingly, the inter-dataset performance is inferior, and several approaches to improve are explored and documented, including the use of angular margin based loss functions. The results demonstrate the importance of such an evaluation and also show that one of the proposed approaches shows promising performance, albeit with significant room for improvement. This raises the need for a standardized dataset to overcome this issue of generalization for real-life applications, as well as the need to encourage future works to concentrate on challenging evaluations instead of pursuing deceptively good intra-dataset performance.

• Introduced and formulated the problem of Bangla sign language (BdSL) alphabet recognition in Zero-shot Learning. • Created a new BdSL semantic attribute representation. • Proposed a seen/unseen split of BdSL signs for the Zero-shot learning approach. • Developed a transfer learning based approach to recognize both one-handed and two handed BdSL. • Compiled and evaluated our model on a large-scale dataset and made this publicly available on GitHub. Bangla, being the fifth most spoken language in the world has its own distinct sign language with two methods (one-handed and two-handed) of representation. However, a standard automatic recognition system of Bangla sign language (BdSL) is still to be achieved. Though widely studied and explored by researchers in the past years, certain unaddressed issues like identifying unseen signs and both types of BdSL or lack of evaluation of the models in versatile environmental conditions demarcate the real-world implementation of the automatic recognition of BdSL. To find a probable solution to the shortcomings in the existing works, this paper proposes two approaches based on conventional transfer learning and contemporary Zero-shot learning (ZSL) for automatic BdSL alphabet recognition of both seen and unseen data. The performance of the proposed system is evaluated for both types of Bangla sign representations as well as on a large dataset with 35,149 images from over 350 subjects, varying in terms of backgrounds, camera angle, light contrast, skin tone, hand size, and orientation. For the ZSL approach, a new semantic descriptor dedicated to BdSL is created and a split of the dataset into seen and unseen classes is proposed. Our model achieved 68.21%, 91.57%, and 54.34% of harmonic mean accuracy, seen accuracy, and zero-shot accuracy with six unseen classes respectively. For the transfer learning-based approach, we found pre-trained DenseNet201 architecture to be the best performing feature extractor and Linear Discriminant Analysis as the best classifier with an overall accuracy of 93.68% on the large dataset after conducting quantitative experimentation on 18 CNN architectures and 21 classifiers. The satisfactory result from our models supports its very probative potential to serve extensively for the hearing and speaking impaired community. [ABSTRACT FROM AUTHOR]

A real-time Bangla Sign Language interpreter can enable more than 200 k hearing and speech-impaired people to the mainstream workforce in Bangladesh. Bangla Sign Language (BdSL) recognition and detection is a challenging topic in computer vision and deep learning research because sign language recognition accuracy may vary on the skin tone, hand orientation, and background. This research has used deep machine learning models for accurate and reliable BdSL Alphabets and Numerals using two well-suited and robust datasets. The dataset prepared in this study comprises of the largest image database for BdSL Alphabets and Numerals in order to reduce inter-class similarity while dealing with diverse image data, which comprises various backgrounds and skin tones. The papers compared classification with and without background images to determine the best working model for BdSL Alphabets and Numerals interpretation. The CNN model trained with the images that had a background was found to be more effective than without background. The hand detection portion in the segmentation approach must be more accurate in the hand detection process to boost the overall accuracy in the sign recognition. It was found that ResNet18 performed best with 99.99% accuracy, precision, F1 score, sensitivity, and 100% specificity, which outperforms the works in the literature for BdSL Alphabets and Numerals recognition. This dataset is made publicly available for researchers to support and encourage further research on Bangla Sign Language Interpretation so that the hearing and speech-impaired individuals can benefit from this research.

Abstract Hearing impaired people have own language called Sign Language but it is difficult for understanding to general people. Sign language is the basic method of communication for deaf people during their everyday of life. Sign digits are also a major part of sign language. So machine translator is necessary to allow them to communicate with general people. For making their language understandable to general people, computer vision based solutions are well known nowadays. In this research work we aims at constructing a model in deep learning approach to recognize Bangla Sign Language (BdSL) digits. In this approach there used Convolutional Neural Network (CNN) to train particular signs with a respective training dataset (Eshara-Lipi) for acquiring our aim. The model trained and tested with respectively 860 training images and 215 (20%) test images of tent classes of digits. Finally, the training model gained about 95% accuracy at recognition of Bangla sign language digits. This model will contribute for moving one step forward to make BdSL machine translator. [ABSTRACT FROM AUTHOR]

Sign language is the comprehensive medium of mass communication for hearing and speaking impaired individuals. As they cannot speak or hear, they are not able to use sound or vocal signals as an information medium for their communication. Rather, they are bound to exchange visual signals to express their feeling in their day-to-day life. For this, they use various body language mainly hand gestures as sign language. Sign language fundamentals can be largely divided into two parts namely digits (numerals) and characters (alphabetical). In this paper, we proposed a hybrid model consisting of a deep transfer learning-based convolutional neural network with a random forest classifier for the automatic recognition of Bangla Sign Language (numerals and alphabets). The overall performance of the presented system is verified on 'Ishara-Bochon' and 'Ishara-Lipi' datasets. 'Ishara-Bochon' and 'Ishara-Lipi' are datasets of isolated numerals and alphabets respectively which are the first complete multipurpose open-access dataset for Bangla Sign Language (BSL). Besides, we also proposed a background elimination algorithm that removes unnecessary features from the sign images. Along with the proposed background elimination technique, the system is able to achieve accuracy, precision, recall, f1-score values of 91.67%, 93.64%, 91.67%, 91.47% for character recognition and 97.33%, 97.89%, 97.33%, 97.37% for digit recognition respectively. The detailed experimental analysis assures the feasibility and effectiveness of the proposed system for BSL recognition. • Demonstration of recent advancements in various Sign Language recognition research. • Employment of proposed background elimination algorithm to remove unwanted features. • Hybridization of transfer learning model with the Random Forest classifier. • Employment of backbone networks pre-trained on ImageNet to handle smaller datasets. • Improvement of evaluation parameters compared to other existing recognition systems. [ABSTRACT FROM AUTHOR]

A real-time Bangla Sign Language interpreter can enable more than 200 k hearing and speech-impaired people to the mainstream workforce in Bangladesh. Bangla Sign Language (BdSL) recognition and detection is a challenging topic in computer vision and deep learning research because sign language recognition accuracy may vary on the skin tone, hand orientation, and background. This research has used deep machine learning models for accurate and reliable BdSL Alphabets and Numerals using two well-suited and robust datasets. The dataset prepared in this study comprises of the largest image database for BdSL Alphabets and Numerals in order to reduce inter-class similarity while dealing with diverse image data, which comprises various backgrounds and skin tones. The papers compared classification with and without background images to determine the best working model for BdSL Alphabets and Numerals interpretation. The CNN model trained with the images that had a background was found to be more effective than without background. The hand detection portion in the segmentation approach must be more accurate in the hand detection process to boost the overall accuracy in the sign recognition. It was found that ResNet18 performed best with 99.99% accuracy, precision, F1 score, sensitivity, and 100% specificity, which outperforms the works in the literature for BdSL Alphabets and Numerals recognition. This dataset is made publicly available for researchers to support and encourage further research on Bangla Sign Language Interpretation so that the hearing and speech-impaired individuals can benefit from this research. [ABSTRACT FROM AUTHOR]