Search

This article presents a novel approach towards solving the problem of Word Sense Disambiguation (WSD) for Bengali Text. The algorithm used in this work is a modification of Lesk Algorithm. In the original algorithm, the overlap between the “context bag” and the “sense bag” items from the lexical resource (WordNet) are calculated using word pair matching. In the current approach the overlap is calculated by adopting semantic similarity measure using the fastText subword embeddings. The approach can efficiently handle unknown wordforms and discover the latent semantics of words. Significant progress has been made in WSD for English and other European Languages. Indian languages like Bengali still pose a formidable challenge. The dataset used for the work is individual sentences from the Bengali Wikipedia which is a huge collection of Bengali text (96 K Webpages with 1700 K sentences), the Indo WordNet for Bengali language and Bengali Online Dictionary. The results of the experiments performed are promising. The target words which have semantically distinct synsets in the WordNet give a high F1 score. The F1 score achieved is 80% which is well over the baseline and shows significant improvement over the other knowledge-based approaches tried on low resource Indian languages. [ABSTRACT FROM AUTHOR]

An attempt is made in this paper to report how a supervised methodology has been adopted for the task of Word Sense Disambiguation (WSD) in Bengali with necessary modifications. At the initial stage, four commonly used supervised methods, Decision Tree (DT), Support Vector Machine (SVM), Artificial Neural Network (ANN) and Naïve Bayes (NB), are developed at the baseline. These algorithms are applied individually on a data set of 13 most frequently used Bengali ambiguous words. On experimental basis, the baseline strategy is modified with two extensions: (a) inclusion of lemmatization process into the system and (b) bootstrapping of the operational process. As a result, the levels of accuracy of the baseline methods are slightly improved, which is a positive signal for the whole process of disambiguation as it opens scope for further modification of the existing method for better result. In this experiment, the data sets are prepared from the Bengali corpus, developed in the Technology Development for Indian Languages (TDIL) project of the Government of India and from the Bengali WordNet, which is developed at the Indian Statistical Institute, Kolkata. The paper reports the challenges and pitfalls of the work that have been closely observed during the experiment. [ABSTRACT FROM AUTHOR]

In this work, Word Sense Disambiguation (WSD) in Bengali language is implemented using unsupervised methodology. In the first phase of this experiment, sentence clustering is performed using Maximum Entropy method and the clusters are labelled with their innate senses by manual intervention, as these sense-tagged clusters could be used as sense inventories for further experiment. In the next phase, when a test data comes to be disambiguated, the Cosine Similarity Measure is used to find the closeness of that test data with the initially sense-tagged clusters. The minimum distance of that test data from a particular sense-tagged cluster assigns the same sense to the test data as that of the cluster it is assigned with. This strategy is considered as the baseline strategy, which produces 35% accurate result in WSD task. Next, two extensions are adopted over this baseline strategy: (a) Principal Component Analysis (PCA) over the feature vector, which produces 52% accuracy in WSD task and (b) Context Expansion of the sentences using Bengali WordNet coupled with PCA, which produces 61% accuracy in WSD task. The data sets that are used in this work are obtained from the Bengali corpus, developed under the Technology Development for the Indian Languages (TDIL) project of the Government of India, and the lexical knowledge base (i.e., the Bengali WordNet) used in the work is developed at the Indian Statistical Institute, Kolkata, under the Indradhanush Project of the DeitY, Government of India. The challenges and the pitfalls of this work are also described in detail in the pre-conclusion section. [ABSTRACT FROM AUTHOR]

Virtual Assistants (VA) and Chatbots have boosted the pace of research in Question Answering (QA) system. QA systems are supposed to return the answers of the questions by processing the backend repository. All the questions and the text in the repositories are in natural languages only. Substantial number of projects are executed for building QA systems in high resource languages. In case of low resource languages, the progress is still in early stage. In this work, we have designed, developed and evaluated the performance of a factoid QA system in a low resource language—Bengali. The system takes the questions from the human and then retrieves all the prospective answers from a multi-domain repository. Based on six parameters, the answers are ranked and returned. Therefore, the performance of the system is evaluated and compared with earlier systems using standard metrics. The algorithm is tested on two repositories. First is the TDIL corpus containing large collection of famous Bengali literature, which was developed in the Technology Development of Indian Languages (TDIL) project. Second is the translated SQuAD which is the Bengali translation of Stanford Question Answering Dataset. The accurate answer is ranked by the system as 1st in 88.23% cases. Accuracy and F1 score are calculated as 97.64% and 98.5%, respectively for TDIL corpus and 97.16% and 98.51% for translated SQuAD based on the performance evaluation by confusion matrix. [ABSTRACT FROM AUTHOR]