Search

The paper presents a study on word sense disambiguation (WSD) in Bengali, one of the less resourced Indian languages. The overall work is carried out in two sequential phases. In the first phase, four well-known approaches, which are often applied for sense disambiguation of words, are studied using the traditional methods. In the course of application, suitable modifications are made as well as implemented for eliciting desired results. In the second stage, a combined approach is proposed based on the results obtained from initial experiments. Within ‘supervised module’ the four commonly used methods, namely, the Decision Tree (DT) method, Support Vector Machine (SVM) method, Artificial Neural Network (ANN) method, and the Naive Bayes (NB) method are used at the baseline for the purpose of classification of senses. These baseline strategies produced 63.84%, 76.9%, 76.23%, and 80.23% accurate results, respectively, when these methods are tested on 13 mostly used Bengali ambiguous words retrieved from a Bengali text corpus. Next, two major modifications are applied on these baseline strategies to increase the level of accuracy: (a) incorporation of Lemmatization process in the system (that produces 68.30%, 79%, 78.23%, and 82.30% accurate results, respectively), and (b) operation of Bootstrapping on the systems (including lemmatization feature), which produces 70.92%, 79.15%, 79.53%, and 83% accuracy, respectively. Next, in a knowledge-based method, the traditional Lesk algorithm is implemented at the baseline which produces 31% accurate result in sense disambiguation. This strategy is further modified by Context Expansion (CE) method in the sentences using the Bengali WordNet to produce 75% accuracy. Within ‘unsupervised module’, the baseline strategy produced a 36.2% accurate result in sense disambiguation task. To enhance the level of performance, two modifications are adopted over this baseline strategy: (a) Principal Component Analysis (PCA) over the feature vector, which produces 51.2% accuracy in WSD task, and (b) Context Expansion of the sentences using the Bengali WordNet with PCA, which produces 61% accuracy in sense disambiguation task. Finally, a combined approach is adopted after considering all the effective aspects of the three methods, and it produces the highest level accuracy (92%) in the task of sense disambiguation.

Automatic text categorization is the operation of sorting out the text documents into pre-defined text categories using some machine learning algorithms. Normally, it defines the most important approaches to organizing and making the use of a large volume of information exists in unstructured form. Nowadays, text categorization is becoming an extensively researched field of text mining and processing of languages. Word sense, semantic relationships among terms, text documents and categories are quite essential in order of enhancing the performances of categorization. Various surveys on text categorization have already been available which involve techniques of various text representation schemes to such extent but do not include several approaches that have been explored in text categorization over the standard techniques. Here, an exhaustive analysis of different text categorization approaches over the conventional approaches has been undertaken. This survey paper explores a wide variety of algorithms used for categorizing text documents and tries to assemble the existing works into three basic fields: conventional methods, fuzzy logic-based methods, deep learning-based methods. Further, conventional methods have been categorized into three fields: text categorization using handcrafted features, text categorization using nature-inspired algorithms and text categorization using graph-based methods. Furthermore, this survey provides a clear idea about the available libraries used for different algorithms, availability of datasets, categorization technologies explored in various non-Indian and Indian languages as well.

Technology has evolved remarkably, which has led to an exponential increase in the availability of digital text documents of disparate domains over the Internet. This makes the retrieval of the information a very much time- and resource-consuming task. Thus, a system that can categorize such documents based on their domains can truly help the users in obtaining the required information with relative ease and also reduce the workload of the search engines. This article presents a text categorization system (CESS) that categorizes text document using newly proposed hybrid features that combines term frequency-inverse document frequency-inverse class frequency and modified chi-square methods. Experiments were performed on real-world Bangla documents from eight domains comprises of 24,29,857 tokens, and the highest accuracy of 99.91% has been obtained with multilayer perceptron-based classification. Also, the experiments were tested on Reuters-21578 and 20 Newsgroups datasets and obtained accuracies of 97.29% and 94.67%, respectively, to show the language-independent nature of the system.

Some of the basic and preliminary ideas of text annotation and text processing techniques, which are normally carried out on a corpus of written and spoken texts are addressed in this chapter. Keeping non-trained linguistic scholars and common linguistic readers in view, we briefly discuss the basic nature and goal of text annotation, describe the purposes of text annotation, and refer to the common maxims of text annotation. A common reader may need these ideas to understand the tools, systems, and techniques of text annotation and processing that are discussed in this book. Next, we report on different types of text annotation, which we apply to written and spoken text corpora. We address these issues keeping in view the theoretical, functional, and referential importance of text annotation and text processing in the analysis and application of a natural language data by man and machine in various domains of linguistics and technology. We also draw theoretical differences between text annotation and text processing to dispel the confusions faced by both academicians and corporate scholars who use annotated and processed texts as an indispensable resource. We look into the present status of text annotation and processing in both resource-rich and resource-poor languages and propose to take the necessary initiative for building raw corpora, annotated corpora, and processed corpora for the resource-poor languages to address the requirements of language technology, linguistics, and other disciplines. Finally, we discuss how the applicational importance and referential relevance of corpora are increased after corpora are annotated with various kinds of linguistic (and extralinguistic) information and processed at various levels.

In this chapter, we describe some principles and rules that we apply during the process of part-of-speech (POS) annotation on a written text corpus. For various practical and theoretical reasons, these principles and rules become useful and relevant for overcoming many unwanted hurdles in POS annotation. We also describe here the strategies that we apply when we formulate some algorithms for the automatic assignment of grammatical values to words in a text. Although we address theoretical and practical issues of POS annotation, we do not talk about technical and computational issues which are important when a computer system is engaged for this kind of work. Since the target readers of this chapter are students of linguistics who have limited or zero exposure to computation, we keep these technical and computational issues beyond this chapter. We primarily focus on some linguistic and theoretical aspects of the problem. We keep these issues relatively simple as an area of general inquiry, so that non-experts and common readers can get ideas about how words in a text are annotated at the POS level, and how the principles and rules are to be used during this work. This chapter has two parts: in Part I, we propose some principles that are to be followed during POS annotation; and in Part II, we define some rules which are to be adopted during the actual work of POS annotation. Supporting examples are taken from English and Bengali text corpus. This chapter imparts necessary guidance to the new generation of scholars about the nature and characteristics of POS annotation and how proper reference to these principles and rules can make the subsequent works of text processing easy and simplified.

The instantaneous magnification of digital texts has made text categorization a prime technique for managing and sorting text data. The goal of this study is to present a lazy learning-based text categorization system using text representation features. Experiments were performed on 14,373 texts from nine different categories procured from various online news corpus and magazines. The experimental results show an encouraging outcome of 96.24% accuracy using the K* algorithm. The performance of the system was also tested with other existing approaches and datasets and our system produced a better result.

In this chapter, we describe a process of lemmatization of inflected nouns in Bengali as a part of lexical processing. Inflected nouns are used at a very high frequency in Bengali texts. We first collect a large number of inflected nouns from a Bengali corpus and compile a noun database. Then we apply a process of lemmatization to separate inflections from nominal bases. There are several intermediate stages in lemmatization which are applied following grammatical mapping rules (GMRs). These rules isolate inflections from nominal bases. The GMRs are first designed manually after analyzing a large set of inflected nouns to collect necessary data and information. At subsequent stages, these GMRs are developed in a machine-readable format so that the lemmatizer can separate the inflections from inflected nouns with the least human intervention. This strategy is proved to be largely successful in the sense that most of the inflected Bengali nouns, which are stored in a noun database, are rightly lemmatized. This multilayered process also generates an exhaustive list of nominal inflections and a large list of lemmatized nouns. At the subsequent stage, nouns are semantically classified for their use in translation, dictionary compilation, lexical decomposition, and language teaching. We have also applied this method to lemmatize inflected pronouns and adjectives which follow a similar pattern of inflection and affixation in Bengali.

Annotating words at the part-of-speech level, either manually or by a machine, is a tough task. It is done effectively when human annotators, as well as computer systems, are properly trained so that they can correctly identify morphological properties and syntactic functions of words in a piece of text. We discuss in this chapter some theoretical aspects and practical issues of part-of-speech (POS) annotation on a written Bengali text corpus. We deliberately avoid all those issues and aspects that are required to design and develop an automatic POS annotation tool for a text, since this is not the goal of this chapter. To keep things simple and within the capacity of those readers who are not well-versed in the application of computers, we address here some of the primary concerns and challenges involved in POS annotation. Starting with the basic concept of POS annotation, we highlight the underlying differences between POS annotation and morphological processing; define the levels and stages of POS annotation; refer to some of the early works on POS annotation; present a generic scheme for POS annotation; and show how a POS annotated text is utilized in various domains and sub-domains of theoretical, descriptive, applied, computational, and cognitive linguistics. The data and information presented in this chapter are primarily meant for the students of those less-advanced languages which still lack linguistic resources like POS annotated texts. The rudimentary ideas and information that are presented in this chapter may be treated as valuable and usable inputs for designing linguistic and computational models for POS annotation in these less-advanced languages.

We apply an analytical method to decompose inflected verbs in Bengali. We treat these verbs as autonomous and isolated linguistic units that are physically free from their contextualized frames of occurrence in sentences. The basic goal behind applying this strategy is to retrieve various kinds of morphological information from inflected Bengali verbs. Other goals are to develop a verb root database, a verb suffix database, and a semi-supervised verb analysis system. To achieve these goals, we apply a rule-based suffix stripping method based on knowledge and information obtained from a manual analysis of a large number of inflected verbs that are used in the Bengali written corpus. Our system first accepts an inflected verb as a lexical input, treats it as a word of an unbroken string of characters, applies stripping rules for root and suffix segmentation, matches segmented parts with respective roots and suffix lists, extracts relevant information from a machine-readable inbuilt dictionary, and displays outputs with information about the constituents that are used to generate the final inflected form. The outputs are root lists, suffix lists, morph lists, grammatical rules, and matching algorithms all of which are necessary for developing a morphological analyzing system for Bengali. The lexical database, morph lists, and metadata that are produced through this method are also useful for developing systems which work for lexical information retrieval and lexical teaching. The method that we use here can be applied for decomposing inflected verbs of other Indian languages by replacing the respective root and suffix databases and concatenation rules.

The morphological processing of words is a lexical analysis process which is used to retrieve various kinds of morphological information from affixed and inflected words. The concept of morphological processing, in the general linguistic discussion, is often mixed up with part-of-speech annotation and syntactic annotation. To make things clear, we define the core concept of morphological processing and discuss some of the methods and approaches that are so far developed and used for processing words across languages. Also, we briefly describe how a morphological processing system is actually applied to analyze affixed and inflected words used in texts. Moreover, we show how processing of inflected words of different parts-of-speech usually put up strong challenges in the generation of right outputs; how orthographic irregularities and inconsistencies posit strong barriers in proper recognition of inflected words; and how detached words and lexical ambiguities can mar the robustness of a morphological processing system in analysis and generation of right outputs. The success of a morphological processing system largely depends on two basic components: (a) its ability in dealing with the inherent morphological complexities of inflected words, and (b) its ability in the management of linguistic data, information, and grammatical mapping rules. We address all these issues in this chapter with reference to Bengali—one of the highly inflected Indian languages.

In this chapter, we define the basic concepts of some non-conventional types of text annotation which are, due to several reasons, not frequently used on corpus texts. This discussion gives readers some preliminary ideas about the ways and means of annotating a text at various levels, beyond the grammatical and syntactic level of annotation, for making a text useful in various domains of linguistics and language engineering. The availability of different types of annotated texts is a blessing for various academic and commercial applications. The primary goals and objectives of each type of corpus annotation are characteristically different. Goals vary depending on the type of a text and scopes for possible utilization in a customized and object-oriented application. Because of marked differences in goals, the processes of corpus annotation vary. Besides, the kind of text considered for annotation plays a decisive role in the selection of annotation type. The non-conventionally annotated texts are not always useful for all kinds of linguistic investigation and studies. They are useful in those contexts where non-standard analysis and interpretation of texts are required for a specific application. That means the application of non-conventional annotation processes on text generates a kind of outputs which are not frequently applied in traditional schemes of language description and analysis. However, such texts become significantly relevant in many advanced areas of applied linguistics and language data management. On the other hand, non-conventional annotation techniques require a different kind of capability in understanding a text, which in return generates a new kind of expertise in text interpretation and information processing and management. Keeping all these aspects in view, in this chapter, we focus on some of the non-typical and non-conventional text annotation types which are not so frequently applied in corpus annotation.

The digital world is flooded with a huge number of documents belonging to multifarious categories. Most of these documents are uncategorized, which is a hindrance to efficient retrieval. In the case of news texts (one of the largest and most common sources of text information), it is often observed that a text does not belong to one particular category and has contents from multiple domains. This demands a text categorization system to segregate it into its respective domains for efficient information retrieval. The main challenge lies in handling the overlap of vocabulary among different domains at the time of categorization, which we have tackled using an approach based on fuzzy logic. In the present work a fuzzy rule inference system is presented, which works with newly proposed statistical features for segregating documents that belong to more than one or an undefined category. The generated model was defuzzified using five different techniques for determining the category of a document and the highest accuracy of 98.63% for the Centroid method was obtained. Experimentation was also carried out on standard English datasets (Reuters-21578 R8 and 20 Newsgroups). We obtain better results than those of reported works, thereby pointing to the language independence of our system.

Categorization of text documents has become a prime task with plenty of applications. It is a process that assigns the text documents to their individual categories based on the contents. In this paper, a long-short-term memory (LSTM) recurrent neural network (RNN) based system along with handcrafted graph-based features is proposed for text categorization. In comparison to the traditional bag of words model for representation of documents, this graph-based model represents each document by a weighted graph for determining the relationships among the tokens. The significance of a token to a text document is determined by the graph-based feature. Experiments were tested on 14,373 Bangla text documents (around 57,22,569 tokens) and a maximum accuracy of 99.21% has been obtained.

espanolEste articulo lleva a cabo un esbozo linguisticode la lenguakhortha, que se habla en el estado de Jharkhand, India. El khortha es el segundo idioma mas hablado despues del hindi en el estado de Jharkhand, con aproximadamente 80 millones de hablantes (segun el Gobierno de India, informes del censo de 2011). La escasez de recursos linguisticos en Khortha jugo un papel vital paramotivarel presente trabajo. La metodologia que se ha adoptadocomprende encuestas linguisticas (Dash &Aman 2015) y resenassobre la literatura anterior delkhortha. El estado actual y el perfil demografico delkhortha sugieren que se usa tambiencomo unalenguade enlace entre las otras comunidades de lenguas indigenas (es decir, el munda, el bedia, elkurmali, etc.). El alcance (uso) del idioma khortha dentro de los diversos dominios (es decir, la administracion, la educacion, los medios de comunicacion, las divisiones sociales y la religion, el poder judicial y la comunicacion interpersonal), como se exponeen el articulo, da una idea clara de su uso e identidad linguistica. Este trabajopuede ser un recurso util para los investigadores a fin de retratar el estado linguistico actual dela lengua. EnglishThis paper describes the linguistic outline of Khortha language, which is spoken in the state of Jharkhand, India. Khortha is the second most spoken language after Hindi in the state of Jharkhand, with approximately 80 million speakers (As per the Govt. ofIndia, census reports 2011). The paucity of the language resources in Khortha played a vital role in motivating us for the present work. The methodology adopted for the present study comprises linguistic field surveys (Dash & Aman 2015) and reviews on theearlier literature of Khortha. The current status and demographic profile of Khortha suggest its usage as a link language among the other indigenous language communities (i.e. Munda, Bedia, Kurmali, etc.) as well. The scope (usage) of the Khortha languagewithin the various domains (i.e. administration, education, mass media, social divisions and religion, judiciary and interpersonal communication), as discussed in the paper, gives a clear idea of its usage and linguistic identity. This paper can be a helpful resource for the researchers in order to portray the current linguistic status of the language.

An attempt is made in this paper to report how a supervised methodology has been adopted for the task of word sense disambiguation in Bangla with necessary modifications. At the initial stage, the Naive Bayes probabilistic model that has been adopted as a baseline method for sense classification, yields moderate result with 81% accuracy when applied on a database of 19 (nineteen) most frequently used Bangla ambiguous words. On experimental basis, the baseline method is modified with two extensions: (a) inclusion of lemmatization process into of the system, and (b) bootstrapping of the operational process. As a result, the level of accuracy of the method is slightly improved up to 84% accuracy, which is a positive signal for the whole process of disambiguation as it opens scope for further modification of the existing method for better result. The data sets that have been used for this experiment include the Bangla POS tagged corpus obtained from the Indian Languages Corpora Initiative, and the Bangla WordNet, an online sense inventory developed at the Indian Statistical Institute, Kolkata. The paper also reports about the challenges and pitfalls of the work that have been closely observed and addressed to achieve expected level of accuracy.

Internet technology has brought a substantial change to our day to day life. Nowadays, we need several digital means to automatically manage, filter, store and retrieve information from text documents. Automated text document analysis and mining are becoming an essential part of computer applications, and thus various classification and clustering approaches are required for carrying out these tasks. The classification of the documents needs to be performed over training datasets, which are further used to train the model to classify text documents into their respective text categories or domains. Thus, text analysis becomes one of the major aspects of text data mining. This chapter presents a graph-based text classification approach for analysing web text documents obtained from various online sources. Sets of text documents are represented as graph sets to which an algorithm of a weighted graph is implemented to extract subgraphs that are further used to develop feature vectors for each text document being classified. A weighted graph algorithm is chosen to extract the most relevant subgraphs, which in turn increases the classification and computational efficiency. Contrary to the traditional vector space model and bag-of-words approaches, the graph-based model performs more efficiently by considering the structural information of the text documents. Graph-based models are becoming an alternative way of representing text because of the ability to encapsulate important facts related to the text, such as the ordering of a term, co-occurrence of terms and relationships among different terms. The proposed model is validated and evaluated by using several popular classification algorithms on the obtained datasets. The performance of the system is stated in terms of precision, recall and F1 measure. Compared with different methodologies adopted by researchers, the proposed approach is found to be accurate and more efficient for classifying text documents of multiple categories.

An attempt is made in this paper to report how a supervised methodology has been adopted for the task of Word Sense Disambiguation (WSD) in Bengali with necessary modifications. At the initial stage, four commonly used supervised methods, Decision Tree (DT), Support Vector Machine (SVM), Artificial Neural Network (ANN) and Naive Bayes (NB), are developed at the baseline. These algorithms are applied individually on a data set of 13 most frequently used Bengali ambiguous words. On experimental basis, the baseline strategy is modified with two extensions: (a) inclusion of lemmatization process into the system and (b) bootstrapping of the operational process. As a result, the levels of accuracy of the baseline methods are slightly improved, which is a positive signal for the whole process of disambiguation as it opens scope for further modification of the existing method for better result. In this experiment, the data sets are prepared from the Bengali corpus, developed in the Technology Development for Indian Languages (TDIL) project of the Government of India and from the Bengali WordNet, which is developed at the Indian Statistical Institute, Kolkata. The paper reports the challenges and pitfalls of the work that have been closely observed during the experiment.

Semantic similarity is a long-standing problem in natural language processing (NLP). It is a topic of great interest as its understanding can provide a look into how human beings comprehend meaning and make associations between words. However, when this problem is looked at from the viewpoint of machine understanding, particularly for under resourced languages, it poses a different problem altogether. In this paper, semantic similarity is explored in Bangla, a less resourced language. For ameliorating the situation in such languages, the most rudimentary method (path-based) and the latest state-of-the-art method (Word2Vec) for semantic similarity calculation were augmented using cross-lingual resources in English and the results obtained are truly astonishing. In the presented paper, two semantic similarity approaches have been explored in Bangla, namely the path-based and distributional model and their cross-lingual counterparts were synthesized in light of the English WordNet and Corpora. The proposed methods were evaluated on a dataset comprising of 162 Bangla word pairs, which were annotated by five expert raters. The correlation scores obtained between the four metrics and human evaluation scores demonstrate a marked enhancement that the cross-lingual approach brings into the process of semantic similarity calculation for Bangla.

With the advent of technology and thrust to create machines with human intelligence, the frontiers in machine learning are being stretched incessantly. With the increasing number of digital information in recent scenario, text classification has gained importance in machine learning task which needs to study texts thoroughly to achieve success. It is a task of assigning a random document to its predefined text class. This paper aims at presenting a methodology for developing an automatic system for solving the problem of classifying Bangla text document into their respective text categories. It introduces a hybrid approach (i.e., PART) for classification of Bangla text documents based on ‘term association’ and ‘term aggregation’ as baseline feature extraction methods. Comparison of results with other classification algorithms shows that this approach can elicit better results than the existing methods.

The continuous increase of information in the web with varying dimensions is becoming difficult for users to filter and analyse them efficiently as it incorporates redundant and irrelevant terms. Managing, filtering and organizing such huge datasets need the classification of text documents to be performed. Text classification is the process of assigning the text documents to their predefined text categories based on the content. The aim of this paper is to explore Cuckoo search optimization (CSO) problem established from the behaviour of cuckoo birds for selection of relevant features by modifying the algorithm. The revised algorithm is named as modified Cuckoo search (MCS) optimization algorithm that can be proved to be useful for developing an efficient text classification system. The proposed method is generated by combining the ability of MCS with the sharpness of Naive Bayes Multinomial (NBM) algorithm for generating proper feature which increases the rate of success. The approach adopted here is tested on 9000 text documents that cover eight different domains fetched from several web sources and obtains encouraging outcome. The results compared with the results from other well-known approaches for text classification task show the effectiveness of the proposed approach as an automatic Bangla text classification system.

This paper aims to develop an automatic text categorization system that classifies Bangla medical and non-medical text documents based on two primary features, that is, word length and the presence of English equivalent words in the text documents. To start with, it has been shown that based on the word length and the number of English equivalent words present in a particular text, Bangla medical text documents can be identified among other text documents of any domain. SGD (Stochastic Gradient Descent) classification algorithm is used and an accuracy of 97.75% has been achieved. Comparisons have also been done with other commonly used classifiers to test the system from which it has been observed that SGD performs better than those classifiers.

The humble goal of this chapter is to refer to some of the achievements in the area of Indian languages corpora generation and lexical databases compilation, which have been done for a few Indian languages within last two and half decades. We shall also try to refer in this chapter some works which are still in the process of continuation for the development of corpora and lexical database for the Indian languages. What is most satisfying is the involvement and active participation of a large number of renowned institutions and individuals of the country in such works due to which these have drawn a considerable amount of attention and approval across the globe. In essence, in this chapter, we shall make a short survey on the development of monolingual corpora and parallel translation corpora, which are developed through some individual attempts or joint enterprise across the country.

In this chapter, we shall make attempt to discuss some of the most common techniques that are often used for processing texts stored in a text corpus. From the early stage of corpus generation and processing, most of these techniques have been quite useful in compiling different types of information about a language as well as formulating some new approaches useful for corpus text analysis and investigation. The important thing is that most of the corpus processing techniques have never been used in language analysis before the advent of the corpus in digital form and application of computer in language data collection and analysis. Another important aspect of this new trend that most of the techniques have strong functional relevance in the present context of language analysis since these techniques give us much better ways to look at the properties of language and utilize them in language analysis and application. We shall briefly discuss some of the most useful text processing techniques such as frequency calculation of characters and words, lexical collocation, concordance of words, keyword in context, local word grouping, and lemmatization. Also, we shall try to show how information and data extracted through the text processing techniques are useful in language description, analysis, and application.

In this chapter, we shall argue and try to explain our claims that a text language corpus, which is being developed for representing a natural language, should be large in the amount of data, multidimensional in text composition, and maximally representative to record all kinds of text varieties found in the language. If we can follow this method, a corpus will be able to overcome the pitfalls of skewedness in text representation and imbalance in text content. The achievement of these properties requires proper treatment of text on the part of the corpus designers as long as the task of data collection and storage continues. In essence, the process of text corpus generation involves various issues like methods of text data selection, hardware to be used in corpus generation, methods adopted for text entry for digitization, nature of corpus generation, the process of corpus data management, and others. We shall discuss some of these issues in some details in this chapter keeping the TDIL Indian language corpus into our purview.

History shows that a machine translation (MT) system with the support of a few linguistic rules is not realistic. A few rules are not sufficient for capturing the wide variety a natural language exhibits in its diverse use. This leads us to argue for a corpus-based machine translation (CBMT) system that desires to rely on a large amount of linguistic data, information, examples, and rules retrieved from corpora. The first benefit of a CBMT system is the development of algorithms for alignment of bilingual text corpus (BTC)—an essential part of an MT system. A BTC generates a new kind of translation support resource that helps in learning through trial, verification, and validation. A CBMT system begins with analysis of translations produced by human to understand and define the internal structures of BTC, completely or partially, to design strategies for machine learning. Analysis of BTC lends heavily to develop aids to translation as we do not expect an MT system to ‘produce’ exact translation but to ‘understand’ how translations are actually produced with linguistic and extralinguistic information. The use of BTC in CBMT is justified on the ground that data and information acquired from BTC are richer than monolingual corpus with regard to information of contextual equivalence between the languages. Thus, a CBMT system earns a unique status by a combination of features of the example-based machine translation (EBMT) and statistics-based machine translation (SBMT) keeping a mutual interface between the two.

Every natural language has a large set of words, which, when these are used in a piece of text, may vary in sense denotation. It has been noted that for ages that context, where these words are found to be used, can play an explicit and active role to influence the words to deviate from the original sense to generate new senses. And these new senses are usually contextualized or context based. The newly acquired senses often vary from the original senses of words usually derived from their origin or etymology. This phenomenon of words in a natural language has several long-standing problems relating to understanding and cognition of word meanings, using word meanings in machine learning as well as presenting word meanings in the dictionary. In this chapter, we shall describe the ‘corpus-based approach’ to deal with the phenomenon of sense understanding of words. We shall try to discuss how the information extracted from a corpus can help us attribute meaning of words to their unique distributional information and contextual environments. While distributional information refers to the frequency distribution of the senses of the words, contextualized environments refer to the setting of occurrence of the words in some particular textual situation. We shall also try to show how we can extract necessary information from the contexts of the words used in a corpus to pick up necessary cues for understanding the actual contextualized senses. To substantiate our arguments, we shall try to draw supporting data, information, and evidence from the Bangla corpus of written texts developed in the TDIL project.

The development of an exhaustive database of scientific and technical terms in a natural language carries tremendous importance in the areas of linguistic resource generation, translation, machine learning, knowledge representation, language standardization, information retrieval, dictionary compilation, language education, text composition, language planning, as well as in many other domains of language technology and mass literacy. Keeping these utilities in mind, in this chapter, we propose for developing a large lexical database of scientific and technical terms in a natural language with the utilization of a corpus. In this work, we propose to adopt an advanced method for systematic collection of scientific and technical terms from a digital language corpus, which is already developed and made available for general access. Since most of the Indian languages are enriched with digital texts that are available on the Internet, it will not be unfair to expect that we can develop a resource of this kind in most of the Indian languages. Following some of the stages of corpus processing discussed in this book (Chap. 5), we can develop an exhaustive database of scientific and technical terms in any of the Indian languages which can be utilized in all possible linguistic activities.

In this chapter, we propose for applying processes like pre-editing and text standardization as some of the essential components of corpus editing and text normalization for making a text corpus ready for access across various domains of linguistics and language technology. Here, we identify some of the basic pre-editing and text standardization tasks, and we describe these works with reference to Bangla text corpus. As the name suggests, text normalization involves diverse tasks of text adjustment and standardization to improve utility of the texts stored in a corpus in manual- and machine-based applications. The methods and the strategies that we propose here to overcome the problems of text normalization are largely tilted toward written text corpus since text normalization activities relating to spoken text corpus usually invoke a new set of operations that hardly match with the normalization processes normally applied on written text corpus. The normalized version of a text not only reduces workload in subsequent utilization of a corpus but also enhances its accessibility by man and machine across all domains where language corpus has application and referential relevance.

In this chapter, we propose for utilizing English Language Corpus (ELC) as a secondary resource for developing English Language Teaching (ELT) materials for teaching English to non-native learners. We argue for using ELC as one of the most authentic representative collections of modern English language from where we can extract necessary linguistic data, appropriate information, and suitable examples to develop some of the basic as well as additional resources of ELT, such as textbooks, reference materials, syllabuses, grammar books, dictionaries, and terminology databases, for the learners. Keeping the requirements of individual learners in mind, we also propose for customizing the materials to make these maximally useful within the broader scheme of computer-assisted language teaching (CALT) method. In our view, the ELT resources developed from the ELC can be far more useful for the learners, because the ELC represents diversified varieties of usage of modern English in real life situations, which the learners need to assimilate to be at par with native speakers across the globe. Also, ELC-based ELT resources can be far more interesting and beneficial to the learners if the ELT instructors carefully utilize these resources keeping in mind the requirements of specific learners. If the ELT learners want to compete in the global frame, then utilization of ELC-based ELT resources is the most useful solution, which we can ignore at the cost of our own peril.

In this chapter, we make attempt to discuss in brief about various types of statistical approaches that are normally used for processing and analyzing a text corpus as well as for obtaining data which may be considered statistically reliable in making some generalized or specific comments on the patterns of occurrence or manner of distribution of linguistic elements in a corpus. Moreover, in this chapter, we try to show how, based on the nature of text used in a corpus, the patterns of quantitative analysis may vary from that of qualitative analysis, although in the long run both types of analysis may be combined together to get a clear picture of the linguistic phenomenon under scrutiny. We also try to give a short history about the use of statistical methods and techniques in the analysis of corpus before and after the introduction of digital corpus as well as describe how descriptive approaches, inferential approaches, and evaluative approaches can be combined together in the act of corpus analysis, linguistics investigation, and inference deduction.

In this chapter, we shall briefly discuss some of the basic issues that are directly linked with text corpus generation in digital form with the involvement of computer in the process. The act of corpus generation asks for consideration of various linguistic and statistical issues and factors which eventually control the entire process of corpus generation. Factors like size of a corpus, choice of text documents, collection of text documents, selection of text samples, sorting of text materials, manner of page sampling and selection, determination of target corpus users, manner of data input, methods of corpus cleaning, management of corpus files are immediate issues that demand utmost attention in corpus generation. Most of these issues are important in the context of text corpus generation not only for advanced languages like English and Spanish but also are of greater importance for poorly resourced languages used in less advanced countries. We shall discuss all these issues in this chapter with reference to some of the Indian languages.