Search

This paper describes the procedures and specific features of application of a new method of automatic classification based on calculation of deviations of stop-words distribution from Zipfian score. To neutralize discrepancies in texts lengths the author describes and applies the text undersampling methodology. The concept of an iterative threshold level is introduced to reduce text dimensionality to several dozen units. To evaluate the method’s efficiency the author has developed discriminative and similarative powers indicators that underlie the generalized efficiency score. Fourteen tests have been conducted, including comparison with the cosine similarity measure, that proved high efficiency of the proposed method for the solution of the tasks of authorship attribution of texts of fiction and clusterization of political texts.

The paper proposes an original methodology of authorship attribution based on the deviations from Zipf distribution and statistical data obtained with the help of a concordance program and computations performed in a table processor. The methodology involves finding distances between input texts and a reference text basing on deviations of stop-words frequencies. The results that have been achieved prove that the proposed methodology allows performing efficient authorship attribution and that it can be used in the educational process to develop student skills and competencies pertaining to natural language processing.

In this paper, we draw a distinction between information and computer sciences in terms of their objects and directions of research. In terms of the operating mode, we distinguish among automatic, automated, semi-automatic, and assistant systems and show that in each application domain there can be different configurations of these systems at different periods of time. In addition, we analyze the engineering, linguistic, and mathematical levels of domain-specific research and development, formulate their problems, and discuss promising directions.

The paper proposes the principles of publicity of historical process, consequency, and paradigmality. Depending on their significance, historical events are classified into critical, epochal, key, and occasional events. Local and global events are differentiated and possible variants of representation of local events at the global level are determined. The paper describes variants of correlation between the new paradigm and the old one, such as total rejection, partial rejection, mergence, and co-existence. It proves that the last variant is more characteristic of computer science. An attempt is made to distinguish “computer science” from “information science.”

This paper describes a methodology for developing a linguistic ontology as a component of a system for automatic analysis of customer opinions about commercial products. The fundamental principles of building ontologies of this type are substantiated, which include the following: the relationship between ontology and grammar; distinguishing parametric and evaluative terms in its structure and classification of evaluative terms into syntactic and semantic ones; the binary relationship between syntactic and semantic terms; the gradation scale of the intensity of evaluations. The cases of the homonymy and synonymy of evaluative terms are analyzed for the first time based on Russian data.

The efficiency of using the chi-square metrics to weigh terms used in text documents is evaluated. The procedure includes the selection and advanced processing of class C and ~C texts, compilation of a reference dictionary and calculation of scores for all the terms in the dictionary, calculation of ?2 coefficients for terms from a class C text, and calculation of the general efficiency factor by the sum of the coefficients found for the terms from the reference dictionary. The weighting by the ?2 formula, odds-ratio (OR) formula, and on the basis of probabilistic variables is analyzed and compared. It was found that the best result is yielded by the OR-based weighting.

This paper describes a method for automatic text classification based on analysing the deviation of the word distribution from Zipf's law, combined with the zonal data-processing approach. Deviation is understood as the difference between the actual numerical score of a word and its score according to Zipf's law. The proposed method involves the division of input and reference texts into J0, J1, and J2 zones, and the creation of a numerical series using the words that are contained in the J0 zone. The constructed numerical series shows the difference between the real scores of words and the scores calculated according to Zipf's law. The proposed method can significantly reduce text dimensionality and thus improve the running speed of automatic text classification.

This paper analyses the method of zonal correlation text analysis based on the comparison of the distribution of word counts in the J 1 zones of two or more texts. The compared texts are divided into the J 0, J 1, and J 2 zones in accordance with the previously developed interpretation of Bradford's law. A comparative analysis of the parameters of words that are contained in the J 1 text zones is performed. The distance, which points to the degree of semantic proximity of the compared texts, is calculated. The described method can be applied for automatic classification and author attribution of texts. In addition, it allows one to obtain adequate results based on a limited number of parameters.

The concept of "linguistic informatics" is introduced in order to refer to a scientific domain that studies the distribution patterns of text information, as well as problems, principles, methods, and algorithms applied for the development of linguistic software and hardware. The key terms and concepts in the related field are investigated; a classification of linguistic software is introduced.

The paper focuses on the structure of predicative possessive constructions described in terms of semantic features distinguished within scope of predication analysis, viz. causative, agentive, inchoative, egressive, continuative, and stative features. Features that are in complementary distribution yield contaminated forms of possessive constructions: inchoative-causative, inchoative-agentive, egressive-causative, egressive-agentive, continuative-causative, continuative-agentive. Causative possessive constructions are taken to be basic since their structure corresponds to the structure of the prototypical situation while the other ones are derived from them by omission of appropriate semantic features. The structure of causative possessive sentences is described in terms of componential analysis.

A classification of lexicographic resources is proposed to support automatic textanalysis systems. Four types of dictionaries are distinguished and described, namely, terminological, terminological-statistical, thesauri, and ontologies. Methods for dictionary generation are divided into static and dynamic, as well as linear and stepwise. Methods for weighting terms are divided into intertextual and intratextual. The features of the TF*IDF algorithm are considered in detail. Two techniques for dictionary generation are described.

The features of different types of decomposition that are employed in preliminary text processing are considered. Linguistic problems of decomposition by clauses via the transformation of communicative and modal planes of a text are discussed. The grammar and algorithms that are required to perform decomposition by clauses are described.

This paper suggests interpreting Bradford's law in terms of a geometric progression; it introduces a constant, which allows simplifying the application of the law, and outlines the methodology for using the law to analyze the data related to various subject areas.

This is a review of methods for the automated analysis of texts. The features of algorithms and programs that are used at the morphological, lexical, syntactical, and discursive levels of a language system are described.

The application domain for the intelligent analysis of texts is described in its specific features. The main stages in the creation and the structure of a linguistic ontology that was intended for a system of automatic analysis of user sentiments about commercial products are considered. The ontology includes syntactic and semantic terms, their subcategories, groups and classes; it enables us to estimate the weighting scores of the evaluation terms that refer to the name of a commercial product indicated in a user's query.

This paper describes an experimental method for automatic text genre recognition based on 45 statistical, lexical, syntactic, positional, and discursive parameters. The suggested method includes: (1) the development of software permitting heterogeneous parameters to be normalized and clustered using the k-means algorithm; (2) the verification of parameters; (3) the selection of the parameters that are the most significant for scientific, newspaper, and artistic texts using two-factor analysis algorithms. Adaptive summarization algorithms have been developed based on these parameters.

This paper considers the existing algorithms and suggests new algorithms for preliminary text processing that permit its quality to be increased, including: the deduction-inversion architecture of decomposition, modified algorithm of bidirectional interference, and morphological analysis based on preliminary annotation with tags of parts of speech.

The paper classifies systems of automatic text summarization and considers in detail the architecture and algorithms in the functioning of systems of a superficial level. It formulates a universal criterion for selection of linguistic units in the process of information search, which is an interpretation of the Zipf law. The problems and prospects in the development of this object domain are considered.

Four modern systems of automatic text summarization are tested on the basis of a model vocabulary composed by subjects. Distribution of terms of the vocabulary in the source text is compared with their distribution in summaries of different length generated by the systems. Principles for evaluation of the efficiency of the current systems of automatic text summarization are described.