Search

Topic modeling is a probabilistic graphical model for discovering latent topics in text corpora by using multinomial distributions of topics over words. Topic labeling is used to assign meaningful labels for the discovered topics. In this paper, we present a new topic labeling method that uses automatic term recognition to discover and assign relevant labels for each topic, i.e., TLATR (Topic Labeling using Automatic Term Recognition). TLATR uses domain-specific multi-terms that appear in the set of documents belonging to a topic. The multi-term having the highest score as determined by the automatic term recognition algorithm is chosen as the label for that topic. To evaluate TLATR, we use two real, publicly available datasets that contain scientific articles’ abstracts. The topic label evaluation is done both automatically and using human annotators. For the automatic evaluation, we use Pointwise Mutual Information, Normalized Pointwise Mutual Information, and document similarity. For human evaluation, we employ the average rating method. Furthermore, we also evaluate the quality of the topic models using the Adjusted Rand Index. To prove that our novel method extracts relevant topic labels, we compare TLATR with two state-of-the-art methods, one supervised and one unsupervised, provided by the NETL Automatic Topic Labelling system. The experimental results show that our method outperforms or provides similar results with both NETL’s supervised and unsupervised approaches.

Motivation We are increasingly accumulating complex omics data that capture different aspects of cellular functioning. A key challenge is to untangle their complexity and effectively mine them for new biomedical information. To decipher this new information, we introduce algorithms based on network embeddings. Such algorithms represent biological macromolecules as vectors in d-dimensional space, in which topologically similar molecules are embedded close in space and knowledge is extracted directly by vector operations. Recently, it has been shown that neural networks used to obtain vectorial representations (embeddings) are implicitly factorizing a mutual information matrix, called Positive Pointwise Mutual Information (PPMI) matrix. Thus, we propose the use of the PPMI matrix to represent the human protein–protein interaction (PPI) network and also introduce the graphlet degree vector PPMI matrix of the PPI network to capture different topological (structural) similarities of the nodes in the molecular network. Results We generate the embeddings by decomposing these matrices with Nonnegative Matrix Tri-Factorization. We demonstrate that genes that are embedded close in these spaces have similar biological functions, so we can extract new biomedical knowledge directly by doing linear operations on their embedding vector representations. We exploit this property to predict new genes participating in protein complexes and to identify new cancer-related genes based on the cosine similarities between the vector representations of the genes. We validate 80% of our novel cancer-related gene predictions in the literature and also by patient survival curves that demonstrating that 93.3% of them have a potential clinical relevance as biomarkers of cancer. Availability and implementation Code and data are available online at https://gitlab.bsc.es/axenos/embedded-omics-data-geometry/. Supplementary information Supplementary data are available at Bioinformatics online.

This chapter opens the second part of the book, which focuses on mathematical models used for representing textual data. As already mentioned in Chap. 1, the basic objective of text mining (and data mining, in general) can be reduced to the discovery and extraction of relevant and valuable information from large volumes of data. In this chapter we will begin our description of text models by presenting methods that are based on the observation of basic properties and regularities in large volumes of text, which are generally referred to as corpus statistics. First, in Sect. 6.1, we describe some fundamental properties of natural language as they are reflected on the textual representation of the language. Then, in Sect. 6.2, we introduce the concept of word co-occurrences, as well as the commonly used measure of pointwise mutual information, for studying dependences between words. In Sect. 6.3, we focus on word co-occurrences at shorter distances while taking also word order into account.

In tertiary education, program education objects (PEDs) are a core component around which all program's activities revolve. This paper presents a data-driven approach to uncover an important aspect of this component that is its correlation to program majors (PMs). It does so by applying three well-known data correlation metrics, namely Pointwise Mutual Information, Correlation Coefficient, and Odds Ratio, to a dataset extracted from self-study reports of a set of Engineering programs. The collected dataset has undergone a preprocessing step to transform it into a suitable representation. This involves data cleaning, data annotation using a set of PEDs labels, and data projection to break down each multi-PEDs label data instances into a number of single PEDs data instances. The results obtained from the application of the three correlation metrics show a remarkable consistency among the three metrics in their evaluation of the correlation between PMs and PEDs. In a subsequent step, a ranking procedure of the PEDs within each PM, based on the obtained PMs-PEDs correlation strength, is applied and then a majority vote among the ranks of the three metrics is performed to obtain an overall rank of the PEDs within each PM. The obtained results show that each PM has a unique pattern of PEDs ranks, which suggests that PM nature plays a key role in determining the PM-PEDs correlation pattern. As a general conclusion, although the obtained results need further investigation on their causality correlation, the obtained quantitative correlations are very beneficial to the academicians particularly when designing new programs or reviewing existing ones.

We study the change of multivariate mutual information among a set of random variables when some common randomness is added to or removed from a subset of the random variables. This is formulated more precisely as two new multiterminal secret key agreement problems that, respectively, ask how one can increase the secrecy capacity efficiently by adding common randomness to a small subset of users, and how one can simplify the source model by removing redundant common randomness that does not contribute to the secrecy capacity. Characterizations and strongly polynomial-time computations are derived for the rates of change, maximum usable increment, and redundancy. These results can be applied to study the communication complexity for secret key agreement.

In this paper, the exponential weighted entropy (EWE) and exponential weighted mutual information (EWMI) are proposed as the more generalized forms of Shannon entropy and mutual information (MI), respectively. They are position-related and causal systems that redefine the foundations of information-theoretic metrics. As the special forms of the weighted entropy and the weighted mutual information, EWE and EWMI have been proved that they preserve nonnegativity and concavity properties similar to Shannon frameworks. They can be adopted as the information measures in spatial interaction modeling. Paralleling with the normalized mutual information (NMI), the normalized exponential weighted mutual information (NEWMI) is also investigated. Image registration experiments demonstrate that EWMI and NEWMI algorithms can achieve higher aligned accuracy than MI and NMI algorithms.

The present paper aims to propose a simple method to realize mutual information maximization for better interpretation and generalization. To train neural networks and obtain better performance, neurons should impartially consider as many input patterns as possible. Simultaneously, and especially for ease of interpretation, they should represent characteristics specific to certain input patterns as faithfully as possible. This contradiction can be solved by introducing mutual information between neurons and input patterns. However, because of the complicated computational procedures associated with mutual information maximization, it has been difficult to apply mutual information maximization to actual problems. Though many simplified methods have been developed so far, they have not necessarily been applied with success, in particular to large-scale practical problems. To further aid simplification, here we propose a new computational method to realize mutual information. One of the main characteristics of this new method is the consideration of multiple neural networks when defining mutual information, thereby simplifying the method. In addition, learning is also simplified by using the indirect, independent, and fast learning of the potential method. This method was applied to two well known data sets: the Australian credit data and on-line popularity data set. The experimental results showed that mutual information could be increased via the present method. In addition, mutual information maximization was accompanied by an increase in generalization and interpretation performance, mainly due to the simple internal representations.

In this paper, we propose a new method to measure the influence of a third variable on the interactions of two variables. The method called transfer mutual information (TMI) is defined by the difference between the mutual information and the partial mutual information. It is established on the assumption that if the presence or the absence of one variable does make change to the interactions of another two variables, then quantifying this change is supposed to be the influence from this variable to those two variables. Moreover, a normalized TMI and other derivatives of the TMI are introduced as well. The empirical analysis including the simulations as well as real-world applications is investigated to examine this measure and to reveal more information among variables.

Formulation of graph similarity in terms of structural information channels.Posing mutual information between graphs in terms of manifold alignment.Efficient bypass estimators of mutual information between graphs using copulas.Outperforming of state-of-the-art graph similarities. Display Omitted The estimation of mutual information between graphs has been an elusive problem until the formulation of graph matching in terms of manifold alignment. Then, graphs are mapped to multi-dimensional sets of points through structure preserving embeddings. Point-wise alignment algorithms can be exploited in this context to re-cast graph matching in terms of point matching. Methods based on bypass entropy estimation must be deployed to render the estimation of mutual information computationally tractable. In this paper the novel contribution is to show how manifold alignment can be combined with copula-based entropy estimators to efficiently estimate the mutual information between graphs. We compare the empirical copula with an Archimedean copula (the independent one) in terms of retrieval/recall after graph comparison. Our experiments show that mutual information built in both choices improves significantly state-of-the art divergences.

We consider the amount of available information about an arbitrary future state of a Gaussian stochastic process. We derive an infinite series for the marginal mutual information in terms of the autocorrelation function. We derive an infinite series for the newly available information for prediction, the conditional mutual information, in terms of the moving average parameters, and directly characterize predictability in terms of sensitivity to random shocks. We apply our results to long memory, or more generally, hyperbolic decay models, and give information-theoretic characterizations of the transition from persistence to anti-persistence, stationary long memory to nonstationarity, and a stationary regime where the mutual information is not summable.

Keyword extraction is a basic text retrieval technique in natural language processing, which can highly summarize text content and reflect the author’s writing purposes. It plays an important role in document retrieval, text classification and data mining. In this paper, we propose a TextRank algorithm based on PMI (pointwise mutual information) weighting for extracting keywords from documents. The initial transition probability of the candidate words is constructed by calculating the PMI between vocabularies, which is used for iterative calculation of the vocabulary graph model within TextRank and keyword extraction. Taking into account the mutual information between the vocabulary in the document set, the word relationship in the single document is corrected, which is helpful to improve the accuracy of document keyword extraction. Experiments show that our method achieves better performance in extracting keywords in large-scale text data.

We propose PsiRec, a novel user preference propagation recommender that incorporates pseudo-implicit feedback for enriching the original sparse implicit feedback dataset. Three of the unique characteristics of PsiRec are: (i) it views user-item interactions as a bipartite graph and models pseudo-implicit feedback from this perspective; (ii) its random walks-based approach extracts graph structure information from this bipartite graph, toward estimating pseudo-implicit feedback; and (iii) it adopts a Skip-gram inspired measure of confidence in pseudo-implicit feedback that captures the pointwise mutual information between users and items. This pseudo-implicit feedback is ultimately incorporated into a new latent factor model to estimate user preference in cases of extreme sparsity. PsiRec results in improvements of 21.5% and 22.7% in terms of Precision@10 and Recall@10 over state-of-the-art Collaborative Denoising Auto-Encoders. Our implementation is available at https://github.com/heyunh2015/PsiRecICDM2018., Accepted by ICDM'18 as a short paper

We introduce a new concept called as the mutual uncertainty between two observables in a given quantum state which enjoys similar features like the mutual information for two random variables. Further, we define the conditional uncertainty as well as conditional variance and show that conditioning on more observable reduces the uncertainty. Given three observables, we prove a 'strong sub-additivity' relation for the conditional uncertainty under certain condition. As an application, we show that using the conditional variance one can detect bipartite higher dimensional entangled states. The efficacy of our detection method lies in the fact that it gives better detection criteria than most of the existing criteria based on geometry of the states. Interestingly, we find that for $N$-qubit product states, the mutual uncertainty is exactly equal to $N-\sqrt{N}$, and if it is other than this value, the state is entangled. We also show that using the mutual uncertainty between two observables, one can detect non-Gaussian steering where Reid's criteria fails to detect. Our results may open up a new direction of exploration in quantum theory and quantum information using the mutual uncertainty, conditional uncertainty and the strong sub-additivity for multiple observables., 9 pages, 1 figure. V3: Contains new results: a) a new entanglement detection criteria and b) a new steering criteria. Accepted for publication in Phys. Rev. A (2018)

What are the distinct ways in which a set of predictor variables can provide information about a target variable? When does a variable provide unique information, when do variables share redundant information, and when do variables combine synergistically to provide complementary information? The redundancy lattice from the partial information decomposition of Williams and Beer provided a promising glimpse at the answer to these questions. However, this structure was constructed using a much criticised measure of redundant information, and despite sustained research, no completely satisfactory replacement measure has been proposed. In this paper, we take a different approach, applying the axiomatic derivation of the redundancy lattice to a single realisation from a set of discrete variables. To overcome the difficulty associated with signed pointwise mutual information, we apply this decomposition separately to the unsigned entropic components of pointwise mutual information which we refer to as the specificity and ambiguity. This yields a separate redundancy lattice for each component. Then based upon an operational interpretation of redundancy, we define measures of redundant specificity and ambiguity enabling us to evaluate the partial information atoms in each lattice. These atoms can be recombined to yield the sought-after multivariate information decomposition. We apply this framework to canonical examples from the literature and discuss the results and the various properties of the decomposition. In particular, the pointwise decomposition using specificity and ambiguity satisfies a chain rule over target variables, which provides new insights into the so-called two-bit-copy example., 31 pages, 10 figures. (v1: preprint; v2: as accepted; v3: title corrected)

Recently, a series of papers addressed the problem of decomposing the information of two random variables into shared information, unique information and synergistic information. Several measures were proposed, although still no consensus has been reached. Here, we compare these proposals with an older approach to define synergistic information based on the projections on exponential families containing only up to k-th order interactions. We show that these measures are not compatible with a decomposition into unique, shared and synergistic information if one requires that all terms are always non-negative (local positivity). We illustrate the difference between the two measures for multivariate Gaussians. EC/FP7/318723

For high dimensional data analytics, feature selection is an indispensable preprocessing step to reduce dimensionality and keep the simplicity and interpretability of models. This is particularly important for fuzzy modeling since fuzzy models are widely recognized for their transparency and interpretability. Despite the substantial work on feature selection, there is little research on determining the optimal number of features for a task. In this paper, we propose a method to help find the optimal number of feature effectively based on mutual information.

The value of Shannon’s mutual information is commonly used to describe the total amount of information that the neural code transfers between the ensemble of stimuli and the ensemble of neural responses. In addition, it is often desirable to know which features of the stimulus or response are most informative. The literature offers several different decompositions of the mutual information into its stimulus or response-specific components, such as the specific surprise or the uncertainty reduction, but the number of mutually distinct measures is in fact infinite. We resolve this ambiguity by requiring the specific information measures to be invariant under invertible coordinate transformations of the stimulus and the response ensembles. We prove that the Kullback–Leibler divergence is then the only suitable measure of the specific information. On a more general level, we discuss the necessity and the fundamental aspects of the coordinate invariance as a selection principle. We believe that our results will encourage further research into invariant statistical methods for the analysis of neural coding.

In this paper, we propose a new kernel-based co-occurrence measure that can be applied to sparse linguistic expressions (e.g., sentences) with a very short learning time, as an alternative to pointwise mutual information (PMI). As well as deriving PMI from mutual information, we derive this new measure from the Hilbert--Schmidt independence criterion (HSIC); thus, we call the new measure the pointwise HSIC (PHSIC). PHSIC can be interpreted as a smoothed variant of PMI that allows various similarity metrics (e.g., sentence embeddings) to be plugged in as kernels. Moreover, PHSIC can be estimated by simple and fast (linear in the size of the data) matrix calculations regardless of whether we use linear or nonlinear kernels. Empirically, in a dialogue response selection task, PHSIC is learned thousands of times faster than an RNN-based PMI while outperforming PMI in accuracy. In addition, we also demonstrate that PHSIC is beneficial as a criterion of a data selection task for machine translation owing to its ability to give high (low) scores to a consistent (inconsistent) pair with other pairs., Accepted by EMNLP 2018

Binary predictors are used in a wide range of crop protection decision-making applications. Such predictors provide a simple analytical apparatus for the formulation of evidence related to risk factors, for use in the process of Bayesian updating of probabilities of crop disease. For diagrammatic interpretation of diagnostic probabilities, the receiver operating characteristic is available. Here, we view binary predictors from the perspective of diagnostic information. After a brief introduction to the basic information theoretic concepts of entropy and expected mutual information, we use an example data set to provide diagrammatic interpretations of expected mutual information, relative entropy, information inaccuracy, information updating, and specific information. Our information graphs also illustrate correspondences between diagnostic information and diagnostic probabilities.

There are various definitions of mutual information. Essentially, these definitions can be divided into two classes: (1) definitions with random variables and (2) definitions with ensembles. However, there are some mathematical flaws in these definitions. For instance, Class 1 definitions either neglect the probability spaces or assume the two random variables have the same probability space. Class 2 definitions redefine marginal probabilities from the joint probabilities. In fact, the marginal probabilities are given from the ensembles and should not be redefined from the joint probabilities. Both Class 1 and Class 2 definitions assume a joint distribution exists. Yet, they all ignore an important fact that the joint or the joint probability measure is not unique. In this paper, we first present a new unified definition of mutual information to cover all the various definitions and to fix their mathematical flaws. Our idea is to define the joint distribution of two random variables by taking the marginal probabilities into consideration. Next, we establish some properties of the newly defined mutual information. We then propose a method to calculate mutual information in machine learning. Finally, we apply our newly defined mutual information to credit scoring.

For modeling of multivariate time series, input variable selection is a key problem. This paper presents the estimation of joint mutual information and its application in input variable selection problems. Mutual information is a commonly used measure for variable selection. To improve the performance of input variable selection, we propose a novel high-dimensional mutual information estimator based on copula entropy, which is estimated by the truncated k -nearest neighbor method. Simulations on high dimensional Gaussian distributions substantiate the effectiveness of the proposed mutual information estimator. A relationship between the joint mutual information and the copula entropy is derived, which is used for joint mutual information estimation. Then the proposed estimator is applied to input variable selection for multivariate time series modeling based on the criterion of max dependency and max–min dependency. A stop criterion is proposed to terminate the selection process automatically. Simulation results show that the input variable selection method works well on both synthetic and real life dataset.

Aim at existing selection algorithm mutual information inaccurate valuation problem, a condition dynamic concept of mutual information. On this basis, the conditions proposed based on dynamic mutual information (CDMI) feature selection algorithm to overcome the traditional mutual information selection process dynamic correlation problem; conditions of dynamic mutual information throughout the selection process is dynamic valuation, those the samples can be identified after each selection features removed so that they no longer participate in conditional mutual information calculation process, accurate measurement sample. Accurate measurement sample on the degree of importance characteristics and at the same time ensure that the characteristics of information content. The experimental results verify the correctness and effectiveness of the algorithm.

Identification of salient features from a time-varying multivariate system plays an important role in scientific data understanding. In this work, we present a unified analysis framework based on mutual information and two of its decomposition: specific and pointwise mutual information to quantify the amount of information content between different value combinations from multiple variables over time. The pointwise mutual information (PMI), computed for each value combination, is used to construct informative scalar fields, which allow close examination of combined and complementary information possessed by multiple variables. Since PMI gives us a way of quantifying information shared among all combinations of scalar values for multiple variables, it is used to identify salient isovalue tuples. Visualization of isosurfaces on those selected tuples depicts combined or complementary relationships in the data. For intuitive interaction with the data, an interactive interface is designed based on the proposed information-theoretic measures. Finally, successful application of the proposed method on two time-varying data sets demonstrates the efficacy of the system.

Starting with a new formulation for the mutual information (MI) between a pair of events, this paper derives alternative upper bounds and extends those to the case of two discrete random variables. Normalized mutual information (NMI) measures are then obtained from those bounds, emphasizing the use of least upper bounds. Conditional NMI measures are also derived for three different events and three different random variables. Since the MI formulation for a pair of events is always nonnegative, it can properly be extended to include weighted MI and NMI measures for pairs of events or for random variables that are analogous to the well-known weighted entropy. This weighted MI is generalized to the case of continuous random variables. Such weighted measures have the advantage over previously proposed measures of always being nonnegative. A simple transformation is derived for the NMI, such that the transformed measures have the value-validity property necessary for making various appropriate comparisons between values of those measures. A numerical example is provided.

Part 3: MAKE Privacy; International audience; We propose, in this paper, to address the issue of measuring the impact of privacy and anonymization techniques, by measuring the data loss between “before” and “after”. The proposed approach focuses therefore on data usability, more than in ensuring that the data is sufficiently anonymized. We use Mutual Information as the measure criterion for this approach, and detail how we propose to measure Mutual Information over non-Euclidean data, in practice, using two possible existing estimators. We test this approach using toy data to illustrate the effects of some well known anonymization techniques on the proposed measure.

Interaction information is one of the multivariate generalizations of mutual information, which expresses the amount information shared among a set of variables, beyond the information, which is shared in any proper subset of those variables. Unlike (conditional) mutual information, which is always non-negative, interaction information can be negative. We utilize this property to find the direction of causal influences among variables in a triangle topology under some mild assumptions.

A good feature selection method should take into account both category information and high-frequency information to select useful features that can effectively display the information of a target. Because basic mutual information (BMI) prefers low-frequency features and ignores high-frequency features, clustering mutual information is proposed, which is based on clustering and makes effective high-frequency features become unique, better integrating category information and useful high-frequency information. Time is an important factor in topic detection and tracking (TDT). In order to improve the performance of TDT, time difference is integrated into clustering mutual information to dynamically adjust the mutual information, and then another algorithm called the dynamic clustering mutual information (DCMI) is given. In order to obtain the optimal subsets to display topics information, an objective function is proposed, which is based on the idea that a good feature subset should have the smallest distance within-class and the largest distance across-class. Experiments on TDT4 corpora using this objective function are performed; then, comparing the performances of BMI, DCMI, and the only existed topic feature selection algorithm Incremental Term Frequency-Inverted Document Frequency (ITF-IDF), these performance information will be displayed by four figures. Computation time of DCMI is previously lower than BMI and ITF-IDF. The optimal normalized-detection performance (Cdet)norm of DCMI is decreased by 0.3044 and 0.0970 compared with those of BMI and ITF-IDF, respectively.

This paper first illustrates that a mutual information index detects and ranks dependence of a wide variety of absolutely continuous families, but the popular association and variance reduction indices fail to serve as such “common metrics”. We then elaborate on some theoretical merits of the mutual information and give several results. The mutual information provides a notion of the utility of dependence for predicting random variables and quantifies how much the joint distribution is more informative about the variables than the independent model. We present insightful partitions of dependence among the components of a random vector, for a class of models recently proposed for dependence of uncorrelated variables, and for the elliptical families. We also recall that the mutual information is not applicable to singular distributions and give some results for a generalized information index for these models. The generalized index is derived for the Marshall–Olkin copula and for a new singular copula that represents the dependence of the consecutive terms of the exponential autoregressive and related processes.

Using a recently discovered method for producing random symbol sequences with prescribed transition counts, we present an exact null hypothesis significance test (NHST) for mutual information between two random variables, the null hypothesis being that the mutual information is zero (i.e., independence). The exact tests reported in the literature assume that data samples for each variable are sequentially independent and identically distributed (iid). In general, time series data have dependencies (Markov structure) that violate this condition. The algorithm given in this paper is the first exact significance test of mutual information that takes into account the Markov structure. When the Markov order is not known or indefinite, an exact test is used to determine an effective Markov order.

Summary Given a suite of potential surveillance operations, we define surveillance optimization as the problem of choosing the operation that gives the minimum expected value of P90 minus P10 (i.e., P90 – P10) of a specified reservoir variable J (e.g., cumulative oil production) that will be obtained by conditioning J to the observed data. Two questions can be posed: (1) Which surveillance operation is expected to provide the greatest uncertainty reduction in J? and (2) What is the expected value of the reduction in uncertainty that would be achieved if we were to undertake each surveillance operation to collect the associated data and then history match the data obtained? In this work, we extend and apply a conceptual idea that we recently proposed for surveillance optimization to 2D and 3D waterflooding problems. Our method is based on information theory in which the mutual information between J and the random observed data vector Dobs is estimated by use of an ensemble of prior reservoir models. This mutual information reflects the strength of the relationship between J and the potential observed data and provides a qualitative answer to Question 1. Question 2 is answered by calculating the conditional entropy of J to generate an approximation of the expected value of the reduction in (P90 – P10) of J. The reliability of our method depends on obtaining a good estimate of the mutual information. We consider several ways to estimate the mutual information and suggest how a good estimate can be chosen. We validate the results of our proposed method with an exhaustive history-matching procedure. The methodology provides an approximate way to decide the data that should be collected to maximize the uncertainty reduction in a specified reservoir variable and to estimate the reduction in uncertainty that could be obtained. We expect this paper will stimulate significant research on the application of information theory and lead to practical methods and workflows for surveillance optimization.

Squared-loss mutual information (SMI) is a robust measure of the statistical dependence between random variables. The sample-based SMI approximator called least-squares mutual information (LSMI) was demonstrated to be useful in performing various machine learning tasks such as dimension reduction, clustering, and causal inference. The original LSMI approximates the pointwise mutual information by using the kernel model, which is a linear combination of kernel basis functions located on paired data samples. Although LSMI was proved to achieve the optimal approximation accuracy asymptotically, its approximation capability is limited when the sample size is small due to an insufficient number of kernel basis functions. Increasing the number of kernel basis functions can mitigate this weak

Mutual information, conditional mutual information and interaction information have been widely used in scientific literature as measures of dependence, conditional dependence and mutual dependence. However, these concepts suffer from several computational issues; they are difficult to estimate in continuous domain, the existing regularised estimators are almost always defined only for real or vector-valued random variables, and these measures address what dependence, conditional dependence and mutual dependence imply in terms of the random variables but not finite realisations. In this paper, we address the issue that given a set of realisations in an arbitrary metric space, what characteristic makes them dependent, conditionally dependent or mutually dependent. With this novel understanding, we develop new estimators of association, conditional association and interaction association. Some attractive properties of these estimators are that they do not require choosing free parameter(s), they are computationally simpler, and they can be applied to arbitrary metric spaces.

Stock price forecasting phenomenon has been majorly made on the basis of quantitative information. Over the time, with the advent of technology, stock forecasting used technical analysis to get more accurate predictions. Until recently, studies have demonstrated that sentiment information hidden in corporate reports can be effectively incorporated to predict short-run stock price returns. Soft computing methods, like neural networks, fuzzy models and support vector regression, have shown great results in the forecasting of stock price due to their ability to model complex non-linear systems.In this paper we propose a hybrid method for stock price predication, which is combinational feature from technical analysis and sentiment analysis (SA). The features of sentiment analysis are based on a Point wise Mutual Information (PMI) and we apply neural network and ε-support vector regression models to predict the yearly change in the stock price.

Feature selection is an important preprocessing step for many high-dimensional regression problems. One of the most common strategies is to select a relevant feature subset based on the mutual information criterion. However, no connection has been established yet between the use of mutual information and a regression error criterion in the machine learning literature. This is obviously an important lack, since minimising such a criterion is eventually the objective one is interested in. This paper demonstrates that under some reasonable assumptions, features selected with the mutual information criterion are the ones minimising the mean squared error and the mean absolute error. On the contrary, it is also shown that the mutual information criterion can fail in selecting optimal features in some situations that we characterise. The theoretical developments presented in this work are expected to lead in practice to a critical and efficient use of the mutual information for feature selection.

The study of interhuman communication requires a more complex framework than Claude E. Shannon's (1948) mathematical theory of communication because "information" is defined in the latter case as meaningless uncertainty. Assuming that meaning cannot be communicated, we extend Shannon's theory by defining mutual redundancy as a positional counterpart of the relational communication of information. Mutual redundancy indicates the surplus of meanings that can be provided to the exchanges in reflexive communications. The information is redundant because it is based on "pure sets" (i.e., without subtraction of mutual information in the overlaps). We show that in the three-dimensional case (e.g., of a triple helix of university-industry-government relations), mutual redundancy is equal to mutual information (Rxyz = Txyz); but when the dimensionality is even, the sign is different. We generalize to the measurement in N dimensions and proceed to the interpretation. Using Niklas Luhmann's (1984-1995) social systems theory and/or Anthony Giddens's (1979, 1984) structuration theory, mutual redundancy can be provided with an interpretation in the sociological case: Different meaning-processing structures code and decode with other algorithms. A surplus of ("absent") options can then be generated that add to the redundancy. Luhmann's "functional (sub)systems" of expectations or Giddens's "rule-resource sets" are positioned mutually, but coupled operationally in events or "instantiated" in actions. Shannon-type information is generated by the mediation, but the "structures" are (re-)positioned toward one another as sets of (potentially counterfactual) expectations. The structural differences among the coding and decoding algorithms provide a source of additional options in reflexive and anticipatory communications.

In this work, we propose a method for automatic analysis of attitude (affect, judgment, and appreciation) in sentiment words. The first stage of the proposed method is an automatic separation of unambiguous affective and judgmental adjectives from those that express appreciation or different attitudes depending on context. In our experiments with machine learning algorithms we employed three feature sets based on Pointwise Mutual Information, word-pattern co-occurrence, and minimal path length. The next stage of the proposed method is to estimate the potentials of miscellaneous adjectives to convey affect, judgment, and appreciation. Based on the sentences automatically collected for each adjective, the algorithm analyses the context of phrases that contain sentiment words by considering morphological tags, high-level concepts, and named entities, and then makes decisions about contextual attitude labels. Finally, the appraisal potentials of a word are calculated based on the number of sentences related to each type of attitude. Our two-stage method was evaluated on two data sets, and promising results were obtained. The performance of our method was also compared with the method from previous work.

Which component is most important for a system's survival? We answer this question by ranking the information relationship between a system and its components. The mutual information (M) measures dependence between the operational states of the system and a component for a mission time as well as between their life lengths. This measure ranks each component in terms of its expected utility for predicting the system's survival. We explore some relationships between the ordering of importance of components by M and by Zellner's Maximal Data Information (MDIP) criterion. For many systems the bivariate distribution of the component and system lifetimes does not have a density with respect to the two-dimensional Lebesgue measure. For these systems, M is not defined, so we use a modification of a mutual information index to cover such situations. Our results for ordering dependence are general in terms of binary structures, sum of random variables, and order statistics.

In this paper, the mathematical structure of information filed is main topic for discussion in this paper, for example, mutual information, information strength, relativity and their characters. The paper firstly defines information transmission function, mean information distance and information plane which are used to measure the strength of information transmission, and also put forward their account formulas.

Mutual information is a widely used performance criterion for filter feature selection. However, despite its popularity and its appealing properties, mutual information is not always the most appropriate criterion. Indeed, contrary to what is sometimes hypothesized in the literature, looking for a feature subset maximizing the mutual information does not always guarantee to decrease the misclassification probability, which is often the objective one is interested in. The first objective of this paper is thus to clearly illustrate this potential inadequacy and to emphasize the fact that the mutual information remains a heuristic, coming with no guarantee in terms of classification accuracy. Through extensive experiments, a deeper analysis of the cases for which the mutual information is not a suitable criterion is then conducted. This analysis allows us to confirm the general interest of the mutual information for feature selection. It also helps us better apprehending the behaviour of mutual information throughout a feature selection process and consequently making a better use of it as a feature selection criterion.

Based on the notion of mutual information between the components of a random vector, we construct, for data reduction reasons, an optimal quantization of the support of its probability measure. More precisely, we propose a simultaneous discretization of the whole set of the components of the random vector which takes into account, as much as possible, the stochastic dependence between them. Examples are presented.

The main aim of this contribution is to define the notions of Kullback-Leibler divergence and conditional mutual information in fuzzy probability spaces and to derive the basic properties of the suggested measures. In particular, chain rules for mutual information of fuzzy partitions and for Kullback-Leibler divergence with respect to fuzzy P-measures are established. In addition, a convexity of Kullback-Leibler divergence and mutual information with respect to fuzzy P-measures is studied.

The penalty term plays an important role in model order selection rules. The Exponentially Embedded Families (EEF) is consistent and effective in model order selection. In this paper we show that the EEF penalty term can be viewed as estimated mutual information (MI) between unknown parameters and received data from Bayesian viewpoints. The finding is a result of an important relationship between Kullback-Leibler Divergence (KLD), signal-to-noise ratio (SNR) and MI in estimation/detection of random signals, which is also introduced.

Let $X$, $Y$ be two correlated discrete random variables. We consider an estimation of $X$ from encoded data $\varphi(Y)$ of $Y$ by some encoder function $\varphi(Y)$. We derive an inequality describing a relation of the correct probability of estimation and the mutual information between $X$ and $\varphi(Y)$. This inequality may be useful for the secure analysis of crypto system when we use the success probability of estimating secret data as a security criterion. It also provides an intuitive meaning of the secrecy exponent in the strong secrecy criterion.

Finding good principles to choose the actions of artificial agents like robots in the most beneficial way to optimize their control of the environment is very much in the focus of current research in the field of intelligent systems. Especially in reinforcement learning, where the agent learns through the direct interaction with the environment, a good choice of actions is essential. We propose a new approach that allows a predictive ranking of different action sets with regard to their influence on the learning performance of an artificial agent. Our approach is based on a measure of control that utilizes the concept of mutual information. To evaluate this approach, we investigate its prediction of the effectiveness of different sets of actions in “mediated interaction” scenarios. Our results indicate that the mutual information-based measure can yield useful predictions on the aptitude of action sets for the learning process.

Today's asset management academia and practice is dominated by mean-variance thinking. In consequence, this leads to the quantification of the dependence structure of asset returns by the covariance or the Pearson's correlation coefficient matrix. However, the respective dependence measures are linear by construction and hence unable to detect non-linear dependencies. This article tackles the described concern with regard to the previous publication of Baitinger and Papenbrock (2017). We introduce the mutual information measure, which is an information-theoretic concept and able to detect linear and non-linear dependencies. Next, correlation-based networks are extensively compared to mutual information-based networks. Lastly, the empirical study of Baitinger and Papenbrock (2017) is replicated using mutual information-based networks.

We present a variant of the Taylor diagram, a type of 2D plot that succinctly shows the relationship between two or more random variables based on their variance and correlation. The Taylor diagram has been adopted by the climate and geophysics communities to produce insightful visualizations, e.g., for intercomparison studies. Our variant, which we call the Mutual Information Diagram, represents the relationship between random variables in terms of their entropy and mutual information, and naturally maps well-known statistical quantities to their information-theoretic counterparts. Our new diagram is able to describe non-linear relationships where linear correlation may fail; it allows for categorical and multi-variate data to be compared; and it incorporates the notion of uncertainty, key in the study of large ensembles of data.