Search

The utility of a given experience, like interacting with a particular friend or tasting a particular food, fluctuates continually according to homeostatic and hedonic principles. Consequently, to maximize reward, an individual must be able to escape or attain outcomes as preferences change, by switching between actions. Recent work on human and artificial intelligence has defined such flexible instrumental control in information theoretic terms and postulated that it may serve as a reward surrogate. Another possibility, however, is that the adaptability afforded by flexible control is tacitly implemented by planning for dynamic changes in outcome values. In the current study, an expected utility model that computes decision values over a range of possible monetary gains and losses associated with sensory outcomes provided the best fit to behavioral choice data and performed best in terms of earned rewards. Moreover, consistent with previous work on perceived control and personality, individual differences in dimensional schizotypy were correlated with behavioral choice preferences in conditions with the greatest and lowest levels of flexible control. These results contribute to a growing literature on the role of instrumental control in goal-directed choice., Competing Interests: Declaration of Competing Interest There are no conflicts of interests., (Copyright © 2022 The Author. Published by Elsevier B.V. All rights reserved.)

With the proliferation of 5G mobile networks within next-generation wireless communication, the design and optimization of 5G networks are progressing in the direction of improving the physical layer security (PLS) paradigm. This phenomenon is due to the fact that traditional methods for the network optimization of PLS fail to adapt new features, technologies, and resource management to diversified demand applications. To improve these methods, future 5G and beyond 5G (B5G) networks will need to rely on new enabling technologies. Therefore, approaches for PLS design and optimization that are based on artificial intelligence (AI) and machine learning (ML) have been corroborated to outperform traditional security technologies. This will allow future 5G networks to be more intelligent and robust in order to significantly improve the performance of system design over traditional security methods. With the objective of advancing future PLS research, this review paper presents an elaborate discussion on the design and optimization approaches of wireless PLS techniques. In particular, we focus on both signal processing and information-theoretic security approaches to investigate the optimization techniques and system designs of PLS strategies. The review begins with the fundamental concepts that are associated with PLS, including a discussion on conventional cryptographic techniques and wiretap channel models. We then move on to discuss the performance metrics and basic optimization schemes that are typically adopted in PLS design strategies. The research directions for secure system designs and optimization problems are then reviewed in terms of signal processing, resource allocation and node/antenna selection. Thereafter, the applications of AI and ML technologies in the optimization and design of PLS systems are discussed. In this context, the ML- and AI-based solutions that pertain to end-to-end physical layer joint optimization, secure resource allocation and signal processing methods are presented. We finally conclude with discussions on future trends and technical challenges that are related to the topics of PLS system design and the benefits of AI technologies.

Background : Ever since the seminal work by McCulloch and Pitts, the theory of neural computation and its philosophical foundation known as 'computationalism' have been central to brain-inspired artificial intelligence (AI) technologies. The present study describes neural dynamics and neural coding approaches to understand the mechanisms of neural computation. The primary focus is to characterize the multiscale nature of logic computations in the brain, which might occur at a single neuron level, between neighboring neurons via synaptic transmission, and at the neural circuit level. Results : For this, we begin the analysis with simple neuron models to account for basic Boolean logic operations at a single neuron level and then move on to the phenomenological neuron models to explain the neural computation from the viewpoints of neural dynamics and neural coding. The roles of synaptic transmission in neural computation are investigated using biologically realistic multi-compartment neuron models: two representative computational entities, CA1 pyramidal neuron in the hippocampus and Purkinje fiber in the cerebellum, are analyzed in the information-theoretic framework. We then construct two-dimensional mutual information maps, which demonstrate that the synaptic transmission can process not only basic AND/OR Boolean logic operations but also the linearly non-separable XOR function. Finally, we provide an overview of the evolutionary algorithm and discuss its benefits in automated neural circuit design for logic operations. Conclusions : This study provides a comprehensive perspective on the multiscale logic operations in the brain from both neural dynamics and neural coding viewpoints. It should thus be beneficial for understanding computational principles of the brain and may help design biologically plausible neuron models for AI devices., (© 2021 The Author(s). Published by BRI.)

Cognition-sensing and responding to the environment-is the unifying principle behind the genetic code, origin of life, evolution, consciousness, artificial intelligence, and cancer. However, the conventional model of biology seems to mistake cause and effect. According to the reductionist view, the causal chain in biology is chemicals → code → cognition. Despite this prevailing view, there are no examples in the literature to show that the laws of physics and chemistry can produce codes, or that codes produce cognition. Chemicals are just the physical layer of any information system. In contrast, although examples of cognition generating codes and codes controlling chemicals are ubiquitous in biology and technology, cognition remains a mystery. Thus, the central question in biology is: What is the nature and origin of cognition? In order to elucidate this pivotal question, we must cultivate a deeper understanding of information flows. Through this lens, we see that biological cognition is volitional (i.e., deliberate, intentional, or knowing), and while technology is constrained by deductive logic, living things make choices and generate novel information using inductive logic. Information has been called "the hard problem of life' and cannot be fully explained by known physical principles (Walker et al., 2017). The present paper uses information theory (the mathematical foundation of our digital age) and Turing machines (computers) to highlight inaccuracies in prevailing reductionist models of biology, and proposes that the correct causation sequence is cognition → code → chemicals., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: The author is founder and shareholder of Natural Code LLC, a firm offering $100,000 for discovery of a natural process that generates code from non-living material without direct human agency. The investment group offers $10 million to secure patent rights to said process. The prize specification is at www.naturalcode.org., (Copyright © 2021 The Author. Published by Elsevier Ltd.. All rights reserved.)

The medical knowledge sharing community provides users with an open platform for accessing medical resources and sharing medical knowledge, treatment experience, and emotions. Compared with the recipients of general commodities, the recipients in the medical knowledge sharing community pay more attention to the intensity or overall evaluation of emotional vocabularies in the comments, such as treatment effects, prices, service attitudes, and other aspects. Therefore, the overall evaluation is not a key factor in medical service comments, but the semantics of the emotional polarity is the key to affect recipients of the medical information. In this paper, we propose an adaptive learning emotion identification method (ALEIM) based on mutual information feature weight, which captures the correlation and redundancy of features. In order to evaluate the proposed method's effectiveness, we use four basic corpus libraries crawled from the Haodf's online platform and employ Taiwan University NTUSD Simplified Chinese Emotion Dictionary for emotion classification. The experimental results show that our proposed ALEIM method has a better performance for the identification of the low-frequency words' redundant features in comments of the online medical knowledge sharing community.

Codebook-based learning provides a flexible way to extract the contents of an image in a data-driven manner for visual recognition. One central task in such frameworks is codeword assignment, which allocates local image descriptors to the most similar codewords in the dictionary to generate histogram for categorization. Nevertheless, existing assignment approaches, e.g., nearest neighbors strategy (hard assignment) and Gaussian similarity (soft assignment), suffer from two problems: 1) too strong Euclidean assumption and 2) neglecting the label information of the local descriptors. To address the aforementioned two challenges, we propose a graph assignment method with maximal mutual information (GAMI) regularization. GAMI takes the power of manifold structure to better reveal the relationship of massive number of local features by nonlinear graph metric. Meanwhile, the mutual information of descriptor-label pairs is ultimately optimized in the embedding space for the sake of enhancing the discriminant property of the selected codewords. According to such objective, two optimization models, i.e., inexact-GAMI and exact-GAMI, are respectively proposed in this paper. The inexact model can be efficiently solved with a closed-from solution. The stricter exact-GAMI nonparametrically estimates the entropy of descriptor-label pairs in the embedding space and thus leads to a relatively complicated but still trackable optimization. The effectiveness of GAMI models are verified on both the public and our own datasets.

Identifying the type of stimuli that attracts human visual attention has been an appealing topic for scientists for many years. In particular, marking the salient regions in images is useful for both psychologists and many computer vision applications. In this paper, we propose a computational approach for producing saliency maps using statistics and machine learning methods. Based on four assumptions, three properties (Feature-Prior, Position-Prior, and Feature-Distribution) can be derived and combined by a simple intersection operation to obtain a saliency map. These properties are implemented by a similarity computation, support vector regression (SVR) technique, statistical analysis of training samples, and information theory using low-level features. This technique is able to learn the preferences of human visual behavior while simultaneously considering feature uniqueness. Experimental results show that our approach performs better in predicting human visual attention regions than 12 other models in two test databases., (© 2014 ARVO.)

We propose a general information-theoretic approach to semi-supervised metric learning called SERAPH (SEmi-supervised metRic leArning Paradigm with Hypersparsity) that does not rely on the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize its entropy on labeled data and minimize its entropy on unlabeled data following entropy regularization. For metric learning, entropy regularization improves manifold regularization by considering the dissimilarity information of unlabeled data in the unsupervised part, and hence it allows the supervised and unsupervised parts to be integrated in a natural and meaningful way. Moreover, we regularize SERAPH by trace-norm regularization to encourage low-dimensional projections associated with the distance metric. The nonconvex optimization problem of SERAPH could be solved efficiently and stably by either a gradient projection algorithm or an EM-like iterative algorithm whose M-step is convex. Experiments demonstrate that SERAPH compares favorably with many well-known metric learning methods, and the learned Mahalanobis distance possesses high discriminability even under noisy environments.

A firing rate map, also known as a tuning curve, describes the nonlinear relationship between a neuron's spike rate and a low-dimensional stimulus (e.g., orientation, head direction, contrast, color). Here we investigate Bayesian active learning methods for estimating firing rate maps in closed-loop neurophysiology experiments. These methods can accelerate the characterization of such maps through the intelligent, adaptive selection of stimuli. Specifically, we explore the manner in which the prior and utility function used in Bayesian active learning affect stimulus selection and performance. Our approach relies on a flexible model that involves a nonlinearly transformed gaussian process (GP) prior over maps and conditionally Poisson spiking. We show that infomax learning, which selects stimuli to maximize the information gain about the firing rate map, exhibits strong dependence on the seemingly innocuous choice of nonlinear transformation function. We derive an alternate utility function that selects stimuli to minimize the average posterior variance of the firing rate map and analyze the surprising relationship between prior parameterization, stimulus selection, and active learning performance in GP-Poisson models. We apply these methods to color tuning measurements of neurons in macaque primary visual cortex.

Mixed vascular and Alzheimer-type dementia and pure Alzheimer's disease are both associated with changes in spoken language. These changes have, however, seldom been subjected to systematic comparison. In the present study, we analyzed language samples obtained during the course of a longitudinal clinical study from patients in whom one or other pathology was verified at post mortem. The aims of the study were twofold: first, to confirm the presence of differences in language produced by members of the two groups using quantitative methods of evaluation; and secondly to ascertain the most informative sources of variation between the groups. We adopted a computational approach to evaluate digitized transcripts of connected speech along a range of language-related dimensions. We then used machine learning text classification to assign the samples to one of the two pathological groups on the basis of these features. The classifiers' accuracies were tested using simple lexical features, syntactic features, and more complex statistical and information theory characteristics. Maximum accuracy was achieved when word occurrences and frequencies alone were used. Features based on syntactic and lexical complexity yielded lower discrimination scores, but all combinations of features showed significantly better performance than a baseline condition in which every transcript was assigned randomly to one of the two classes. The classification results illustrate the word content specific differences in the spoken language of the two groups. In addition, those with mixed pathology were found to exhibit a marked reduction in lexical variation and complexity compared to their pure AD counterparts.

We attempt to demonstrate the effectiveness of multiple points of view toward neural networks. By restricting ourselves to two points of view of a neuron, we propose a new type of information-theoretic method called "cooperation-controlled learning." In this method, individual and collective neurons are distinguished from one another, and we suppose that the characteristics of individual and collective neurons are different. To implement individual and collective neurons, we prepare two networks, namely, cooperative and uncooperative networks. The roles of these networks and the roles of individual and collective neurons are controlled by the cooperation parameter. As the parameter is increased, the role of cooperative networks becomes more important in learning, and the characteristics of collective neurons become more dominant. On the other hand, when the parameter is small, individual neurons play a more important role. We applied the method to the automobile and housing data from the machine learning database and examined whether explicit class boundaries could be obtained. Experimental results showed that cooperation-controlled learning, in particular taking into account information on input units, could be used to produce clearer class structure than conventional self-organizing maps.

In some pattern analysis problems, there exists expert knowledge, in addition to the original data involved in the classification process. The vast majority of existing approaches simply ignore such auxiliary (privileged) knowledge. Recently a new paradigm-learning using privileged information-was introduced in the framework of SVM+. This approach is formulated for binary classification and, as typical for many kernel-based methods, can scale unfavorably with the number of training examples. While speeding up training methods and extensions of SVM+ to multiclass problems are possible, in this paper we present a more direct novel methodology for incorporating valuable privileged knowledge in the model construction phase, primarily formulated in the framework of generalized matrix learning vector quantization. This is done by changing the global metric in the input space, based on distance relations revealed by the privileged information. Hence, unlike in SVM+, any convenient classifier can be used after such metric modification, bringing more flexibility to the problem of incorporating privileged information during the training. Experiments demonstrate that the manipulation of an input space metric based on privileged data improves classification accuracy. Moreover, our methods can achieve competitive performance against the SVM+ formulations.

Expression Quantitative Trait Locus (eQTL) analysis is a powerful tool to study the biological mechanisms linking the genotype with gene expression. Such analyses can identify genomic locations where genotypic variants influence the expression of genes, both in close proximity to the variant (cis-eQTL), and on other chromosomes (trans-eQTL). Many traditional eQTL methods are based on a linear regression model. In this study, we propose a novel method by which to identify eQTL associations with information theory and machine learning approaches. Mutual Information (MI) is used to describe the association between genetic marker and gene expression. MI can detect both linear and non-linear associations. What's more, it can capture the heterogeneity of the population. Advanced feature selection methods, Maximum Relevance Minimum Redundancy (mRMR) and Incremental Feature Selection (IFS), were applied to optimize the selection of the affected genes by the genetic marker. When we applied our method to a study of apoE-deficient mice, it was found that the cis-acting eQTLs are stronger than trans-acting eQTLs but there are more trans-acting eQTLs than cis-acting eQTLs. We compared our results (mRMR.eQTL) with R/qtl, and MatrixEQTL (modelLINEAR and modelANOVA). In female mice, 67.9% of mRMR.eQTL results can be confirmed by at least two other methods while only 14.4% of R/qtl result can be confirmed by at least two other methods. In male mice, 74.1% of mRMR.eQTL results can be confirmed by at least two other methods while only 18.2% of R/qtl result can be confirmed by at least two other methods. Our methods provide a new way to identify the association between genetic markers and gene expression. Our software is available from supporting information.

Discovering the structure underlying observed data is a recurring problem in machine learning with important applications in neuroscience. It is also a primary function of the brain. When data can be actively collected in the context of a closed action-perception loop, behavior becomes a critical determinant of learning efficiency. Psychologists studying exploration and curiosity in humans and animals have long argued that learning itself is a primary motivator of behavior. However, the theoretical basis of learning-driven behavior is not well understood. Previous computational studies of behavior have largely focused on the control problem of maximizing acquisition of rewards and have treated learning the structure of data as a secondary objective. Here, we study exploration in the absence of external reward feedback. Instead, we take the quality of an agent's learned internal model to be the primary objective. In a simple probabilistic framework, we derive a Bayesian estimate for the amount of information about the environment an agent can expect to receive by taking an action, a measure we term the predicted information gain (PIG). We develop exploration strategies that approximately maximize PIG. One strategy based on value-iteration consistently learns faster than previously developed reward-free exploration strategies across a diverse range of environments. Psychologists believe the evolutionary advantage of learning-driven exploration lies in the generalized utility of an accurate internal model. Consistent with this hypothesis, we demonstrate that agents which learn more efficiently during exploration are later better able to accomplish a range of goal-directed tasks. We will conclude by discussing how our work elucidates the explorative behaviors of animals and humans, its relationship to other computational models of behavior, and its potential application to experimental design, such as in closed-loop neurophysiology studies.

We present a general analysis of return maximization in reinforcement learning. This analysis does not require assumptions of Markovianity, stationarity, and ergodicity for the stochastic sequential decision processes of reinforcement learning. Instead, our analysis assumes the asymptotic equipartition property fundamental to information theory, providing a substantially different view from that in the literature. As our main results, we show that return maximization is achieved by the overlap of typical and best sequence sets, and we present a class of stochastic sequential decision processes with the necessary condition for return maximization. We also describe several examples of best sequences in terms of return maximization in the class of stochastic sequential decision processes, which satisfy the necessary condition., (Copyright © 2011 Elsevier Ltd. All rights reserved.)

Many trace elements (TE) occur naturally in marine environments and accomplish decisive functions in humans to maintain good health. Mytilus galloprovincialis (MG) is a rich source of TE, but since it is grown near industrial outfalls, they become polluted with elevated levels of TE concentration and serve as biomarkers of pollution. As bioremediation is increasingly reliant on machine learning data processing techniques, we propose the information theoretic concept of using MG for bioremediation. The in situ bioremediation in MG is accomplished by reduction in concentration of TE by the technique of determinant inequalities and the maximization of Mutual Information (MI) without adding any chemical element externally. We bring out the superiority of our technique of MI over that of Principal Component Analysis (PCA) in predicting lower concentration for bioremediation of Cd and Pb in MG., (Copyright © 2011 Elsevier Ltd. All rights reserved.)

Medial representations of shapes are useful due to their use of an object-centered coordinate system that directly captures intuitive notions of shape such as thickness, bending, and elongation. However, it is well known that an object's medial axis transform (MAT) is unstable with respect to small perturbations of its boundary. This instability results in additional, unwanted branches in the skeletons, which must be pruned in order to recover the portions of the skeletons arising purely from the uncorrupted shape information. Almost all approaches to skeleton pruning compute a significance measure for each branch according to some heuristic criteria, and then prune the least significant branches first. Current approaches to branch significance computation can be classified as either local, solely using information from a neighborhood surrounding each branch, or global, using information about the shape as a whole. In this paper, we propose a third, groupwise approach to branch significance computation. We develop a groupwise skeletonization framework that yields a fuzzy significance measure for each branch, derived from information provided by the group of shapes. We call this framework the Groupwise Medial Axis Transform (G-MAT). We propose and evaluate four groupwise methods for computing branch significance and report superior performance compared to a recent, leading method. We measure the performance of each pruning algorithm using denoising, classification, and within-class skeleton similarity measures. This research has several applications, including object retrieval and shape analysis.

For many learning tasks the duration of the data collection can be greater than the time scale for changes of the underlying data distribution. The question we ask is how to include the information that data are aging. Ad hoc methods to achieve this include the use of validity windows that prevent the learning machine from making inferences based on old data. This introduces the problem of how to define the size of validity windows. In this brief, a new adaptive Bayesian inspired algorithm is presented for learning drifting concepts. It uses the analogy of validity windows in an adaptive Bayesian way to incorporate changes in the data distribution over time. We apply a theoretical approach based on information geometry to the classification problem and measure its performance in simulations. The uncertainty about the appropriate size of the memory windows is dealt with in a Bayesian manner by integrating over the distribution of the adaptive window size. Thus, the posterior distribution of the weights may develop algebraic tails. The learning algorithm results from tracking the mean and variance of the posterior distribution of the weights. It was found that the algebraic tails of this posterior distribution give the learning algorithm the ability to cope with an evolving environment by permitting the escape from local traps.

This paper discusses an information theoretic approach of designing sparse kernel adaptive filters. To determine useful data to be learned and remove redundant ones, a subjective information measure called surprise is introduced. Surprise captures the amount of information a datum contains which is transferable to a learning system. Based on this concept, we propose a systematic sparsification scheme, which can drastically reduce the time and space complexity without harming the performance of kernel adaptive filters. Nonlinear regression, short term chaotic time-series prediction, and long term time-series forecasting examples are presented.

We propose a new matrix learning scheme to extend relevance learning vector quantization (RLVQ), an efficient prototype-based classification algorithm, toward a general adaptive metric. By introducing a full matrix of relevance factors in the distance measure, correlations between different features and their importance for the classification scheme can be taken into account and automated, and general metric adaptation takes place during training. In comparison to the weighted Euclidean metric used in RLVQ and its variations, a full matrix is more powerful to represent the internal structure of the data appropriately. Large margin generalization bounds can be transferred to this case, leading to bounds that are independent of the input dimensionality. This also holds for local metrics attached to each prototype, which corresponds to piecewise quadratic decision boundaries. The algorithm is tested in comparison to alternative learning vector quantization schemes using an artificial data set, a benchmark multiclass problem from the UCI repository, and a problem from bioinformatics, the recognition of splice sites for C. elegans.

According to the integrated information theory, the quantity of consciousness is the amount of integrated information generated by a complex of elements, and the quality of experience is specified by the informational relationships it generates. This paper outlines a framework for characterizing the informational relationships generated by such systems. Qualia space (Q) is a space having an axis for each possible state (activity pattern) of a complex. Within Q, each submechanism specifies a point corresponding to a repertoire of system states. Arrows between repertoires in Q define informational relationships. Together, these arrows specify a quale -- a shape that completely and univocally characterizes the quality of a conscious experience. Phi -- the height of this shape -- is the quantity of consciousness associated with the experience. Entanglement measures how irreducible informational relationships are to their component relationships, specifying concepts and modes. Several corollaries follow from these premises. The quale is determined by both the mechanism and state of the system. Thus, two different systems having identical activity patterns may generate different qualia. Conversely, the same quale may be generated by two systems that differ in both activity and connectivity. Both active and inactive elements specify a quale, but elements that are inactivated do not. Also, the activation of an element affects experience by changing the shape of the quale. The subdivision of experience into modalities and submodalities corresponds to subshapes in Q. In principle, different aspects of experience may be classified as different shapes in Q, and the similarity between experiences reduces to similarities between shapes. Finally, specific qualities, such as the "redness" of red, while generated by a local mechanism, cannot be reduced to it, but require considering the entire quale. Ultimately, the present framework may offer a principled way for translating qualitative properties of experience into mathematics.

We present a new, geometric approach for determining the probability density of the intensity values in an image. We drop the notion of an image as a set of discrete pixels, and assume a piecewise-continuous representation. The probability density can then be regarded as being proportional to the area between two nearby isocontours of the image surface. Our paper extends this idea to joint densities of image pairs. We demonstrate the application of our method to affine registration between two or more images using information theoretic measures such as mutual information. We show cases where our method outperforms existing methods such as simple histograms, histograms with partial volume interpolation, Parzen windows, etc. under fine intensity quantization for affine image registration under significant image noise. Furthermore, we demonstrate results on simultaneous registration of multiple images, as well as for pairs of volume datasets, and show some theoretical properties of our density estimator. Our approach requires the selection of only an image interpolant. The method neither requires any kind of kernel functions (as in Parzen windows) which are unrelated to the structure of the image in itself, nor does it rely on any form of sampling for density estimation.

An essential component of many medical image analysis protocols is the establishment and manipulation of feature correspondences. These image features can assume such forms spanning the range of functions of individual or regional pixel intensities to geometric structures extracted as a preprocessing segmentation step. Many algorithms focusing on the latter set of salient features attempt to reduce these structures to such geometric primitives as surfaces, curves and/or points for correspondence-based study. Although the latter geometric primitive forms the basis of many of these algorithms, unrealistic constraints such as assumptions of identical cardinality between point-sets hinder general usage. Furthermore, the local structure for certain point-sets derived from segmentation processes is often ignored. In this paper, we introduce a family of novel information-theoretic measures for pooint-set registration derived as a generalization of the well-known Shannon entropy known as the Havrda-Charvat-Tsallis entropy. This divergence measure permits a fine-tuning between robustness and sensitivity emphasis. In addition, we employ a directly manipulated free-form deformation (DMFFD) transformation model, a recently developed variant of the well-known FFD transformation model.

Competitive learning neural networks are regarded as a powerful tool for online data clustering to represent a non-stationary probability distribution with a fixed number of weight vectors. One difficulty in practical applications of competitive learning neural networks to online data clustering is that most of them require heuristically-predetermined threshold parameters for balancing a trade-off between convergence accuracy, i.e. error minimization performance, and speed of adaptation to the changes in source statistics. Although adaptation acceleration is achievable by relocating a "useless" node so that it becomes useful, excessive relocation often disturbs error minimization. Hence, both of the adaptation speed and the error minimization performance sensitively depend on threshold parameters to determine whether a node should be relocated or not. In general, it is difficult to know adequate threshold parameters a priori. This paper proposes a novel criterion for decision making of node relocation without heuristically predetermined thresholds. According to the proposed criterion, a node is relocated only if the relocation task improves partial distortion entropy, which is an online optimality metric reliable from the viewpoint of error minimization. Hence, node relocation is carried out without disturbing error minimization. As a result, both quick adaptation and error minimization are simultaneously accomplished without any carefully predefined parameters. Experimental results clarify the validity of the proposed criterion. Competitive learning with the criterion is clearly superior to other representative algorithms in terms of both quick adaptation and error minimization performance.

Motivation: Despite advances in the gene annotation process, the functions of a large portion of gene products remain insufficiently characterized. In addition, the in silico prediction of novel Gene Ontology (GO) annotations for partially characterized gene functions or processes is highly dependent on reverse genetic or functional genomic approaches. To our knowledge, no prediction method has been demonstrated to be highly accurate for sparsely annotated GO terms (those associated to fewer than 10 genes)., Results: We propose a novel approach, information theory-based semantic similarity (ITSS), to automatically predict molecular functions of genes based on existing GO annotations. Using a 10-fold cross-validation, we demonstrate that the ITSS algorithm obtains prediction accuracies (precision 97%, recall 77%) comparable to other machine learning algorithms when compared in similar conditions over densely annotated portions of the GO datasets. This method is able to generate highly accurate predictions in sparsely annotated portions of GO, where previous algorithms have failed. As a result, our technique generates an order of magnitude more functional predictions than previous methods. A 10-fold cross validation demonstrated a precision of 90% at a recall of 36% for the algorithm over sparsely annotated networks of the recent GO annotations (about 1400 GO terms and 11,000 genes in Homo sapiens). To our knowledge, this article presents the first historical rollback validation for the predicted GO annotations, which may represent more realistic conditions than more widely used cross-validation approaches. By manually assessing a random sample of 100 predictions conducted in a historical rollback evaluation, we estimate that a minimum precision of 51% (95% confidence interval: 43-58%) can be achieved for the human GO Annotation file dated 2003., Availability: The program is available on request. The 97,732 positive predictions of novel gene annotations from the 2005 GO Annotation dataset and other supplementary information is available at http://phenos.bsd.uchicago.edu/ITSS/., Supplementary Information: Supplementary data are available at Bioinformatics online.

We propose a new information-theoretic metric, the symmetric Kullback-Leibler divergence (sKL-divergence), to measure the difference between two water diffusivity profiles in high angular resolution diffusion imaging (HARDI). Water diffusivity profiles are modeled as probability density functions on the unit sphere, and the sKL-divergence is computed from a spherical harmonic series, which greatly reduces computational complexity. Adjustment of the orientation of diffusivity functions is essential when the image is being warped, so we propose a fast algorithm to determine the principal direction of diffusivity functions using principal component analysis (PCA). We compare sKL-divergence with other inner-product based cost functions using synthetic samples and real HARDI data, and show that the sKL-divergence is highly sensitive in detecting small differences between two diffusivity profiles and therefore shows promise for applications in the nonlinear registration and multisubject statistical analysis of HARDI data.

The purpose of this study was to evaluate image similarity measures employed in an information-theoretic computer-assisted detection (IT-CAD) scheme. The scheme was developed for content-based retrieval and detection of masses in screening mammograms. The study is aimed toward an interactive clinical paradigm where physicians query the proposed IT-CAD scheme on mammographic locations that are either visually suspicious or indicated as suspicious by other cuing CAD systems. The IT-CAD scheme provides an evidence-based, second opinion for query mammographic locations using a knowledge database of mass and normal cases. In this study, eight entropy-based similarity measures were compared with respect to retrieval precision and detection accuracy using a database of 1820 mammographic regions of interest. The IT-CAD scheme was then validated on a separate database for false positive reduction of progressively more challenging visual cues generated by an existing, in-house mass detection system. The study showed that the image similarity measures fall into one of two categories; one category is better suited to the retrieval of semantically similar cases while the second is more effective with knowledge-based decisions regarding the presence of a true mass in the query location. In addition, the IT-CAD scheme yielded a substantial reduction in false-positive detections while maintaining high detection rate for malignant masses.

Bayesian statistics and information theory are used to analyze the reliability of extracting chemical shift parameters from spinning sideband patterns of spin 1/2 systems. Efficient code has been written to calculate the two-dimensional posterior probability as a function of the chemical shift anisotropy, delta, and the asymmetry parameter, eta, given the sideband intensities and the signal-to-noise ratio. This method has the advantage of assuming only that the noise in the sideband intensities is distributed as a Gaussian. It assumes nothing about the distribution of the values of parameters delta and eta, which are shown in some cases to be highly non-Gaussian. The utility of Bayesian analysis is demonstrated on 1D slow-spinning MAS spectra and on sideband patterns extracted from 2D PASS spectra. Previous investigations have shown that there is an optimal range of spinning frequencies for determining delta. In this study, information theory is used to determine the signal-to-noise ratio dependence of the entropy in delta, eta, and total entropy in spinning sideband spectra. The entropy is a measure of the information content of a probability distribution. When the entropy is zero, there is perfect information on a system, while if it is infinite, there is no information on the system. It is found that for all values of eta and for signal-to-noise ratios in the range 50-1000, an entropy minimum in nudelta/nur occurs for values 1.5

Decomposition methods are well-known techniques for solving quadratic programming (QP) problems arising in support vector machines (SVMs). In each iteration of a decomposition method, a small number of variables are selected and a QP problem with only the selected variables is solved. Since large matrix computations are not required, decomposition methods are applicable to large QP problems. In this paper, we will make a rigorous analysis of the global convergence of general decomposition methods for SVMs. We first introduce a relaxed version of the optimality condition for the QP problems and then prove that a decomposition method reaches a solution satisfying this relaxed optimality condition within a finite number of iterations under a very mild condition on how to select variables.

This paper presents a new framework for sensor reliability evaluation in classification problems based on evidence theory (or the Dempster-Shafer theory of belief functions). The evaluation is treated as a two-stage training process. First, the authors assess the static reliability from a training set by comparing the sensor classification readings with the actual values of data, which are both represented by belief functions. Information content contained in the actual values of each target is extracted to determine its influence on the evaluation. Next, considering the ability of the sensor to understand a dynamic working environment, the dynamic reliability is evaluated by measuring the degree of consensus among a group of sensors. Finally, the authors discuss why and how to combine these two kinds of reliabilities. A significant improvement using the authors' method is observed in numerical simulations as compared with the recently proposed method.

A classification system typically consists of both a feature extractor (preprocessor) and a classifier. These two components can be trained either independently or simultaneously. The former option has an implementation advantage since the extractor need only be trained once for use with any classifier, whereas the latter has an advantage since it can be used to minimize classification error directly. Certain criteria, such as Minimum Classification Error, are better suited for simultaneous training, whereas other criteria, such as Mutual Information, are amenable for training the feature extractor either independently or simultaneously. Herein, an information-theoretic criterion is introduced and is evaluated for training the extractor independently of the classifier. The proposed method uses nonparametric estimation of Renyi's entropy to train the extractor by maximizing an approximation of the mutual information between the class labels and the output of the feature extractor. The evaluations show that the proposed method, even though it uses independent training, performs at least as well as three feature extraction methods that train the extractor and classifier simultaneously.

We present a novel, variational and statistical approach for shape registration. Shapes of interest are implicitly embedded in a higher-dimensional space of distance transforms. In this implicit embedding space, registration is formulated in a hierarchical manner: the Mutual Information criterion supports various transformation models and is optimized to perform global registration; then, a B-spline-based Incremental Free Form Deformations (IFFD) model is used to minimize a Sum-of-Squared-Differences (SSD) measure and further recover a dense local nonrigid registration field. The key advantage of such framework is twofold: (1) it naturally deals with shapes of arbitrary dimension (2D, 3D, or higher) and arbitrary topology (multiple parts, closed/open) and (2) it preserves shape topology during local deformation and produces local registration fields that are smooth, continuous, and establish one-to-one correspondences. Its invariance to initial conditions is evaluated through empirical validation, and various hard 2D/3D geometric shape registration examples are used to show its robustness to noise, severe occlusion, and missing parts. We demonstrate the power of the proposed framework using two applications: one for statistical modeling of anatomical structures, another for 3D face scan registration and expression tracking. We also compare the performance of our algorithm with that of several other well-known shape registration algorithms.

In this paper the influence of intensity clustering and shading correction on mutual information based image registration is studied. Instead of the generally used equidistant re-binning, we use k-means clustering in order to achieve a more natural binning of the intensity distribution. Secondly, image inhomogeneities occurring notably in MR images can have adverse effects on the registration. We use a shading correction method in order to reduce these effects. The method is validated on clinical MR, CT and PET images, as well as synthetic MR images. It is shown that by employing clustering with inhomogeneity correction the number of misregistrations is reduced without loss of accuracy thus increasing robustness as compared to the standard non-inhomogeneity corrected and equidistant binning based registration.

We propose two information theoretic similarity measures that allow to incorporate tissue class information in non-rigid image registration. The first measure assumes that tissue class probabilities have been assigned to each of the images to be registered by prior segmentation of both of them. One image is then non-rigidly deformed to match the other such that the fuzzy overlap of corresponding voxel object labels becomes similar to the ideal case whereby the tissue probability maps of both images are identical. Image similarity is assessed during registration by the divergence between the ideal and actual joint class probability distributions of both images. A second registration measure is proposed that applies in case a segmentation is available for only one of the images, for instance an atlas image that is to be matched to a study image to guide the segmentation thereof. Intensities in one image are matched to the fuzzy class labels in the other image by minimizing the conditional entropy of the intensities in the first image given the class labels in the second image. We derive analytic expressions for the gradient of each measure with respect to individual voxel displacements to derive a force field that drives the registration process, which is regularized by a viscous fluid model. The performance of the class-based measures is evaluated in the context of non-rigid inter-subject registration and atlas-based segmentation of MR brain images and compared with maximization of mutual information using only intensity information. Our results demonstrate that incorporation of class information in the registration measure significantly improves the overlap between corresponding tissue classes after non-rigid matching. The methods proposed here open new perspectives for integrating segmentation and registration in a single process, whereby the output of one is used to guide the other.

The paper deals with the concept of relevance learning in learning vector quantization and classification. Recent machine learning approaches with the ability of metric adaptation but based on different concepts are considered in comparison to variants of relevance learning vector quantization. We compare these methods with respect to their theoretical motivation and we demonstrate the differences of their behavior for several real world data sets.

In this paper, we combine discrete and continuous image models with information-theoretic-based criteria for unsupervised hierarchical image-set clustering. The continuous image modeling is based on mixture of Gaussian densities. The unsupervised image-set clustering is based on a generalized version of a recently introduced information-theoretic principle, the information bottleneck principle. Images are clustered such that the mutual information between the clusters and the image content is maximally preserved. Experimental results demonstrate the performance of the proposed framework for image clustering on a large image set. Information theoretic tools are used to evaluate cluster quality. Particular emphasis is placed on the application of the clustering for efficient image search and retrieval.

The detection of image features is an essential component of medical image processing, and has wide-ranging applications including adaptive filtering, segmentation, and registration. In this paper, we present an information-theoretic approach to feature detection in ultrasound images. Ultrasound images are corrupted by speckle noise, which is a disruptive random pattern that obscures the features of interest. Using theoretical probability density functions of the speckle intensity distributions, we derive analytic expressions that measure the distance between distributions taken from different regions in an ultrasound image and use these distances to detect features. We compare the technique to classic gradient-based feature detection methods.

A new training algorithm called the approximated maximum mutual information (AMMI) is proposed to improve the accuracy of myoelectric speech recognition using hidden Markov models (HMMs). Previous studies have demonstrated that automatic speech recognition can be performed using myoelectric signals from articulatory muscles of the face. Classification of facial myoelectric signals can be performed using HMMs that are trained using the maximum likelihood (ML) algorithm; however, this algorithm maximizes the likelihood of the observations in the training sequence, which is not directly associated with optimal classification accuracy. The AMMI training algorithm attempts to maximize the mutual information, thereby training the HMMs to optimize their parameters for discrimination. Our results show that AMMI training consistently reduces the error rates compared to these by the ML training, increasing the accuracy by approximately 3% on average.

In this paper, we present a new information-theoretic approach to image segmentation. We cast the segmentation problem as the maximization of the mutual information between the region labels and the image pixel intensities, subject to a constraint on the total length of the region boundaries. We assume that the probability densities associated with the image pixel intensities within each region are completely unknown a priori, and we formulate the problem based on nonparametric density estimates. Due to the nonparametric structure, our method does not require the image regions to have a particular type of probability distribution and does not require the extraction and use of a particular statistic. We solve the information-theoretic optimization problem by deriving the associated gradient flows and applying curve evolution techniques. We use level-set methods to implement the resulting evolution. The experimental results based on both synthetic and real images demonstrate that the proposed technique can solve a variety of challenging image segmentation problems. Futhermore, our method, which does not require any training, performs as good as methods based on training.

Supervised learning is conventionally performed with pairwise input-output labeled data. After the training procedure, the adaptive system's weights are fixed while the testing procedure with unlabeled data is performed. Recently, in an attempt to improve classification performance unlabeled data has been exploited in the machine learning community. In this paper, we present an information theoretic learning (ITL) approach based on density divergence minimization to obtain an extended training algorithm using unlabeled data during the testing. The method uses a boosting-like algorithm with an ITL based cost function. Preliminary simulations suggest that the method has the potential to improve the performance of classifiers in the application phase.

In this paper, we introduce costs into the framework of information maximization and try to maximize the ratio of information to its associated cost. We have shown that competitive learning is realized by maximizing mutual information between input patterns and competitive units. One shortcoming of the method is that maximizing information does not necessarily produce representations faithful to input patterns. Information maximizing primarily focuses on some parts of input patterns that are used to distinguish between patterns. Therefore, we introduce the cost, which represents average distance between input patterns and connection weights. By minimizing the cost, final connection weights reflect input patterns well. We applied the method to a political data analysis, a voting attitude problem and a Wisconsin cancer problem. Experimental results confirmed that, when the cost was introduced, representations faithful to input patterns were obtained. In addition, improved generalization performance was obtained within a relatively short learning time.

We have proposed earlier the incremental internode communication method to reduce the communication cost as well as the time of the learning process in artificial neural networks (ANNs). In this paper, the limited precision incremental communication method is applied to a class of recurrent neural networks, the adaptive resonance theory 2 (ART2) networks. Simulation studies are carried out to examine the effects of the incremental communication method on the convergence behavior of ART2 networks. We have found that, 7-13-b precision is sufficient to obtain almost the same results as those with full (32-b) precision conventional communication. A theoretical error analysis is also carried out to analyze the effects of the limited precision incremental communication. The simulation and analytical results show that the limited precision errors are bounded and do not seriously degrade the convergence of ART2 networks. Therefore, the incremental communication can be incorporated in parallel and special-purpose very large scale integration (VLSI) implementations of the ART2 networks.

The non-rigid registration of a group of images shares a common feature with building a model of a group of images: a dense, consistent correspondence across the group. Image registration aims to find the correspondence, while modelling requires it. This paper presents the theoretical framework required to unify these two areas, providing a groupwise registration algorithm, where the inherently groupwise model of the image data becomes an integral part of the registration process. The performance of this algorithm is evaluated by extending the concepts of generalisability and specificity from shape models to image models. This provides an independent metric for comparing registration algorithms of groups of images. Experimental results on MR data of brains for various pairwise and groupwise registration algorithms is presented, and demonstrates the feasibility of the combined registration/modelling framework, as well as providing quantitative evidence for the superiority of groupwise approaches to registration.

Under-complete models, which derive lower dimensional representations of input data, are valuable in domains in which the number of input dimensions is very large, such as data consisting of a temporal sequence of images. This paper presents the under-complete product of experts (UPoE), where each expert models a one-dimensional projection of the data. Maximum-likelihood learning rules for this model constitute a tractable and exact algorithm for learning under-complete independent components. The learning rules for this model coincide with approximate learning rules proposed earlier for under-complete independent component analysis (UICA) models. This paper also derives an efficient sequential learning algorithm from this model and discusses its relationship to sequential independent component analysis (ICA), projection pursuit density estimation, and feature induction algorithms for additive random field models. This paper demonstrates the efficacy of these novel algorithms on high-dimensional continuous datasets.

In this paper, we introduce and investigate a new adaptive equalization method based on minimizing approximate negentropy of the estimation error for a finite-length equalizer. We consider an approximate negentropy using nonpolynomial expansions of the estimation error as a new performance criterion to improve performance of a linear equalizer based on minimizing minimum mean squared error (MMSE). Negentropy includes higher order statistical information and its minimization provides improved converge, performance and accuracy compared to traditional methods such as MMSE in terms of bit error rate (BER). The proposed negentropy minimization (NEGMIN) equalizer has two kinds of solutions, the MMSE solution and the other one, depending on the ratio of the normalization parameters. The NEGMIN equalizer has best BER performance when the ratio of the normalization parameters is properly adjusted to maximize the output power(variance) of the NEGMIN equalizer. Simulation experiments show that BER performance of the NEGMIN equalizer with the other solution than the MMSE one has similar characteristics to the adaptive minimum bit error rate (AMBER) equalizer. The main advantage of the proposed equalizer is that it needs significantly fewer training symbols than the AMBER equalizer. Furthermore, the proposed equalizer is more robust to nonlinear distortions than the MMSE equalizer.

Recent publications have proposed various information-theoretic learning (ITL) criteria based on Renyi's quadratic entropy with nonparametric kernel-based density estimation as alternative performance metrics for both supervised and unsupervised adaptive system training. These metrics, based on entropy and mutual information, take into account higher order statistics unlike the mean-square error (MSE) criterion. The drawback of these information-based metrics is the increased computational complexity, which underscores the importance of efficient training algorithms. In this paper, we examine familiar advanced-parameter search algorithms and propose modifications to allow training of systems with these ITL criteria. The well known algorithms tailored here for ITL include various improved gradient-descent methods, conjugate gradient approaches, and the Levenberg-Marquardt (LM) algorithm. Sample problems and metrics are presented to illustrate the computational efficiency attained by employing the proposed algorithms.

This paper derives a family of differential learning rules that optimize the Shannon entropy at the output of an adaptive system via kernel density estimation. In contrast to parametric formulations of entropy, this nonparametric approach assumes no particular functional form of the output density. We address problems associated with quantized data and finite sample size, and implement efficient maximum likelihood techniques for optimizing the regularizer. We also develop a normalized entropy estimate that is invariant with respect to affine transformations, facilitating optimization of the shape, rather than the scale, of the output density. Kernel density estimates are smooth and differentiable; this makes the derived entropy estimates amenable to manipulation by gradient descent. The resulting weight updates are surprisingly simple and efficient learning rules that operate on pairs of input samples. They can be tuned for data-limited or memory-limited situations, or modified to give a fully online implementation.

In this paper, we regard the sequence of returns as outputs from a parametric compound source. Utilizing the fact that the coding rate of the source shows the amount of information about the return, we describe l-learning algorithms based on the predictive coding idea for estimating an expected information gain concerning future information and give a convergence proof of the information gain. Using the information gain, we propose the ratio w of return loss to information gain as a new criterion to be used in probabilistic action-selection strategies. In experimental results, we found that our w-based strategy performs well compared with the conventional Q-based strategy.

This paper reports a study on the problem of the blind simultaneous extraction of specific groups of independent components from a linear mixture. This paper first presents a general overview and unification of several information theoretic criteria for the extraction of a single independent component. Then, our contribution fills the theoretical gap that exists between extraction and separation by presenting tools that extend these criteria to allow the simultaneous blind extraction of subsets with an arbitrary number of independent components. In addition, we analyze a family of learning algorithms based on Stiefel manifolds and the natural gradient ascent, present the nonlinear optimal activations (score) functions, and provide new or extended local stability conditions. Finally, we illustrate the performance and features of the proposed approach by computer-simulation experiments.

A new information-theoretic learning algorithm for kernel-based topographic map formation is introduced. In the one-dimensional case, the algorithm is aimed at uniformizing the cumulative distribution of the kernel mixture densities by maximizing its differential entropy. A nonparametric differential entropy estimator is used on which normalized gradient ascent is performed. Both differentiable and nondifferentiable kernels are in principle supported, such as Gaussian and rectangular (on/off) kernels. The relation is shown with joint entropy maximization of the kernel outputs. The learning algorithm's performance is assessed and compared with the theoretically optimal performance. A fixed-point rule is derived for the case of heterogeneous kernel mixtures. Finally, an extension of the algorithm to the multidimensional case is suggested.