Search

Application of interpretable machine learning techniques on medical datasets facilitate early and fast diagnoses, along with getting deeper insight into the data. Furthermore, the transparency of these models increase trust among application domain experts. Medical datasets face common issues such as heterogeneous measurements, imbalanced classes with limited sample size, and missing data, which hinder the straightforward application of machine learning techniques. In this paper we present a family of prototype-based (PB) interpretable models which are capable of handling these issues. The models introduced in this contribution show comparable or superior performance to alternative techniques applicable in such situations. However, unlike ensemble based models, which have to compromise on easy interpretation, the PB models here do not. Moreover we propose a strategy of harnessing the power of ensembles while maintaining the intrinsic interpretability of the PB models, by averaging the model parameter manifolds. All the models were evaluated on a synthetic (publicly available dataset) in addition to detailed analyses of two real-world medical datasets (one publicly available). Results indicated that the models and strategies we introduced addressed the challenges of real-world medical data, while remaining computationally inexpensive and transparent, as well as similar or superior in performance compared to their alternatives.

Funder: Global Alliance, The early stages of Alzheimer’s disease (AD) involve interactions between multiple pathophysiological processes. Although these processes are well studied, we still lack robust tools to predict individualised trajectories of disease progression. Here, we employ a robust and interpretable machine learning approach to combine multimodal biological data and predict future pathological tau accumulation. In particular, we use machine learning to quantify interactions between key pathological markers (β-amyloid, medial temporal lobe atrophy, tau and APOE 4) at mildly impaired and asymptomatic stages of AD. Using baseline non-tau markers we derive a prognostic index that: (a) stratifies patients based on future pathological tau accumulation, (b) predicts individualised regional future rate of tau accumulation, and (c) translates predictions from deep phenotyping patient cohorts to cognitively normal individuals. Our results propose a robust approach for fine scale stratification and prognostication with translation impact for clinical trial design targeting the earliest stages of AD.

The early stages of Alzheimer's disease (AD) involve interactions between multiple pathophysiological processes. Although these processes are well studied, we still lack robust tools to predict individualised trajectories of disease progression. Here, we employ a robust and interpretable machine learning approach to combine multimodal biological data and predict future pathological tau accumulation. In particular, we use machine learning to quantify interactions between key pathological markers (β-amyloid, medial temporal lobe atrophy, tau and APOE 4) at mildly impaired and asymptomatic stages of AD. Using baseline non-tau markers we derive a prognostic index that: (a) stratifies patients based on future pathological tau accumulation, (b) predicts individualised regional future rate of tau accumulation, and (c) translates predictions from deep phenotyping patient cohorts to cognitively normal individuals. Our results propose a robust approach for fine scale stratification and prognostication with translation impact for clinical trial design targeting the earliest stages of AD., This work was supported by grants to: Z.K. from the Biotechnology and Biological Sciences Research Council (H012508 and BB/P021255/1), Alan Turing Institute (TU/B/000095), Wellcome Trust (205067/Z/16/Z, 221633/Z/20/Z), Royal Society (INF\R2\202107); Z.K. and W.J.J. from the Global Alliance; W.J.J. US National Institute on Aging (AG034570, AG062542, AG024904). Data collection and sharing for this project was funded by the Alzheimer's Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012).

In modern business processes, the amount of data collected has increased substantially in recent years. Because this data can potentially yield valuable insights, automated knowledge extraction based on process mining has been proposed, among other techniques, to provide users with intuitive access to the information contained therein. At present, the majority of technologies aim to reconstruct explicit business process models. These are directly interpretable but limited concerning the integration of diverse and real-valued information sources. On the other hand, Machine Learning (ML) benefits from the vast amount of data available and can deal with high-dimensional sources, yet it has rarely been applied to being used in processes. In this contribution, we evaluate the capability of modern Transformer architectures as well as more classical ML technologies of modeling process regularities, as can be quantitatively evaluated by their prediction capability. In addition, we demonstrate the capability of attentional properties and feature relevance determination by highlighting features that are crucial to the processes’ predictive abilities. We demonstrate the efficacy of our approach using five benchmark datasets and show that the ML models are capable of predicting critical outcomes and that the attention mechanisms or XAI components offer new insights into the underlying processes.

Pretraining language models on large text corpora is a common practice in natural language processing. Fine-tuning of these models is then performed to achieve the best results on a variety of tasks. In this paper, we investigate the problem of catastrophic forgetting in transformer neural networks and question the common practice of fine-tuning with a flat learning rate for the entire network in this context. We perform a hyperparameter optimization process to find learning rate distributions that are better than a flat learning rate. We combine the learning rate distributions thus found and show that they generalize to better performance with respect to the problem of catastrophic forgetting. We validate these learning rate distributions with a variety of NLP benchmarks from the GLUE dataset. The source code is open-source and free software, available at https://github.com/TheMody/NAS-CatastrophicForgetting.

The use of primary care electronic health records for research is abundant. The benefits gained from utilising such records lies in their size, longitudinal data collection and data quality. However, the use of such data to undertake high quality epidemiological studies, can lead to significant challenges particularly in dealing with misclassification, variation in coding and the significant effort required to pre-process the data in a meaningful format for statistical analysis. In this paper, we describe a methodology to aid with the extraction and processing of such databases, delivered by a novel software programme; the “Data extraction for epidemiological research” (DExtER). The basis of DExtER relies on principles of extract, transform and load processes. The tool initially provides the ability for the healthcare dataset to be extracted, then transformed in a format whereby data is normalised, converted and reformatted. DExtER has a user interface designed to obtain data extracts specific to each research question and observational study design. There are facilities to input the requirements for; eligible study period, definition of exposed and unexposed groups, outcome measures and important baseline covariates. To date the tool has been utilised and validated in a multitude of settings. There have been over 35 peer-reviewed publications using the tool, and DExtER has been implemented as a validated public health surveillance tool for obtaining accurate statistics on epidemiology of key morbidities. Future direction of this work will be the application of the framework to linked as well as international datasets and the development of standardised methods for conducting electronic pre-processing and extraction from datasets for research purposes. Electronic supplementary material The online version of this article (10.1007/s10654-020-00677-6) contains supplementary material, which is available to authorized users.

The study of low-dimensional, noisy manifolds embedded in a higher dimensional space has been extremely useful in many applications, from the chemical analysis of multi-phase flows to simulations of galactic mergers. Building a probabilistic model of the manifolds has helped in describing their essential properties and how they vary in space. However, when the manifold is evolving through time, a joint spatio-temporal modelling is needed, in order to fully comprehend its nature. We propose a first-order Markovian process that propagates the spatial probabilistic model of a manifold at fixed time, to its adjacent temporal stages. The proposed methodology is demonstrated using a particle simulation of an interacting dwarf galaxy to describe the evolution of a cavity generated by a Supernova

Level crossings have a function to let the traffic cross the railroad from one side to the other. In the Netherlands, 2300 level crossings are spread out over the country, playing a significant role in daily traffic. Currently, there isn’t an accurate estimation of the arrival time of trains at level crossings while it plays an important role in traffic flow management in intelligent transport systems. This paper presents a state-of-the-art deep learning model for predicting the arrival time of trains at level crossings using spatial and temporal aspects, external attributes, and multi-task learning. The spatial and temporal aspects incorporate geographical and historical travel data and the attributes provide specific information about a train route. Using multi-task learning all the information is combined and an arrival time prediction is made both for the entire route as for sub-parts of that route. Experimental results show that on average, the error is only 281 s with an average trip time of one hour. The model is able to accurately predict the arrival time at level crossings for various time steps in advance. The source code is available at https://github.com/basbuijse/train-arrival-time-estimator.

Topological data analysis tools enjoy increasing popularity in a wide range of applications. However, due to computational complexity, processing large number of samples of higher dimensionality quickly becomes infeasible. We propose a novel sub-sampling strategy inspired by Coulomb’s law to decrease the number of data points in d-dimensional point clouds while preserving its Homology. The method is not only capable of reducing the memory and computation time needed for the construction of different types of simplicial complexes but also preserves the size of the voids in d-dimensions, which is crucial e.g. for astronomical applications. We demonstrate and compare the strategy in several synthetic scenarios and an astronomical particle simulation of a Jellyfish galaxy for the detection of superbubbles (supernova signatures).

Making predictions about future events relies on interpreting streams of information that may initially appear incomprehensible. This skill relies on extracting regular patterns in space and time by mere exposure to the environment (i.e., without explicit feedback). Yet, we know little about the functional brain networks that mediate this type of statistical learning. Here, we test whether changes in the processing and connectivity of functional brain networks due to training relate to our ability to learn temporal regularities. By combining behavioral training and functional brain connectivity analysis, we demonstrate that individuals adapt to the environment's statistics as they change over time from simple repetition to probabilistic combinations. Further, we show that individual learning of temporal structures relates to decision strategy. Our fMRI results demonstrate that learning-dependent changes in fMRI activation within and functional connectivity between brain networks relate to individual variability in strategy. In particular, extracting the exact sequence statistics (i.e., matching) relates to changes in brain networks known to be involved in memory and stimulus-response associations, while selecting the most probable outcomes in a given context (i.e., maximizing) relates to changes in frontal and striatal networks. Thus, our findings provide evidence that dissociable brain networks mediate individual ability in learning behaviorally-relevant statistics.

The increasing occurrence of ordinal data, mainly sociodemographic, led to a renewed research interest in ordinal regression, i.e. the prediction of ordered classes. Besides model accuracy, the interpretation of these models itself is of high relevance, and existing approaches therefore enforce e.g. model sparsity. For high dimensional or highly correlated data, however, this might be misleading due to strong variable dependencies. In this contribution, we aim for an identification of feature relevance bounds which - besides identifying all relevant features - explicitly differentiates between strongly and weakly relevant features., preprint of a paper accepted for oral presentation at the 27th European Symposium on Artificial Neural Networks (ESANN 2019)

We present Extended Riemannian Stochastic Derivative-Free Optimization (Extended RSDFO), a novel population-based stochastic optimization algorithm on Riemannian manifolds that addresses the locality and implicit assumptions of manifold optimization in the literature. We begin by investigating the Information Geometrical structure of statistical model over Riemannian manifolds. This establishes a geometrical framework of Extended RSDFO using both the statistical geometry of the decision space and the Riemannian geometry of the search space. We construct locally inherited probability distribution via an orientation-preserving diffeomorphic bundle morphism, and then extend the information geometrical structure to mixture densities over totally bounded subsets of manifolds. The former relates the information geometry of the decision space and the local point estimations on the search space manifold. The latter overcomes the locality of parametric probability distributions on Riemannian manifolds. We then construct Extended RSDFO and study its structure and properties from a geometrical perspective. We show that Extended RSDFO's expected fitness improves monotonically and it's global eventual convergence in finitely many steps on connected compact Riemannian manifolds. Extended RSDFO is compared to state-of-the-art manifold optimization algorithms on multi-modal optimization problems over a variety of manifolds. In particular, we perform a novel synthetic experiment on Jacob's ladder to motivate and necessitate manifold optimization. Jacob's ladder is a non-compact manifold of countably infinite genus, which cannot be expressed as polynomial constraints and does not have a global representation in an ambient Euclidean space. Optimization problems on Jacob's ladder thus cannot be addressed by traditional (constraint) optimization methods on Euclidean spaces., Comment: The present abstract is slightly altered from the PDF version due to the limitation "The abstract field cannot be longer than 1,920 characters"

Astronomical observations already produce vast amounts of data through a new generation of telescopes that cannot be analyzed manually. Next-generation telescopes such as the Large Synoptic Survey Telescope and the Square Kilometer Array are planned to become operational in this decade and the next, and will increase the data volume by many orders of magnitude. The increased spatial, temporal and spectral resolution afford a powerful magnifying lens on the physical processes that underlie the data but, at the same time, generate unprecedented complexity hard to exploit for knowledge extraction. It is therefore imperative to develop machine intelligence, machine learning (ML) in particular, suitable for processing the amount and variety of astronomical data that will be collected, and capable of answering scientific questions based on the data. Astronomical data exhibit the usual challenges associated with 'big data' such as immense volumes, high dimensionality, missing or highly distorted observations. In addition, astronomical data can exhibit large continuous observational gaps, very low signal-to-noise ratio and the need to distinguish between true missing data and non-detections due to upper limits). There are strict laws of physics behind the data production which can be assimilated into ML mechanisms to improve over general off-the-shelf state-of-the-art methods. Significant progress in the face of these challenges can be achieved only via the new discipline of Astroinformatics: a symbiosis of diverse disciplines, such as ML, probabilistic modeling, astronomy and astrophysics, statistics, distributed computing and natural computation. This editorial summarizes the contents of a soon to appear Focus Issue of the PASP on Machine Learning in Astronomy and Astrophysics (with contributions by 69 authors representing 15 countries, from 6 continents)., Comment: 11 pages

Time series data is composed of observations of one or more variables along a time period. By analyzing the variability of the variables we can reveal patterns that repeat or that are correlated, which helps to understand the behaviour of the variables over time. Our method finds frequent distributions of a target variable in time series data and discovers relationships between frequent distributions in consecutive time intervals. The frequent distributions are found using a new method, and relationships between them are found using association rules mining.

Parameterized state space models in the form of recurrent networks are often used in machine learning to learn from data streams exhibiting temporal dependencies. To break the black box nature of such models it is important to understand the dynamical features of the input driving time series that are formed in the state space. We propose a framework for rigorous analysis of such state representations in vanishing memory state space models such as echo state networks (ESN). In particular, we consider the state space a temporal feature space and the readout mapping from the state space a kernel machine operating in that feature space. We show that: (1) The usual ESN strategy of randomly generating input-to-state, as well as state coupling leads to shallow memory time series representations, corresponding to cross-correlation operator with fast exponentially decaying coefficients; (2) Imposing symmetry on dynamic coupling yields a constrained dynamic kernel matching the input time series with straightforward exponentially decaying motifs or exponentially decaying motifs of the highest frequency; (3) Simple cycle high-dimensional reservoir topology specified only through two free parameters can implement deep memory dynamic kernels with a rich variety of matching motifs. We quantify richness of feature representations imposed by dynamic kernels and demonstrate that for dynamic kernel associated with cycle reservoir topology, the kernel richness undergoes a phase transition close to the edge of stability., Comment: 45 pages, 17 figures, accepted

The Classifier kNN is largely used in Human Activity Recognition systems. Research efforts have proposed methods to decrease the high computational costs of the original kNN by focusing, e.g., on approximate kNN solutions such as the ones relying on Locality-sensitive Hashing (LSH). However, embedded kNN implementations need to address the target device memory constraints and power/energy consumption savings. One of the important aspects is the constraint regarding the maximum number of instances stored in the kNN learning process (being it offline or online and incremental). This paper presents simple, energy/computationally efficient and real-time feasible schemes to maintain a maximum number of learning instances stored by kNN. Experiments in the context of HAR show the efficiency of our best approaches, and their capability to avoid the kNN storage runs out of training instances for a given activity, a situation not prevented by typical default schemes.

Many machine learning models can be attacked with adversarial examples, i.e. inputs close to correctly classified examples that are classified incorrectly. However, most research on adversarial attacks to date is limited to vectorial data, in particular image data. In this contribution, we extend the field by introducing adversarial edit attacks for tree-structured data with potential applications in medicine and automated program analysis. Our approach solely relies on the tree edit distance and a logarithmic number of black-box queries to the attacked classifier without any need for gradient information. We evaluate our approach on two programming and two biomedical data sets and show that many established tree classifiers, like tree-kernel-SVMs and recursive neural networks, can be attacked effectively., accepted at the 20th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL)

Qualitative clinical assessments of the recovery of awareness after severe brain injury require an assessor to differentiate purposeful behavior from spontaneous behavior. As many such behaviors are minimal and inconsistent, behavioral assessments are susceptible to diagnostic errors. Advanced neuroimaging tools can bypass behavioral responsiveness and reveal evidence of covert awareness and cognition within the brains of some patients, thus providing a means for more accurate diagnoses, more accurate prognoses, and, in some instances, facilitated communication. The majority of reports to date have employed the neuroimaging methods of functional magnetic resonance imaging, positron emission tomography, and electroencephalography (EEG). However, each neuroimaging method has its own advantages and disadvantages (e.g., signal resolution, accessibility, etc.). Here, we describe a burgeoning technique of non-invasive optical neuroimaging-functional near-infrared spectroscopy (fNIRS)-and review its potential to address the clinical challenges of prolonged disorders of consciousness. We also outline the potential for simultaneous EEG to complement the fNIRS signal and suggest the future directions of research that are required in order to realize its clinical potential.

Extracting the statistics of event streams in natural environments is critical for interpreting current events and predicting future ones. The brain is known to rapidly find structure and meaning in unfamiliar streams of sensory experience, often by mere exposure to the environment (i.e., without explicit feedback). Yet, we know little about the brain pathways that support this type of statistical learning. Here, we test whether changes in white-matter (WM) connectivity due to training relate to our ability to extract temporal regularities. By combining behavioral training and diffusion tensor imaging (DTI), we demonstrate that humans adapt to the environment's statistics as they change over time from simple repetition to probabilistic combinations. In particular, we show that learning relates to the decision strategy that individuals adopt when extracting temporal statistics. We next test for learning-dependent changes in WM connectivity and ask whether they relate to individual variability in decision strategy. Our DTI results provide evidence for dissociable WM pathways that relate to individual strategy: extracting the exact sequence statistics (i.e., matching) relates to connectivity changes between caudate and hippocampus, while selecting the most probable outcomes in a given context (i.e., maximizing) relates to connectivity changes between prefrontal, cingulate and basal ganglia (caudate, putamen) regions. Thus, our findings provide evidence for distinct cortico-striatal circuits that show learning-dependent changes of WM connectivity and support individual ability to learn behaviorally-relevant statistics.

Background: The urinary steroid metabolome is considered the fingerprint of adrenal gland function. Novel methods using mass spectrometry profiling have seen the advent of a new era for metabolomics with powerful implications for both diagnostics and discovery. Its interpretation is difficult and performed by few specialists with the expertise to do so. This makes it a relatively inaccessible tool for the majority of Clinical Endocrinologists. Objective: To create an automatous method for accurate diagnosis and differentiation of inborn steroidogenic disorders using 34 distinct measured urinary steroid metabolites, that is accurate, reproducible and suitable for high-throughput use. Methods: Using GC/MS, 829 healthy control urines were analysed (302 neonates and infant, 149 children, 18 adolescents, 326 adults, 34 unknown age) and baseline urine from 118 newly diagnosed with inborn steroidogenic disorders (P450 oxidoreductase deficiency, 21 hydroxylase deficiency, 5a reductase deficiency, 17bHSD3 deficiency, 17 hydroxylase deficiency, 3bHSD2 deficiency, 11b hydroxylase type 1 and type 2 deficiency and cyt B5 deficiency). We custom-designed an interpretable machine learning technique, Angle Learning Vector Quantisation, designed to distinguish these conditions using the urinary steroid metabolome. We looked at all possible steroid ratios and by means of ANOVA reduced this to 165 most informative steroid ratios. Using these, the method is able to computationally determine a reduced list of the most relevant ratios to differentiate specific disorders. The method runs independent of sex and age information, method of urine collection (spot, nappy, 24 h collection), and compensates for missing measurements. Results: Our machine learning method was able to predict an affected urine vs a healthy urine with a sensitivity of 100% and specificity of 97%. For our three most prevalent conditions (PORD, SRD5A2 and CYP21A2), the method correctly identified the specific condition in 96% of cases. Where it incorrectly identified the condition, it was mistaken for a biologically very similar one. Conclusion: We have developed a novel machine learning which is highly sensitive and specific. With further validation, it has potential for application in routine clinical practice.

Predicting investors reactions to financial and political news is important for the early detection of stock market jitters. Evidence from several recent studies suggests that online social media could improve prediction of stock market movements. However, utilizing such information to predict strong stock market fluctuations has not been explored so far. In this work, we propose a novel event detection method on Twitter, tailored to detect financial and political events that influence a specific stock market. The proposed approach applies a bursty topic detection method on a stream of tweets related to finance or politics followed by a classification process which filters-out events that do not influence the examined stock market. We train our classifier to recognise real events by using solely information about stock market volatility, without the need of manual labeling. We model Twitter events as feature vectors that encompass a rich variety of information, such as the geographical distribution of tweets, their polarity, information about their authors as well as information about bursty words associated with the event. We show that utilizing only information about tweets polarity, like most previous studies, results in wasting important information. We apply the proposed method on high-frequency intra-day data from the Greek and Spanish stock market and we show that our financial event detector successfully predicts most of the stock market jitters.

Indefinite similarity measures can be frequently found in bio-informatics by means of alignment scores, but are also common in other fields like shape measures in image retrieval. Lacking an underlying vector space, the data are given as pairwise similarities only. The few algorithms available for such data do not scale to larger datasets. Focusing on probabilistic batch classifiers, the Indefinite Kernel Fisher Discriminant (iKFD) and the Probabilistic Classification Vector Machine (PCVM) are both effective algorithms for this type of data but, with cubic complexity. Here we propose an extension of iKFD and PCVM such that linear runtime and memory complexity is achieved for low rank indefinite kernels. Employing the Nystr\"om approximation for indefinite kernels, we also propose a new almost parameter free approach to identify the landmarks, restricted to a supervised learning problem. Evaluations at several larger similarity data from various domains show that the proposed methods provides similar generalization capabilities while being easier to parametrize and substantially faster for large scale data.

Indefinite similarity measures can be frequently found in bio-informatics by means of alignment scores. Lacking an underlying vector space, the data are given as pairwise similarities only. Indefinite Kernel Fisher Discriminant (iKFD) is a very effective classifier for this type of data but has cubic complexity and does not scale to larger problems. Here we propose an extension of iKFD such that linear runtime and memory complexity is achieved for low rank indefinite kernels. Evaluation at several larger similarity data from various domains shows that the proposed method provides similar generalization capabilities while being substantially faster for large scale data.

Ordinal classification refers to classification problems in which the classes have a natural order imposed on them because of the nature of the concept studied. Some ordinal classification approaches perform a projection from the input space to 1-dimensional (latent) space that is partitioned into a sequence of intervals (one for each class). Class identity of a novel input pattern is then decided based on the interval its projection falls into. This projection is trained only indirectly as part of the overall model fitting. As with any latent model fitting, direct construction hints one may have about the desired form of the latent model can prove very useful for obtaining high quality models. The key idea of this paper is to construct such a projection model directly, using insights about the class distribution obtained from pairwise distance calculations. The proposed approach is extensively evaluated with eight nominal and ordinal classifiers methods, ten real world ordinal classification datasets, and four different performance measures. The new methodology obtained the best results in average ranking when considering three of the performance metrics, although significant differences are found only for some of the methods. Also, after observing other methods internal behaviour in the latent space, we conclude that the internal projection do not fully reflect the intra-class behaviour of the patterns. Our method is intrinsically simple, intuitive and easily understandable, yet, highly competitive with state-of-the-art approaches to ordinal classification

The emergence of large scaled sensor networks facilitates the collection of large amounts of real-time data to monitor and control complex engineering systems. However, in many cases the collected data may be incomplete or inconsistent, while the underlying environment may be time-varying or un-formulated. In this paper, we have developed an innovative cognitive fault diagnosis framework that tackles the above challenges. This framework investigates fault diagnosis in the model space instead of in the signal space. Learning in the model space is implemented by fitting a series of models using a series of signal segments selected with a rolling window. By investigating the learning techniques in the fitted model space, faulty models can be discriminated from healthy models using one-class learning algorithm. The framework enables us to construct fault library when unknown faults occur, which can be regarded as cognitive fault isolation. This paper also theoretically investigates how to measure the pairwise distance between two models in the model space and incorporates the model distance into the learning algorithm in the model space. The results on three benchmark applications and one simulated model for the Barcelona water distribution network have confirmed the effectiveness of the proposed framework.

In Bioinformatics, Audic and Claverie were among the first to systematically study the influence of random fluctuations and sampling size on the reliability of digital expression profile data. For a transcript representing a small fraction of the library and a large number N of clones, the probability of observing x tags of the same gene will be well-approximated by the Poisson distribution parametrised by its mean (and variance) m>0, where the unknown parameter m signifies the number of transcripts of the given type (tag) per N clones in the cDNA library. On an abstract level, to determine whether a gene is differentially expressed or not, one has two numbers generated from two distinct Poisson distributions and based on this (extremely sparse) sample one has to decide whether the two Poisson distributions are identical or not. This can be used e.g. to determine equivalence of Poisson photon sources (up to time shift) in gravitational lensing. Each Poisson distribution is represented by a single measurement only, which is, of course, from a purely statistical standpoint very problematic. The key instrument of the Audic-Claverie approach is a distribution P over tag counts y in one library informed by the tag count x in the other library, under the null hypothesis that the tag counts are generated from the same but unknown Poisson distribution. P is obtained by Bayesian averaging (infinite mixture) of all possible Poisson distributions with mixing proportions equal to the posteriors (given x) under the flat prior over m. We ask: Given that the tag count samples from SAGE libraries are *extremely* limited, how useful actually is the Audic-Claverie methodology? We rigorously analyse the A-C statistic P that forms a backbone of the methodology and represents our knowledge of the underlying tag generating process based on one observation. We show will that the A-C statistic P and the underlying Poisson distribution of the tag counts share the same mode structure. Moreover, the K-L divergence from the true unknown Poisson distribution to the A-C statistic is minimised when the A-C statistic is conditioned on the mode of the Poisson distribution. Most importantly (and perhaps rather surprisingly), the expectation of this K-L divergence never exceeds 1/2 bit! This constitutes a rigorous quantitative argument, extending the previous empirical Monte Carlo studies, that supports the wide spread use of Audic-Claverie method, even though by their very nature, the SAGE libraries represent very sparse samples.

We derive a novel derivative based version of kernelized Generalized Learning Vector Quantization (KGLVQ) as an effective, easy to interpret, prototype based and kernelized classifier. It is called D-KGLVQ and we provide generalization error bounds, experimental results on real world data, showing that D-KGLVQ is competitive with KGLVQ and the SVM on UCI data and additionally show that automatic parameter adaptation for the used kernels simplifies the learning.

We study the problem of estimating the time delay between two signals representing delayed, irregularly sampled and noisy versions of the same underlying pattern. We propose a kernel-based technique in the context of an astronomical problem, namely estimating the time delay between two gravitationally lensed signals from a distant quasar. We test the algorithm on several artificial data sets, and also on real astronomical observations. By carrying out a statistical analysis of the results we present a detailed comparison of our method with the most popular methods for time delay estimation in astrophysics. Our method yields more accurate and more stable time delay estimates. Our methodology can be readily applied to current state-of-the-art optical monitoring data in astronomy, but can also be applied in other disciplines involving similar time series data.

It has been argued that a single two-dimensional visualization plot may not be sufficient to capture all of the interesting aspects of complex data sets, and therefore a hierarchical visualization system is desirable. In this paper we extend an existing locally linear hierarchical visualization system PhiVis ¸iteBishop98a in several directions: bf(1) We allow for em non-linear projection manifolds. The basic building block is the Generative Topographic Mapping (GTM). bf(2) We introduce a general formulation of hierarchical probabilistic models consisting of local probabilistic models organized in a hierarchical tree. General training equations are derived, regardless of the position of the model in the tree. bf(3) Using tools from differential geometry we derive expressions for local directional curvatures of the projection manifold. Like PhiVis, our system is statistically principled and is built interactively in a top-down fashion using the EM algorithm. It enables the user to interactively highlight those data in the ancestor visualization plots which are captured by a child model. We also incorporate into our system a hierarchical, locally selective representation of magnification factors and directional curvatures of the projection manifolds. Such information is important for further refinement of the hierarchical visualization plot, as well as for controlling the amount of regularization imposed on the local models. We demonstrate the principle of the approach on a toy data set and apply our system to two more complex 12- and 18-dimensional data sets.

An interactive hierarchical Generative Topographic Mapping (HGTM) ¸iteHGTM has been developed to visualise complex data sets. In this paper, we build a more general visualisation system by extending the HGTM visualisation system in 3 directions: bf (1) We generalize HGTM to noise models from the exponential family of distributions. The basic building block is the Latent Trait Model (LTM) developed in ¸iteKabanpami. bf (2) We give the user a choice of initializing the child plots of the current plot in either em interactive, or em automatic mode. In the interactive mode the user interactively selects ``regions of interest'' as in ¸iteHGTM, whereas in the automatic mode an unsupervised minimum message length (MML)-driven construction of a mixture of LTMs is employed. bf (3) We derive general formulas for magnification factors in latent trait models. Magnification factors are a useful tool to improve our understanding of the visualisation plots, since they can highlight the boundaries between data clusters. The unsupervised construction is particularly useful when high-level plots are covered with dense clusters of highly overlapping data projections, making it difficult to use the interactive mode. Such a situation often arises when visualizing large data sets. We illustrate our approach on a toy example and apply our system to three more complex real data sets.

We analyse how the Generative Topographic Mapping (GTM) can be modified to cope with missing values in the training data. Our approach is based on an Expectation -Maximisation (EM) method which estimates the parameters of the mixture components and at the same time deals with the missing values. We incorporate this algorithm into a hierarchical GTM. We verify the method on a toy data set (using a single GTM) and a realistic data set (using a hierarchical GTM). The results show our algorithm can help to construct informative visualisation plots, even when some of the training points are corrupted with missing values.

It has been argued that a single two-dimensional visualization plot may not be sufficient to capture all of the interesting aspects of complex data sets, and therefore a hierarchical visualization system is desirable. In this paper we extend an existing locally linear hierarchical visualization system PhiVis ¸iteBishop98a in several directions: bf(1) We allow for em non-linear projection manifolds. The basic building block is the Generative Topographic Mapping. bf(2) We introduce a general formulation of hierarchical probabilistic models consisting of local probabilistic models organized in a hierarchical tree. General training equations are derived, regardless of the position of the model in the tree. bf(3) Using tools from differential geometry we derive expressions for local directional curvatures of the projection manifold. Like PhiVis, our system is statistically principled and is built interactively in a top-down fashion using the EM algorithm. It enables the user to interactively highlight those data in the parent visualization plot which are captured by a child model. We also incorporate into our system a hierarchical, locally selective representation of magnification factors and directional curvatures of the projection manifolds. Such information is important for further refinement of the hierarchical visualization plot, as well as for controlling the amount of regularization imposed on the local models. We demonstrate the principle of the approach on a toy data set and apply our system to two more complex 12- and 19-dimensional data sets.

In this paper we investigate the potential of the analysis of noisy non-stationary time series by quantizing it into streams of discrete symbols and applying finite-memory symbolic predictors. The main argument is that careful quantization can reduce the noise in the time series to make model estimation more amenable given limited numbers of samples that can be drawn due to the non-stationarity in the time series. As a main application area we study the use of such an analysis in a realistic setting involving financial forecasting and trading. In particular, using historical data, we simulate the trading of straddles on the financial indexes DAX and FTSE 100 on a daily basis, based on predictions of the daily volatility differences in the underlying indexes. We propose a parametric, data-driven quantization scheme which transforms temporal patterns in the series of daily volatility changes into grammatical and statistical patterns in the corresponding symbolic streams. As symbolic predictors operating on the quantized streams we use the classical fixed-order Markov models, variable memory length Markov models and a novel variation of fractal-based predictors introduced in its original form in (Tino, 2000b). The fractal-based predictors are designed to efficiently use deep memory. We compare the symbolic models with continuous techniques such as time-delay neural networks with continuous and categorical outputs, and GARCH models. Our experiments strongly suggest that the robust information reduction achieved by quantizing the real-valued time series is highly beneficial. To deal with non-stationarity in financial daily time series, we propose two techniques that combine ``sophisticated" models fitted on the training data with a fixed set of simple-minded symbolic predictors not using older (and potentially misleading) data in the training set. Experimental results show that by quantizing the volatility differences and then using symbolic predictive models, market makers can generate a statistically significant excess profit. However, with respect to our prediction and trading techniques, the option market on the DAX does seem to be efficient for traders and non-members of the stock exchange. There is a potential for traders to make an excess profit on the FTSE 100. We also mention some interesting observations regarding the memory structure in the studied series of daily volatility differences. (author's abstract), Series: Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

Motivated by previous findings that discretization of financial time series can effectively filter the data and reduce the noise, this experimental study compares the trading performance of predictive models based on different modelling paradigms in a realistic setting. Different methods ranging from real-valued time series models to predictive models on a symbolic level are applied to predict the daily change in volatility of two major stock indices. The predicted volatility changes are interpreted as trading signals for buying or selling a straddle portfolio on the underlying stock index. Profits realized by this trading strategy are tested for statistical significance taking into account transaction costs. The results indicate that symbolic information processing is a promising approach to financial prediction tasks undermining the hypothesis of efficient captial markets. (author's abstract), Series: Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

We propose a novel approach to constructing predictive models on long complex symbolic sequences. The models are constructed by first transforming the training sequence n-block structure into a spatial structure of points in a unit hypercube. The transformation between the symbolic and Euclidean spaces embodies a natural smoothness assumption (n-blocks with long common suffices are likely to produce similar continuations) in that the longer is the common suffix shared by any two n-blocks, the closer lie their point representations. Finding a set of prediction contexts is then formulated as a resource allocation problem solved by vector quantizing the spatial representation of the training sequence n-block structure. Our predictive models are similar in spirit to variable memory length Markov models (VLMMs). We compare the proposed models with both the classical and variable memory length Markov models on two chaotic symbolic sequences with different levels of subsequence distribution structure. Our models have equal or better modeling performance, yet, their construction is more intuitive (unlike in VLMMs, we have a clear idea about the size of the model under construction) and easier to automize (construction of our models can be done in a completely self-organized manner, which is shown to be problematic in the case of VLMMs). (author's abstract), Series: Working Papers SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

We suggest a recurrent neural network (RNN) model with a recurrent part corresponding to iterative function systems (IFS) introduced by Barnsley [1] as a fractal image compression mechanism. The key idea is that 1) in our model we avoid learning the RNN state part by having non-trainable connections between the context and recurrent layers (this makes the training process less problematic and faster), 2) the RNN state part codes the information processing states in the symbolic input stream in a well-organized and intuitively appealing way. We show that there is a direct correspondence between the Rényi entropy spectra characterizing the input stream and the spectra of Renyi generalized dimensions of activations inside the RNN state space. We test both the new RNN model with IFS dynamics and its conventional counterpart with trainable recurrent part on two chaotic symbolic sequences. In our experiments, RNNs with IFS dynamics outperform the conventional RNNs with respect to information theoretic measures computed on the training and model generated sequences. (author's abstract), Series: Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

Jeffrey proposed a graphic representation of DNA sequences using Barnsley's iterative function systems. In spite of further developments in this direction (Oliver et. al, 1993), (Roman-Roldan et. al, 1994), (Li, 1997), the proposed graphic representation of DNA sequences has been lacking a rigorous connection between its spatial scaling characteristics and the statistical characteristics of the DNA sequences themselves. We 1) generalize Jeffrey's graphic representation to accommodate (possibly infinite) sequences over an arbitrary finite number of symbols, 2) establish a direct correspondence between the statistical characterization of symbolic sequences via Renyi entropy spectra and the multifractal characteristics (Renyi generalized dimensions) of the sequences' spatial representations, 3) show that for general symbolic dynamical systems, the multifractal fH-spectra in the sequence space coincide with the fH-spectra on spatial sequence representations. (author's abstract), Series: Working Papers SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

We consider the problem of predicting the direction of daily volatility changes in the Dow Jones Industrial Average (DJIA). This is accomplished by quantizing a series of historic volatility changes into a symbolic stream over 2 or 4 symbols. We compare predictive performance of the classical fixed-order Markov models with that of a novel approach to variable memory length prediction (called prediction fractal machine, or PFM) which is able to select very specific deep prediction contexts (whenever there is a sufficient support for such contexts in the training data). We learn that daily volatility changes of the DJIA only exhibit rather shallow finite memory structure. On the other hand, a careful selection of quantization cut values can strongly enhance predictive power of symbolic schemes. Results on 12 non-overlapping epochs of the DJIA strongly suggest that PFMs can outperform both traditional Markov models and (continuous-valued) GARCH models in the task of predicting volatility one time-step ahead. (author's abstract), Series: Working Papers SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

We present an algorithm for the visualisation of time series. To that end we employ echo state networks to convert time series into a suitable vector representation which is capable of capturing the latent dynamics of the time series. Subsequently, the obtained vector representations are put through an autoencoder and the visualisation is constructed using the activations of the bottleneck. The crux of the work lies with defining an objective function that quantifies the reconstruction error of these representations in a principled manner. We demonstrate the method on synthetic and real data., Comment: Published in ESANN 2015

Alzheimer's disease (AD) is characterised by a dynamic process of neurocognitive changes from normal cognition to mild cognitive impairment (MCI) and progression to dementia. However, not all individuals with MCI develop dementia. Predicting whether individuals with MCI will decline (i.e. progressive MCI) or remain stable (i.e. stable MCI) is impeded by patient heterogeneity due to comorbidities that may lead to MCI diagnosis without progression to AD. Despite the importance of early diagnosis of AD for prognosis and personalised interventions, we still lack robust tools for predicting individual progression to dementia. Here, we propose a novel trajectory modelling approach based on metric learning (Generalised Metric Learning Vector Quantization) that mines multimodal data from MCI patients in the Alzheimer's disease Neuroimaging Initiative (ADNI) cohort to derive individualised prognostic scores of cognitive decline due to AD. We develop an integrated biomarker generation- using partial least squares regression- and classification methodology that extends beyond binary patient classification into discrete subgroups (i.e. stable vs. progressive MCI), determines individual profiles from baseline (i.e. cognitive or biological) data and predicts individual cognitive trajectories (i.e. change in memory scores from baseline). We demonstrate that a metric learning model trained on baseline cognitive data (memory, executive function, affective measurements) discriminates stable vs. progressive MCI individuals with high accuracy (81.4%), revealing an interaction between cognitive (memory, executive functions) and affective scores that may relate to MCI comorbidity (e.g. affective disturbance). Training the model to perform the same binary classification on biological data (mean cortical β-amyloid burden, grey matter density, APOE 4) results in similar prediction accuracy (81.9%). Extending beyond binary classifications, we develop and implement a trajectory modelling approach that shows significantly better performance in predicting individualised rate of future cognitive decline (i.e. change in memory scores from baseline), when the metric learning model is trained with biological (r = -0.68) compared to cognitive (r = -0.4) data. Our trajectory modelling approach reveals interpretable and interoperable markers of progression to AD and has strong potential to guide effective stratification of individuals based on prognostic disease trajectories, reducing MCI patient misclassification, that is critical for clinical practice and discovery of personalised interventions.

We present a probabilistic generative approach for constructing topographic maps of light curves from eclipsing binary stars. The model defines a low-dimensional manifold of local noise models induced by a smooth non-linear mapping from a low-dimensional latent space into the space of probabilistic models of the observed light curves. The local noise models are physical models that describe how such light curves are generated. Due to the principled probabilistic nature of the model, a cost function arises naturally and the model parameters are fitted via MAP estimation using the Expectation-Maximisation algorithm. Once the model has been trained, each light curve may be projected to the latent space as the the mean posterior probability over the local noise models. We demonstrate our approach on a dataset of artificially generated light curves and on a dataset comprised of light curves from real observations., Comment: 10 pages, 5 figures. Accepted for the (refereed) proceedings of the International Conference on Artificial Neural Networks (ICANN 2009), September 14-17, Limassol, Cyprus (http://www.kios.org.cy/ICANN09/)

During my PhD studies, I collaborated in an exciting project between astronomeers and computer scientists. We developed machine learning strategies to detect, extract, and denoise manifolds embedded in potentially large amounts of background noise in higher dimensions. Examples of such a dataset are filaments in an N-body simulation of the Cosmic web or streams of a jellyfish-like dwarf galaxy. Furthermore, we proposed a subsampling technique that preserves the crucial topological information of large astronomical datasets. Extracting informationfrom large data sets, such as discovering bubbles in a dwarf galaxy simulation, is often resource-demanding. As a result, the proposed subsampling is of utmost importance. Besides, to further continue the study, the position of particles on the border of bubbles should be determined. We also suggest a pipeline to recover a tight boundary around each bubble and finally, we conduct an extensive analysis of the temperature, velocity of expansion, age, and other physical features of the discovered bubbles through several snapshots of the dwarf galaxy simulation.

Understanding in which circumstances office workers take rest breaks is important for delivering effective mobile notifications and make inferences about their daily lifestyle, e.g., whether they are active and/or have a sedentary life. Previous studies designed for office workers show the effectiveness of rest breaks for preventing work-related conditions. In this article, we propose a hybrid personalised model involving a kernel density estimation model and a generalised linear mixed model to model office workers’ available moments for rest breaks during working hours. We adopt the experience-based sampling method through which we collected office workers’ responses regarding their availability through a mobile application with contextual information extracted by means of the mobile phone sensors. The experiment lasted 10 workdays and involved 19 office workers with a total of 528 responses. Our results show that time, location, ringer mode, and activity are effective features for predicting office workers’ availability. Our method can address sparse sample issues for building individual predictive behavioural models based on limited and unbalanced data. In particular, the proposed method can be considered as a potential solution to the “cold-start problem,” i.e., the negative impact of the lack of individual data when a new application is installed.

Interactive machine learning (ML) allows a music performer to digitally represent musical actions (via gestural interfaces) and affect their musical output in real-time. Processing musical actions (termed performance gestures) with ML is useful because it predicts and maps often-complex biometric data. ML models can therefore be used to create novel interactions with musical systems, game-engines, and networked analogue devices. Wekinator is a free open-source software for ML (based on the Waikato Environment for Knowledge Analysis – WEKA - framework) which has been widely used, since 2009, to build supervised predictive models when developing real-time interactive systems. This is because it is accessible in its format (i.e. a graphical user interface – GUI) and simplified approach to ML. Significantly, it allows model training via gestural interfaces through demonstration. However, Wekinator offers the user several models to build predictive systems with. This paper explores which ML models (in Wekinator) are the most useful for predicting an output in the context of interactive music composition. We use two performance gestures for piano, with opposing datasets, to train available ML models, investigate compositional outcomes and frame the investigation. Our results show ML model choice is important for mapping performance gestures because of disparate mapping accuracies and behaviours found between all Wekinator ML models.

Dictionary based classifiers are a family of algorithms for time series classification (TSC) that focus on capturing the frequency of pattern occurrences in a time series. The ensemble based Bag of Symbolic Fourier Approximation Symbols (BOSS) was found to be a top performing TSC algorithm in a recent evaluation, as well as the best performing dictionary based classifier. However, BOSS does not scale well. We evaluate changes to the way BOSS chooses classifiers for its ensemble, replacing its parameter search with random selection. This change allows for the easy implementation of contracting (setting a build time limit for the classifier) and check-pointing (saving progress during the classifiers build). We achieve a significant reduction in build time without a significant change in accuracy on average when compared to BOSS by creating a fixed size weighted ensemble selecting the best performers from a randomly chosen parameter set. Our experiments are conducted on datasets from the recently expanded UCR time series archive. We demonstrate the usability improvements to randomised BOSS with a case study using a large whale acoustics dataset for which BOSS proved infeasible.

We present novel, low-cost and non-invasive potential diagnostic biomarkers of schizophrenia. They are based on the ‘mirror-game’, a coordination task in which two partners are asked to mimic each other’s hand movements. In particular, we use the patient’s solo movement, recorded in the absence of a partner, and motion recorded during interaction with an artificial agent, a computer avatar or a humanoid robot. In order to discriminate between the patients and controls, we employ statistical learning techniques, which we apply to nonverbal synchrony and neuromotor features derived from the participants’ movement data. The proposed classifier has 93% accuracy and 100% specificity. Our results provide evidence that statistical learning techniques, nonverbal movement coordination and neuromotor characteristics could form the foundation of decision support tools aiding clinicians in cases of diagnostic uncertainty., Mirror game test could detect schizophrenia A new test of movement and social interaction could detect markers of schizophrenia, and help to diagnose and manage the condition. In an effort to establish reliable indicators of schizophrenia, Piotr Slowinski at the University of Exeter, UK and colleagues developed a test that could detect deficits in movement and social interactions, both characteristics of the disorder. They asked people to perform movements alone, and to mirror the movements of a computer avatar or a humanoid robot. Automated analysis of the movements allowed to distinguish people with schizophrenia from healthy participants with accuracy and specificity slightly better than clinical interviews and comparable to test based on much more expensive neuroimaging methods. The technique could help with diagnosis of schizophrenia and to monitor patients’ responses to treatment, but needs to be tested in clinical trials before being applied in clincal practice.