Search

The mathematician and bio‐scientist Sam Karlin is quoted stating that "The purpose of models is not to fit the data but to sharpen the question". In this paper, we describe a journey between questions, models and data analysis to reach specific goals. This journey is typical in industrial, engineering, biology and social science applications. It contrasts regulated clinical research where a statistical analysis plan is declared before data collection. We consider random forests, ridge regression, lasso and elastic nets. To make our point, we use a case study of 63 sensors collected in the testing of an electronic system. The paper lists a sequence of questions and how they were tackled by statistical analysis to meet the analysis goal. Eventually, we were able to provide a robust parsimonious and effective model for predicting the system condition using a subset of the 63 sensors. In handling this problem, we develop and apply several innovative methods and insights that can prove useful in other contexts. [ABSTRACT FROM AUTHOR]

This paper deals with measuring the influence of observations on the results obtained with CART classification trees. To define the influence of individuals on the analysis, we use influence measures to propose criterions to quantify the sensitivity of the CART classification tree analysis. The proposals are based on predictions and use jackknife trees. The analysis is extended to the pruned sequences of CART trees to produce CART specific notions of influence. Using the framework of influence functions, distributional results are derived. A numerical example, the well known spam dataset, is presented to illustrate the notions developed throughout the paper. A real dataset relating the administrative classification of cities surrounding Paris, France, to the characteristics of their tax revenues distribution, is finally analyzed using the new influence-based tools. [ABSTRACT FROM AUTHOR]

We present two strategies for detecting patterns and clusters in high-dimensional timedependent functional data. The use on wavelet-based similarity measures, since wavelets are well suited for identifying highly discriminant local time and scale features. The multiresolution aspect of the wavelet transform provides a time-scale decomposition of the signals allowing to visualize and to cluster the functional data into homogeneous groups. For each input function, through its empirical orthogonal wavelet transform the first strategy uses the distribution of energy across scales to generate a representation that can be sufficient to make the signals well distinguishable. Our new similarity measure combined with an efficient feature selection technique in the wavelet domain is then used within more or less classical clustering algorithms to effectively differentiate among high-dimensional populations. The second strategy uses a similarity measure between the whole time-scale representations that is based on wavelet-coherence tools. The clustering is then performed using a k-centroid algorithm starting from these similarities. Practical performance is illustrated through simulations as well as daily profiles of the French electricity power demand [ABSTRACT FROM AUTHOR]

Smart grids require flexible data driven forecasting methods. We propose clustering tools for bottom-up short-term load forecasting. We focus on individual consumption data analysis which plays a major role for energy management and electricity load forecasting. The first section is dedicated to the industrial context and a review of individual electrical data analysis. Then, we focus on hierarchical time-series for bottom-up forecasting. The idea is to decompose the global signal and obtain disaggregated forecasts in such a way that their sum enhances the prediction. This is done in three steps: identify a rather large number of super-consumers by clustering their energy profiles, generate a hierarchy of nested partitions and choose the one that minimize a prediction criterion. Using a nonparametric model to handle forecasting, and wavelets to define various notions of similarity between load curves, this disaggregation strategy gives a 16% improvement in forecasting accuracy when applied to French individual consumers. Then, this strategy is implemented using R—the free software environment for statistical computing—so that it can scale when dealing with massive datasets. The proposed solution is to make the algorithm scalable combine data storage, parallel computing and double clustering step to define the super-consumers. The resulting software is openly available. [ABSTRACT FROM AUTHOR]