Search

Real-world applications often require complicated pipeline that do not progress sequentially. For example, many experiments have demonstrated that bagging is a powerful method to improve model performance. Bagging can be thought of as a non-sequential pipeline where a learner is replicated, each separate learner is trained and makes predictions, and their results are combined. This is non-sequential as data is not flowing sequentially through the pipeline but is instead passed to all learners (who may then subsample the data) and then recombined, thus creating a pipeline where operations have multiple inputs and outputs. Pipeline operations also have hyperparameters that can be set and tuned to improve model performance. Moreover the choice of operations to include in a pipeline can also be tuned, known as combined algorithm selection and hyperparameter optimization (CASH). This chapter looks at more advanced uses of mlr3pipelines. This is put into practice by demonstrating how to build a bagging and stacking pipeline from scratch, as well as how to access common pipelines that are readily available in mlr3pipelines. The chapter then looks at tuning pipelines and CASH.

The increasing availability of data and software frameworks to create predictive models has allowed the widespread adoption of machine learning in many applications. However, high predictive performance of such models often comes at the cost of interpretability. Machine learning interpretation methods can be useful for several purposes: 1) gaining global insights into a model (e.g., feature importance); 2) model improvement if flaws were identified (e.g., unexpected reliance on a certain feature); 3) understanding individual predictions. Several model-agnostic methods have been developed including feature permutation, Shapleys, and LIME. This chapter presents the packages iml, counterfactuals, and DALEX, which implement model-agnostic interpretation methods. Throughout the chapter an xgboost is trained on the german credit dataset to understand how predictions are made and why. The chapter starts with discussing the iml package and the theory behind the discussed methods, as well as how to practically use the interface. It then moves to counterfactuals and the benefits of counterfactual analysis, including methods What-If and MOC. Finally, DALEX is introduced, which includes similar methods to iml but with a different design, hence users can make use of either package depending on their design preference.

Feature selection, also known as variable or descriptor selection, is the process of finding a subset of features to use with a given task and learner. Finding the optimal set of features can improve predictive performance, reduce noise in data, and make models easier to interpret. There are multiple ways to perform feature selection, including with filters and wrappers. Filters calculate a value for each feature which can then be used to select them, for example selecting features where the feature value is greater than some value Y. Wrappers fit models on selected feature subsets and evaluate their performance to then pick the optimal subset – sequential forward selection is a well-known example. This chapter introduces mlr3fselect and mlr3filters for feature selection and filter methods. The design of these packages is similar to mlr3tuning so the interface should feel familiar to readers who read Chapter 4. Filters are first discussed including feature importance and embedded methods special cases. Then wrappers are discussed, including automating the process with nested resampling.

Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time-consuming and irreproducible manual process of trial-and-error to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization. This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction.

Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness., Comment: Published at ACM TELO

Automated hyperparameter optimization (HPO) has gained great popularity and is an important ingredient of most automated machine learning frameworks. The process of designing HPO algorithms, however, is still an unsystematic and manual process: Limitations of prior work are identified and the improvements proposed are -- even though guided by expert knowledge -- still somewhat arbitrary. This rarely allows for gaining a holistic understanding of which algorithmic components are driving performance, and carries the risk of overlooking good algorithmic design choices. We present a principled approach to automated benchmark-driven algorithm design applied to multifidelity HPO (MF-HPO): First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to common HPO algorithms, and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration, in some ways simpler than established methods, performs very well as long as some critical configuration parameters have the right value., Comment: * Equal Contributions

Spatial and spatiotemporal machine-learning models require a suitable framework for their model assessment, model selection, and hyperparameter tuning, in order to avoid error estimation bias and over-fitting. This contribution reviews the state-of-the-art in spatial and spatiotemporal cross-validation, and introduces the {R} package {mlr3spatiotempcv} as an extension package of the machine-learning framework {mlr3}. Currently various {R} packages implementing different spatiotemporal partitioning strategies exist: {blockCV}, {CAST}, {skmeans} and {sperrorest}. The goal of {mlr3spatiotempcv} is to gather the available spatiotemporal resampling methods in {R} and make them available to users through a simple and common interface. This is made possible by integrating the package directly into the {mlr3} machine-learning framework, which already has support for generic non-spatiotemporal resampling methods such as random partitioning. One advantage is the use of a consistent nomenclature in an overarching machine-learning toolkit instead of a varying package-specific syntax, making it easier for users to choose from a variety of spatiotemporal resampling methods. This package avoids giving recommendations which method to use in practice as this decision depends on the predictive task at hand, the autocorrelation within the data, and the spatial structure of the sampling design or geographic objects being studied., Comment: 35 pages, 15 Figures, 1 Table

Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming and unreproducible manual trial-and-error process to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods, e.g., based on resampling error estimation for supervised machine learning, can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization. This work is accompanied by an appendix that contains information on specific software packages in R and Python, as well as information and recommended hyperparameter search spaces for specific learning algorithms. We also provide notebooks that demonstrate concepts from this work as supplementary files.

Fitting models with high predictive accuracy that include all relevant but no irrelevant or redundant features is a challenging task on data sets with similar (e.g. highly correlated) features. We propose the approach of tuning the hyperparameters of a predictive model in a multi-criteria fashion with respect to predictive accuracy and feature selection stability. We evaluate this approach based on both simulated and real data sets and we compare it to the standard approach of single-criteria tuning of the hyperparameters as well as to the state-of-the-art technique "stability selection". We conclude that our approach achieves the same or better predictive performance compared to the two established approaches. Considering the stability during tuning does not decrease the predictive accuracy of the resulting models. Our approach succeeds at selecting the relevant features while avoiding irrelevant or redundant features. The single-criteria approach fails at avoiding irrelevant or redundant features and the stability selection approach fails at selecting enough relevant features for achieving acceptable predictive accuracy. For our approach, for data sets with many similar features, the feature selection stability must be evaluated with an adjusted stability measure, that is, a measure that considers similarities between features. For data sets with only few similar features, an unadjusted stability measure suffices and is faster to compute.

Cost-sensitive feature selection describes a feature selection problem, where features raise individual costs for inclusion in a model. These costs allow to incorporate disfavored aspects of features, e.g. failure rates of as measuring device, or patient harm, in the model selection process. Random Forests define a particularly challenging problem for feature selection, as features are generally entangled in an ensemble of multiple trees, which makes a post hoc removal of features infeasible. Feature selection methods therefore often either focus on simple pre-filtering methods, or require many Random Forest evaluations along their optimization path, which drastically increases the computational complexity. To solve both issues, we propose Shallow Tree Selection, a novel fast and multivariate feature selection method that selects features from small tree structures. Additionally, we also adapt three standard feature selection algorithms for cost-sensitive learning by introducing a hyperparameter-controlled benefit-cost ratio criterion (BCR) for each method. In an extensive simulation study, we assess this criterion, and compare the proposed methods to multiple performance-based baseline alternatives on four artificial data settings and seven real-world data settings. We show that all methods using a hyperparameterized BCR criterion outperform the baseline alternatives. In a direct comparison between the proposed methods, each method indicates strengths in certain settings, but no one-fits-all solution exists. On a global average, we could identify preferable choices among our BCR based methods. Nevertheless, we conclude that a practical analysis should never rely on a single method only, but always compare different approaches to obtain the best results., Comment: Corrected minor typo in Figure 1, Added ancillary files

As machine learning has become increasingly popular over the last few decades, so too has the number of machine learning interfaces for implementing these models. Whilst many R libraries exist for machine learning, very few offer extended support for survival analysis. This is problematic considering its importance in fields like medicine, bioinformatics, economics, engineering, and more. mlr3proba provides a comprehensive machine learning interface for survival analysis and connects with mlr3's general model tuning and benchmarking facilities to provide a systematic infrastructure for survival modeling and evaluation., Comment: Submitted to Bioinformatics

A novel machine learning optimization process coined Restrictive Federated Model Selection (RFMS) is proposed under the scenario, for example, when data from healthcare units can not leave the site it is situated on and it is forbidden to carry out training algorithms on remote data sites due to either technical or privacy and trust concerns. To carry out a clinical research under this scenario, an analyst could train a machine learning model only on local data site, but it is still possible to execute a statistical query at a certain cost in the form of sending a machine learning model to some of the remote data sites and get the performance measures as feedback, maybe due to prediction being usually much cheaper. Compared to federated learning, which is optimizing the model parameters directly by carrying out training across all data sites, RFMS trains model parameters only on one local data site but optimizes hyper-parameters across other data sites jointly since hyper-parameters play an important role in machine learning performance. The aim is to get a Pareto optimal model with respective to both local and remote unseen prediction losses, which could generalize well across data sites. In this work, we specifically consider high dimensional data with shifted distributions over data sites. As an initial investigation, Bayesian Optimization especially multi-objective Bayesian Optimization is used to guide an adaptive hyper-parameter optimization process to select models under the RFMS scenario. Empirical results show that solely using the local data site to tune hyper-parameters generalizes poorly across data sites, compared to methods that utilize the local and remote performances. Furthermore, in terms of dominated hypervolumes, multi-objective Bayesian Optimization algorithms show increased performance across multiple data sites among other candidates.

Cet ouvrage présente une synthèse des inondations remarquables recensées en France en 2011, à l’occasion de l’application de la Directive européenne sur les inondations. Il permet de disposer d’éléments d’information sur environ 175 inondations et couvre une période de plus de deux cents ans, de 1770 à 2011, avec plus de la moitié des événements après 1980. Il complète ainsi utilement la précédente synthèse nationale réalisée par Champion (1858-1864).

Cet ouvrage présente une synthèse des inondations remarquables recensées en France en 2011, à l’occasion de l’application de la Directive européenne sur les inondations. Il permet de disposer d’éléments d’information sur environ 175 inondations et couvre une période de plus de deux cents ans, de 1770 à 2011, avec plus de la moitié des événements après 1980. Il complète ainsi utilement la précédente synthèse nationale réalisée par Champion (1858-1864).

Cet ouvrage présente une synthèse des inondations remarquables recensées en France en 2011, à l’occasion de l’application de la Directive européenne sur les inondations. Il permet de disposer d’éléments d’information sur environ 175 inondations et couvre une période de plus de deux cents ans, de 1770 à 2011, avec plus de la moitié des événements après 1980. Il complète ainsi utilement la précédente synthèse nationale réalisée par Champion (1858-1864).

Cet ouvrage présente une synthèse des inondations remarquables recensées en France en 2011, à l’occasion de l’application de la Directive européenne sur les inondations. Il permet de disposer d’éléments d’information sur environ 175 inondations et couvre une période de plus de deux cents ans, de 1770 à 2011, avec plus de la moitié des événements après 1980. Il complète ainsi utilement la précédente synthèse nationale réalisée par Champion (1858-1864).

Cet ouvrage présente une synthèse des inondations remarquables recensées en France en 2011, à l’occasion de l’application de la Directive européenne sur les inondations. Il permet de disposer d’éléments d’information sur environ 175 inondations et couvre une période de plus de deux cents ans, de 1770 à 2011, avec plus de la moitié des événements après 1980. Il complète ainsi utilement la précédente synthèse nationale réalisée par Champion (1858-1864).

Cet ouvrage présente une synthèse des inondations remarquables recensées en France en 2011, à l’occasion de l’application de la Directive européenne sur les inondations. Il permet de disposer d’éléments d’information sur environ 175 inondations et couvre une période de plus de deux cents ans, de 1770 à 2011, avec plus de la moitié des événements après 1980. Il complète ainsi utilement la précédente synthèse nationale réalisée par Champion (1858-1864).

An original hydraulics-based Bayesian method is introduced for developing stage-fall-discharge (SFD) rating curves used at twin-gauge stations and estimating their uncertainties. A channel control with variable energy slope due to variable backwater is considered in combination with a backwater-unaffected control, usually another channel control. Succession between backwater-affected and backwater-unaffected controls is solved based on the continuity of the stage-discharge relation. The difference between the reference levels at the two stations is estimated as another uncertain parameter of the SFD model. Results at one typical twin-gauge station affected by the backwater of a run-of-the-river dam are presented. The accuracy and uncertainty of predicted discharges appear to be acceptable even when the uncertainty of the transition between backwater-affected and backwater-unaffected controls is high.

Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create and leverage these benchmarking suites. These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R. OpenML benchmarking suites (a) are easy to use through standardized data formats, APIs, and client libraries; (b) come with extensive meta-information on the included datasets; and (c) allow benchmarks to be shared and reused in future studies. We then present a first, carefully curated and practical benchmarking suite for classification: the OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18). Finally, we discuss use cases and applications which demonstrate the usefulness of OpenML benchmarking suites and the OpenML-CC18 in particular., Comment: Accepted for publication in the Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS 2021)

OpenML is an online machine learning platform where researchers can easily share data, machine learning tasks and experiments as well as organize them online to work and collaborate more efficiently. In this paper, we present an R package to interface with the OpenML platform and illustrate its usage in combination with the machine learning R package mlr. We show how the OpenML package allows R users to easily search, download and upload data sets and machine learning tasks. Furthermore, we also show how to upload results of experiments, share them with others and download results from other users. Beyond ensuring reproducibility of results, the OpenML platform automates much of the drudge work, speeds up research, facilitates collaboration and increases the users' visibility online.

Streamflow data are used for important environmental and economic decisions, such asspecifying and regulating minimum flows, managing water supplies, and planning for flood hazards.Despite significant uncertainty in most flow data, the flow series for these applications are oftencommunicated and used without uncertainty information. In this commentary, we argue that properanalysis of uncertainty in river flow data can reduce costs and promote robust conclusions in watermanagement applications. We substantiate our argument by providing case studies from Norway and NewZealand where streamflow uncertainty analysis has uncovered economic costs in the hydropower industry,improved public acceptance of a controversial water management policy, and tested the accuracy of waterquality trends. We discuss the need for practical uncertainty assessment tools that generate multiple flowseries realizations rather than simple error bounds. Although examples of such tools are in development,considerable barriers for uncertainty analysis and communication still exist for practitioners, and futureresearch must aim to provide easier access and usability of uncertainty estimates. We conclude that flowuncertainty analysis is critical for good water management decisions.

Ce guide méthodologique, réalisé pour le compte du ministère chargé de l’Écologie a pour objectif d’améliorer la qualité des études hydrologiques, d’assurer une certaine homogénéité dans le mode de détermination des crues de référence à l’échelle du territoire national, et d’exploiter au mieux les informations hydrologiques disponibles sur les événements passés. Il dresse un inventaire des différentes méthodes probabilistes de prédétermination des crues avec une série d’exemples d’application et une partie bibliographique. Il est destiné aux services de l’État chargés de l’instruction des PPR, et aux bureaux d’études et étudiants concernés par le risque inondation.

Ce guide méthodologique, réalisé pour le compte du ministère chargé de l’Écologie a pour objectif d’améliorer la qualité des études hydrologiques, d’assurer une certaine homogénéité dans le mode de détermination des crues de référence à l’échelle du territoire national, et d’exploiter au mieux les informations hydrologiques disponibles sur les événements passés. Il dresse un inventaire des différentes méthodes probabilistes de prédétermination des crues avec une série d’exemples d’application et une partie bibliographique. Il est destiné aux services de l’État chargés de l’instruction des PPR, et aux bureaux d’études et étudiants concernés par le risque inondation.

This document provides and in-depth introduction to the mlr framework for machine learning experiments in R.

Cet ouvrage est consacré au projet "Historique Isère et torrents affluents" dont le principe est de valoriser l'information historique pour une meilleure connaissance du risque inondation. Une équipe interdisciplinaire (hydrologie, géomorphologie, hydraulique, histoire, archives) a appliqué cette démarche sur le secteur de la moyenne Isère et deux petits affluents torrentiels, le Manival et le Merdaret. Les auteurs commentent les principaux résultats obtenus au terme de deux années d'étude : un inventaire pratiquement exhaustif des sources de documents relatifs à ces cours d'eau, une exploitation des sources d'archives à des fins méthodologiques, une modélisation hydraulique des crues de l'Isère sur le secteur Pontcharra Grenoble, un premier bilan de la recherche et de l'utilisation de l'information historique relative au cas spécifique des torrents.

Cet ouvrage est consacré au projet "Historique Isère et torrents affluents" dont le principe est de valoriser l'information historique pour une meilleure connaissance du risque inondation. Une équipe interdisciplinaire (hydrologie, géomorphologie, hydraulique, histoire, archives) a appliqué cette démarche sur le secteur de la moyenne Isère et deux petits affluents torrentiels, le Manival et le Merdaret. Les auteurs commentent les principaux résultats obtenus au terme de deux années d'étude : un inventaire pratiquement exhaustif des sources de documents relatifs à ces cours d'eau, une exploitation des sources d'archives à des fins méthodologiques, une modélisation hydraulique des crues de l'Isère sur le secteur Pontcharra Grenoble, un premier bilan de la recherche et de l'utilisation de l'information historique relative au cas spécifique des torrents.

Cet ouvrage est consacré au projet "Historique Isère et torrents affluents" dont le principe est de valoriser l'information historique pour une meilleure connaissance du risque inondation. Une équipe interdisciplinaire (hydrologie, géomorphologie, hydraulique, histoire, archives) a appliqué cette démarche sur le secteur de la moyenne Isère et deux petits affluents torrentiels, le Manival et le Merdaret. Les auteurs commentent les principaux résultats obtenus au terme de deux années d'étude : un inventaire pratiquement exhaustif des sources de documents relatifs à ces cours d'eau, une exploitation des sources d'archives à des fins méthodologiques, une modélisation hydraulique des crues de l'Isère sur le secteur Pontcharra Grenoble, un premier bilan de la recherche et de l'utilisation de l'information historique relative au cas spécifique des torrents.

Cet ouvrage est consacré au projet "Historique Isère et torrents affluents" dont le principe est de valoriser l'information historique pour une meilleure connaissance du risque inondation. Une équipe interdisciplinaire (hydrologie, géomorphologie, hydraulique, histoire, archives) a appliqué cette démarche sur le secteur de la moyenne Isère et deux petits affluents torrentiels, le Manival et le Merdaret. Les auteurs commentent les principaux résultats obtenus au terme de deux années d'étude : un inventaire pratiquement exhaustif des sources de documents relatifs à ces cours d'eau, une exploitation des sources d'archives à des fins méthodologiques, une modélisation hydraulique des crues de l'Isère sur le secteur Pontcharra Grenoble, un premier bilan de la recherche et de l'utilisation de l'information historique relative au cas spécifique des torrents.

Cet ouvrage propose une procédure simplifiée pour réaliser une analyse de séries hydrologiques dans un cadre multifractal. Après un rappel de la théorie des multifractales, il décrit un modèle permettant d’identifier les propriétés d’invariance d’échelle, et présente des exemples d’application à des séries temporelles de pluie et de débit. Il intéressera les enseignants et chercheurs de la discipline tant au niveau national qu’international.

Cet ouvrage propose une procédure simplifiée pour réaliser une analyse de séries hydrologiques dans un cadre multifractal. Après un rappel de la théorie des multifractales, il décrit un modèle permettant d’identifier les propriétés d’invariance d’échelle, et présente des exemples d’application à des séries temporelles de pluie et de débit. Il intéressera les enseignants et chercheurs de la discipline tant au niveau national qu’international.

Cet ouvrage propose une procédure simplifiée pour réaliser une analyse de séries hydrologiques dans un cadre multifractal. Après un rappel de la théorie des multifractales, il décrit un modèle permettant d’identifier les propriétés d’invariance d’échelle, et présente des exemples d’application à des séries temporelles de pluie et de débit. Il intéressera les enseignants et chercheurs de la discipline tant au niveau national qu’international.