Search

The No Free Lunch Theorem tells us that no algorithm can beat other algorithms on all types of problems. The algorithm selection structure is proposed to select the most suitable algorithm from a set of algorithms for an unknown optimization problem. This paper introduces an innovative algorithm selection approach called the CNN-HT, which is a two-stage algorithm selection framework. In the first stage, a Convolutional Neural Network (CNN) is employed to classify problems. In the second stage, the Hypothesis Testing (HT) technique is used to suggest the best-performing algorithm based on the statistical analysis of the performance metric of algorithms that address various problem categories. The two-stage approach can adapt to different algorithm combinations without the need to retrain the entire model, and modifications can be made in the second stage only, which is an improvement of one-stage approaches. To provide a more general structure for the classification model, we adopt Exploratory Landscape Analysis (ELA) features of the problem as input and utilize feature selection techniques to reduce the redundant ones. In problem classification, the average accuracy of classifying problems using CNN is 96%, which demonstrates the advantages of CNN compared to Random Forest and Support Vector Machines. After feature selection, the accuracy increases to 98.8%, further improving the classification performance while reducing the computational cost. This demonstrates the effectiveness of the first stage of the CNN-HT method, which provides a basis for algorithm selection. In the experiments, CNN-HT shows the advantages of the second stage algorithm as well as good performance with better average rankings in different algorithm combinations compared to the individual algorithms and another algorithm combination approach. [ABSTRACT FROM AUTHOR]

In practical data mining, a wide range of classification algorithms is employed for prediction tasks. However, selecting the best algorithm poses a challenging task for machine learning practitioners and experts, primarily due to the inherent variability in the characteristics of classification problems, referred to as datasets, and the unpredictable performance of these algorithms. Dataset characteristics are quantified in terms of meta-features, while classifier performance is evaluated using various performance metrics. The assessment of classifiers through empirical methods across multiple classification datasets, while considering multiple performance metrics, presents a computationally expensive and time-consuming obstacle in the pursuit of selecting the optimal algorithm. Furthermore, the scarcity of sufficient training data, denoted by dimensions representing the number of datasets and the feature space described by meta-feature perspectives, adds further complexity to the process of algorithm selection using classical machine learning methods. This research paper presents an integrated framework called eML-CBR that combines edge edge-ML and case-based reasoning methodologies to accurately address the algorithm selection problem. It adapts a multi-level, multi-view case-based reasoning methodology, considering data from diverse feature dimensions and the algorithms from multiple performance aspects, that distributes computations to both cloud edges and centralized nodes. On the edge, the first-level reasoning employs machine learning methods to recommend a family of classification algorithms, while at the second level, it recommends a list of the top-k algorithms within that family. This list is further refined by an algorithm conflict resolver module. The eML-CBR framework offers a suite of contributions, including integrated algorithm selection, multi-view meta-feature extraction, innovative performance criteria, improved algorithm recommendation, data scarcity mitigation through incremental learning, and an open-source CBR module, reshaping research paradigms. The CBR module, trained on 100 datasets and tested with 52 datasets using 9 decision tree algorithms, achieved an accuracy of 94% for correct classifier recommendations within the top k=3 algorithms, making it highly suitable for practical classification applications. [ABSTRACT FROM AUTHOR]

Featured Application: The artifact developed in this article is applicable to AGV and production planning optimization problems that align with the principles of the job-shop scheduling problem (JSSP) and the flexible job-shop scheduling problem (FJSSP). Artificial intelligence is considered to be a significant technology for driving the future evolution of smart manufacturing environments. At the same time, automated guided vehicles (AGVs) play an essential role in manufacturing systems due to their potential to improve internal logistics by increasing production flexibility. Thereby, the productivity of the entire system relies on the quality of the schedule, which can achieve production cost savings by minimizing delays and the total makespan. However, traditional scheduling algorithms often have difficulties in adapting to changing environment conditions, and the performance of a selected algorithm depends on the individual scheduling problem. Therefore, this paper aimed to analyze the scheduling problem classes of AGVs by applying design science research to develop an algorithm selection approach. The designed artifact addressed a catalogue of characteristics that used several machine learning algorithms to find the optimal solution strategy for the intended scheduling problem. The contribution of this paper is the creation of an algorithm selection method that automatically selects a scheduling algorithm, depending on the problem class and the algorithm space. In this way, production efficiency can be increased by dynamically adapting the AGV schedules. A computational study with benchmark literature instances unveiled the successful implementation of constraint programming solvers for solving JSSP and FJSSP scheduling problems and machine learning algorithms for predicting the most promising solver. The performance of the solvers strongly depended on the given problem class and the problem instance. Consequently, the overall production performance increased by selecting the algorithms per instance. A field experiment in the learning factory at Reutlingen University enabled the validation of the approach within a running production scenario. [ABSTRACT FROM AUTHOR]

This paper presents MachSMT, an algorithm selection tool for Satisfiability Modulo Theories (SMT) solvers. MachSMT supports the entirety of the SMT-LIB language and standardized SMT-LIB theories, and is easy to extend with support for new theories. MachSMT deploys machine learning methods to construct both empirical hardness models and pairwise ranking comparators over state-of-the-art SMT solvers. Given an input formula in SMT-LIB format, MachSMT leverages these learnt models to output a ranking of solvers based on predicted runtimes. We provide an extensive empirical evaluation of MachSMT to demonstrate the efficiency and efficacy of MachSMT over three broad usage scenarios on theories and theory combinations of practical relevance (e.g., bit-vectors, (non)linear integer and real arithmetic, arrays, and floating-point arithmetic). First, we deploy MachSMT on state-of-the-art solvers in SMT-COMP 2019 and 2020. We observe MachSMT frequently improves on the best performing solvers in the competition, winning 57 divisions outright, with up to a 99.4 % improvement in PAR-2 score. Second, we evaluate MachSMT to select configurations from a single underlying solver. We observe that MachSMT solves 898 more benchmarks and up to a 93.4 % improvement in PAR-2 score across 23 configurations of the SMT solver cvc5. Last, we evaluate MachSMT on domain-specific problems, namely network verification with simple domain-specific features, and observe an improvement of 77.3 % in PAR-2 score. [ABSTRACT FROM AUTHOR]

The recent evolution of machine learning (ML) algorithms and the high level of expertise required to use them have fuelled the demand for non-experts solutions. The selection of an appropriate algorithm and the configuration of its hyperparameters is among the most complicated tasks while applying ML to new problems. It necessitates well awareness and knowledge of ML algorithms. The algorithm selection problem (ASP) is defined as the process of identifying the algorithm (s) that can deliver top performance for a particular problem, task, and evaluation measure. In this context, meta-learning is one of the approaches to achieve this objective by using prior learning experiences to assist the learning process on unseen problems and tasks. As a data-driven approach, appropriate data characterization is of vital importance for the meta-learning. Nonetheless, the recent literature witness a variety of data characterization techniques including simple, statistical and information theory based measures. However, their quality still needs to be improved. In this paper, a new Autoencoder-kNN (AeKNN) based meta-model with built-in latent features extraction is proposed. The approach is aimed to extract new characterizations of the data, with lower dimensionality but more significant and meaningful features. AeKNN internally uses a deep autoencoder as a latent features extractor from a set of existing meta-features induced from the dataset. From this new features vectors the computed distances are more significant, thus providing a way to accurately recommending top-performing pipelines for previously unseen datasets. In an application on a large-scale hyperparameters optimization task for 400 real world datasets with varying schemas as a meta-learning task, we show that AeKNN offers considerable improvements of the classical kNN as well as traditional meta-models in terms of performance. [ABSTRACT FROM AUTHOR]

With the rapid development of artificial intelligence, the selection of algorithms that meet application requirements from feasible algorithms has become a critical problem to be solved urgently in various fields, that is, the algorithm selection problem. The approach based on meta-learning is an important way to solve the algorithm selection problem, which is widely applied in algorithm selection research and achieves good results. The approach selects appropriate algorithms by constructing the mapping model from problem features to candidate algorithms performance, mainly including the steps of extracting meta-features, calculating candidate algorithms performance, constructing meta-dataset and training meta-model, etc. Firstly, this paper expounds the concept and framework of algorithm selection based on meta-learning, and reviews related surveys. Secondly, it summarizes the research progress from three aspects: meta-features, meta-learners and meta-model performance measures, introduces typical methods and compares the advantages, disadvantages and application scope of different types of methods. Then, it outlines the application of algorithm selection based on meta-learning in different learning tasks. Next, it utilizes 140 classification datasets, 9 candidate classification algorithms and 5 performance indicators to conduct algorithm selection experiments to compare the performance of different algorithm selection methods. Finally, it analyzes the current challenges and problems, and discusses future development directions. [ABSTRACT FROM AUTHOR]

A wide variety of methods and techniques with multiple characteristics are used in solving replenishment, production and distribution planning problems. Selecting a solution method (either a solver or an algorithm) when attempting to solve an optimization problem involves considerable difficulty. Identifying the best solution method among the many available ones is a complex activity that depends partly on human experts or a random trial-and-error procedure. This paper addresses the challenge of recommending a solution method for replenishment, production and distribution planning problems by proposing a decision-making tool for algorithm selection based on the fuzzy TOPSIS approach. This approach considers a collection of the different most commonly used solution methods in the literature, including distinct types of algorithms and solvers. To evaluate a solution method, 13 criteria were defined that all address several important dimensions when solving a planning problem, such as the computational difficulty, scheduling knowledge, mathematical knowledge, algorithm knowledge, mathematical modeling software knowledge and expected computational performance of the solution methods. An illustrative example is provided to demonstrate how planners apply the approach to select a solution method. A sensitivity analysis is also performed to examine the effect of decision maker biases on criteria ratings and how it may affect the final selection. The outcome of the approach provides planners with an effective and systematic decision support tool to follow the process of selecting a solution method. [ABSTRACT FROM AUTHOR]

Predictive maintenance refers to predicting malfunctions using data from monitoring equipment and process performance measurements. Machine learning algorithms and techniques are often used to analyze equipment monitoring data. Machine learning is the process in which a computer can work more precisely by collecting and analyzing data. It is often the case that machine learning algorithms use supervised learning, in which labelled data is used to feed the algorithm. However, there are many supervised machine learning algorithms available. Therefore, choosing the best-supervised machine learning algorithm to resolve predictive maintenance issues is not trivial. This paper aims to increase the performance of predictive maintenance and achieve its goals by selecting the most suitable supervised machine learning algorithm. Based on the most commonly used criteria in research articles, we selected three supervised machine learning algorithms from a comparative study: Random forest, Decision tree and KNN. We then tested selected algorithms on data from real-world and simulation scenarios. Finally, we conducted the experiment based on vibration analysis and reliability evaluation. We noticed that Random forests and Decision trees obtained slightly the same performance. KNN is a better classification algorithm for extensive volumes of data; on the contrary, Random forest performs better in the case of small datasets. [ABSTRACT FROM AUTHOR]

This paper presents the cognitive module of the Cognitive Architecture for Artificial Intelligence (CAAI) in cyber-physical production systems (CPPS). The goal of this architecture is to reduce the implementation effort of artificial intelligence (AI) algorithms in CPPS. Declarative user goals and the provided algorithm-knowledge base allow the dynamic pipeline orchestration and configuration. A big data platform (BDP) instantiates the pipelines and monitors the CPPS performance for further evaluation through the cognitive module. Thus, the cognitive module is able to select feasible and robust configurations for process pipelines in varying use cases. Furthermore, it automatically adapts the models and algorithms based on model quality and resource consumption. The cognitive module also instantiates additional pipelines to evaluate algorithms from different classes on test functions. CAAI relies on well-defined interfaces to enable the integration of additional modules and reduce implementation effort. Finally, an implementation based on Docker, Kubernetes, and Kafka for the virtualization and orchestration of the individual modules and as messaging technology for module communication is used to evaluate a real-world use case. [ABSTRACT FROM AUTHOR]

Algorithm selection is the task of choosing an algorithm from a given set of candidate algorithms when faced with a particular problem instance. Algorithm selection via machine learning (ML) has recently been successfully applied for various problem classes, including computationally hard problems such as SAT. In this paper, we study algorithm selection for software validation, i.e., the task of choosing a software validation tool for a given validation instance. A validation instance consists of a program plus properties to be checked on it. The application of machine learning techniques to this task first of all requires an appropriate representation of software. To this end, we propose a dedicated kernel function, which compares two programs in terms of their similarity, thus making the algorithm selection task amenable to kernel-based machine learning methods. Our kernel operates on a graph representation of source code mixing elements of control-flow and program-dependence graphs with abstract syntax trees. Thus, given two such representations as input, the kernel function yields a real-valued score that can be interpreted as a degree of similarity. We experimentally evaluate our kernel in two learning scenarios, namely a classification and a ranking problem: (1) selecting between a verification and a testing tool for bug finding (i.e., property violation), and (2) ranking several verification tools, from presumably best to worst, for property proving. The evaluation, which is based on data sets from the annual software verification competition SV-COMP, demonstrates our kernel to generalize well and to achieve rather high prediction accuracy, both for the classification and the ranking task. [ABSTRACT FROM AUTHOR]

Item Response Theory (IRT) has been proposed within the field of Educational Psychometrics to assess student ability as well as test question difficulty and discrimination power. More recently, IRT has been applied to evaluate machine learning algorithm performance on a single classification dataset, where the student is now an algorithm, and the test question is an observation to be classified by the algorithm. In this paper we present a modified IRT-based framework for evaluating a portfolio of algorithms across a repository of datasets, while simultaneously eliciting a richer suite of characteristics - such as algorithm consistency and anomalousness - that describe important aspects of algorithm performance. These characteristics arise from a novel inversion and reinterpretation of the traditional IRT model without requiring additional dataset feature computations. We test this framework on algorithm portfolios for a wide range of applications, demonstrating the broad applicability of this method as an insightful algorithm evaluation tool. Furthermore, the explainable nature of IRT parameters yield an increased understanding of algorithm portfolios. [ABSTRACT FROM AUTHOR]

Copyright of Radio Electronics, Computer Science, Control is the property of Zaporizhzhia National Technical University and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

We investigate an important research question for solving the car sequencing problem, that is, which characteristics make an instance hard to solve? To do so, we carry out an instance space analysis for the car sequencing problem, by extracting a vector of problem features to characterize an instance. In order to visualize the instance space, the feature vectors are projected onto a 2-D space using dimensionality reduction techniques. The resulting 2-D visualizations provide new insights into the characteristics of the instances used for testing and how these characteristics influence the behaviours of an optimization algorithm. This analysis guides us in constructing a new set of benchmark instances with a range of instance properties. We demonstrate that these new instances are more diverse than the previous benchmarks, including some instances that are significantly more difficult to solve. We introduce two new algorithms for solving the car sequencing problem and compare them with four existing methods from the literature. Our new algorithms are shown to perform competitively for this problem but no single algorithm can outperform all others over all instances. This observation motivates us to build an algorithm selection model based on machine learning, to identify the niche in the instance space that an algorithm is expected to perform well on. Our analysis helps to understand problem hardness and select an appropriate algorithm for solving a given car sequencing problem instance. [ABSTRACT FROM AUTHOR]

Today, the algorithm selection paradigm has become one of the promising approaches in the field of optimization problems. Its main goal is to solve each case of an optimization problem with the most accurate algorithm using machine learning techniques. This paper treats the issue of the algorithm selection for the Single Machine Scheduling Problem with Early/Tardy jobs by adapting three meta-heuristics from the state-of-the-art, namely genetic algorithm, particle swarm optimization, and tabu search. In the proposed framework, we combine the running time and the cost function to get a new performance criterion. A large set composed of 98000 instances of the problem is generated with 12 features characterizing each instance. We carry a statistical comparison of the implemented meta-heuristics, and we evaluate 10 classifiers. It can be deduced that the Dagging algorithm combined with the Random Forest is the most likely to be the best classifier, which achieves 88.44% of the maximum accuracy.

In this paper, we propose a graph classification approach for automatically determining whether to use a monolithic or a decomposition-based solution method. In this approach, an optimization problem is represented as a graph that captures the structural and functional coupling among the variables and constraints of the problem via an appropriate set of features. Given this representation, a graph classifier can be built to assist a solver in selecting the best solution strategy for a given problem with respect to some metric of choice. The proposed approach is used to develop a classifier that determines whether a convex Mixed Integer Nonlinear Programming problem should be solved using branch and bound or the outer approximation algorithm. Finally, it is shown how the learned classifier can be incorporated into existing mixed integer optimization solvers. • A deep learning framework is proposed to decide when to use a decomposition. • A feature-based graph representation of optimization problems is proposed. • The proposed approach is applied to Mixed Integer Nonlinear optimization problems. • Integration of the proposed approach with optimization solvers is discussed. [ABSTRACT FROM AUTHOR]

Classification is the key and most widely studied paradigm in machine learning community. The selection of appropriate classification algorithm for a particular problem is a challenging task, formally known as algorithm selection problem (ASP) in literature. It is increasingly becoming focus of research in machine learning community. Meta-learning has demonstrated substantial success in solving ASP, especially in the domain of classification. Considerable progress has been made in classification algorithm recommendation and researchers have proposed various methods in literature that tackles ASP in many different ways in meta-learning setup. Yet there is a lack of survey and comparative study that critically analyze, summarize and assess the performance of existing methods. To fill these gaps, in this paper we first present a literature survey of classification algorithm recommendation methods. The survey shed light on the motivational reasons for pursuing classifier selection through meta-learning and comprehensively discusses the different phases of classifier selection based on a generic framework that is formed as an outcome of reviewing prior works. Subsequently, we critically analyzed and summarized the existing studies from the literature in three important dimensions i.e., meta-features, meta-learner and meta-target. In the second part of this paper, we present extensive comparative evaluation of all the prominent methods for classifier selection based on 17 classification algorithms and 84 benchmark datasets. The comparative study quantitatively assesses the performance of classifier selection methods and highlight the limitations and strengths of meta-features, meta-learners and meta-target in classification algorithm recommendation system. Finally, we conclude this paper by identifying current challenges and suggesting future work directions. We expect that this work will provide baseline and a solid overview of state of the art works in this domain to new researchers, and will steer future research in this direction.

Artificial intelligence is considered to be a significant technology for driving the future evolution of smart manufacturing environments. At the same time, automated guided vehicles (AGVs) play an essential role in manufacturing systems due to their potential to improve internal logistics by increasing production flexibility. Thereby, the productivity of the entire system relies on the quality of the schedule, which can achieve production cost savings by minimizing delays and the total makespan. However, traditional scheduling algorithms often have difficulties in adapting to changing environment conditions, and the performance of a selected algorithm depends on the individual scheduling problem. Therefore, this paper aimed to analyze the scheduling problem classes of AGVs by applying design science research to develop an algorithm selection approach. The designed artifact addressed a catalogue of characteristics that used several machine learning algorithms to find the optimal solution strategy for the intended scheduling problem. The contribution of this paper is the creation of an algorithm selection method that automatically selects a scheduling algorithm, depending on the problem class and the algorithm space. In this way, production efficiency can be increased by dynamically adapting the AGV schedules. A computational study with benchmark literature instances unveiled the successful implementation of constraint programming solvers for solving JSSP and FJSSP scheduling problems and machine learning algorithms for predicting the most promising solver. The performance of the solvers strongly depended on the given problem class and the problem instance. Consequently, the overall production performance increased by selecting the algorithms per instance. A field experiment in the learning factory at Reutlingen University enabled the validation of the approach within a running production scenario.

This paper presents a meta-algorithm for approximating the Pareto optimal set of costly black-box multiobjective optimization problems given a limited number of objective function evaluations. The key idea is to switch among different algorithms during the optimization search based on the predicted performance of each algorithm at the time. Algorithm performance is modeled using a machine learning technique based on the available information. The predicted best algorithm is then selected to run for a limited number of evaluations. The proposed approach is tested on several benchmark problems and the results are compared against those obtained using any one of the candidate algorithms alone. [ABSTRACT FROM AUTHOR]

The process of extracting valuable and novel insights from raw data involves a series of complex steps. In the realm of Automated Machine Learning (AutoML), a significant research focus is on automating aspects of this process, specifically tasks like selecting algorithms and optimising their hyper-parameters. A particularly challenging task in AutoML is automatic workflow composition (AWC). AWC aims to identify the most effective sequence of data preprocessing and machine learning algorithms, coupled with their best hyper-parameters, for a specific dataset. However, existing AWC methods are limited in how many and in what ways they can combine algorithms within a workflow. Addressing this gap, this paper introduces EvoFlow , a grammar-based evolutionary approach for AWC. EvoFlow enhances the flexibility in designing workflow structures, empowering practitioners to select algorithms that best fit their specific requirements. EvoFlow stands out by integrating two innovative features. First, it employs a suite of genetic operators, designed specifically for AWC, to optimise both the structure of workflows and their hyper-parameters. Second, it implements a novel updating mechanism that enriches the variety of predictions made by different workflows. Promoting this diversity helps prevent the algorithm from overfitting. With this aim, EvoFlow builds an ensemble whose workflows differ in their misclassified instances. To evaluate EvoFlow 's effectiveness, we carried out empirical validation using a set of classification benchmarks. We begin with an ablation study to demonstrate the enhanced performance attributable to EvoFlow 's unique components. Then, we compare EvoFlow with other AWC approaches, encompassing both evolutionary and non-evolutionary techniques. Our findings show that EvoFlow 's specialised genetic operators and updating mechanism substantially outperform current leading methods in predictive performance. Additionally, EvoFlow is capable of discovering workflow structures that other approaches in the literature have not considered. • A grammar-guided genetic programming approach for automated workflow composition. • The grammar support workflows with more flexible structure than previous methods. • An ensemble of workflows is built to guarantee diversity in terms of predictions. • Our method outperforms the existing methods in the literature. • Discovery of a wider spectrum of solutions than the baseline evolutionary method. [ABSTRACT FROM AUTHOR]

Subject of Research. The paper deals with research of clustering algorithms for hyperparameters optimization used in machine learning. Model selection problem is comprehensively studied, and the need of the tradeoff between exploration and exploitation is identified. Thus, the problem is reduced to multi-armed bandit problem. Method. The paper presented the approach for simultaneous algorithm selection and hyperparameters optimization. We used solution of the Multiarmed Bandit problem and considered Softmax- and UCB1-based algorithm variants in combination with different reward functions. Main Results. Experiments on various datasets from UCI repository were carried out. The results of experiments confirmed that proposed algorithms in general achieve significantly better results than exhaustive search method. It also helped to determine the most promising version of the algorithm we propose. Practical Relevance. The suggested algorithm can be successfully used for model selection and configuration for clustering algorithms, and can be applied in a wide range of clustering tasks in various areas, including biology, psychology, and image analysis.

Abstract In practical data mining, a wide range of classification algorithms is employed for prediction tasks. However, selecting the best algorithm poses a challenging task for machine learning practitioners and experts, primarily due to the inherent variability in the characteristics of classification problems, referred to as datasets, and the unpredictable performance of these algorithms. Dataset characteristics are quantified in terms of meta-features, while classifier performance is evaluated using various performance metrics. The assessment of classifiers through empirical methods across multiple classification datasets, while considering multiple performance metrics, presents a computationally expensive and time-consuming obstacle in the pursuit of selecting the optimal algorithm. Furthermore, the scarcity of sufficient training data, denoted by dimensions representing the number of datasets and the feature space described by meta-feature perspectives, adds further complexity to the process of algorithm selection using classical machine learning methods. This research paper presents an integrated framework called eML-CBR that combines edge edge-ML and case-based reasoning methodologies to accurately address the algorithm selection problem. It adapts a multi-level, multi-view case-based reasoning methodology, considering data from diverse feature dimensions and the algorithms from multiple performance aspects, that distributes computations to both cloud edges and centralized nodes. On the edge, the first-level reasoning employs machine learning methods to recommend a family of classification algorithms, while at the second level, it recommends a list of the top-k algorithms within that family. This list is further refined by an algorithm conflict resolver module. The eML-CBR framework offers a suite of contributions, including integrated algorithm selection, multi-view meta-feature extraction, innovative performance criteria, improved algorithm recommendation, data scarcity mitigation through incremental learning, and an open-source CBR module, reshaping research paradigms. The CBR module, trained on 100 datasets and tested with 52 datasets using 9 decision tree algorithms, achieved an accuracy of 94% for correct classifier recommendations within the top k=3 algorithms, making it highly suitable for practical classification applications.

Despite the advances made in the last decade in automated planning, no planner outperforms all the others in every known benchmark domain. This observation motivates the idea of selecting different planning algorithms for different domains. Moreover, the planners' performances are affected by the structure of the search space, which depends on the encoding of the considered domain. In many domains, the performance of a planner can be improved by exploiting additional knowledge, for instance, in the form of macro-operators or entanglements. In this paper we propose ASAP, an automatic Algorithm Selection Approach for Planning that: (i) for a given domain initially learns additional knowledge, in the form of macro-operators and entanglements, which is used for creating different encodings of the given planning domain and problems, and (ii) explores the 2 dimensional space of available algorithms, defined as encodings-planners couples, and then (iii) selects the most promising algorithm for optimising either the runtimes or the quality of the solution plans. [ABSTRACT FROM AUTHOR]

Instance Space Analysis (ISA) is a recently developed methodology to (a) support objective testing of algorithms and (b) assess the diversity of test instances. Representing test instances as feature vectors, the ISA methodology extends Rice’s 1976 Algorithm Selection Problem framework to enable visualization of the entire space of possible test instances, and gain insights into how algorithm performance is affected by instance properties. Rather than reporting algorithm performance on average across a chosen set of test problems, as is standard practice, the ISA methodology offers a more nuanced understanding of the unique strengths and weaknesses of algorithms across different regions of the instance space that may otherwise be hidden on average. It also facilitates objective assessment of any bias in the chosen test instances and provides guidance about the adequacy of benchmark test suites. This article is a comprehensive tutorial on the ISA methodology that has been evolving over several years, and includes details of all algorithms and software tools that are enabling its worldwide adoption in many disciplines. A case study comparing algorithms for university timetabling is presented to illustrate the methodology and tools. [ABSTRACT FROM AUTHOR]

In real world images, many algorithms for adaptive contours detection exist and various improvements to the contours detection have been proposed. The reason for such diversity is that real world images contains heterogeneous mixtures of features and each of the available algorithms exploits some of these features. Thus, depending on the image, different algorithms shows different quality of result. In this paper we propose a method that improves the result adaptive contours detection by using an algorithm selection approach. Previous methods using the algorithm selection approach have been focusing only on images with a particular class of features (artificial, cellular) because of the complexity of real world images. In order to successfully solve this problem we first determine a set of distinctive features of each algorithm using machine learning. Then using these distinctive features we teach an algorithm selector to select best algorithm when a set of features is provided. Finally, we propose a method to split the input image into sub regions that are selected in such a manner that improves the quality of the image processing result. The proposed algorithm is verified on the set of benchmarks and its performance is comparable and better in many cases than the currently best contour detection algorithms. [ABSTRACT FROM AUTHOR]

This paper presents a probabilistic framework based on Bayesian theory for the performance prediction and selection of an optimal segmentation algorithm. The framework models the optimal algorithm selection process as one that accounts for the information content of an input image as well as the behavioral properties of a particular candidate segmentation algorithm. The input image information content is measured in terms of image features while the candidate segmentation algorithm’s behavioral characteristics are captured through the use of segmentation quality features. Gaussian probability distribution models are used to learn the required relationships between the extracted image and algorithm features and the framework tested on the Berkeley Segmentation Dataset using four candidate segmentation algorithms. [ABSTRACT FROM AUTHOR]

This paper addresses the question of allocating computational resources among a set of algorithms to achieve the best performance on scheduling problems. Our primary motivation in addressing this problem is to reduce the expertise needed to apply optimization technology. Therefore, we investigate algorithm control techniques that make decisions based only on observations of the improvement in solution quality achieved by each algorithm. We call our approach “low knowledge” since it does not rely on complex prediction models, either of the problem domain or of algorithm behavior. We show that a low-knowledge approach results in a system that achieves significantly better performance than all of the pure algorithms without requiring additional human expertise. Furthermore the low-knowledge approach achieves performance equivalent to a perfect high-knowledge classification approach. [ABSTRACT FROM AUTHOR]

Abstract The recent evolution of machine learning (ML) algorithms and the high level of expertise required to use them have fuelled the demand for non-experts solutions. The selection of an appropriate algorithm and the configuration of its hyperparameters is among the most complicated tasks while applying ML to new problems. It necessitates well awareness and knowledge of ML algorithms. The algorithm selection problem (ASP) is defined as the process of identifying the algorithm (s) that can deliver top performance for a particular problem, task, and evaluation measure. In this context, meta-learning is one of the approaches to achieve this objective by using prior learning experiences to assist the learning process on unseen problems and tasks. As a data-driven approach, appropriate data characterization is of vital importance for the meta-learning. Nonetheless, the recent literature witness a variety of data characterization techniques including simple, statistical and information theory based measures. However, their quality still needs to be improved. In this paper, a new Autoencoder-kNN (AeKNN) based meta-model with built-in latent features extraction is proposed. The approach is aimed to extract new characterizations of the data, with lower dimensionality but more significant and meaningful features. AeKNN internally uses a deep autoencoder as a latent features extractor from a set of existing meta-features induced from the dataset. From this new features vectors the computed distances are more significant, thus providing a way to accurately recommending top-performing pipelines for previously unseen datasets. In an application on a large-scale hyperparameters optimization task for 400 real world datasets with varying schemas as a meta-learning task, we show that AeKNN offers considerable improvements of the classical kNN as well as traditional meta-models in terms of performance.

The No Free Lunch Theorem tells us that no algorithm can beat other algorithms on all types of problems. The algorithm selection structure is proposed to select the most suitable algorithm from a set of algorithms for an unknown optimization problem. This paper introduces an innovative algorithm selection approach called the CNN-HT, which is a two-stage algorithm selection framework. In the first stage, a Convolutional Neural Network (CNN) is employed to classify problems. In the second stage, the Hypothesis Testing (HT) technique is used to suggest the best-performing algorithm based on the statistical analysis of the performance metric of algorithms that address various problem categories. The two-stage approach can adapt to different algorithm combinations without the need to retrain the entire model, and modifications can be made in the second stage only, which is an improvement of one-stage approaches. To provide a more general structure for the classification model, we adopt Exploratory Landscape Analysis (ELA) features of the problem as input and utilize feature selection techniques to reduce the redundant ones. In problem classification, the average accuracy of classifying problems using CNN is 96%, which demonstrates the advantages of CNN compared to Random Forest and Support Vector Machines. After feature selection, the accuracy increases to 98.8%, further improving the classification performance while reducing the computational cost. This demonstrates the effectiveness of the first stage of the CNN-HT method, which provides a basis for algorithm selection. In the experiments, CNN-HT shows the advantages of the second stage algorithm as well as good performance with better average rankings in different algorithm combinations compared to the individual algorithms and another algorithm combination approach.

Abstract A wide range of clustering algorithms exists, most of them expose many hyperparameters, on which clustering partition quality depends. Simultaneous algorithm (model) selection and its hyperparameters optimization is considered to be a sophisticated task, which is known according to some sources as combined algorithm selection and hyperparameter optimization. In this paper, we focus on problem of selecting a clustering algorithm and its hyperparameter vector simultaneously given a dataset in order to achieve the best partition quality. We propose a method for selecting a proper clustering algorithm and its hyperparameter vector using reinforcement learning. Instead of tuning hyperparameters for all available clustering algorithms and selecting one showing the best performance, we make them to compete for time that they can use for optimizing their own hyperparameters. In our algorithm, we use a framework for multi-armed bandit problem, which is a special case of reinforcement learning. Each clustering algorithm is considered as an arm in the multi-armed bandit setting, while assigning a time budget to optimize hyperparameters of a clustering algorithm is considered as playing the corresponding arm. We conducted series of experiments for comparing out reinforcement learning approach to the classical exhaustive search approach. We conducted experiments on 20 datasets from UCI Repository such as Iris, haberman, krvskp, glass and other. We use 19 cluster validity indices to validate the clusters, built by selected and configured algorithm. As a hyperparameter optimization algorithm, we used SMAC. Our approach managed to improve model selection and hyperparameter optimization process, by sustaining the exploration-exploitation trade-off and spending available time budget more wisely. [ABSTRACT FROM AUTHOR]

Constraint Programming (CP) is a powerful technique for solving large-scale combinatorial problems. Solving a problem proceeds in two distinct phases: modelling and solving. Effective modelling has a huge impact on the performance of the solving process. Even with the advance of modern automated modelling tools, search spaces involved can be so vast that problems can still be difficult to solve. To further constrain the model, a more aggressive step that can be taken is the addition of streamliner constraints, which are not guaranteed to be sound but are designed to focus effort on a highly restricted but promising portion of the search space. Previously, producing effective streamlined models was a manual, difficult and time-consuming task. This paper presents a completely automated process to the generation, search and selection of streamliner portfolios to produce a substantial reduction in search effort across a diverse range of problems. The results demonstrate a marked improvement in performance for both Chuffed, a CP solver with clause learning, and lingeling, a modern SAT solver. [ABSTRACT FROM AUTHOR]

In this paper, we investigate how systemic errors due to random sampling impact on automated algorithm selection for bound-constrained, single-objective, continuous black-box optimization. We construct a machine learning-based algorithm selector, which uses exploratory landscape analysis features as inputs. We test the accuracy of the recommendations experimentally using resampling techniques and the hold-one-instance-out and hold-one-problem-out validation methods. The results demonstrate that the selector remains accurate even with sampling noise, although not without trade-offs. [ABSTRACT FROM AUTHOR]

Algorithm selection (AS), selecting the algorithm best suited for a particular problem instance, is acknowledged to be a key issue to make the best out of algorithm portfolios. This paper presents a collaborative filtering approach to AS. Collaborative filtering, popularized by the Netflix challenge, aims to recommend the items that a user will most probably like, based on the previous items she liked, and the items that have been liked by other users. As first noted by Stern et al. [47] , algorithm selection can be formalized as a collaborative filtering problem, by considering that a problem instance “likes better“ the algorithms that achieve better performance on this particular instance. Two merits of collaborative filtering (CF) compared to the mainstream algorithm selection (AS) approaches are the following. Firstly, mainstream AS requires extensive and computationally expensive experiments to learn a performance model, with all algorithms launched on all problem instances, whereas CF can exploit a sparse matrix, with a few algorithms launched on each problem instance. Secondly, AS learns a performance model as a function of the initial instance representation, whereas CF builds latent factors to describe algorithms and instances, and uses the associated latent metrics to recommend algorithms for a specific problem instance. A main contribution of the proposed algorithm recommender Alors system is to handle the cold start problem – emitting recommendations for a new problem instance – through the non-linear modeling of the latent factors based on the initial instance representation, extending the linear approach proposed by Stern et al. [47] . The merits and generality of Alors are empirically demonstrated on the ASLib [6] and OpenML [53] benchmarks. [ABSTRACT FROM AUTHOR]

We propose a novel technique for algorithm-selection, applicable to optimisation domains in which there is implicit sequential information encapsulated in the data, e.g., in online bin-packing. Specifically we train two types of recurrent neural networks to predict a packing heuristic in online bin-packing, selecting from four well-known heuristics. As input, the RNN methods only use the sequence of item-sizes. This contrasts to typical approaches to algorithm-selection which require a model to be trained using domain-specific instance features that need to be first derived from the input data. The RNN approaches are shown to be capable of achieving within 5% of the oracle performance on between 80.88 and 97.63% of the instances, depending on the dataset. They are also shown to outperform classical machine learning models trained using derived features. Finally, we hypothesise that the proposed methods perform well when the instances exhibit some implicit structure that results in discriminatory performance with respect to a set of heuristics. We test this hypothesis by generating fourteen new datasets with increasing levels of structure, and show that there is a critical threshold of structure required before algorithm-selection delivers benefit. [ABSTRACT FROM AUTHOR]

AutoML aims at best configuring learning systems automatically. It contains core subtasks of algorithm selection and hyper-parameter tuning. Previous approaches considered searching in the joint hyper-parameter space of all algorithms, which forms a huge but redundant space and causes an inefficient search. We tackle this issue in a cascaded algorithm selection way, which contains an upper-level process of algorithm selection and a lower-level process of hyper-parameter tuning for algorithms. While the lower-level process employs an anytime tuning approach, the upper-level process is naturally formulated as a multi-armed bandit, deciding which algorithm should be allocated one more piece of time for the lower-level tuning. To achieve the goal of finding the best configuration, we propose the Extreme-Region Upper Confidence Bound (ER-UCB) strategy. Unlike UCB bandits that maximize the mean of feedback distribution, ER-UCB maximizes the extreme-region of feedback distribution. We first consider stationary distributions and propose the ER-UCB-S algorithm that has $O(K\ln n)$ O (K ln n) regret upper bound with $K$ K arms and $n$ n trials. We then extend to non-stationary settings and propose the ER-UCB-N algorithm that has $O(Kn^\nu)$ O (K n ν) regret upper bound, where $\frac{2}{3}<\nu <1$ 2 3 < ν < 1 . Finally, empirical studies on synthetic and AutoML tasks verify the effectiveness of ER-UCB-S/N by their outperformance in corresponding settings. [ABSTRACT FROM AUTHOR]

Machine learning (ML) models have been used to improve the quality of different physical design steps, such as timing analysis, clock tree synthesis, and routing. However, so far very few works have addressed the problem of algorithm selection during physical design, which can drastically reduce the computational effort of some steps. This work proposes a legalization algorithm selection framework using deep convolutional neural networks (CNNs). To extract features, we used snapshots of circuit placements and used transfer learning to train the models using pretrained weights of the Squeezenet architecture. By doing so, we can greatly reduce the training time and required data even though the pretrained weights come from a different problem. We performed extensive experimental analysis of ML models, providing details on how we chose the parameters of our model, such as CNN architecture, learning rate, and number of epochs. We evaluated the proposed framework by training a model to select between different legalization algorithms according to cell displacement and wirelength variation. The trained models achieved an average $F$ -score of 0.98 when predicting cell displacement and 0.83 when predicting wirelength variation. When integrated into the physical design flow, the cell displacement model achieved the best results on 15 out of 16 designs, while the wirelength variation model achieved that for 10 out of 16 designs, being better than any individual legalization algorithm. Finally, using the proposed ML model for algorithm selection resulted in a speedup of up to $10\times $ compared to running all the algorithms separately. [ABSTRACT FROM AUTHOR]

This paper considers the challenge of recognizing how the properties of a graph determine the performance of graph coloring algorithms. In particular, we examine how spectral properties of the graph make the graph coloring task easy or hard. We address the question of when an exact algorithm is likely to provide a solution in a reasonable computation time, when we are better off using a quick heuristic, and how the properties of the graph influence algorithm performance. A new methodology is introduced to visualize a collection of graphs in an instance space defined by their properties, with an optimal feature selection approach included to ensure that the instance space preserves the topology of an algorithm performance metric. In this instance space we can reveal how different algorithms perform across a variety of instance classes with various properties. We demonstrate the methodology using a sophisticated branch and bound exact algorithm, and the faster heuristic approaches of integer rounding of a linear programming relaxation of the graph coloring formulation, as well as a greedy degree-saturation heuristic. Our results demonstrate that spectral properties of the graph instances can provide useful descriptions of when each of these algorithms will provide the best solution for a given computational effort. [ABSTRACT FROM AUTHOR]