Search

Dissertação de mestrado em Informatics Engineering, In the last years, the number of Machine Learning algorithms and their parameters has increased significantly. This allows for more accurate models to be found, but it also increases the complexity of the task of training a model, as the search space expands significantly. As datasets keep growing in size, traditional approaches based on extensive search start to become costly in terms of computational resources and time, especially in data streaming scenarios. With this growth, new challenges in Machine Learning started to appear. The speed at which data arrives and different ways of storing data are forcing organizations to address and explore new ways of adapting fast enough so their ML models don’t become obsolete. This dissertation aims to develop an approach based on meta-learning that tackles two main challenges: predict ing the performance metrics of a future model and recommending the best algorithm/configuration for training a model for a specific Machine Learning problem. Throughout this dissertation, all the study objectives and questions, along with the relevant contextualization will be exposed. The proposed solution, when compared to an AutoML approach is up to 130x faster and only 2% worse in terms of average model quality, showing it is a good solution for scenarios in which models need to be updated regularly, such as in streaming scenarios with Big Data, in which some accuracy can be traded for a much shorter model training time., Nos últimos anos, o número de algoritmos de Machine Learning e seus parâmetros aumentou significativamente. Isso permite que modelos mais precisos sejam encontrados, mas também aumenta a complexidade da tarefa de treinar um modelo, pois o espaço de busca expande-se significativamente. À medida que os conjuntos de dados continuam a crescer em tamanho, abordagens tradicionais baseadas em uma pesquisa extensiva começam a se tornar caras em termos de recursos computacionais e tempo, especialmente em cenários de streaming de dados. Com esse crescimento, novos desafios no Machine Learning começaram a aparecer. A velocidade com que os dados chegam e as diferentes maneiras de armazenar dados estão a forçar as organizações a abordar e explorar novas formas de se adaptar rápido o suficiente para que os seus modelos de ML não se tornem obsoletos. Esta dissertação visa desenvolver uma abordagem baseada em Meta-Learning que aborda dois desafios principais: prever as métricas de desempenho de um modelo futuro e recomendar o melhor algoritmo/configuração para treinar um modelo para um problema específico de Machine Learning. Ao longo desta dissertação, serão expostos todos os objetivos e questões do estudo, juntamente com a contextualização relevante. A solução proposta, quando comparada a uma abordagem AutoML é até 130x mais rápida e apenas 2% pior em termos de qualidade média do modelo, mostrando que é uma boa solução para cenários em que os modelos precisam ser atualizados regularmente, como em cenários de streaming com Big Data, em que alguma precisão pode ser negociada por um tempo de treino de modelo muito menor.

Image segmentation is a key issue in image processing. New image segmentation algorithms have been proposed in the last years. However, there is no optimal algorithm for every image processing task. The selection of the most suitable algorithm usually occurs by testing every possible algorithm or using knowledge from previous problems. These processes can have a high computational cost. Meta-learning has been successfully used in the machine learning research community for the recommendation of the most suitable machine learning algorithm for a new dataset. We believe that meta-learning can also be useful to select the most suitable image segmentation algorithm. This hypothesis is investigated in this paper. For such, we perform experiments with eight segmentation algorithms from two approaches using a segmentation benchmark of 300 images and 2100 augmented images. The experimental results showed that meta-learning can recommend the most suitable segmentation algorithm with more than 80% of accuracy for one group of algorithms and with 69% for the other group, overcoming the baselines used regarding recommendation and segmentation performance.

Trabalho de Projeto do Mestrado Integrado em Engenharia Biomédica apresentado à Faculdade de Ciências e Tecnologia As aplicações de aprendizagem-máquina são desafiadas pelos fatores de complexidade dos dados. Estes são responsáveis pela degradação da qualidade dos dados, sendo que lidar com estes fatores é uma tarefa importante para evitar a degradação do desempenho de classificadores. Dentro dos fatores de complexidade, o desequilíbrio de classes, que é característico em diversas bases de dados biomédicas, normalmente é abordado com algoritmos de pré-processamento, que são eficazes em melhorar o desempenho de tarefas de classificação.Dado que a seleção do algoritmo mais indicado para lidar com o desequilíbrio de classes muitas vezes é baseada em abordagens de "força-bruta", sistemas de recomendação têm sido desenvolvidos de forma a providenciar a estratégia ótima a utilizar para um dado problema, baseado nas meta-características do conjunto de dados. No entanto, embora diversos sistemas de recomendação tenham sido bem-sucedidos, estes não têm a capacidade de fornecer conhecimento interpretável, uma vez que apenas a entrada (conjunto de dados) e a saída (estratégia recomendada) destes sistemas são conhecidas.De forma a solucionar este problema, o objetivo da presente dissertação é estudar as relações entre meta-características dos dados e algoritmos de pré-processamento no desempenho de classificadores. Para alcançar os objetivos, uma metodologia de meta-aprendizagem foi desenvolvida, baseada em "Exceptional Preferences Mining", que demonstrou ser apropriada para fornecer condições interpretáveis, referentes às relações entre as meta-características dos dados e o ranking de algoritmos de pré-processamento. Em adição, uma nova métrica é proposta com a finalidade de salientar os subgrupos onde grandes variações são observadas, no desempenho de vários algoritmos de pré-processamento.As experiências realizadas incluem 163 bases de dados, pré-processadas com 9 estratégias a nível dos dados, de onde meta-características provenientes de 8 grupos foram extraídas. Os resultados mais relevantes salientam que a utilização de uma estratégia para lidar com o desequilíbrio de classes pode nem sempre ser necessária e que não existe uma relação evidente com a proporção de pontos entre as classes maioritária e minoritária, mas sim com a associação do desequilíbrio de classes com outros fatores de complexidade. Adicionalmente, os domínios de aplicação de estratégias para lidar com distribuições assimétricas de classes são individualmente descritas, para além de outros resultados úteis para o desenvolvimento de novos sistemas de recomendação. Machine learning applications are challenged by data difficulty factors, which are responsible for the degradation of data quality and dealing with them is a demanding task. Among the difficulty factors, class imbalance, which is noticeable in many biomedical databases, is often tackled with preprocessing algorithms that effectively improve classification performance.Since the selection of an imbalance strategy for a problem often encompasses "brute-force" approaches, recommendation systems have been developed to provide optimal imbalance strategies for the problem at hand, based on the meta-characteristics of the dataset. However, despite the success of such systems, arguably these do not provide any insightful information, since only the inputs (datasets) and outputs (recommended imbalance strategies) of these systems are provided.Addressing this issue, the purpose of this dissertation is to provide a study of the relations between data meta-characteristics and imbalance strategies in the performance of classifiers. To this end, a meta-learning-based framework was developed, based on Exceptional Preferences Mining, which has proven to be suitable to deliver interpretable conditions, concerning the relations between data meta-characteristics and the ranking of preprocessing algorithms. Additionally, a novel metric was proposed, which is suitable to highlight the subgroups where steep performance variations are observable, among the performance of imbalance strategies.The experiments considered 163 datasets, where meta-features from 8 groups were extracted and preprocessed with 9 data-level imbalance strategies. The main findings include that employing an imbalance strategy may not always be required and that there is no evident relation with the imbalance ratio, rather with the association of imbalance with other difficulty factors. Moreover, the domains of application of individual imbalance strategies are described, among other findings suitable for the design of novel recommendation systems.

Hierarchical classification is a special type of classification task where the class labels are organised into a hierarchy, with more generic class labels being ancestors of more specific ones. Meta-learning for classification-algorithm recommendation consists of recommending to the user a classification algorithm, from a pool of candidate algorithms, for a dataset, based on the past performance of the candidate algorithms in other datasets. Meta-learning is normally used in conventional, non-hierarchical classification. By contrast, this paper proposes a meta-learning approach for more challenging task of hierarchical classification, and evaluates it in a large number of bioinformatics datasets. Hierarchical classification is especially relevant for bioinformatics problems, as protein and gene functions tend to be organised into a hierarchy of class labels. This work proposes meta-learning approach for recommending the best hierarchical classification algorithm to a hierarchical classification dataset. This work’s contributions are: 1) proposing an algorithm for splitting hierarchical datasets into new datasets to increase the number of meta-instances, 2) proposing meta-features for hierarchical classification, and 3) interpreting decision-tree meta-models for hierarchical classification algorithm recommendation., {"references":["L. Schietgat, C. Vens, J. Struyf, H. Blockeel, D. Kocev, and S. Dzeroski,\n\"Predicting gene function using hierarchical multi-label decision tree\nensembles.\" BMC Bioinformatics, vol. 11, no. 2, pp. 1–14, Jan. 2010.","D. Delen, G. Walker, and A. Kadam, \"Predicting breast cancer\nsurvivability: a comparison of three data mining methods,\" Artificial\nIntelligence in Medicine, vol. 34, no. 2, pp. 113–127, 2005.","C. N. Silla Jr. and A. A. Freitas, \"A Survey of Hierarchical Classification\nAcross Different Application Domains,\" Data Mining and Knowledge\nDiscovery, vol. 44, no. 1-2, pp. 31–72, 2011.","P. Brazdil, C. G. Carrier, C. Soares, and R. Vilalta, Metalearning:\nApplications to data mining. Springer, 2008.","C. Vens, L. Schietgat, J. Struyf, H. Blockeel, and D. Kocev, \"Predicting\nGene Function using Predictive Clustering Trees,\" BMC Bioinformatics,\nvol. 11, no. 2, pp. 1–25, 2010.","D. Koller and M. Sahami, \"Hierarchically Classifying Documents Using\nVery Few Words,\" in Proceedings of the 14th International Conference\non Machine Learning, ser. ICML '97. San Francisco, CA, USA:\nMorgan Kaufmann Publishers Inc., 1997, pp. 170—-178.","M. A. Harris, J. Clark, A. Ireland, J. Lomax et al., \"The Gene Ontology\n(GO) database and informatics resource.\" Nucleic Acids Research,\nvol. 32, pp. D258–61, Jan. 2004.","H. Blockeel, M. Bruynooghe, S. Dzeroski, J. Ramon, and J. Struyf,\n\"Hierarchical Multi-Classification,\" in Proceedings of the ACM SIGKDD\n2002 workshop on multi-relational data mining (MRDM 2002), 2002,\npp. 21–35.","C. Vens, J. Struyf, L. Schietgat, S. Dzeroski, and H. Blockeel, \"Decision\nTrees for Hierarchical Multi-label Classification,\" Machine Learning,\nvol. 73, no. 2, pp. 185–214, Aug. 2008.\n[10] F. Fabris and A. A. Freitas, \"Dependency Network Methods\nfor Hierarchical Multi-label Classification of Gene Functions,\"\nin Proceedings of the 2014 IEEE International Conference on\nComputational Intelligence and Data Mining, Orlando, Florida, Dec.\n2014, pp. 241–248.\n[11] F. Fabris, A. Freitas, and J. Tullet, \"An Extensive Empirical\nComparison of Probabilistic Hierarchical Classifiers in Datasets of\nAgeing-Related Genes,\" IEEE/ACM transactions on computational\nbiology and bioinformatics/IEEE, ACM, pp. 1–14, dec 2015. [Online].\nAvailable: http://europepmc.org/abstract/MED/26661786\n[12] F. Fabris and A. A. Freitas, \"A Novel Extended Hierarchical Dependence\nNetwork Method Based on non-Hierarchical Predictive Classes and\nApplications to Ageing-Related Data,\" in Proceedings of the 2015\nIEEE 27th International Conference on Tools with Artificial Intelligence\n(ICTAI). IEEE, 2015, pp. 294–301.\n[13] L. d. C. Merschmann and A. A. Freitas, \"An Extended Local\nHierarchical Classifier for Prediction of Protein and Gene Functions,\"\nin Data Warehousing and Knowledge Discovery, ser. Lecture Notes in\nComputer Science. Springer, 2013, vol. 8057, pp. 159–171.\n[14] A. A. Freitas, \"Comprehensible Classification Models - a position\npaper,\" ACM SIGKDD Explor. Newsl., vol. 15, no. 1, pp. 1–10, 2014.\n[15] A. Vellido, J. D. Mart´ın-Guerrero, and P. J. Lisboa, \"Making machine\nlearning models interpretable,\" in In Proc. European Symposium on\nArtificial Neural Networks, Computational Intelligence and Machine\nLearning, vol. 12, 2012, pp. 163–172.\n[16] K. Boyd, K. H. Eng, and C. D. Page, \"Area Under the Precision-Recall\nCurve: Point Estimates and Confidence Intervals,\" in Machine Learning\nand Knowledge Discovery in Databases, ser. Lecture Notes in Computer\nScience. Springer, 2013, vol. 8190, pp. 451–466.\n[17] Y. Peng, P. A. Flach, C. Soares, and P. B. Brazdil, \"Improved dataset\ncharacterisation for meta-learning,\" ser. Lecture Notes in Computer\nScience. Springer, 2002, vol. 2534, pp. 141–152.\n[18] R. Leite and Pavel Brazdil, \"Active Testing Strategy to Predict the\nBest Classification Algorithm via Sampling and Meta-Learning,\" in\nProceedings of the 2010 conference on ECAI 2010: 19th European\nConference on Artificial Intelligence. IOS Press, 2010, pp. 309–314.\n[19] Q. Sun and B. Pfahringer, \"Pairwise meta-rules for better\nmeta-learning-based algorithm ranking,\" Machine Learning, vol. 93,\nno. 1, pp. 141–161, jul 2013.\n[20] J. N. van Rijn, S. M. Abdulrahman, P. Brazdil, and J. Vanschoren, \"Fast\nalgorithm selection using learning curves,\" in International Symposium\non Intelligent Data Analysis. Springer, 2015, pp. 298–309.\n[21] R. Leite, P. Brazdil, and J. Vanschoren, \"Selecting classification\nalgorithms with active testing,\" in Machine Learning and Data Mining\nin Pattern Recognition, ser. Lecture Notes in Computer Science, 2012,\nvol. 7376, pp. 117–131.\n[22] S. M. Abdulrahman and P. Brazdil, \"Measures for combining accuracy\nand time for meta-learning,\" in Proceedings of the 2014 International\nConference on Meta-learning and Algorithm Selection (MLAS'14), vol.\n1201, 2014, pp. 49–50.\n[23] I. Partalas, R. Babbar, E. Gaussier, and C. Amblard, \"Adaptive classifier\nselection in large-scale hierarchical classification,\" in Lecture Notes in\nComputer Science, vol. 7665, no. 3, 2012, pp. 612–619.\n[24] G. Tsoumakas, I. Katakis, and I. Vlahavas, \"Mining Multi-label Data,\"\nin Data Mining and Knowledge Discovery Handbook, O. Maimon and\nL. Rokach, Eds., 2010, pp. 667–685.\n[25] A. Ruepp, A. Zollner, D. Maier, K. Albermann, J. Hani et al., \"The\nFunCat, a functional annotation scheme for systematic classification of\nproteins from whole genomes,\" Nucleic Acids Research, vol. 32, no. 18,\npp. 5539–5545, 2004.\n[26] R. Tacutu, T. Craig, A. Budovsky, D. Wuttke, G. Lehmann,\nD. Taranukha, J. Costa, V. E. Fraifeld, and J. a. P. de Magalh˜aes,\n\"Human Ageing Genomic Resources: integrated databases and tools for\nthe biology and genetics of ageing.\" Nucleic Acids Research, vol. 41,\nno. Database issue, pp. D1027–D1033, Jan. 2013.\n[27] F. Fabris and A. A. Freitas, \"New KEGG pathway-based interpretable\nfeatures for classifying ageing-related mouse proteins,\" Bioinformatics,\nvol. 32, no. 19, pp. 2988–2995, jun 2016.\n[28] \"HMC Software and Datasets,\" https://dtai.cs.kuleuven.be/clus/\nhmcdatasets/, accessed: 2016-09-23.\n[29] \"Other Bioinformatics Datasets, including ageing-related datasets with\nGO and FunCat classes,\" https://www.cs.kent.ac.uk/people/rpg/ff79/\nFabris Datasets.tar.gz, accessed: 2016-09-23.\n[30] M. Lichman, \"UCI machine learning repository\nhttp://archive.ics.uci.edu/ml,\" 2013. [Online]. Available: http:\n//archive.ics.uci.edu/ml\n[31] B. E. Boser, I. M. Guyon, and V. N. Vapnik, \"A training algorithm for\noptimal margin classifiers,\" in Proceedings of the Fifth Annual Workshop\non Computational Learning Theory, ser. COLT '92. New York, NY,\nUSA: ACM, 1992, pp. 144–152.\n[32] J. R. Quinlan, C4.5: Programs for Machine Learning. San Francisco,\nCA, USA: Morgan Kaufmann Publishers Inc., 1993.\n[33] I. H. Witten and E. Frank, Data Mining: Practical Machine Learning\nTools and Techniques with Java Implementations. San Francisco, CA,\nUSA: Morgan Kaufmann Publishers Inc., 2000.\n[34] C.-C. Chang and C.-J. Lin, \"LIBSVM: A library for support vector\nmachines,\" ACM Transactions on Intelligent Systems and Technology,\nvol. 2, no. 3, pp. 1–27, 2011.\n[35] T. D. Gautheir, \"Detecting Trends Using Spearman's Rank Correlation\nCoefficient,\" Environmental Forensics, vol. 2, no. 4, pp. 359–362, 2001.\n[36] P. B. Brazdil, C. Soares, and J. P. Da Costa, \"Ranking learning\nalgorithms: Using IBL and meta-learning on accuracy and time results,\"\nMachine Learning, vol. 50, no. 3, pp. 251–277, 2003.\n[37] J. Demsar, \"Statistical Comparisons of Classifiers over Multiple Data\nSets,\" Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006."]}