Search

With the growing popularity of machine learning, implementations of the environment for developing and maintaining these models, called MLOps, are becoming more common. The number of publications in this area is relatively small, although growing rapidly. Our goal was to review the current state of the literature in the MLOps area and answer the following research questions: What classes of tools are used in MLOps environments? Which tool implementations are the most popular? What processes are implemented within MLOps? What metrics are used to measure the effectiveness of MLOps implementation? Based on this review, we identified classes of tools included in the MLOps architecture, along with their most popular implementations. While some tools originate from DevOps practices, others, such as Model Orchestrators, Feature Stores, and Model Repositories, are unique to MLOps. We propose a reference MLOps architecture based on these findings and outline the stages of the model production process. We also sought metrics that would allow us to assess and compare the effectiveness of MLOps practices, but unfortunately, we were unable to find a satisfactory answer in this area.

Background: Methods for accurately predicting the prognosis of patients with recurrent hepatolithiasis (RH) after biliary surgery are lacking. This study aimed to develop a model that dynamically predicts the risk of hepatolithiasis recurrence using a machine-learning (ML) approach based on multiple clinical high-order correlation data. Materials and methods: Data from patients with RH who underwent surgery at five centres between January 2015 and December 2020 were collected and divided into training and testing sets. Nine predictive models, which we named the Correlation Analysis and Recurrence Evaluation System (CARES), were developed and compared using machine learning (ML) methods to predict the patients' dynamic recurrence risk within 5 post-operative years. We adopted a k-fold cross validation with k = 10 and tested model performance on a separate testing set. The area under the receiver operating characteristic curve was used to evaluate the performance of the models, and the significance and direction of each predictive variable were interpreted and justified based on Shapley Additive Explanations. Results: Models based on ML methods outperformed those based on traditional regression analysis in predicting the recurrent risk of patients with RH, with Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM) showing the best performance, both yielding an AUC (Area Under the receiver operating characteristic Curve) of∼0.9 or higher at predictions. These models were proved to have even better performance on testing sets than in a 10-fold cross validation, indicating that the model was not overfitted. The SHAP method revealed that immediate stone clearance, final stone clearance, number of previous surgeries, and preoperative CA19-9 index were the most important predictors of recurrence after reoperation in RH patients. An online version of the CARES model was implemented. Conclusion: The CARES model was firstly developed based on ML methods and further encapsulated into an online version for predicting the recurrence of patients with RH after hepatectomy, which can guide clinical decision-making and personalised postoperative surveillance. [ABSTRACT FROM AUTHOR]

Background: Discharge date prediction plays a crucial role in healthcare management, enabling efficient resource allocation and patient care planning. Accurate estimation of the discharge date can optimize hospital operations and facilitate better patient outcomes. Materials and methods: In this study, we employed a systematic approach to develop a discharge date prediction model. We collaborated closely with clinical experts to identify relevant data elements that contribute to the prediction accuracy. Feature engineering was used to extract predictive features from both structured and unstructured data sources. XGBoost, a powerful machine learning algorithm, was employed for the prediction task. Furthermore, the developed model was seamlessly integrated into a widely used Electronic Medical Record (EMR) system, ensuring practical usability. Results: The model achieved a performance surpassing baseline estimates by up to 35.68% in the F1-score. Post-deployment, the model demonstrated operational value by aligning with MS GMLOS and contributing to an 18.96% reduction in excess hospital days. Conclusions: Our findings highlight the effectiveness and potential value of the developed discharge date prediction model in clinical practice. By improving the accuracy of discharge date estimations, the model has the potential to enhance healthcare resource management and patient care planning. Additional research endeavors should prioritize the evaluation of the model's long-term applicability across diverse scenarios and the comprehensive analysis of its influence on patient outcomes. [ABSTRACT FROM AUTHOR]

Machine learning (ML) revolutionized traditional machine fault detection and identification (FDI), as complex-structured models with well-designed unsupervised learning strategies can detect abnormal patterns from abundant data, which significantly reduces the total cost of ownership. However, their opaqueness raised human concern and intrigued the eXplainable artificial intelligence (XAI) concept. Furthermore, the development of ML-based FDI models can be improved fundamentally with machine learning operations (MLOps) guidelines, enhancing reproducibility and operational quality. This study proposes a framework for the continuous development of ML-based FDI solutions, which contains a general structure to simultaneously visualize and check the performance of the ML model while directing the resource-efficient development process. A use case is conducted on sensor data of a hydraulic system with a simple long short-term memory (LSTM) network. Proposed XAI principles and tools supported the model engineering and monitoring, while additional system optimization can be made regarding input data preparation, feature selection, and model usage. Suggested MLOps principles help developers create a minimum viable solution and involve it in a continuous improvement loop. The promising result motivates further adoption of XAI and MLOps while endorsing the generalization of modern ML-based FDI applications with the HITL concept. [ABSTRACT FROM AUTHOR]

BackgroundMethods for accurately predicting the prognosis of patients with recurrent hepatolithiasis (RH) after biliary surgery are lacking. This study aimed to develop a model that dynamically predicts the risk of hepatolithiasis recurrence using a machine-learning (ML) approach based on multiple clinical high-order correlation data.Materials and methodsData from patients with RH who underwent surgery at five centres between January 2015 and December 2020 were collected and divided into training and testing sets. Nine predictive models, which we named the Correlation Analysis and Recurrence Evaluation System (CARES), were developed and compared using machine learning (ML) methods to predict the patients’ dynamic recurrence risk within 5 post-operative years. We adopted a k-fold cross validation with k = 10 and tested model performance on a separate testing set. The area under the receiver operating characteristic curve was used to evaluate the performance of the models, and the significance and direction of each predictive variable were interpreted and justified based on Shapley Additive Explanations.ResultsModels based on ML methods outperformed those based on traditional regression analysis in predicting the recurrent risk of patients with RH, with Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM) showing the best performance, both yielding an AUC (Area Under the receiver operating characteristic Curve) of∼0.9 or higher at predictions. These models were proved to have even better performance on testing sets than in a 10-fold cross validation, indicating that the model was not overfitted. The SHAP method revealed that immediate stone clearance, final stone clearance, number of previous surgeries, and preoperative CA19-9 index were the most important predictors of recurrence after reoperation in RH patients. An online version of the CARES model was implemented.ConclusionThe CARES model was firstly developed based on ML methods and further encapsulated into an online version for predicting the recurrence of patients with RH after hepatectomy, which can guide clinical decision-making and personalised postoperative surveillance.

This paper presents DeepTSF, a comprehensive machine learning operations (MLOps) framework aiming to innovate time series forecasting through workflow automation and codeless modeling. DeepTSF automates key aspects of the machine learning (ML) lifecycle, making it an ideal tool for data scientists and MLops engineers engaged in ML and deep learning (DL)-based forecasting. DeepTSF empowers users with a robust and user-friendly solution, while it is designed to seamlessly integrate with existing data analysis workflows, providing enhanced productivity and compatibility. The framework offers a front-end user interface (UI) suitable for data scientists, as well as other higher-level stakeholders, enabling comprehensive understanding through insightful visualizations and evaluation metrics. DeepTSF also prioritizes security through identity management and access authorization mechanisms. The application of DeepTSF in real-life use cases of the I-NERGY project has already proven DeepTSF’s efficacy in DL-based load forecasting, showcasing its significant added value in the electrical power and energy systems domain.

BackgroundDischarge date prediction plays a crucial role in healthcare management, enabling efficient resource allocation and patient care planning. Accurate estimation of the discharge date can optimize hospital operations and facilitate better patient outcomes.Materials and methodsIn this study, we employed a systematic approach to develop a discharge date prediction model. We collaborated closely with clinical experts to identify relevant data elements that contribute to the prediction accuracy. Feature engineering was used to extract predictive features from both structured and unstructured data sources. XGBoost, a powerful machine learning algorithm, was employed for the prediction task. Furthermore, the developed model was seamlessly integrated into a widely used Electronic Medical Record (EMR) system, ensuring practical usability.ResultsThe model achieved a performance surpassing baseline estimates by up to 35.68% in the F1-score. Post-deployment, the model demonstrated operational value by aligning with MS GMLOS and contributing to an 18.96% reduction in excess hospital days.ConclusionsOur findings highlight the effectiveness and potential value of the developed discharge date prediction model in clinical practice. By improving the accuracy of discharge date estimations, the model has the potential to enhance healthcare resource management and patient care planning. Additional research endeavors should prioritize the evaluation of the model's long-term applicability across diverse scenarios and the comprehensive analysis of its influence on patient outcomes.

Machine learning (ML) revolutionized traditional machine fault detection and identification (FDI), as complex-structured models with well-designed unsupervised learning strategies can detect abnormal patterns from abundant data, which significantly reduces the total cost of ownership. However, their opaqueness raised human concern and intrigued the eXplainable artificial intelligence (XAI) concept. Furthermore, the development of ML-based FDI models can be improved fundamentally with machine learning operations (MLOps) guidelines, enhancing reproducibility and operational quality. This study proposes a framework for the continuous development of ML-based FDI solutions, which contains a general structure to simultaneously visualize and check the performance of the ML model while directing the resource-efficient development process. A use case is conducted on sensor data of a hydraulic system with a simple long short-term memory (LSTM) network. Proposed XAI principles and tools supported the model engineering and monitoring, while additional system optimization can be made regarding input data preparation, feature selection, and model usage. Suggested MLOps principles help developers create a minimum viable solution and involve it in a continuous improvement loop. The promising result motivates further adoption of XAI and MLOps while endorsing the generalization of modern ML-based FDI applications with the HITL concept.

Purpose: In recent years, the use of Steady-State Visual Evoked Potentials (SSVEPs) in Brain-Computer Interface (BCI) systems has dramatically increased across several fields, such as rehabilitation, cognitive impairment, and brain disease or disorder detection, as well as artificial limbs, wheelchairs, and biomechanical systems. In this study, a novel method is proposed to help scientists develop more efficient BCI systems for Machine Learning Operations (MLOps). This study proposed a state-of-the-art method for detecting SSVEP-based stimulation frequencies with statistical models to design an optimal BCI system. Materials and Methods: In this study, the Canonical Correlation Analysis (CCA) method has been implemented to extract features from the accessible-to-the-public Tsinghua University Benchmark dataset. A limited number of subjects are being studied. After completing feature selection methods and selecting the best subset of features using a specified feature selection method, the classification of the best features using machine learning-based classification methods has been completed. Furthermore, it is assumed that scientists will design and implement a system specifically for subjects. Models work for subjects independently. However, because model training is subject-specific, we must execute the proposed methods on each subject separately. Results: The findings indicate that the novel suggested BCI system achieves an average accuracy of 83±9% in stimulation detection, which is higher than that of the traditional CCA approach with an accuracy of 80±11% (p

In the era of big data analytics and AI applications, data provenance is as important as ever, particularly as applications emerge in vital industries like healthcare. Additionally, as the suites of tools and packages grow exponentially, code transparency and experiment record keeping are essential to ensuring full traceability of AI and ML models. This manuscript presents an open-source Machine Learning Operations (MLOps) Template that provides a consistent framework to support collaborative development and improve efficiency. The template provides a robust and reliable software structure incorporating essential development aspects. These tools include automated code documentation, built-in package management, experiment tracking, configuration and logging infrastructure, and more. The template is built on an agglomeration of best practices gleaned from industry and academia alike, providing a great starting point for any ML/AI project.

In the defense sector, artificial intelligence (AI) and machine learning (ML) have been used to analyse and decipher massive volumes of data, namely for target recognition, surveillance, threat detection and cybersecurity, autonomous vehicles and drones guidance, and language translation. However, there are key points that have been identified as barriers or challenges, especially related to data curation. For this reason, and also due to the need for quick response, the defense sector is looking for AI technologies capable of successfully processing and extracting results from huge amounts of unlabelled or very poorly labelled data. This paper presents an in-depth review of AI/ML algorithms for unsupervised or poorly supervised data, and machine learning operations (MLOps) techniques that are suitable for the defense industry. The algorithms are divided according to their nature, meaning that they either focus on techniques, or on applications. Techniques can belong to the supervision spectrum, or focus on explainability. Applications are either focused on text processing or computer vision. MLOps techniques, tools and practices are then discussed, revealing approaches and reporting experiences with the objective of declaring how to make the operationalization of ML integrated systems more efficient. Despite many contributions from several researchers and industry, further efforts are required to construct substantially robust and reliable models and supporting infrastructures for AI systems, which are reliable and suitable for the defense sector. This review brings up-to-date information regarding AI algorithms and MLOps that will be helpful for future research in the field. [ABSTRACT FROM AUTHOR]

In clinical trial monitoring, substantial resources are allocated to perform source data verification (SDV). SDV ensures accurate and reliable transcription of trial participant information. Clinical site visits are typically scheduled at a fixed frequency for SDV, without objectively factoring in individual site workload. This often results in wasted resources and directly influences clinical trial cost. We leveraged historical data from several hundred clinical trials to predict SDV workload at trial sites using machine learning. Specifically, we adopted cross industry standard process for data mining (CRISP-DM) process model and devised a novel deep learning pipeline for longitudinal clinical trial data. The deep learning pipeline, which comprises recurrent neural network-based encoder and decoder, ingests multivariate sequence data from study sites and predicts SDV workload for future months. We also developed an efficient model enhancement workflow, in a data science platform that facilitates machine learning operations best practices, for timely adaptation of new features and changes. Several enhancement iterations have been performed since the launch of first SDV workload prediction model, resulting in a more accurate latest model compared to previous versions. We discuss these enhancements in the context of CRISP-DM phases. In conclusion, the SDV workload prediction model has enabled informed planning and optimization of resources within clinical trial operations.

Information Technology involves solutions for many kinds of industries and organizations, offering conditions for solving problems of different types and complexities. Artificial Intelligence, and more specifically applications that considers Machine Learning (ML) and Software Technology are part of these solutions for solving problems, including solutions for solving problems that involve smart cities approach. In order to present frameworks that deal with the operationalization of Machine Learning and Software technology, this article is based on the study and evaluation of frameworks that involve Machine Learning Operations (MLOps) and microservices. Specifically, three frameworks that integrate ML algorithms with microservices are evaluated based on a bibliographical review in scientific journals of relevance to the area. From an exploratory analysis of these frameworks, it was possible to highlight their main objectives, their benefits, and their ability to offer solutions that favor the large-scale use of Machine Learning algorithms in problem solving. The main results are highlighted in the article through a qualitative analysis that considers six evaluation criteria, such as: capacity for sharing resources, scope of use by users, and use in a cloud environment. The results achieved are satisfactory since the work allows, through a qualitative view of the evaluated frameworks, a perspective of how the integration of MLOps and microservices has been carried out, its benefits and possible results achieved through this integration. [ABSTRACT FROM AUTHOR]