Search

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks. [ABSTRACT FROM AUTHOR]

Many imputation methods have been developed over the years and tested mostly under ideal settings. Surprisingly, there is no detailed research on how imputation methods perform when the idealized assumptions about the distribution of data and/or model assumptions are partly not fulfilled. This research looks into the susceptibility of imputation techniques, particularly in relation to outliers, misclassifications, and incorrect model specifications. This is crucial knowledge about how well the methods convince in everyday life because, in reality, conditions are usually not ideal, and model assumptions may not hold. The data may not fit the defined models well. Outliers distort the estimates, and misclassifications reduce the quality of most imputation methods. Several different evaluation measures are discussed, from comparing imputed values with true values or comparing certain statistics, from the performance of classifiers to the variance of estimated parameters. Some well-known imputation methods are compared based on real data and simulations. It turns out that robust conditional imputation methods outperform other methods for real data and simulation settings. [ABSTRACT FROM AUTHOR]

This paper addresses some classes of combined and separate imputation methods (CSIMs) of the population mean under stratified simple random sampling (SSRS) along with their characteristics. To the best of our knowledge, these imputation methods (IMs) have yet not been studied by any author under SSRS, hence these IMs are called ‘novel’. In addition, the existing CSIMs are distinguished as the members of the suggested CSIMs, respectively. The theoretical conditions under which the proposed IMs perform better are obtained by comparing the proposed IMs with the existing IMs. To validate the theoretical findings, the numerical and simulation studies are conducted on real and artificial populations, respectively.

Numerous real-world data contain missing values, while in contrast, most Machine Learning (ML) algorithms assume complete datasets. For this reason, several imputation algorithms have been proposed to predict and fill in the missing values. Given the advances in predictive modeling algorithms tuned in an Automated Machine Learning context (AutoML) setting, a question that naturally arises is to what extent sophisticated imputation algorithms (e.g., Neural Network based) are really needed, or we can obtain a descent performance using simple methods like Mean/Mode (MM). In this article, we experimentally compare six state-of-the-art representatives of different imputation algorithmic families from an AutoML predictive modeling perspective, including a feature selection step and combined algorithm and hyper-parameter selection. We used a commercial AutoML tool for our experiments, in which we included the selected imputation methods. Experiments ran on 25 binary classification real-world incomplete datasets with missing values and 10 binary classification complete datasets in which synthetic missing values are introduced according to different missingness mechanisms, at varying missing frequencies. The main conclusion drawn from our experiments is that the best method on average is the Denoise AutoEncoder on real-world datasets and the MissForest in simulated datasets, followed closely by MM. In addition, binary indicator variables encoding missingness patterns actually improve predictive performance, on average. Last, although there are cases where Neural-Network-based imputation significantly improves predictive performance, this comes at a great computational cost and requires measuring all feature values to impute new samples. [ABSTRACT FROM AUTHOR]

This paper addresses some classes of combined and separate imputation methods (CSIMs) of the population mean under stratified simple random sampling (SSRS) along with their characteristics. To the best of our knowledge, these imputation methods (IMs) have yet not been studied by any author under SSRS, hence these IMs are called 'novel'. In addition, the existing CSIMs are distinguished as the members of the suggested CSIMs, respectively. The theoretical conditions under which the proposed IMs perform better are obtained by comparing the proposed IMs with the existing IMs. To validate the theoretical findings, the numerical and simulation studies are conducted on real and artificial populations, respectively. [ABSTRACT FROM AUTHOR]

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks.

Multivariate time series (MTS) are captured in a great variety of real-world applications. However, analysing and modeling the data for classification and forecasting purposes can become very challenging if values are missing in the data set. The need for imputation methods, to fill the gaps in MTS, is well known. Thus, a great variaty of algorithms for solving this task has been proposed in the literature. However, the research community is constantly working on the development of advanced algorithms, that fulfill the special requirements of multidimensional temporal data, since most of the existing imputation methods treat MTS as ordinary structured data and fail to model the temporal relationships within and between sequences of observations. The main emphasis of MTS imputation research is currently put on deep learning (DL) models, especially those making use of generative adversarial networks (GANs). In our survey, we present our categorization of imputation algorithms, especially of GAN-models. We included eighteen different GAN-models designed for the MTS imputation task, which we introduce in detail. We provide a comparison of the models including their performance regarding MTS imputation, based on our findings in the literature. The following points can be considered the most important findings from our survey: The research on GAN-based imputation models for MTS has gained momentum in the last years across different domains, therewhile showing the effectiveness of these methods. The latest trend in the research area is the incorporation of attention mechanisms into the algorithms. Nevertheless, there are open research challenges, e.g. the transferability of models across data sets from different domains.

Scientists not only engage in research, but also write articles based on their work, and naturally aim to submit their articles to prestigious, well-received, and highly regarded journals or conferences. One way to measure the prestige of a journal is to use the Journal Impact Factor (JIF), but in order to make best use of this information one needs to better understand the relationships among various JIF-related attributes. However, in recent years JIF values have been missing or unavailable for some journals, due to several objective and subjective factors, which significantly impacts the ability to forecast such values in the following years. In this article we study the factors that directly affect the ranking of journals and can be used to forecast the appropriate values in the next few years, in order to help researchers find the right place to submit their articles in certain journals.

Imputation methods are popular tools that allow for a wide range of subsequent analyses on complete data sets. However, in order for these analyses to be trustworthy, it is important that the imputation procedure reflects the true distribution of the unobserved data sufficiently well. This raises the question how well different imputation methods can reproduce multivariate correlations, associations or even the entire multivariate distribution. The paper gives first answers to this question by means of an extensive comparative simulation study. In particular, we evaluate the multivariate distributional accuracy for six state-of-the art imputation algorithms with respect to different measures and give practical recommendations. [ABSTRACT FROM AUTHOR]

A high level of data quality has always been a concern for many applications based on machine learning, including clinical decision support systems, weather forecasting, traffic predictions, and many others. A very limited amount of work is devoted to exploiting the missing values for effective imputation and better prediction. This paper introduces a unique approach to predicting and imputing missing data fields in the multivariate dataset such as numerical, categorical, and unstructured. The proposed imputation method is a multi-model scheme based on the joint approach of natural language processing (NLP) encoders, machine learning-driven feature extractors, and a sequential regression imputation technique to predict missing values. The proposed system is robust and scalable without requiring extensive engineering. The validation of the model is done on the benchmarked clinical dataset of heart disease obtained from UCI. The results show that the proposed methods achieve better imputation accuracy and require significantly less time than other missing data imputation methods. [ABSTRACT FROM AUTHOR]

The occurrence of missing observations is nearly unavoidable in longitudinal studies where repeated measurements are taken over time on the same subject who may miss appointments or drop out during the study period. In this article, we use the Gaussian estimating objective function to estimate the regression and correlation parameters and handle missing data using multiple imputation. The estimation of these parameters is carried out simultaneously using the iterative Newton-Raphson algorithm and the expectation-maximization algorithm. These ideas are implemented using two real data sets and both algorithms showed comparable results with respect to the standard errors of the parameters of interest. [ABSTRACT FROM AUTHOR]

Incomplete data sets with different data types are difficult to handle, but regularly to be found in practical clustering tasks. Therefore in this paper, two procedures for clustering mixed-type data with missing values are derived and analyzed in a simulation study with respect to the factors of partition, prototypes, imputed values, and cluster assignment. Both approaches are based on the k-prototypes algorithm (an extension of k-means), which is one of the most common clustering methods for mixed-type data (i.e., numerical and categorical variables). For k-means clustering of incomplete data, the k-POD algorithm recently has been proposed, which imputes the missings with values of the associated cluster center. We derive an adaptation of the latter and additionally present a cluster aggregation strategy after multiple imputation. It turns out that even a simplified and time-saving variant of the presented method can compete with multiple imputation and subsequent pooling. [ABSTRACT FROM AUTHOR]

Many datasets in statistical analyses contain missing values. As omitting observations containing missing entries may lead to information loss or greatly reduce the sample size, imputation is usually preferable. However, imputation can also introduce bias and impact the quality and validity of subsequent analysis. Focusing on binary classification problems, we analyzed how missing value imputation under MCAR as well as MAR missingness with different missing patterns affects the predictive performance of subsequent classification. To this end, we compared imputation methods such as several MICE variants, missForest, Hot Deck as well as mean imputation with regard to the classification performance achieved with commonly used classifiers such as Random Forest, Extreme Gradient Boosting, Support Vector Machine and regularized logistic regression. Our simulation results showed that Random Forest based imputation (i.e., MICE Random Forest and missForest) performed particularly well in most scenarios studied. In addition to these two methods, simple mean imputation also proved to be useful, especially when many features (covariates) contained missing values. [ABSTRACT FROM AUTHOR]

Cardiovascular disease (CVD) is a severe public health concern globally. Early and accurate CVD diagnosis is a difficult task but a necessary endeavour required to prevent further damage and protect patients' lives. Machine Learning (ML)-based Clinical Decision Support Systems (CDSS) have the potential to assist healthcare providers in making accurate CVD diagnoses and treatments. Clinical data usually contains missing values (MVs); hence, the incorporated imputation techniques for ML have become a critical consideration when working with real-world medical datasets. Furthermore, removing instances with MVs will lead to essential data loss and produce incorrect results. To overcome these issues, this paper proposes an efficient and reliable CDSS with Ensemble Two-Fold Classification (ETC) framework for classifying heart diseases. The effectiveness of the proposed ETC framework using different supervised ML algorithms is evaluated with four distinct imputation methods for handling MVs over the standard benchmark dataset, viz., the University of California, Irwin (UCI). Experimental results show that our proposed ETC framework with the k-Nearest Neighbors(k-NN) imputation method achieves better classification accuracy of 0.9999 and a lesser error rate of 0.0989 compared to other imputation methods and classifiers with similar execution times. [ABSTRACT FROM AUTHOR]

As a useful tool for business research, PLS-SEM is widely adopted for the assessment of causal-predictive relationships of models when developing and testing theories. Nevertheless, the less error-prone techniques for handling missing data are routinely ignored by PLS-SEM researchers. In this paper, we propose an imputation method, called EM PLS-SEM, to deal with missing values in PLS-SEM. The method takes advantages of the estimation procedure of PLS-SEM to reach the goal of filling the missing elements with values that are most likely to appear. Numerical studies verify that the proposed method outperforms other alternatives in data completion and model fitting. [ABSTRACT FROM AUTHOR]

Numerous species of Earth's biota are at risk of extinction and wildlife conservation is more important than ever. Reliable baseline data are essential for wildlife management to inform on the numbers and distribution of endangered species. A promising non-invasive and cost-effective method for monitoring endangered species is the Footprint Identification Technology (FIT). It lends itself to both conservation research as well as citizen science and can be combined with other data collection methods. FIT extracts and uses morphometrics from animal footprints to create geometric profiles that are analyzed in customized classification models. These can identify species, sex and individual. The ability to identify individuals can then be used to predict various population parameters including size, distribution, and growth rate. FIT has been developed and published for several species, but it requires high quality footprints. Perfect prints are not always easy to find in the field as various factors can influence their quality. In this paper, we demonstrate that geometrical profiles derived from poor quality footprints can be seen as datasets with missing values. Missing values are a common problem in various disciplines, and well-established strategies to impute missing values are widely available. We conducted two experiments to see whether such an approach could widen the application of the FIT method. The experiments were designed to test the hypothesis that population sizes can be underestimated when incomplete footprints are discarded from the data. We artificially introduced different proportions of missing values in datasets with geometric profiles of five different species for which FIT models have been published. We also analyzed a new dataset of geometric profiles of cheetah (Acinonyx jubatus) footprints not meeting the standard FIT requirements. We demonstrated that excluding incomplete footprints led to an underestimation of the known population. As an alternative to discarding footprints, we compared different imputation techniques as data pre-processing steps by comparing the performance of resulting FIT models. When imputation was chosen instead, we could show that FIT models with imputed geometric profiles were not significantly less accurate in predicting individual ID or population size even with high rates of missing values. We believe that our findings can be generalized, and the results indicate that imperfect footprints can contribute to the robustness of the FIT method and that this approach is particularly applicable when few good-quality footprints are available. We therefore highly recommend including imputation of imperfect footprints as a data pre-processing step. [ABSTRACT FROM AUTHOR]

The accuracy of the statistical learning model depends on the learning technique used which in turn depends on the dataset's values. In most research studies, the existence of missing values (MVs) is a vital problem. In addition, any dataset with MVs cannot be used for further analysis or with any data driven tool especially when the percentage of MVs are high. In this paper, the authors propose a novel algorithm for dealing with MVs depending on the feature selection (FS) of similarity classifier with fuzzy entropy measure. The proposed algorithm imputes MVs in cumulative order. The candidate feature to be manipulated is selected using similarity classifier with Parkash's fuzzy entropy measure. The predictive model to predict MVs within the candidate feature is the Bayesian Ridge Regression (BRR) technique. Furthermore, any imputed features will be incorporated within the BRR equation to impute the MVs in the next chosen incomplete feature. The proposed algorithm was compared against some practical state-of-the-art imputation methods by conducting an experiment on four medical datasets which were gathered from several databases repository with MVs generated from the three missingness mechanisms. The evaluation metrics of mean absolute error (MAE), root mean square error (RMSE) and coefficient of determination (R2 score) were used to measure the performance. The results exhibited that performance vary depending on the size of the dataset, amount of MVs and the missingness mechanism type. Moreover, compared to other methods, the results showed that the proposed method gives better accuracy and less error in most cases. [ABSTRACT FROM AUTHOR]

The purpose of this study is to implement the ensemble self-organizing maps (ESOM) method to impute missing values at the preprocessing data stage, which is an important stage when making predictions or classifications. The Ensemble Self Organizing Maps (E-SOM) is the development of the SOM imputation method, in which the E-SOM method is implemented by applying an ensemble framework using several SOMs to improve generalization capabilities. In this study, the E-SOM imputation method is implemented in South African heart disease data using random forest as a classification model. The results of the model evaluation showed that for accuracy in testing data, the Random Forest model formed from E-SOM imputed data yields better accuracy values than the Random Forest model formed from SOM-imputed data for variations of 36, 49, 64, and 81 neurons, while for variation of 25 neurons both models produce the same accuracy value. From the variation of the number of ensembles applied, the E-SOM imputation method with a combination of 81 neurons and 15 ensemble numbers produced a Random Forest model with the most optimal value of accuracy. [ABSTRACT FROM AUTHOR]

Medical doctors may struggle to diagnose dementia, particularly when clinical test scores are missing or incorrect. In case of any doubts, both morphometrics and demographics are crucial when examining dementia in medicine. This study aims to impute and verify clinical test scores with brain MRI analysis and additional demographics, thereby proposing a decision support system that improves diagnosis and prognosis in an easy-to-understand manner. Therefore, we impute the missing clinical test score values by unsupervised dementia-related user-based collaborative filtering to minimize errors. By analyzing succession rates, we propose a reliability scale that can be utilized for the consistency of existing clinical test scores. The complete base of 816 ADNI1-screening samples was processed, and a hybrid set of 603 features was handled. Moreover, the detailed parameters in use, such as the best neighborhood and input features were evaluated for further comparative analysis. Overall, certain collaborative filtering configurations outperformed alternative state-of-the-art imputation techniques. The imputation system and reliability scale based on the proposed methodology are promising for supporting the clinical tests. [ABSTRACT FROM AUTHOR]

Medical doctors may struggle to diagnose dementia, particularly when clinical test scores are missing or incorrect. In case of any doubts, both morphometrics and demographics are crucial when examining dementia in medicine. This study aims to impute and verify clinical test scores with brain MRI analysis and additional demographics, thereby proposing a decision support system that improves diagnosis and prognosis in an easy-to-understand manner. Therefore, we impute the missing clinical test score values by unsupervised dementia-related user-based collaborative filtering to minimize errors. By analyzing succession rates, we propose a reliability scale that can be utilized for the consistency of existing clinical test scores. The complete base of 816 ADNI1-screening samples was processed, and a hybrid set of 603 features was handled. Moreover, the detailed parameters in use, such as the best neighborhood and input features were evaluated for further comparative analysis. Overall, certain collaborative filtering configurations outperformed alternative state-of-the-art imputation techniques. The imputation system and reliability scale based on the proposed methodology are promising for supporting the clinical tests.

Dealing with missing data poses a challenge in Principal Component Analysis (PCA) since the most common algorithms are not designed to handle them. Several approaches have been proposed to solve the missing value problem in PCA, such as Imputation based on SVD (I-SVD), where missing entries are filled by imputation and updated in every iteration until convergence of the PCA model, and the adaptation of the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm, able to work skipping the missing entries during the least-squares estimation of scores and loadings. However, some limitations have been reported for both approaches. On the one hand, convergence of the I-SVD algorithm can be very slow for data sets with a high percentage of missing data. On the other hand, the orthogonality properties among scores and loadings might be lost when using NIPALS. To solve these issues and perform PCA of data sets with missing values without the need of imputation steps, a novel algorithm called Orthogonalized-Alternating Least Squares (O-ALS) is proposed. The O-ALS algorithm is an alternating least-squares algorithm that estimates the scores and loadings subject to the Gram-Schmidt orthogonalization constraint. The way to estimate scores and loadings is adapted to work only with the available information. In this study, the performance of O-ALS is tested and compared with NIPALS and I-SVD in simulated data sets and in a real case study. The results show that O-ALS is an accurate and fast algorithm to analyze data with any percentage and distribution pattern of missing entries, being able to provide correct scores and loadings in cases where I-SVD and NIPALS do not perform satisfactorily. • The novel O-ALS algorithm is proposed to perform PCA on data with missing values. • O-ALS efficiently estimates scores and loadings while preserving orthogonality. • O-ALS does not need any imputation step. • The algorithm adapts to any kind of percentage and patterns of missing values. [ABSTRACT FROM AUTHOR]

Ship data obtained through the maritime sector will inevitably have missing values and outliers, which will adversely affect the subsequent study. Many existing methods for missing data imputation cannot meet the requirements of ship data quality, especially in cases of high missing rates. In this paper, a missing data imputation method based on generative adversarial networks (GANs) is proposed. The generative adversarial imputation network (GAIN) is improved using the Wasserstein distance and gradient penalty to handle missing values. Meanwhile, the data preprocessing process is optimized by combining knowledge from the ship domain, such as using isolation forests for anomaly detection. Statistical analysis of ship data is also conducted, including correlation analysis of ship design parameters, analysis of outliers, and analysis of missing data types. These analyses provide the basis for the proposed model. In a case study of 8167 bulk carriers, the proposed model outperformed the missing forest (MF) and polynomial fitting (PF) models, with an average error reduction of 2.4% and 6.3%, respectively. The proposed model also showed stable performance in cases of high missing rates. This study provides a new approach for estimating or imputing critical parameters of ships.

Many datasets in statistical analyses contain missing values. As omitting observations containing missing entries may lead to information loss or greatly reduce the sample size, imputation is usually preferable. However, imputation can also introduce bias and impact the quality and validity of subsequent analysis. Focusing on binary classification problems, we analyzed how missing value imputation under MCAR as well as MAR missingness with different missing patterns affects the predictive performance of subsequent classification. To this end, we compared imputation methods such as several MICE variants, missForest, Hot Deck as well as mean imputation with regard to the classification performance achieved with commonly used classifiers such as Random Forest, Extreme Gradient Boosting, Support Vector Machine and regularized logistic regression. Our simulation results showed that Random Forest based imputation (i.e., MICE Random Forest and missForest) performed particularly well in most scenarios studied. In addition to these two methods, simple mean imputation also proved to be useful, especially when many features (covariates) contained missing values.

Background As databases grow larger, it becomes harder to fully control their collection, and they frequently come with missing values. These large databases are well suited to train machine learning models, e.g. for forecasting or to extract biomarkers in biomedical settings. Such predictive approaches can use discriminative—rather than generative—modeling and thus open the door to new missing-values strategies. Yet existing empirical evaluations of strategies to handle missing values have focused on inferential statistics. Results Here we conduct a systematic benchmark of missing-values strategies in predictive models with a focus on large health databases: 4 electronic health record datasets, 1 population brain imaging database, 1 health survey, and 2 intensive care surveys. Using gradient-boosted trees, we compare native support for missing values with simple and state-of-the-art imputation prior to learning. We investigate prediction accuracy and computational time. For prediction after imputation, we find that adding an indicator to express which values have been imputed is important, suggesting that the data are missing not at random. Elaborate missing-values imputation can improve prediction compared to simple strategies but requires longer computational time on large data. Learning trees that model missing values—with missing incorporated attribute—leads to robust, fast, and well-performing predictive modeling. Conclusions Native support for missing values in supervised machine learning predicts better than state-of-the-art imputation with much less computational cost. When using imputation, it is important to add indicator columns expressing which values have been imputed. [ABSTRACT FROM AUTHOR]

Data collection within a real-world environment may be compromised by several factors such as data-logger malfunctions and communication errors, during which no data is collected. As a consequence, appropriate tools are required to handle the missing values when analysing and processing such data. This problem is often tackled via matrix decomposition. While it has been successfully applied in a wide range of applications, in this work we report an issue that has been neglected in literature and "degenerates" the quality of the imputations obtained by matrix decomposition in multivariate time-series (with smooth evolution). Briefly, the problem consists of the misalignment of the matrix decomposition result: the missing values imputations fall within an incorrect range of values and the transitions between observed and imputed values are not smooth. We address this problem by proposing a post-processing alignment strategy. According to our experiments, the post-processing adjustment substantially improves the accuracy of the imputations (when the misalignment occurs). Moreover, the results also suggest that the misalignment occurs mostly when dealing with a small number of time-series due to lack of generalization ability. [ABSTRACT FROM AUTHOR]

In many real-world applications scientists are often confronted with the problem of incomplete datasets due to several reasons. The direct analysis of datasets with missing values in attributes inevitably results in inaccurate learning models and erroneous results. Facing effectively the challenge of missing values is an essential step of the data mining process. Imputation is often employed to overcome the shortcomings incurred by missing data during the pre-process stage of data analysis. Therefore, a plethora of statistical and machine learning methods have been proposed and employed with a view to imputing the missing values in incomplete data with their potential or actual values. In this context, the main objective of this paper is to put forward an iterative stepwise imputation method based on the semi-supervised learning approach, called IRSSI. Semi-supervised methods have proved to be particularly effective for exploiting incomplete or partially labeled data with regard to the values of the target attribute. The proposed algorithm was experimentally evaluated on real-world benchmark datasets and artificially generated datasets using different high ratios of missing data. The experimental results demonstrate the efficiency of IRSSI algorithm compared to typical imputation methods.

Currently, accurate traffic flow analysis and modeling are important key steps for intelligent transportation system (ITS). Missing traffic flow data are one of the most critical issues in the application of ITS. In this study, a hybrid method combining fuzzy rough set (FRS) and fuzzy neural network (FNN) is proposed for imputation of missing traffic data. Firstly, FNN is used for data classification, then the K-Nearest Neighbor (KNN) method is used to determine the optimal number of data used to estimate missing data in each category, and finally the fuzzy rough set is used to impute missing values. In order to validate the imputation performance of the proposed hybrid method, the traffic flow data collected from the loop detectors at different time intervals on roadway network are used in model calibration and validation. Three common indicators, including RMSE (root mean square error), R (correlation coefficient) and RA (relative accuracy), are used to evaluate the imputation performance under different data missing ratios. A model comparison is conducted between proposed imputation method and several widely used models including average-based and regression-based methods. The results show that the proposed method is superior to the traditional method for the traffic flow data collected at different time intervals with different missing ratios, which also further demonstrate its effectivity and validity. [ABSTRACT FROM AUTHOR]

The rapid, increasing adoption of businesses to deliver their services in Software as a Service (SaaS) products in the marketplace presents selection challenges to users. Recently, major cloud service providers such as Amazon Web Services and Microsoft Azure have introduced well-architected frameworks that assess SaaS products in different pillars (also referred to herein as features). Furthermore, customers leave feedback on these features after using SaaS products. However, they do not provide feedback on all the features of a product, which renders the reviews unusable to prospective users needing to assess a product's quality before committing to it. Our study addresses this drawback by imputing or inferring the intensity of the customer's feedback on features that they do not mention in their reviews. Specifically, we propose threshold-based nearest neighbors (T-NN) as an extension of the conventional k-nearest neighbor approach to determine the missing sentiment intensity score of a feature from the values of its other features. We evaluate the proposed approach in two different systems and compare our results with seven other data imputation techniques. The results show that the proposed T-NN approach performs better than the other imputation approaches on the SaaS sentiment dataset. [ABSTRACT FROM AUTHOR]

Missing data is a common problem confronted by researchers in machine learning applications. Missing values affect both the performance of analysis tools, as well as the quality of the drawn decisions. This research aims to analyze the impact of four missing data treatment methods on the predictive accuracy of the C4.5 decision tree algorithm. It also investigates the imputation accuracy of each imputation method using a single dataset with missing values presented in a single variable. The work was performed under three missing data assumptions, namely, Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) with three missingness' rates: 5%, 10%, and 15%. The methods used to treat the missing data are: lite-wise deletion, mean/mode imputation, K-nearest neighbor imputation, and decision tree imputation. The results of the experiments showed that the C4.5 classifier achieved better performance under the MCAR assumption. While the mean/mode imputation has the highest C4.5 predictive accuracy under MAR and MNAR assumptions. The k-nearest neighbor method obtained the most accurate imputation result under the MCAR assumption, whereas mean/mode imputation was the most accurate method under the MAR assumption. On the other hand, the lowest imputation accuracy levels were achieved under the MNAR assumption attributed to the mean/mode imputation method. [ABSTRACT FROM AUTHOR]

Label-free shotgun proteomics is an important tool in biomedical research, where tandem mass spectrometry with data-dependent acquisition (DDA) is frequently used for protein identification and quantification. However, the DDA datasets contain a significant number of missing values (MVs) that severely hinders proper analysis. Existing literature suggests that different imputation methods should be used for the two types of MVs: missing completely at random or missing not at random. However, the simulated or biased datasets utilized by most of such studies offer few clues about the composition and thus proper imputation of MVs in real-life proteomic datasets. Moreover, the impact of imputation methods on downstream differential expression analysis—a critical goal for many biomedical projects—is largely undetermined. In this study, we investigated public DDA datasets of various tissue/sample types to determine the composition of MVs in them. We then developed simulated datasets that imitate the MV profile of real-life datasets. Using such datasets, we compared the impact of various popular imputation methods on the analysis of differentially expressed proteins. Finally, we make recommendations on which imputation method(s) to use for proteomic data beyond just DDA datasets. [ABSTRACT FROM AUTHOR]

Abstract Background LC-MS technology makes it possible to measure the relative abundance of numerous molecular features of a sample in single analysis. However, especially non-targeted metabolite profiling approaches generate vast arrays of data that are prone to aberrations such as missing values. No matter the reason for the missing values in the data, coherent and complete data matrix is always a pre-requisite for accurate and reliable statistical analysis. Therefore, there is a need for proper imputation strategies that account for the missingness and reduce the bias in the statistical analysis. Results Here we present our results after evaluating nine imputation methods in four different percentages of missing values of different origin. The performance of each imputation method was analyzed by Normalized Root Mean Squared Error (NRMSE). We demonstrated that random forest (RF) had the lowest NRMSE in the estimation of missing values for Missing at Random (MAR) and Missing Completely at Random (MCAR). In case of absent values due to Missing Not at Random (MNAR), the left truncated data was best imputed with minimum value imputation. We also tested the different imputation methods for datasets containing missing data of various origin, and RF was the most accurate method in all cases. The results were obtained by repeating the evaluation process 100 times with the use of metabolomics datasets where the missing values were introduced to represent absent data of different origin. Conclusion Type and rate of missingness affects the performance and suitability of imputation methods. RF-based imputation method performs best in most of the tested scenarios, including combinations of different types and rates of missingness. Therefore, we recommend using random forest-based imputation for imputing missing metabolomics data, and especially in situations where the types of missingness are not known in advance.

In this article, Gradient Boosting (gb) is introduced as an easily adaptable machine learning method to fill gaps caused by missing or erroneous data in meteorological time series. The gb routine is applied on a large data set of hourly time series of the measurands: air temperature, wind speed and relative humidity for station-based observations in Germany covering the period from 1951 to 2015. Our analysis shows that Gradient Boosting produces small errors by estimating missing values with median RMSE for temperature of 0.73 °C, wind speed of 0.82 m/s, relative humidity of 4.3 %, respectively. The comparison between the results achieved by Gradient Boosting with results from other gap filling techniques like neural networks or multiple linear regression shows considerably better statistics. The comparison clearly shows that the Gradient Boosting approach outperforms the other techniques particularly in calculation time, performance and the handling of missing predictor values.