Search

The study aimed to examine the effect of the percentage of missing values on the matching of the items of the graded model. To achieve this, the experimental approach was followed in conducting the study, as hypothetical data were used to simulate the experimental conditions represented by the percentage of missing values. It has three levels (5%, 10%, and 20%). Moreover, the compensation method is represented by simple regression. The study found that there were no statistically significant differences in the percentage of items matching the graded model according to the missing percentages (5% and 10%). Moreover, there were statistically significant differences in the percentage of items matching the graded model according to the missing percentages (20%). Therefore, we must use the different nonparametric item response theory (IRT) models because of the good information, it provides to researchers about the test and its items without missing items suitable for this purpose and prejudice--the cumulative response of the subjects. [ABSTRACT FROM AUTHOR]

A crucial input into causal inference is the imputed counterfactual outcome. Imputation error can arise because of sampling uncertainty from estimating the prediction model using the untreated observations, or from out-of-sample information not captured by the model. While the literature has focused on sampling uncertainty, it vanishes with the sample size. Often overlooked is the possibility that the out-of-sample error can be informative about the missing counterfactual outcome if it is mutually or serially correlated. Motivated by the best linear unbiased predictor (BLUP) of Goldberger in a time series setting, we propose an improved predictor of potential outcome when the errors are correlated. The proposed PUP is practical as it is not restricted to linear models, can be used with consistent estimators already developed, and improves mean-squared error for a large class of strong mixing error processes. Ignoring predictability in the errors can distort conditional inference. However, the precise impact will depend on the choice of estimator as well as the realized values of the residuals. [ABSTRACT FROM AUTHOR]

Diabetes is thought to be the most common illness in underdeveloped nations. Early detection and competent medical care are crucial steps in reducing the effects of diabetes. Examining the signs associated with diabetes is one of the most effective ways to identify the condition. The problem of missing data is not very well investigated in existing works. In addition, existing studies on diabetes detection lack accuracy and robustness. The available datasets frequently contain missing information for the automated detection of diabetes, which might negatively impact machine learning model performance. This work suggests an automated diabetes prediction method that achieves high accuracy and effectively manages missing variables in order to address this problem. The proposed strategy employs a stacked ensemble voting classifier model with three machine learning models. and a KNN Imputer to handle missing values. Using the KNN imputer, the suggested model performs exceptionally well, with accuracy, precision, recall, F1 score, and MCC of 98.59%, 99.26%, 99.75%, 99.45%, and 99.24%, respectively. In two scenarios one with missing values eliminated and the other with KNN imputer, the study thoroughly compared the suggested model with seven other machine learning techniques. The outcomes demonstrate the superiority of the suggested model over current state-of-the-art methods and confirm its efficacy. This work demonstrates the capability of KNN imputer and looks at the problem of missing values for diabetes detection. Medical professionals can utilize the results to improve care for diabetes patients and discover problems early. [ABSTRACT FROM AUTHOR]

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks. [ABSTRACT FROM AUTHOR]

Background and objectives: Comprehending the research dataset is crucial for obtaining reliable and valid outcomes. Health analysts must have a deep comprehension of the data being analyzed. This comprehension allows them to suggest practical solutions for handling missing data, in a clinical data source. Accurate handling of missing values is critical for producing precise estimates and making informed decisions, especially in crucial areas like clinical research. With data's increasing diversity and complexity, numerous scholars have developed a range of imputation techniques. To address this, we conducted a systematic review to introduce various imputation techniques based on tabular dataset characteristics, including the mechanism, pattern, and ratio of missingness, to identify the most appropriate imputation methods in the healthcare field. Materials and methods: We searched four information databases namely PubMed, Web of Science, Scopus, and IEEE Xplore, for articles published up to September 20, 2023, that discussed imputation methods for addressing missing values in a clinically structured dataset. Our investigation of selected articles focused on four key aspects: the mechanism, pattern, ratio of missingness, and various imputation strategies. By synthesizing insights from these perspectives, we constructed an evidence map to recommend suitable imputation methods for handling missing values in a tabular dataset. Results: Out of 2955 articles, 58 were included in the analysis. The findings from the development of the evidence map, based on the structure of the missing values and the types of imputation methods used in the extracted items from these studies, revealed that 45% of the studies employed conventional statistical methods, 31% utilized machine learning and deep learning methods, and 24% applied hybrid imputation techniques for handling missing values. Conclusion: Considering the structure and characteristics of missing values in a clinical dataset is essential for choosing the most appropriate data imputation technique, especially within conventional statistical methods. Accurately estimating missing values to reflect reality enhances the likelihood of obtaining high-quality and reusable data, contributing significantly to precise medical decision-making processes. Performing this review study creates a guideline for choosing the most appropriate imputation methods in data preprocessing stages to perform analytical processes on structured clinical datasets. Highlights: • The evidence map emphasized the importance of considering missing data characteristics when choosing an imputation method, providing insights for researchers in method selection. • The distinction between statistical and learning-based approaches highlighted the strengths and considerations of each method based on data structure. • Understanding missing data structures can enhance the quality and reliability of data imputation techniques, and improve medical decision-making accuracy. • Simulation studies are crucial for validating imputation techniques and enhancing their robustness in practical applications. • Considering missing data mechanisms, patterns, and ratios can aid researchers in making informed decisions on selecting appropriate imputation methods, leading to high-quality, and reusable data for precise medical decision-making. [ABSTRACT FROM AUTHOR]

Sample entropy embeds time series into m-dimensional spaces and estimates entropy based on the distances between points in these spaces. However, when samples can be considered as missing or invalid, defining distance in the embedding space becomes problematic. Preprocessing techniques, such as deletion or interpolation, can be employed as a solution, producing time series without missing or invalid values. While deletion ignores missing values, interpolation replaces them using approximations based on neighboring points. This paper proposes a novel approach for the computation of sample entropy when values are considered as missing or invalid. The proposed algorithm accommodates points in the m-dimensional space and handles them there. A theoretical and experimental comparison of the proposed algorithm with deletion and interpolation demonstrates several advantages over these other two approaches. Notably, the deviation of the expected sample entropy value for the proposed methodology consistently proves to be lowest one. [ABSTRACT FROM AUTHOR]

Multiple imputation of missing values is a key step in data analytics and a standard process in data science. Nonlinear imputation methods come into play whenever the linear relationship between a response and predictors cannot be linearized by transformations of variables, adding interactions, or using, e.g., quadratic terms. Generalized additive models (GAM) and its extension, GAMLSS—where each parameter of the distribution, such as mean, variance, skewness, and kurtosis, can be represented as a function of predictors, are widely used nonlinear methods. However, non-robust methods such as standard GAM’s and GAMLSS’s can be swayed by outliers, leading to outlier-driven imputations. This can apply concerning both representative outliers—those true yet unusual values of your population—and non-representative outliers, which are mere measurement errors. Robust (imputation) methods effectively manage outliers and exhibit resistance to their influence, providing a more reliable approach to dealing with missing data. The innovative solution of the proposed new imputation algorithm tackles three major challenges related to robustness. (1) A robust bootstrap method is employed to handle model uncertainty during the imputation of a random sample. (2) The approach incorporates robust fitting techniques to enhance accuracy. (3) It effectively considers imputation uncertainty in a resilient manner. Furthermore, any complex model for any variable with missingness can be considered and run through the algorithm. For the real-world data sets used and the simulation study conducted, the novel algorithm imputeRobust which includes robust methods for imputation with GAM’s demonstrates superior performance compared to existing imputation methods using GAMLSS. Limitations pertain to the imputation of categorical variables using robust techniques. [ABSTRACT FROM AUTHOR]

Abstract Diabetes is thought to be the most common illness in underdeveloped nations. Early detection and competent medical care are crucial steps in reducing the effects of diabetes. Examining the signs associated with diabetes is one of the most effective ways to identify the condition. The problem of missing data is not very well investigated in existing works. In addition, existing studies on diabetes detection lack accuracy and robustness. The available datasets frequently contain missing information for the automated detection of diabetes, which might negatively impact machine learning model performance. This work suggests an automated diabetes prediction method that achieves high accuracy and effectively manages missing variables in order to address this problem. The proposed strategy employs a stacked ensemble voting classifier model with three machine learning models. and a KNN Imputer to handle missing values. Using the KNN imputer, the suggested model performs exceptionally well, with accuracy, precision, recall, F1 score, and MCC of 98.59%, 99.26%, 99.75%, 99.45%, and 99.24%, respectively. In two scenarios one with missing values eliminated and the other with KNN imputer, the study thoroughly compared the suggested model with seven other machine learning techniques. The outcomes demonstrate the superiority of the suggested model over current state-of-the-art methods and confirm its efficacy. This work demonstrates the capability of KNN imputer and looks at the problem of missing values for diabetes detection. Medical professionals can utilize the results to improve care for diabetes patients and discover problems early.

Abstract Background and objectives Comprehending the research dataset is crucial for obtaining reliable and valid outcomes. Health analysts must have a deep comprehension of the data being analyzed. This comprehension allows them to suggest practical solutions for handling missing data, in a clinical data source. Accurate handling of missing values is critical for producing precise estimates and making informed decisions, especially in crucial areas like clinical research. With data's increasing diversity and complexity, numerous scholars have developed a range of imputation techniques. To address this, we conducted a systematic review to introduce various imputation techniques based on tabular dataset characteristics, including the mechanism, pattern, and ratio of missingness, to identify the most appropriate imputation methods in the healthcare field. Materials and methods We searched four information databases namely PubMed, Web of Science, Scopus, and IEEE Xplore, for articles published up to September 20, 2023, that discussed imputation methods for addressing missing values in a clinically structured dataset. Our investigation of selected articles focused on four key aspects: the mechanism, pattern, ratio of missingness, and various imputation strategies. By synthesizing insights from these perspectives, we constructed an evidence map to recommend suitable imputation methods for handling missing values in a tabular dataset. Results Out of 2955 articles, 58 were included in the analysis. The findings from the development of the evidence map, based on the structure of the missing values and the types of imputation methods used in the extracted items from these studies, revealed that 45% of the studies employed conventional statistical methods, 31% utilized machine learning and deep learning methods, and 24% applied hybrid imputation techniques for handling missing values. Conclusion Considering the structure and characteristics of missing values in a clinical dataset is essential for choosing the most appropriate data imputation technique, especially within conventional statistical methods. Accurately estimating missing values to reflect reality enhances the likelihood of obtaining high-quality and reusable data, contributing significantly to precise medical decision-making processes. Performing this review study creates a guideline for choosing the most appropriate imputation methods in data preprocessing stages to perform analytical processes on structured clinical datasets.

Class imbalance in a dataset leads to erroneous outcomes that engrave the learning techniques and high misclassification cost in the minority class. Along with class imbalance, missing values present in a disproportionate amount in a dataset create a great hindrance to the effective performance of a method. The ensemble method, where multiple methods are ensembled, tackles such issues and shows good results compared to the performance of individual methods. In this paper, a hybridization of multiple imputation and one-class bagging ensemble approach is proposed that handles datasets having both class imbalance and missing values. An in-depth analysis of this approach is studied and the effectiveness of the class imbalance is also presented. To tackle the misclassification of minority samples and missing values, factor-based multiple imputation oversampling technique is used and one-class classifier is ensembled to increase the performance of the class imbalance datasets. Experiments are performed using a one-class support vector machine classifier and the results are evaluated using metrics: Recall (Detection rate), Specificity, f-measure, g-mean, AUC, and Precision. The proposed approach yields a 6.3% improvement in Recall, whereas Specificity, f-measure, g-mean, AUC, and Precision show that the proposed approach improves by 4.92%, 11.3%, 9.4%, 8.3%, and 8.03%, respectively. [ABSTRACT FROM AUTHOR]

Many imputation methods have been developed over the years and tested mostly under ideal settings. Surprisingly, there is no detailed research on how imputation methods perform when the idealized assumptions about the distribution of data and/or model assumptions are partly not fulfilled. This research looks into the susceptibility of imputation techniques, particularly in relation to outliers, misclassifications, and incorrect model specifications. This is crucial knowledge about how well the methods convince in everyday life because, in reality, conditions are usually not ideal, and model assumptions may not hold. The data may not fit the defined models well. Outliers distort the estimates, and misclassifications reduce the quality of most imputation methods. Several different evaluation measures are discussed, from comparing imputed values with true values or comparing certain statistics, from the performance of classifiers to the variance of estimated parameters. Some well-known imputation methods are compared based on real data and simulations. It turns out that robust conditional imputation methods outperform other methods for real data and simulation settings. [ABSTRACT FROM AUTHOR]

Instantaneous peak flows (IPFs) are often required to derive design values for sizing various hydraulic structures, such as culverts, bridges, and small dams/levees, in addition to informing several water resources management-related activities. Compared to mean daily flows (MDFs), which represent averaged flows over a period of 24 h, information on IPFs is often missing or unavailable in instrumental records. In this study, conventional methods for estimating IPFs from MDFs are evaluated and new methods based on the nonlinear regression framework and machine learning architectures are proposed and evaluated using streamflow records from all Canadian hydrometric stations with natural and regulated flow regimes. Based on a robust model selection criterion, it was found that multiple methods are suitable for estimating IPFs from MDFs, which precludes the idea of a single universal method. The performance of machine learning-based methods was also found reasonable compared to conventional and regression-based methods. To build on the strengths of individual methods, the fusion modeling concept from the machine learning area was invoked to synthesize outputs of multiple methods. The study findings are expected to be useful to the climate change adaptation community, which currently heavily relies on MDFs simulated by hydrologic models. HIGHLIGHTS New methods for estimating instantaneous peak flows from mean daily flows.; Data completion by filling in missing values of instantaneous peak flows.; Reliable estimation of design flood magnitudes.; Machine learning inspired fusion modeling to synthesize outputs of multiple methods.; Fulfilled a critical need of the climate change impact analysis and adaptation community.;

This paper addresses some classes of combined and separate imputation methods (CSIMs) of the population mean under stratified simple random sampling (SSRS) along with their characteristics. To the best of our knowledge, these imputation methods (IMs) have yet not been studied by any author under SSRS, hence these IMs are called ‘novel’. In addition, the existing CSIMs are distinguished as the members of the suggested CSIMs, respectively. The theoretical conditions under which the proposed IMs perform better are obtained by comparing the proposed IMs with the existing IMs. To validate the theoretical findings, the numerical and simulation studies are conducted on real and artificial populations, respectively.

Numerous real-world data contain missing values, while in contrast, most Machine Learning (ML) algorithms assume complete datasets. For this reason, several imputation algorithms have been proposed to predict and fill in the missing values. Given the advances in predictive modeling algorithms tuned in an Automated Machine Learning context (AutoML) setting, a question that naturally arises is to what extent sophisticated imputation algorithms (e.g., Neural Network based) are really needed, or we can obtain a descent performance using simple methods like Mean/Mode (MM). In this article, we experimentally compare six state-of-the-art representatives of different imputation algorithmic families from an AutoML predictive modeling perspective, including a feature selection step and combined algorithm and hyper-parameter selection. We used a commercial AutoML tool for our experiments, in which we included the selected imputation methods. Experiments ran on 25 binary classification real-world incomplete datasets with missing values and 10 binary classification complete datasets in which synthetic missing values are introduced according to different missingness mechanisms, at varying missing frequencies. The main conclusion drawn from our experiments is that the best method on average is the Denoise AutoEncoder on real-world datasets and the MissForest in simulated datasets, followed closely by MM. In addition, binary indicator variables encoding missingness patterns actually improve predictive performance, on average. Last, although there are cases where Neural-Network-based imputation significantly improves predictive performance, this comes at a great computational cost and requires measuring all feature values to impute new samples. [ABSTRACT FROM AUTHOR]

DNA methylation plays an essential role in regulating gene activity, modulating disease risk, and determining treatment response. We can obtain insight into methylation patterns at a single-nucleotide level via next-generation sequencing technologies. However, complex features inherent in the data obtained via these technologies pose challenges beyond the typical big data problems. Identifying differentially methylated cytosines (dmc) or regions is one such challenge. We have developed DMCFB, an efficient dmc identification method based on Bayesian functional regression, to tackle these challenges. Using simulations, we establish that DMCFB outperforms current methods and results in better smoothing and efficient imputation. We analyzed a dataset of patients with acute promyelocytic leukemia and control samples. With DMCFB, we discovered many new dmcs and, more importantly, exhibited enhanced consistency of differential methylation within islands and their adjacent shores. Additionally, we detected differential methylation at more of the binding sites of the fused gene involved in this cancer. [ABSTRACT FROM AUTHOR]

The site regression model (SREG) is utilized by plant breeders for analyzing multi-environment trials (MET) to examine the relationships among test environments, genotypes (G), and genotype-by-environment interactions (GE). SREG explores a matrix of G and GE by performing a singular value decomposition on the residuals matrix from a one-way ANOVA, requiring complete data. As missing values are common in MET, we propose two new imputation methods that implement an Expectation Maximization algorithm to fit the SREG model. To evaluate the impact on SREG model parameter estimation of these proposed methods and other competing imputation methods available, we conducted two studies using different approaches. One study involved simulated data while the other used a real dataset. In both studies, different measures to verify whether the joint effect of G plus GE is altered by imputation of data, and the reproducibility of missing data were evaluated. We also incorporated situations not commonly addressed in the literature, such as non-random structures of missing values and a big data situation. The proposed procedures provided estimators with good performance, maintaining superiority in several aspects studied, even when the competing imputation methods did not achieve convergence. Therefore, the new methods enabled incomplete MET data to be effectively analyzed by a SREG model. [ABSTRACT FROM AUTHOR]

Satellite time series data, widely used for land cover classification, often contain missing values due to cloud contamination, which can negatively affect classification. Numerous strategies have been developed to reconstruct the missing values to produce regular time series for machine learning classifiers, among which the compositing followed by the linear interpolation is most widely used. However, the classification improvement of linear interpolation for land cover classification has not been examined. Recently developed deep learning models such as long short term memory (LSTM) and Transformer allow such examination as they can classify time series with missing values. In this study, we compared the time series composites with missing values (without linear interpolation) and the linearly interpolated time series composites (without missing values) for land cover classification. About 18 thousand Harmonized Landsat Sentinel-2 (HLS) images acquired over Amur River Basin of China (890,308 km2) in 2021 were composited to 14 16-day periods. Two time series composites were classified, i.e., (i) the 16-day composites without interpolation that have on average 15.35% 16-day periods with missing values and (ii) the linearly interpolated 16-day composites with no missing values. The classifications showed that (1) between classifications with and without linear interpolation there was < 0.2% overall accuracy differences for the bidirectional LSTM (Bi-LSTM) and < 0.5% for the Transformer both of which were smaller than model training randomness; and (2) the computation time can be saved using composites without linear interpolation. The findings suggested that it is unnecessary to use the time-consuming linear interpolation in Bi-LSTM and Transformer-based land cover classifications. The findings were confirmed by experiments for sensitivity to the number of cloud-free composites and to different classification legends using crop type classifications. It implied the linear interpolation algorithm cannot reconstruct reliable time series for land cover classifications and historical use of such method is more about mitigating the inability of traditional classifiers to handle missing values rather than improving classifications. Linear interpolation is not necessary for LSTM and Transformer with capability to handle missing values. The training datasets and developed codes in this study are made publicly available. [ABSTRACT FROM AUTHOR]

Chronic kidney disease (CKD) is a major health concern today, requiring early and accurate diagnosis. Machine learning has emerged as a powerful tool for disease detection, and medical professionals are increasingly using ML classifier algorithms to identify CKD early. This study explores the application of advanced machine learning techniques on a CKD dataset obtained from the University of California, UC Irvine Machine Learning repository. The research introduces TrioNet, an ensemble model combining extreme gradient boosting, random forest, and extra tree classifier, which excels in providing highly accurate predictions for CKD. Furthermore, K nearest neighbor (KNN) imputer is utilized to deal withmissing values while synthetic minority oversampling (SMOTE) is used for class-imbalance problems. To ascertain the efficacy of the proposed model, a comprehensive comparative analysis is conducted with various machine learning models. The proposed TrioNet using KNN imputer and SMOTE outperformed other models with 98.97% accuracy for detectingCKD. This in-depth analysis demonstrates the model's capabilities and underscores its potential as a valuable tool in the diagnosis of CKD. [ABSTRACT FROM AUTHOR]

Linear regression (LR) is vastly used in data analysis for continuous outcomes in biomedicine and epidemiology. Despite its popularity, LR is incompatible with missing data, which frequently occur in health sciences. For parameter estimation, this shortcoming is usually resolved by complete‐case analysis or imputation. Both work‐arounds, however, are inadequate for prediction, since they either fail to predict on incomplete records or ignore missingness‐induced reduction in prediction accuracy and rely on (unrealistic) assumptions about the missing mechanism. Here, we derive adaptive predictor‐set linear model (aps‐lm), capable of making predictions for incomplete data without the need for imputation. It is derived by using a predictor‐selection operation, the Moore–Penrose pseudoinverse, and the reduced QR decomposition. aps‐lm is an LR generalization that inherently handles missing values. It is applied on a reference data set, where complete predictors and outcome are available, and yields a set of privacy‐preserving parameters. In a second stage, these are shared for making predictions of the outcome on external data sets with missing entries for predictors without imputation. Moreover, aps‐lm computes prediction errors that account for the pattern of missing values even under extreme missingness. We benchmark aps‐lm in a simulation study. aps‐lm showed greater prediction accuracy and reduced bias compared to popular imputation strategies under a wide range of scenarios including variation of sample size, goodness of fit, missing value type, and covariance structure. Finally, as a proof‐of‐principle, we apply aps‐lm in the context of epigenetic aging clocks, linear models that predict a person's biological age from epigenetic data with promising clinical applications. [ABSTRACT FROM AUTHOR]

This paper addresses some classes of combined and separate imputation methods (CSIMs) of the population mean under stratified simple random sampling (SSRS) along with their characteristics. To the best of our knowledge, these imputation methods (IMs) have yet not been studied by any author under SSRS, hence these IMs are called 'novel'. In addition, the existing CSIMs are distinguished as the members of the suggested CSIMs, respectively. The theoretical conditions under which the proposed IMs perform better are obtained by comparing the proposed IMs with the existing IMs. To validate the theoretical findings, the numerical and simulation studies are conducted on real and artificial populations, respectively. [ABSTRACT FROM AUTHOR]

Network analysis has gained popularity as an approach to investigate psychological constructs. However, there are currently no guidelines for applied researchers when encountering missing values. In this simulation study, we compared the performance of a two-step EM algorithm with separated steps for missing handling and regularization, a combined direct EM algorithm, and pairwise deletion. We investigated conditions with varying network sizes, numbers of observations, missing data mechanisms, and percentages of missing values. These approaches are evaluated with regard to recovering population networks in terms of loss in the precision matrix, edge set identification and network statistics. The simulation showed adequate performance only in conditions with large samples ( n ≥ 500 ) or small networks (p = 10). Comparing the missing data approaches, the direct EM appears to be more sensitive and superior in nearly all chosen conditions. The two-step EM yields better results when the ratio of n/p is very large – being less sensitive but more specific. Pairwise deletion failed to converge across numerous conditions and yielded inferior results overall. Overall, direct EM is recommended in most cases, as it is able to mitigate the impact of missing data quite well, while modifications to two-step EM could improve its performance. [ABSTRACT FROM AUTHOR]

Robust low‐rank tensor completion plays an important role in multidimensional data analysis against different degradations, such as sparse noise, and missing entries, and has a variety of applications in image processing and computer vision. In this paper, an optimization model for low‐rank tensor completion problems is proposed and a block coordinate descent algorithm is developed to solve this model. It is shown that for one of the subproblems, the closed‐form solution exists and for the other, a Riemannian conjugate gradient algorithm is used. In particular, when all elements are known, that is, no missing values, the block coordinate descent is simplified in the sense that both subproblems have closed‐form solutions. The convergence analysis is established without requiring the latter subproblem to be solved exactly. Numerical experiments illustrate that the proposed model with the algorithm is feasible and effective. [ABSTRACT FROM AUTHOR]

Missing values (NA) often occur in cancer research, which may be due to reasons such as data protection, data loss, or missing follow-up data. Such incomplete patient information can have an impact on prediction models and other data analyses. Imputation methods are a tool for dealing with NA. Cancer data is often presented in an ordered categorical form, such as tumour grading and staging, which requires special methods. This work compares mode imputation, k nearest neighbour (knn) imputation, and, in the context of Multiple Imputation by Chained Equations (MICE), logistic regression model with proportional odds (mice_polr) and random forest (mice_rf) on a real-world prostate cancer dataset provided by the Cancer Registry of Rhineland-Palatinate in Germany. Our dataset contains relevant information for the risk classification of patients and the time between date of diagnosis and date of death. For the imputation comparison, we use Rubin's (1974) Missing Completely At Random (MCAR) mechanism to remove 10%, 20%, 30%, and 50% observations. The results are evaluated and ranked based on the accuracy per patient. Mice_rf performs significantly best for each percentage of NA, followed by knn, and mice_polr performs significantly worst. Furthermore, our findings indicate that the accuracy of imputation methods increases with a lower number of categories, a relatively even proportion of patients in the categories, or a majority of patients in a particular category. [ABSTRACT FROM AUTHOR]

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks.

PLC (Programmable Logic Controllers) based data collection is integral to industrial automation and data acquisition processes. The pipeline – between the device and the database – is a complex system with many components and often causes some time delay, due to the synchronization and the applied hardware and software components. This delay can also lead to data loss on the database site. In this study, we aimed to examine this problem connected to the synchronized behavior of four counter networks within the TIA software for PLCs, particularly focusing on the implications of a 1 Hz (Hz) clock frequency on counter synchronization. Meticulous experiments were conducted and the integration of Node-RED, as an instrumental tool in industrial automation, facilitated counter-behavior monitoring. Recorded values of counter and timestamps were meticulously stored in four separate databases (MSSQL, MySQL, MongoDB, and Apache Cassandra) for comprehensive data analysis. Throughout the experiments, inconsistencies in the counter values were encountered, leading to the discovery of missing values that varied across the ten tests. To detect the reason, a unique delay calculation method was developed. According to our results, we were able to do post-synchronization with millisecond-level accuracy. It can help reveal the missing values that depend on the Node-RED and the PLC cycle time differences.

Characterizing probability distributions of geotechnical data plays an important role in data-centric geotechnics. On the one hand, geotechnical data are Multivariate, Uncertain, and Irregular (MUI), where the irregular characteristic implies that asymmetry and/or multimodality are often observed in the histograms of geotechnical data, so the corresponding probability distribution is Multivariate, Asymmetric, and Multimodal (MAM). On the other hand, many geotechnical datasets are unavoidably subjected to the issue of modelling and prediction stages missing values (called "dual-stage missing values"), so characterising the MAM distribution of geotechnical data with dual-stage missing values becomes an essential task. There are three fundamental difficulties for this purpose. The first is on joint Probability Density Function (PDF) modelling for a MAM distribution given data with modelling stage missing values. Many traditional and advanced approaches collapse in the presence of MAM distributions and modelling stages missing values, respectively. The second is on joint PDF prediction for a MAM distribution given data with prediction stage missing values. The third is on Credible Region (CR) construction of a MAM distribution as there is no unique CR of a MAM distribution given an exceedance probability only. We propose the three-stage BAyeSIan Copula-based Highest density region/contour (BASIC-H). Stage-1 constructs the posterior distribution of data with modelling stage missing values based on Copula theory and Bayesian inference. Stage-2 derives the posterior predictive distribution of data with prediction stage missing values based on marginalisation and conditionalisation of the posterior distribution. Stage-3 constructs the CRs for the posterior and predictive distributions adopting the reasonable constraint imposed by the Highest Density Region (HDR). Examples using simulated data, CLAY/10/7490 and CLAY/5/345 are presented to illustrate the capability of the proposed BASIC-H. [ABSTRACT FROM AUTHOR]

The accuracy of most classification methods is significantly affected by missing values. Therefore, this study aimed to propose a data imputation method to handle missing values through the application of nearest neighbor data and fuzzy membership function as well as to compare the results with standard methods. A total of five datasets related to classification problems obtained from the UCI Machine Learning Repository were used. The results showed that the proposed method had higher accuracy than standard imputation methods. Moreover, triangular method performed better than Gaussian fuzzy membership function. This showed that the combination of nearest neighbor data and fuzzy membership function was more effective in handling missing values and improving classification accuracy. [ABSTRACT FROM AUTHOR]

In the big data context, it is very frequent to manage the analysis of missing values. This is especially relevant in the field of statistical analysis, where this represents a thorny issue. This study proposes a strategy for data enrichment in presence of sparse matrices. The research objective consists in the evaluation of a possible distinction of behaviour among observations in sparse matrices with missing data. After selecting among the multiple imputation methods, an innovative technique will be presented to impute missing observations as a negative position or a neutral opinion. This method has been applied to a dataset measuring the interaction between users and social network pages for some Italian newspapers. [ABSTRACT FROM AUTHOR]

Missing values are a prevalent issue, frequently leading to a considerable decline in the quality of data. Therefore, it becomes imperative to adeptly manage missing data. This study presents a technique for imputing missing values that incorporates Kernelized Fuzzy C-Means (KFCM) clustering. The technique introduced is termed RFKFCM, which combines the benefits of Random Forest (RF) and KFCM algorithms. The proposed RFKFCM's performance is evaluated through a comparative analysis with five state-of-the-art imputation techniques (mean, LI, KNN, FCM, and LIFCM) across four widely used real-world datasets obtained from the UCI repository. Additionally, experiments investigating the impact of missing values affirm the robustness of the proposed technique across different missing rates using Friedman's statistical test for each dataset by comparing their mean rank. The experimental results indicate that our proposed technique performs significantly better than the existing imputation techniques based on Root mean squared error (RMSE) & Mean absolute error (MAE) for these datasets. [ABSTRACT FROM AUTHOR]

Metabolomics as a research field and a set of techniques is to study the entire small molecules in biological samples. Metabolomics is emerging as a powerful tool generally for precision medicine. Particularly, integration of microbiome and metabolome has revealed the mechanism and functionality of microbiome in human health and disease. However, metabolomics data are very complicated. Preprocessing/pretreating and normalizing procedures on metabolomics data are usually required before statistical analysis. In this review article, we comprehensively review various methods that are used to preprocess and pretreat metabolomics data, including MS-based data and NMR -based data preprocessing, dealing with zero and/or missing values and detecting outliers, data normalization, data centering and scaling, data transformation. We discuss the advantages and limitations of each method. The choice for a suitable preprocessing method is determined by the biological hypothesis, the characteristics of the data set, and the selected statistical data analysis method. We then provide the perspective of their applications in the microbiome and metabolome research.

Multivariate time series (MTS) are captured in a great variety of real-world applications. However, analysing and modeling the data for classification and forecasting purposes can become very challenging if values are missing in the data set. The need for imputation methods, to fill the gaps in MTS, is well known. Thus, a great variaty of algorithms for solving this task has been proposed in the literature. However, the research community is constantly working on the development of advanced algorithms, that fulfill the special requirements of multidimensional temporal data, since most of the existing imputation methods treat MTS as ordinary structured data and fail to model the temporal relationships within and between sequences of observations. The main emphasis of MTS imputation research is currently put on deep learning (DL) models, especially those making use of generative adversarial networks (GANs). In our survey, we present our categorization of imputation algorithms, especially of GAN-models. We included eighteen different GAN-models designed for the MTS imputation task, which we introduce in detail. We provide a comparison of the models including their performance regarding MTS imputation, based on our findings in the literature. The following points can be considered the most important findings from our survey: The research on GAN-based imputation models for MTS has gained momentum in the last years across different domains, therewhile showing the effectiveness of these methods. The latest trend in the research area is the incorporation of attention mechanisms into the algorithms. Nevertheless, there are open research challenges, e.g. the transferability of models across data sets from different domains.

Scientists not only engage in research, but also write articles based on their work, and naturally aim to submit their articles to prestigious, well-received, and highly regarded journals or conferences. One way to measure the prestige of a journal is to use the Journal Impact Factor (JIF), but in order to make best use of this information one needs to better understand the relationships among various JIF-related attributes. However, in recent years JIF values have been missing or unavailable for some journals, due to several objective and subjective factors, which significantly impacts the ability to forecast such values in the following years. In this article we study the factors that directly affect the ranking of journals and can be used to forecast the appropriate values in the next few years, in order to help researchers find the right place to submit their articles in certain journals.

Objective: Diabetes ranks as the most prevalent ailment in developing nations. Vital steps to mitigate the consequences of diabetes include early detection and expert medical intervention. A highly effective approach for identifying diabetes involves assessing the specific indicators associated with this condition. When it comes to automated diabetes detection, frequently encountered datasets frequently exhibit gaps in data, which can markedly impact the effectiveness of machine learning models. Methods: The aim of this study is to propose an automated method for predicting diabetes, with a focus on appropriately dealing with missing data and improving accuracy. The proposed framework makes use of K-Nearest Neighbour (KNN) imputed features along with a Tri-ensemble voting classifier model. Results: By incorporating the KNN imputer, the presented model demonstrates impressive performance metrics, including an accuracy of 97.49%, precision of 98.16%, recall of 99.35%, and an F1 score of 98.84%. The study conducted a thorough comparison of this proposed model against seven alternative machine learning algorithms, assessing them under two conditions: one with omitted missing values and another with the KNN imputer applied. These findings support the proposed model’s efficacy, highlighting its superiority over currently established state-of-the-art techniques. Conclusion: This research explores the problem of missing data in diabetes diagnosis and highlights the efficacy of the KNN-imputed technique. The results are promising for healthcare practitioners as they could facilitate early detection and improve the quality of diabetic patient care.

Sample entropy embeds time series into m-dimensional spaces and estimates entropy based on the distances between points in these spaces. However, when samples can be considered as missing or invalid, defining distance in the embedding space becomes problematic. Preprocessing techniques, such as deletion or interpolation, can be employed as a solution, producing time series without missing or invalid values. While deletion ignores missing values, interpolation replaces them using approximations based on neighboring points. This paper proposes a novel approach for the computation of sample entropy when values are considered as missing or invalid. The proposed algorithm accommodates points in the m-dimensional space and handles them there. A theoretical and experimental comparison of the proposed algorithm with deletion and interpolation demonstrates several advantages over these other two approaches. Notably, the deviation of the expected sample entropy value for the proposed methodology consistently proves to be lowest one.

In the field of machine learning, robust model performance is essential for accurate predictions and informed decision-making. One critical challenge that hampers the performance of machine learning algorithms is the presence of missing data. Missing values are ubiquitous in real-world datasets and can substantially impact the performance of predictive models. This study explored the impact of increasing levels of missing values on the performance of machine learning models. Simulated samples with missing values ranging from 5% to 50% were generated, and various models were evaluated accordingly. The results demonstrated a consistent trend of deteriorating model performance as the amount of missing values increases. Higher levels of missing values lead to decreased accuracy scores across all models. Among the models evaluated, decision trees (DT) and random forests (RF) consistently demonstrated high accuracy scores across all sampling techniques, showcasing their robustness in handling missing values. Logistic regression (LR) also performed relatively well, showing consistent performance across different levels of missing values. On the other hand, stochastic gradient descent classifier (SGDC), K-nearest neighbours (kNN), and naive Bayes (NB) models consistently exhibited lower accuracy scores across all sampling techniques, indicating limitations in handling missing values even when the dataset was more balanced. Furthermore, the study highlights the superiority of the SMOTE (Synthetic Minority OVER-sampling Technique) sampling technique compared to the UNDER-sampling approach. Models trained using SMOTE consistently achieved higher accuracy scores across all levels of missing values. This suggests that SMOTE sampling effectively handles im-balanced datasets and enhances classification performance, particularly when dealing with missing values. As the quest for accurate predictions gains paramount importance, addressing the pervasive challenge of missing data emerges as a cornerstone for unlocking the true potential of machine learning in real-world applications. [ABSTRACT FROM AUTHOR]

Pressure ulcers (PUs) are a prevalent skin disease affecting patients with impaired mobility and in high-risk groups. These ulcers increase patients' suffering, medical expenses, and burden on medical staff. This study introduces a clinical decision support system and verifies it for predicting real-time PU occurrences within the intensive care unit (ICU) by using MIMIC-IV and in-house ICU data. We develop various machine learning (ML) and deep learning (DL) models for predicting PU occurrences in real time using the MIMIC-IV and validate using the MIMIC-IV and Kangwon National University Hospital (KNUH) dataset. To address the challenge of missing values in time series, we propose a novel recurrent neural network model, GRU-D++. This model outperformed other experimental models by achieving the area under the receiver operating characteristic curve (AUROC) of 0.945 for the on-time prediction and AUROC of 0.912 for 48h in-advance prediction. Furthermore, in the external validation with the KNUH dataset, the fine-tuned GRU-D++ model demonstrated superior performances, achieving an AUROC of 0.898 for on-time prediction and an AUROC of 0.897 for 48h in-advance prediction. The proposed GRU-D++, designed to consider temporal information and missing values, stands out for its predictive accuracy. Our findings suggest that this model can significantly alleviate the workload of medical staff and prevent the worsening of patient conditions by enabling timely interventions for PUs in the ICU. [ABSTRACT FROM AUTHOR]

In real-world scenarios, most of the collected data are disorganised, and the probability of getting messy data is more in the data collection. However, retrospective data collected from the health sector have reasonable missing values. For example, an Arthritis Profile Data (APD) dataset collected from a clinical laboratory has several missing values. This incomplete dataset is converted to a reasonable complete dataset using imputation methods. There are several approaches to missingness in the literature, but how well they work depends on the missingness techniques. Unfortunately, there has not been enough investigation into missing data on autoimmune diseases. Although many studies have discussed and compared different imputation techniques, some have limited their comparison without in-depth analyses of the features' imputed and original values. Moreover, some studies lack the identity of the missing category of the features in the dataset. Therefore, this paper fills the gap by describing and exploring APD dataset analysis. The objective is to find a suitable imputation technique for the APD dataset. Compared with other imputation techniques, such as mean imputation, median imputation, mode imputation, random value imputer, kNN imputer, MICE imputer and random forest imputer, the random value imputer and mode imputation techniques are suggested to be the best using the proposed degree of proximity and degree of residual procedures respectively. [ABSTRACT FROM AUTHOR]

Imputation methods are popular tools that allow for a wide range of subsequent analyses on complete data sets. However, in order for these analyses to be trustworthy, it is important that the imputation procedure reflects the true distribution of the unobserved data sufficiently well. This raises the question how well different imputation methods can reproduce multivariate correlations, associations or even the entire multivariate distribution. The paper gives first answers to this question by means of an extensive comparative simulation study. In particular, we evaluate the multivariate distributional accuracy for six state-of-the art imputation algorithms with respect to different measures and give practical recommendations. [ABSTRACT FROM AUTHOR]

As the availability of omics data has increased in the last few years, more multi‐omics data have been generated, that is, high‐dimensional molecular data consisting of several types such as genomic, transcriptomic, or proteomic data, all obtained from the same patients. Such data lend themselves to being used as covariates in automatic outcome prediction because each omics type may contribute unique information, possibly improving predictions compared to using only one omics data type. Frequently, however, in the training data and the data to which automatic prediction rules should be applied, the test data, the different omics data types are not available for all patients. We refer to this type of data as block‐wise missing multi‐omics data. First, we provide a literature review on existing prediction methods applicable to such data. Subsequently, using a collection of 13 publicly available multi‐omics data sets, we compare the predictive performances of several of these approaches for different block‐wise missingness patterns. Finally, we discuss the results of this empirical comparison study and draw some tentative conclusions. This article is categorized under:Applications of Computational Statistics > Genomics/Proteomics/GeneticsApplications of Computational Statistics > Health and Medical Data/InformaticsStatistical and Graphical Methods of Data Analysis > Analysis of High Dimensional Data [ABSTRACT FROM AUTHOR]

The measurement of latent traits and investigation of relations between these and a potentially large set of explaining variables is typical in psychology, economics, and the social sciences. Corresponding analysis often relies on surveyed data from large-scale studies involving hierarchical structures and missing values in the set of considered covariates. This paper proposes a Bayesian estimation approach based on the device of data augmentation that addresses the handling of missing values in multilevel latent regression models. Population heterogeneity is modeled via multiple groups enriched with random intercepts. Bayesian estimation is implemented in terms of a Markov chain Monte Carlo sampling approach. To handle missing values, the sampling scheme is augmented to incorporate sampling from the full conditional distributions of missing values. We suggest to model the full conditional distributions of missing values in terms of non-parametric classification and regression trees. This offers the possibility to consider information from latent quantities functioning as sufficient statistics. A simulation study reveals that this Bayesian approach provides valid inference and outperforms complete cases analysis and multiple imputation in terms of statistical efficiency and computation time involved. An empirical illustration using data on mathematical competencies demonstrates the usefulness of the suggested approach. [ABSTRACT FROM AUTHOR]

The accuracy of most classification methods is significantly affected by missing values. Therefore, this study aimed to propose a data imputation method to handle missing values through the application of nearest neighbor data and fuzzy membership function as well as to compare the results with standard methods. A total of five datasets related to classification problems obtained from the UCI Machine Learning Repository were used. The results showed that the proposed method had higher accuracy than standard imputation methods. Moreover, triangular method performed better than Gaussian fuzzy membership function. This showed that the combination of nearest neighbor data and fuzzy membership function was more effective in handling missing values and improving classification accuracy.

Ellenberg indicator values (EIV) are widely used in vegetation ecology, but the values for many species in Southeastern Europe are not available due to incomplete knowledge of their ecology: it is therefore of paramount importance to estimate missing values in existing databases. The entire EIV set for a single species can be missing or a single EIV can be missing for species for which other indicator values are available. Our aim here is to provide a simple method to impute missing values for species who have missing data in a single or multiple EIV. For this purpose, we adopt a multiple imputation procedure and compare a number of imputation methods on the basis of two datasets: i) “indices”, the set of 9 Ellenberg indicators taken from literature, available for 10,824 species and ii) “vegetation”, a set describing the physical and climatic characteristics (Light, Temperature, Continentality, Soil moisture, Nitrogen, Soil pH, Hemeroby index, Humidity, Organic_matter) of 29,935 relevés from Southeastern Europe where at least one tree species is present. The imputation methods we considered are: k-Nearest Neighbour, multiple linear regression (with or without collinearity correction), Reprediction Algorithm, Weighted Averaging (WA) and Weighted Averaging Partial Least Squares (WAPLS) regression. The different methods of imputation were compared by looking at the output produced and its deviation from the “true” observed values for a set of species with known EIVs. We have considered a set of species with known EIVs and proceeded to multiple imputation using the methods above; as a measure of performance we adopted the mean squared error (MSE) estimate, and expert judgement of ecological consistency. Models based on Regression and k-Nearest Neighbour seem to outperform the others. On the contrary, Reprediction algorithm in its different forms: produced less satisfactory results.Imputation of missing values is generally based on expert knowledge or on some variant of weighted averaging (also known as Hill’s method). Here we show that other methods may be more effective and should be appropriately considered by vegetation scientists, since those may allow the application of EIVs in other biogeographic regions.

DNA methylation plays an essential role in regulating gene activity, modulating disease risk, and determining treatment response. We can obtain insight into methylation patterns at a single-nucleotide level via next-generation sequencing technologies. However, complex features inherent in the data obtained via these technologies pose challenges beyond the typical big data problems. Identifying differentially methylated cytosines (dmc) or regions is one such challenge. We have developed DMCFB, an efficient dmc identification method based on Bayesian functional regression, to tackle these challenges. Using simulations, we establish that DMCFB outperforms current methods and results in better smoothing and efficient imputation. We analyzed a dataset of patients with acute promyelocytic leukemia and control samples. With DMCFB, we discovered many new dmcs and, more importantly, exhibited enhanced consistency of differential methylation within islands and their adjacent shores. Additionally, we detected differential methylation at more of the binding sites of the fused gene involved in this cancer.

An emergency information response system (EIRS) is a system that utilizes various intelligence technologies to effectively handle various emergencies and provide decision support for decision-makers. As critical success factors (CSFs) in EIRS play a vital role in emergency management, it is necessary to study the CSFs of EIRS. Most previous studies applied the Decision Experiment and Decision-Making Trial and Evaluation Laboratory (DEMATEL) method with complete evaluation information to identify CSFs. Due to the complexity of the decision-making environment when identifying CSFs of EIRS, decision-makers sometimes cannot provide complete evaluation information during the decision-making process. To fill this gap, this paper provided a Similar-DEMATEL method to impute the missing values and identify CSFs of EIRS, which may avoid the dilemma of decision distortion and make decision-making results more accurate. It is found that the factors of Information mining capability, Equipment support capability, Monitoring and early warning capability, and Organization participation capability are the CSFs in EIRS. Our proposed method differs from previous research, such as the mean imputation method, to impute the missing values. We compared the differences between the proposed method and the mean imputation method and gave the advantages of the proposed method. Our method focuses more on uncertain decision-making environments, which is conducive to improving the efficiency of EIRS in emergency management, and therefore it is more widely adopted. [ABSTRACT FROM AUTHOR]

Abstract The usual Minimum Covariance Determinant (MCD) estimator of a covariance matrix is robust against casewise outliers. These are cases (that is, rows of the data matrix) that behave differently from the majority of cases, raising suspicion that they might belong to a different population. On the other hand, cellwise outliers are individual cells in the data matrix. When a row contains one or more outlying cells, the other cells in the same row still contain useful information that we wish to preserve. We propose a cellwise robust version of the MCD method, called cellMCD. Its main building blocks are observed likelihood and a penalty term on the number of flagged cellwise outliers. It possesses good breakdown properties. We construct a fast algorithm for cellMCD based on concentration steps (C-steps) that always lower the objective. The method performs well in simulations with cellwise outliers, and has high finite-sample efficiency on clean data. It is illustrated on real data with visualizations of the results. Supplementary materials for this article are available online. [ABSTRACT FROM AUTHOR]