Search

The study aimed to examine the effect of the percentage of missing values on the matching of the items of the graded model. To achieve this, the experimental approach was followed in conducting the study, as hypothetical data were used to simulate the experimental conditions represented by the percentage of missing values. It has three levels (5%, 10%, and 20%). Moreover, the compensation method is represented by simple regression. The study found that there were no statistically significant differences in the percentage of items matching the graded model according to the missing percentages (5% and 10%). Moreover, there were statistically significant differences in the percentage of items matching the graded model according to the missing percentages (20%). Therefore, we must use the different nonparametric item response theory (IRT) models because of the good information, it provides to researchers about the test and its items without missing items suitable for this purpose and prejudice--the cumulative response of the subjects. [ABSTRACT FROM AUTHOR]

A crucial input into causal inference is the imputed counterfactual outcome. Imputation error can arise because of sampling uncertainty from estimating the prediction model using the untreated observations, or from out-of-sample information not captured by the model. While the literature has focused on sampling uncertainty, it vanishes with the sample size. Often overlooked is the possibility that the out-of-sample error can be informative about the missing counterfactual outcome if it is mutually or serially correlated. Motivated by the best linear unbiased predictor (BLUP) of Goldberger in a time series setting, we propose an improved predictor of potential outcome when the errors are correlated. The proposed PUP is practical as it is not restricted to linear models, can be used with consistent estimators already developed, and improves mean-squared error for a large class of strong mixing error processes. Ignoring predictability in the errors can distort conditional inference. However, the precise impact will depend on the choice of estimator as well as the realized values of the residuals. [ABSTRACT FROM AUTHOR]

Diabetes is thought to be the most common illness in underdeveloped nations. Early detection and competent medical care are crucial steps in reducing the effects of diabetes. Examining the signs associated with diabetes is one of the most effective ways to identify the condition. The problem of missing data is not very well investigated in existing works. In addition, existing studies on diabetes detection lack accuracy and robustness. The available datasets frequently contain missing information for the automated detection of diabetes, which might negatively impact machine learning model performance. This work suggests an automated diabetes prediction method that achieves high accuracy and effectively manages missing variables in order to address this problem. The proposed strategy employs a stacked ensemble voting classifier model with three machine learning models. and a KNN Imputer to handle missing values. Using the KNN imputer, the suggested model performs exceptionally well, with accuracy, precision, recall, F1 score, and MCC of 98.59%, 99.26%, 99.75%, 99.45%, and 99.24%, respectively. In two scenarios one with missing values eliminated and the other with KNN imputer, the study thoroughly compared the suggested model with seven other machine learning techniques. The outcomes demonstrate the superiority of the suggested model over current state-of-the-art methods and confirm its efficacy. This work demonstrates the capability of KNN imputer and looks at the problem of missing values for diabetes detection. Medical professionals can utilize the results to improve care for diabetes patients and discover problems early. [ABSTRACT FROM AUTHOR]

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks. [ABSTRACT FROM AUTHOR]

Background and objectives: Comprehending the research dataset is crucial for obtaining reliable and valid outcomes. Health analysts must have a deep comprehension of the data being analyzed. This comprehension allows them to suggest practical solutions for handling missing data, in a clinical data source. Accurate handling of missing values is critical for producing precise estimates and making informed decisions, especially in crucial areas like clinical research. With data's increasing diversity and complexity, numerous scholars have developed a range of imputation techniques. To address this, we conducted a systematic review to introduce various imputation techniques based on tabular dataset characteristics, including the mechanism, pattern, and ratio of missingness, to identify the most appropriate imputation methods in the healthcare field. Materials and methods: We searched four information databases namely PubMed, Web of Science, Scopus, and IEEE Xplore, for articles published up to September 20, 2023, that discussed imputation methods for addressing missing values in a clinically structured dataset. Our investigation of selected articles focused on four key aspects: the mechanism, pattern, ratio of missingness, and various imputation strategies. By synthesizing insights from these perspectives, we constructed an evidence map to recommend suitable imputation methods for handling missing values in a tabular dataset. Results: Out of 2955 articles, 58 were included in the analysis. The findings from the development of the evidence map, based on the structure of the missing values and the types of imputation methods used in the extracted items from these studies, revealed that 45% of the studies employed conventional statistical methods, 31% utilized machine learning and deep learning methods, and 24% applied hybrid imputation techniques for handling missing values. Conclusion: Considering the structure and characteristics of missing values in a clinical dataset is essential for choosing the most appropriate data imputation technique, especially within conventional statistical methods. Accurately estimating missing values to reflect reality enhances the likelihood of obtaining high-quality and reusable data, contributing significantly to precise medical decision-making processes. Performing this review study creates a guideline for choosing the most appropriate imputation methods in data preprocessing stages to perform analytical processes on structured clinical datasets. Highlights: • The evidence map emphasized the importance of considering missing data characteristics when choosing an imputation method, providing insights for researchers in method selection. • The distinction between statistical and learning-based approaches highlighted the strengths and considerations of each method based on data structure. • Understanding missing data structures can enhance the quality and reliability of data imputation techniques, and improve medical decision-making accuracy. • Simulation studies are crucial for validating imputation techniques and enhancing their robustness in practical applications. • Considering missing data mechanisms, patterns, and ratios can aid researchers in making informed decisions on selecting appropriate imputation methods, leading to high-quality, and reusable data for precise medical decision-making. [ABSTRACT FROM AUTHOR]

Sample entropy embeds time series into m-dimensional spaces and estimates entropy based on the distances between points in these spaces. However, when samples can be considered as missing or invalid, defining distance in the embedding space becomes problematic. Preprocessing techniques, such as deletion or interpolation, can be employed as a solution, producing time series without missing or invalid values. While deletion ignores missing values, interpolation replaces them using approximations based on neighboring points. This paper proposes a novel approach for the computation of sample entropy when values are considered as missing or invalid. The proposed algorithm accommodates points in the m-dimensional space and handles them there. A theoretical and experimental comparison of the proposed algorithm with deletion and interpolation demonstrates several advantages over these other two approaches. Notably, the deviation of the expected sample entropy value for the proposed methodology consistently proves to be lowest one. [ABSTRACT FROM AUTHOR]

Multiple imputation of missing values is a key step in data analytics and a standard process in data science. Nonlinear imputation methods come into play whenever the linear relationship between a response and predictors cannot be linearized by transformations of variables, adding interactions, or using, e.g., quadratic terms. Generalized additive models (GAM) and its extension, GAMLSS—where each parameter of the distribution, such as mean, variance, skewness, and kurtosis, can be represented as a function of predictors, are widely used nonlinear methods. However, non-robust methods such as standard GAM’s and GAMLSS’s can be swayed by outliers, leading to outlier-driven imputations. This can apply concerning both representative outliers—those true yet unusual values of your population—and non-representative outliers, which are mere measurement errors. Robust (imputation) methods effectively manage outliers and exhibit resistance to their influence, providing a more reliable approach to dealing with missing data. The innovative solution of the proposed new imputation algorithm tackles three major challenges related to robustness. (1) A robust bootstrap method is employed to handle model uncertainty during the imputation of a random sample. (2) The approach incorporates robust fitting techniques to enhance accuracy. (3) It effectively considers imputation uncertainty in a resilient manner. Furthermore, any complex model for any variable with missingness can be considered and run through the algorithm. For the real-world data sets used and the simulation study conducted, the novel algorithm imputeRobust which includes robust methods for imputation with GAM’s demonstrates superior performance compared to existing imputation methods using GAMLSS. Limitations pertain to the imputation of categorical variables using robust techniques. [ABSTRACT FROM AUTHOR]

Abstract Diabetes is thought to be the most common illness in underdeveloped nations. Early detection and competent medical care are crucial steps in reducing the effects of diabetes. Examining the signs associated with diabetes is one of the most effective ways to identify the condition. The problem of missing data is not very well investigated in existing works. In addition, existing studies on diabetes detection lack accuracy and robustness. The available datasets frequently contain missing information for the automated detection of diabetes, which might negatively impact machine learning model performance. This work suggests an automated diabetes prediction method that achieves high accuracy and effectively manages missing variables in order to address this problem. The proposed strategy employs a stacked ensemble voting classifier model with three machine learning models. and a KNN Imputer to handle missing values. Using the KNN imputer, the suggested model performs exceptionally well, with accuracy, precision, recall, F1 score, and MCC of 98.59%, 99.26%, 99.75%, 99.45%, and 99.24%, respectively. In two scenarios one with missing values eliminated and the other with KNN imputer, the study thoroughly compared the suggested model with seven other machine learning techniques. The outcomes demonstrate the superiority of the suggested model over current state-of-the-art methods and confirm its efficacy. This work demonstrates the capability of KNN imputer and looks at the problem of missing values for diabetes detection. Medical professionals can utilize the results to improve care for diabetes patients and discover problems early.

Abstract Background and objectives Comprehending the research dataset is crucial for obtaining reliable and valid outcomes. Health analysts must have a deep comprehension of the data being analyzed. This comprehension allows them to suggest practical solutions for handling missing data, in a clinical data source. Accurate handling of missing values is critical for producing precise estimates and making informed decisions, especially in crucial areas like clinical research. With data's increasing diversity and complexity, numerous scholars have developed a range of imputation techniques. To address this, we conducted a systematic review to introduce various imputation techniques based on tabular dataset characteristics, including the mechanism, pattern, and ratio of missingness, to identify the most appropriate imputation methods in the healthcare field. Materials and methods We searched four information databases namely PubMed, Web of Science, Scopus, and IEEE Xplore, for articles published up to September 20, 2023, that discussed imputation methods for addressing missing values in a clinically structured dataset. Our investigation of selected articles focused on four key aspects: the mechanism, pattern, ratio of missingness, and various imputation strategies. By synthesizing insights from these perspectives, we constructed an evidence map to recommend suitable imputation methods for handling missing values in a tabular dataset. Results Out of 2955 articles, 58 were included in the analysis. The findings from the development of the evidence map, based on the structure of the missing values and the types of imputation methods used in the extracted items from these studies, revealed that 45% of the studies employed conventional statistical methods, 31% utilized machine learning and deep learning methods, and 24% applied hybrid imputation techniques for handling missing values. Conclusion Considering the structure and characteristics of missing values in a clinical dataset is essential for choosing the most appropriate data imputation technique, especially within conventional statistical methods. Accurately estimating missing values to reflect reality enhances the likelihood of obtaining high-quality and reusable data, contributing significantly to precise medical decision-making processes. Performing this review study creates a guideline for choosing the most appropriate imputation methods in data preprocessing stages to perform analytical processes on structured clinical datasets.

Class imbalance in a dataset leads to erroneous outcomes that engrave the learning techniques and high misclassification cost in the minority class. Along with class imbalance, missing values present in a disproportionate amount in a dataset create a great hindrance to the effective performance of a method. The ensemble method, where multiple methods are ensembled, tackles such issues and shows good results compared to the performance of individual methods. In this paper, a hybridization of multiple imputation and one-class bagging ensemble approach is proposed that handles datasets having both class imbalance and missing values. An in-depth analysis of this approach is studied and the effectiveness of the class imbalance is also presented. To tackle the misclassification of minority samples and missing values, factor-based multiple imputation oversampling technique is used and one-class classifier is ensembled to increase the performance of the class imbalance datasets. Experiments are performed using a one-class support vector machine classifier and the results are evaluated using metrics: Recall (Detection rate), Specificity, f-measure, g-mean, AUC, and Precision. The proposed approach yields a 6.3% improvement in Recall, whereas Specificity, f-measure, g-mean, AUC, and Precision show that the proposed approach improves by 4.92%, 11.3%, 9.4%, 8.3%, and 8.03%, respectively. [ABSTRACT FROM AUTHOR]

Many imputation methods have been developed over the years and tested mostly under ideal settings. Surprisingly, there is no detailed research on how imputation methods perform when the idealized assumptions about the distribution of data and/or model assumptions are partly not fulfilled. This research looks into the susceptibility of imputation techniques, particularly in relation to outliers, misclassifications, and incorrect model specifications. This is crucial knowledge about how well the methods convince in everyday life because, in reality, conditions are usually not ideal, and model assumptions may not hold. The data may not fit the defined models well. Outliers distort the estimates, and misclassifications reduce the quality of most imputation methods. Several different evaluation measures are discussed, from comparing imputed values with true values or comparing certain statistics, from the performance of classifiers to the variance of estimated parameters. Some well-known imputation methods are compared based on real data and simulations. It turns out that robust conditional imputation methods outperform other methods for real data and simulation settings. [ABSTRACT FROM AUTHOR]

Instantaneous peak flows (IPFs) are often required to derive design values for sizing various hydraulic structures, such as culverts, bridges, and small dams/levees, in addition to informing several water resources management-related activities. Compared to mean daily flows (MDFs), which represent averaged flows over a period of 24 h, information on IPFs is often missing or unavailable in instrumental records. In this study, conventional methods for estimating IPFs from MDFs are evaluated and new methods based on the nonlinear regression framework and machine learning architectures are proposed and evaluated using streamflow records from all Canadian hydrometric stations with natural and regulated flow regimes. Based on a robust model selection criterion, it was found that multiple methods are suitable for estimating IPFs from MDFs, which precludes the idea of a single universal method. The performance of machine learning-based methods was also found reasonable compared to conventional and regression-based methods. To build on the strengths of individual methods, the fusion modeling concept from the machine learning area was invoked to synthesize outputs of multiple methods. The study findings are expected to be useful to the climate change adaptation community, which currently heavily relies on MDFs simulated by hydrologic models. HIGHLIGHTS New methods for estimating instantaneous peak flows from mean daily flows.; Data completion by filling in missing values of instantaneous peak flows.; Reliable estimation of design flood magnitudes.; Machine learning inspired fusion modeling to synthesize outputs of multiple methods.; Fulfilled a critical need of the climate change impact analysis and adaptation community.;

This paper addresses some classes of combined and separate imputation methods (CSIMs) of the population mean under stratified simple random sampling (SSRS) along with their characteristics. To the best of our knowledge, these imputation methods (IMs) have yet not been studied by any author under SSRS, hence these IMs are called ‘novel’. In addition, the existing CSIMs are distinguished as the members of the suggested CSIMs, respectively. The theoretical conditions under which the proposed IMs perform better are obtained by comparing the proposed IMs with the existing IMs. To validate the theoretical findings, the numerical and simulation studies are conducted on real and artificial populations, respectively.

Copyright of Dirasat: Educational Sciences is the property of University of Jordan and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Numerous real-world data contain missing values, while in contrast, most Machine Learning (ML) algorithms assume complete datasets. For this reason, several imputation algorithms have been proposed to predict and fill in the missing values. Given the advances in predictive modeling algorithms tuned in an Automated Machine Learning context (AutoML) setting, a question that naturally arises is to what extent sophisticated imputation algorithms (e.g., Neural Network based) are really needed, or we can obtain a descent performance using simple methods like Mean/Mode (MM). In this article, we experimentally compare six state-of-the-art representatives of different imputation algorithmic families from an AutoML predictive modeling perspective, including a feature selection step and combined algorithm and hyper-parameter selection. We used a commercial AutoML tool for our experiments, in which we included the selected imputation methods. Experiments ran on 25 binary classification real-world incomplete datasets with missing values and 10 binary classification complete datasets in which synthetic missing values are introduced according to different missingness mechanisms, at varying missing frequencies. The main conclusion drawn from our experiments is that the best method on average is the Denoise AutoEncoder on real-world datasets and the MissForest in simulated datasets, followed closely by MM. In addition, binary indicator variables encoding missingness patterns actually improve predictive performance, on average. Last, although there are cases where Neural-Network-based imputation significantly improves predictive performance, this comes at a great computational cost and requires measuring all feature values to impute new samples. [ABSTRACT FROM AUTHOR]

DNA methylation plays an essential role in regulating gene activity, modulating disease risk, and determining treatment response. We can obtain insight into methylation patterns at a single-nucleotide level via next-generation sequencing technologies. However, complex features inherent in the data obtained via these technologies pose challenges beyond the typical big data problems. Identifying differentially methylated cytosines (dmc) or regions is one such challenge. We have developed DMCFB, an efficient dmc identification method based on Bayesian functional regression, to tackle these challenges. Using simulations, we establish that DMCFB outperforms current methods and results in better smoothing and efficient imputation. We analyzed a dataset of patients with acute promyelocytic leukemia and control samples. With DMCFB, we discovered many new dmcs and, more importantly, exhibited enhanced consistency of differential methylation within islands and their adjacent shores. Additionally, we detected differential methylation at more of the binding sites of the fused gene involved in this cancer. [ABSTRACT FROM AUTHOR]

The site regression model (SREG) is utilized by plant breeders for analyzing multi-environment trials (MET) to examine the relationships among test environments, genotypes (G), and genotype-by-environment interactions (GE). SREG explores a matrix of G and GE by performing a singular value decomposition on the residuals matrix from a one-way ANOVA, requiring complete data. As missing values are common in MET, we propose two new imputation methods that implement an Expectation Maximization algorithm to fit the SREG model. To evaluate the impact on SREG model parameter estimation of these proposed methods and other competing imputation methods available, we conducted two studies using different approaches. One study involved simulated data while the other used a real dataset. In both studies, different measures to verify whether the joint effect of G plus GE is altered by imputation of data, and the reproducibility of missing data were evaluated. We also incorporated situations not commonly addressed in the literature, such as non-random structures of missing values and a big data situation. The proposed procedures provided estimators with good performance, maintaining superiority in several aspects studied, even when the competing imputation methods did not achieve convergence. Therefore, the new methods enabled incomplete MET data to be effectively analyzed by a SREG model. [ABSTRACT FROM AUTHOR]

Satellite time series data, widely used for land cover classification, often contain missing values due to cloud contamination, which can negatively affect classification. Numerous strategies have been developed to reconstruct the missing values to produce regular time series for machine learning classifiers, among which the compositing followed by the linear interpolation is most widely used. However, the classification improvement of linear interpolation for land cover classification has not been examined. Recently developed deep learning models such as long short term memory (LSTM) and Transformer allow such examination as they can classify time series with missing values. In this study, we compared the time series composites with missing values (without linear interpolation) and the linearly interpolated time series composites (without missing values) for land cover classification. About 18 thousand Harmonized Landsat Sentinel-2 (HLS) images acquired over Amur River Basin of China (890,308 km2) in 2021 were composited to 14 16-day periods. Two time series composites were classified, i.e., (i) the 16-day composites without interpolation that have on average 15.35% 16-day periods with missing values and (ii) the linearly interpolated 16-day composites with no missing values. The classifications showed that (1) between classifications with and without linear interpolation there was < 0.2% overall accuracy differences for the bidirectional LSTM (Bi-LSTM) and < 0.5% for the Transformer both of which were smaller than model training randomness; and (2) the computation time can be saved using composites without linear interpolation. The findings suggested that it is unnecessary to use the time-consuming linear interpolation in Bi-LSTM and Transformer-based land cover classifications. The findings were confirmed by experiments for sensitivity to the number of cloud-free composites and to different classification legends using crop type classifications. It implied the linear interpolation algorithm cannot reconstruct reliable time series for land cover classifications and historical use of such method is more about mitigating the inability of traditional classifiers to handle missing values rather than improving classifications. Linear interpolation is not necessary for LSTM and Transformer with capability to handle missing values. The training datasets and developed codes in this study are made publicly available. [ABSTRACT FROM AUTHOR]

Chronic kidney disease (CKD) is a major health concern today, requiring early and accurate diagnosis. Machine learning has emerged as a powerful tool for disease detection, and medical professionals are increasingly using ML classifier algorithms to identify CKD early. This study explores the application of advanced machine learning techniques on a CKD dataset obtained from the University of California, UC Irvine Machine Learning repository. The research introduces TrioNet, an ensemble model combining extreme gradient boosting, random forest, and extra tree classifier, which excels in providing highly accurate predictions for CKD. Furthermore, K nearest neighbor (KNN) imputer is utilized to deal withmissing values while synthetic minority oversampling (SMOTE) is used for class-imbalance problems. To ascertain the efficacy of the proposed model, a comprehensive comparative analysis is conducted with various machine learning models. The proposed TrioNet using KNN imputer and SMOTE outperformed other models with 98.97% accuracy for detectingCKD. This in-depth analysis demonstrates the model's capabilities and underscores its potential as a valuable tool in the diagnosis of CKD. [ABSTRACT FROM AUTHOR]

Linear regression (LR) is vastly used in data analysis for continuous outcomes in biomedicine and epidemiology. Despite its popularity, LR is incompatible with missing data, which frequently occur in health sciences. For parameter estimation, this shortcoming is usually resolved by complete‐case analysis or imputation. Both work‐arounds, however, are inadequate for prediction, since they either fail to predict on incomplete records or ignore missingness‐induced reduction in prediction accuracy and rely on (unrealistic) assumptions about the missing mechanism. Here, we derive adaptive predictor‐set linear model (aps‐lm), capable of making predictions for incomplete data without the need for imputation. It is derived by using a predictor‐selection operation, the Moore–Penrose pseudoinverse, and the reduced QR decomposition. aps‐lm is an LR generalization that inherently handles missing values. It is applied on a reference data set, where complete predictors and outcome are available, and yields a set of privacy‐preserving parameters. In a second stage, these are shared for making predictions of the outcome on external data sets with missing entries for predictors without imputation. Moreover, aps‐lm computes prediction errors that account for the pattern of missing values even under extreme missingness. We benchmark aps‐lm in a simulation study. aps‐lm showed greater prediction accuracy and reduced bias compared to popular imputation strategies under a wide range of scenarios including variation of sample size, goodness of fit, missing value type, and covariance structure. Finally, as a proof‐of‐principle, we apply aps‐lm in the context of epigenetic aging clocks, linear models that predict a person's biological age from epigenetic data with promising clinical applications. [ABSTRACT FROM AUTHOR]

This paper addresses some classes of combined and separate imputation methods (CSIMs) of the population mean under stratified simple random sampling (SSRS) along with their characteristics. To the best of our knowledge, these imputation methods (IMs) have yet not been studied by any author under SSRS, hence these IMs are called 'novel'. In addition, the existing CSIMs are distinguished as the members of the suggested CSIMs, respectively. The theoretical conditions under which the proposed IMs perform better are obtained by comparing the proposed IMs with the existing IMs. To validate the theoretical findings, the numerical and simulation studies are conducted on real and artificial populations, respectively. [ABSTRACT FROM AUTHOR]

Network analysis has gained popularity as an approach to investigate psychological constructs. However, there are currently no guidelines for applied researchers when encountering missing values. In this simulation study, we compared the performance of a two-step EM algorithm with separated steps for missing handling and regularization, a combined direct EM algorithm, and pairwise deletion. We investigated conditions with varying network sizes, numbers of observations, missing data mechanisms, and percentages of missing values. These approaches are evaluated with regard to recovering population networks in terms of loss in the precision matrix, edge set identification and network statistics. The simulation showed adequate performance only in conditions with large samples ( n ≥ 500 ) or small networks (p = 10). Comparing the missing data approaches, the direct EM appears to be more sensitive and superior in nearly all chosen conditions. The two-step EM yields better results when the ratio of n/p is very large – being less sensitive but more specific. Pairwise deletion failed to converge across numerous conditions and yielded inferior results overall. Overall, direct EM is recommended in most cases, as it is able to mitigate the impact of missing data quite well, while modifications to two-step EM could improve its performance. [ABSTRACT FROM AUTHOR]

Robust low‐rank tensor completion plays an important role in multidimensional data analysis against different degradations, such as sparse noise, and missing entries, and has a variety of applications in image processing and computer vision. In this paper, an optimization model for low‐rank tensor completion problems is proposed and a block coordinate descent algorithm is developed to solve this model. It is shown that for one of the subproblems, the closed‐form solution exists and for the other, a Riemannian conjugate gradient algorithm is used. In particular, when all elements are known, that is, no missing values, the block coordinate descent is simplified in the sense that both subproblems have closed‐form solutions. The convergence analysis is established without requiring the latter subproblem to be solved exactly. Numerical experiments illustrate that the proposed model with the algorithm is feasible and effective. [ABSTRACT FROM AUTHOR]

Missing values (NA) often occur in cancer research, which may be due to reasons such as data protection, data loss, or missing follow-up data. Such incomplete patient information can have an impact on prediction models and other data analyses. Imputation methods are a tool for dealing with NA. Cancer data is often presented in an ordered categorical form, such as tumour grading and staging, which requires special methods. This work compares mode imputation, k nearest neighbour (knn) imputation, and, in the context of Multiple Imputation by Chained Equations (MICE), logistic regression model with proportional odds (mice_polr) and random forest (mice_rf) on a real-world prostate cancer dataset provided by the Cancer Registry of Rhineland-Palatinate in Germany. Our dataset contains relevant information for the risk classification of patients and the time between date of diagnosis and date of death. For the imputation comparison, we use Rubin's (1974) Missing Completely At Random (MCAR) mechanism to remove 10%, 20%, 30%, and 50% observations. The results are evaluated and ranked based on the accuracy per patient. Mice_rf performs significantly best for each percentage of NA, followed by knn, and mice_polr performs significantly worst. Furthermore, our findings indicate that the accuracy of imputation methods increases with a lower number of categories, a relatively even proportion of patients in the categories, or a majority of patients in a particular category. [ABSTRACT FROM AUTHOR]

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks.

PLC (Programmable Logic Controllers) based data collection is integral to industrial automation and data acquisition processes. The pipeline – between the device and the database – is a complex system with many components and often causes some time delay, due to the synchronization and the applied hardware and software components. This delay can also lead to data loss on the database site. In this study, we aimed to examine this problem connected to the synchronized behavior of four counter networks within the TIA software for PLCs, particularly focusing on the implications of a 1 Hz (Hz) clock frequency on counter synchronization. Meticulous experiments were conducted and the integration of Node-RED, as an instrumental tool in industrial automation, facilitated counter-behavior monitoring. Recorded values of counter and timestamps were meticulously stored in four separate databases (MSSQL, MySQL, MongoDB, and Apache Cassandra) for comprehensive data analysis. Throughout the experiments, inconsistencies in the counter values were encountered, leading to the discovery of missing values that varied across the ten tests. To detect the reason, a unique delay calculation method was developed. According to our results, we were able to do post-synchronization with millisecond-level accuracy. It can help reveal the missing values that depend on the Node-RED and the PLC cycle time differences.

Characterizing probability distributions of geotechnical data plays an important role in data-centric geotechnics. On the one hand, geotechnical data are Multivariate, Uncertain, and Irregular (MUI), where the irregular characteristic implies that asymmetry and/or multimodality are often observed in the histograms of geotechnical data, so the corresponding probability distribution is Multivariate, Asymmetric, and Multimodal (MAM). On the other hand, many geotechnical datasets are unavoidably subjected to the issue of modelling and prediction stages missing values (called "dual-stage missing values"), so characterising the MAM distribution of geotechnical data with dual-stage missing values becomes an essential task. There are three fundamental difficulties for this purpose. The first is on joint Probability Density Function (PDF) modelling for a MAM distribution given data with modelling stage missing values. Many traditional and advanced approaches collapse in the presence of MAM distributions and modelling stages missing values, respectively. The second is on joint PDF prediction for a MAM distribution given data with prediction stage missing values. The third is on Credible Region (CR) construction of a MAM distribution as there is no unique CR of a MAM distribution given an exceedance probability only. We propose the three-stage BAyeSIan Copula-based Highest density region/contour (BASIC-H). Stage-1 constructs the posterior distribution of data with modelling stage missing values based on Copula theory and Bayesian inference. Stage-2 derives the posterior predictive distribution of data with prediction stage missing values based on marginalisation and conditionalisation of the posterior distribution. Stage-3 constructs the CRs for the posterior and predictive distributions adopting the reasonable constraint imposed by the Highest Density Region (HDR). Examples using simulated data, CLAY/10/7490 and CLAY/5/345 are presented to illustrate the capability of the proposed BASIC-H. [ABSTRACT FROM AUTHOR]

The accuracy of most classification methods is significantly affected by missing values. Therefore, this study aimed to propose a data imputation method to handle missing values through the application of nearest neighbor data and fuzzy membership function as well as to compare the results with standard methods. A total of five datasets related to classification problems obtained from the UCI Machine Learning Repository were used. The results showed that the proposed method had higher accuracy than standard imputation methods. Moreover, triangular method performed better than Gaussian fuzzy membership function. This showed that the combination of nearest neighbor data and fuzzy membership function was more effective in handling missing values and improving classification accuracy. [ABSTRACT FROM AUTHOR]

In the big data context, it is very frequent to manage the analysis of missing values. This is especially relevant in the field of statistical analysis, where this represents a thorny issue. This study proposes a strategy for data enrichment in presence of sparse matrices. The research objective consists in the evaluation of a possible distinction of behaviour among observations in sparse matrices with missing data. After selecting among the multiple imputation methods, an innovative technique will be presented to impute missing observations as a negative position or a neutral opinion. This method has been applied to a dataset measuring the interaction between users and social network pages for some Italian newspapers. [ABSTRACT FROM AUTHOR]

Missing values are a prevalent issue, frequently leading to a considerable decline in the quality of data. Therefore, it becomes imperative to adeptly manage missing data. This study presents a technique for imputing missing values that incorporates Kernelized Fuzzy C-Means (KFCM) clustering. The technique introduced is termed RFKFCM, which combines the benefits of Random Forest (RF) and KFCM algorithms. The proposed RFKFCM's performance is evaluated through a comparative analysis with five state-of-the-art imputation techniques (mean, LI, KNN, FCM, and LIFCM) across four widely used real-world datasets obtained from the UCI repository. Additionally, experiments investigating the impact of missing values affirm the robustness of the proposed technique across different missing rates using Friedman's statistical test for each dataset by comparing their mean rank. The experimental results indicate that our proposed technique performs significantly better than the existing imputation techniques based on Root mean squared error (RMSE) & Mean absolute error (MAE) for these datasets. [ABSTRACT FROM AUTHOR]

Metabolomics as a research field and a set of techniques is to study the entire small molecules in biological samples. Metabolomics is emerging as a powerful tool generally for precision medicine. Particularly, integration of microbiome and metabolome has revealed the mechanism and functionality of microbiome in human health and disease. However, metabolomics data are very complicated. Preprocessing/pretreating and normalizing procedures on metabolomics data are usually required before statistical analysis. In this review article, we comprehensively review various methods that are used to preprocess and pretreat metabolomics data, including MS-based data and NMR -based data preprocessing, dealing with zero and/or missing values and detecting outliers, data normalization, data centering and scaling, data transformation. We discuss the advantages and limitations of each method. The choice for a suitable preprocessing method is determined by the biological hypothesis, the characteristics of the data set, and the selected statistical data analysis method. We then provide the perspective of their applications in the microbiome and metabolome research.

Multivariate time series (MTS) are captured in a great variety of real-world applications. However, analysing and modeling the data for classification and forecasting purposes can become very challenging if values are missing in the data set. The need for imputation methods, to fill the gaps in MTS, is well known. Thus, a great variaty of algorithms for solving this task has been proposed in the literature. However, the research community is constantly working on the development of advanced algorithms, that fulfill the special requirements of multidimensional temporal data, since most of the existing imputation methods treat MTS as ordinary structured data and fail to model the temporal relationships within and between sequences of observations. The main emphasis of MTS imputation research is currently put on deep learning (DL) models, especially those making use of generative adversarial networks (GANs). In our survey, we present our categorization of imputation algorithms, especially of GAN-models. We included eighteen different GAN-models designed for the MTS imputation task, which we introduce in detail. We provide a comparison of the models including their performance regarding MTS imputation, based on our findings in the literature. The following points can be considered the most important findings from our survey: The research on GAN-based imputation models for MTS has gained momentum in the last years across different domains, therewhile showing the effectiveness of these methods. The latest trend in the research area is the incorporation of attention mechanisms into the algorithms. Nevertheless, there are open research challenges, e.g. the transferability of models across data sets from different domains.