Search

Time series data are pivotal in diverse fields such as finance, meteorology, and health data analysis. Accurate analysis of these data is crucial for identifying temporal trends and making informed decisions. However, frequent occurrences of missing values, often due to device failures or data collection errors, pose a significant challenge. In this work, we introduce TS-Pothole, a method for imputing missing values in univariate time series. This method leverages cyclic pattern analysis and a recursive strategy to handle univariate datasets in which missing values are distributed both continuously and randomly. We evaluate TS-Pothole on four real-world datasets representing different configurations of missing values, and assess its performance in terms of accuracy and execution speed. In particular, we compare our approach with state-of-the-art methods, such as GANs and autoencoders. Our experiments show that TS-Pothole outperforms such methods by providing more accurate (up to 1.5 times) and faster (up to 2 times) imputations, even as the proportion of missing data increases, representing the best alternative in handling univariate time series with missing values. [ABSTRACT FROM AUTHOR]

In many application settings, data have missing entries, which makes subsequent analyses challenging. An abundant literature addresses missing values in an inferential framework, aiming at estimating parameters and their variance from incomplete tables. Here, we consider supervised-learning settings: predicting a target when missing values appear in both training and test data. We first rewrite classic missing values results for this setting. We then show the consistency of two approaches, test-time multiple imputation and single imputation in prediction. A striking result is that the widely-used method of imputing with a constant prior to learning is consistent when missing values are not informative. This contrasts with inferential settings where mean imputation is frowned upon as it distorts the distribution of the data. The consistency of such a popular simple approach is important in practice. Finally, to contrast procedures based on imputation prior to learning with procedures that optimize the missing-value handling for prediction, we consider decision trees. Indeed, decision trees are among the few methods that can tackle empirical risk minimization with missing values, due to their ability to handle the half-discrete nature of incomplete variables. After comparing empirically different missing values strategies in trees, we recommend using the "missing incorporated in attribute" method as it can handle both non-informative and informative missing values. [ABSTRACT FROM AUTHOR]

Limited research has assessed the accuracy of imputation methods in aerobiological datasets. We conducted a simulation study to evaluate, for the first time, the effectiveness of Gappy Singular Value Decomposition (GSVD), a data-driven approach, comparing it with the moving mean interpolation, a statistical approach. Utilizing complete pollen data from two monitoring stations in northeastern Italy for 2022, we randomly generated missing data considering the combination of various proportions (5%, 10%, 25%) and gap lengths (3, 5, 7, 10 days). We imputed 4800 time series using the GSVD algorithm, specifically implemented for this study, and the moving mean algorithm of the "AeRobiology" R package. We assessed imputation accuracy by calculating the Root Mean Square Error and employed multiple linear regression models to identify factors independently affecting the error (e.g. pollen variability, simulation settings). The results showed that the GSVD was as good as the well-established moving mean method and demonstrated its strong generalization capabilities across different data types. However, the imputation error was primarily influenced by pollen characteristics and location, regardless of the imputation method used. High variability in pollen concentrations and the distribution of missing data negatively affected imputation accuracy. In conclusion, we introduced and tested a novel imputation method, demonstrating comparable performance to the statistical approach in aerobiological data reconstruction. These findings contribute to advancing aerobiological data analysis, highlighting the need for improving imputation methods. [ABSTRACT FROM AUTHOR]

Systematically missing data in distributed data networks presents practical and methodological challenges. Failure to handle it appropriately can bias statistical inference. Multiple imputations can be used to address systematic missingness. However, when data from different study sites cannot be pooled into a unified file, conventional imputation approaches become unavailable due to the absence of a basis for imputation. To address such challenges, we introduce an imputation method based on conditional quantiles – conditional quantile imputation (CQI) – which involves four steps: (i) estimating 99 quantiles for the systematically missing variable in studies with observed data; (ii) deriving a weighted average of regression coefficients across studies and transmitting it to sites with systematically missing data; (iii) imputing the systematically missing values based on observed data and the set of regression coefficients from step ii; and (iv) combining estimates of the substantive outcome model across imputations using Rubin's rules. We evaluate CQI in different simulation scenarios and illustrate it with an applied data example. We conclude that CQI can be a suitable approach for the imputation of systematically missing data when data from multiple studies cannot be pooled. [ABSTRACT FROM AUTHOR]

In various fields of information examination, for example, AI, profoundly getting the hang of missing information is a typical issue. Missing qualities should be tended to since they can adversely affect the exactness and adequacy of prescient models. This research investigates how data discretization affects deep learning methods for filling the missing values in datasets with continuous features. They provide a unique method for imputing missing values using deep neural networks (DNNs) called extravagant expectation maximization-deep neural network (EEM-DNN). This approach discretizes continuous features into separate intervals initially. This is justified by treating the issue of missing value imputation as a classification work, with the missing values being considered a distinct class. A DNN, designed explicitly for imputation, is then trained using the discretized data. The expectation maximization concepts are incorporated into the network architecture, and as a result, the network iteratively improves its imputation predictions. They run comprehensive experiments on several datasets from different fields to gauge the efficacy of the suggested strategy. The effectiveness of EEM-DNN is compared to that of other imputation approaches, such as traditional imputation techniques and deep learning methods without data discretization. Our findings show that data discretization significantly enhances imputation accuracy. In terms of imputation accuracy and prediction performance on downstream tasks, the EEM-DNN method regularly performs better than alternative methods. It also examines if various discretization techniques affect the overall imputation process. They find that the trade-off between bias and variance in imputed data depends on the discretization method selected. This highlights the significance of choosing a suitable discretization approach depending on the unique properties of the dataset. [ABSTRACT FROM AUTHOR]

Motivation Integrating multiple omics datasets can significantly advance our understanding of disease mechanisms, physiology, and treatment responses. However, a major challenge in multi-omics studies is the disparity in sample sizes across different datasets, which can introduce bias and reduce statistical power. To address this issue, we propose a novel framework, OmicsNMF, designed to impute missing omics data and enhance disease phenotype prediction. OmicsNMF integrates Generative Adversarial Networks (GANs) with Non-Negative Matrix Factorization (NMF). NMF is a well-established method for uncovering underlying patterns in omics data, while GANs enhance the imputation process by generating realistic data samples. This synergy aims to more effectively address sample size disparity, thereby improving data integration and prediction accuracy. Results For evaluation, we focused on predicting breast cancer subtypes using the imputed data generated by our proposed framework, OmicsNMF. Our results indicate that OmicsNMF consistently outperforms baseline methods. We further assessed the quality of the imputed data through survival analysis, revealing that the imputed omics profiles provide significant prognostic power for both overall survival and disease-free status. Overall, OmicsNMF effectively leverages GANs and NMF to impute missing samples while preserving key biological features. This approach shows potential for advancing precision oncology by improving data integration and analysis. Availability and implementation Source code is available at: https://github.com/compbiolabucf/OmicsNMF. [ABSTRACT FROM AUTHOR]

The integration of multiple survey, administrative, and third-party data offers the opportunity to innovate and improve survey estimation via statistical modeling. With decreasing response rates and increasing interest for more timely and geographically detailed estimates, imputation methodology that combines multiple data sources to adjust for low unit response and allow for more detailed publication levels, including geographic estimates, is both timely and necessary. Motivated by the Advance Monthly Retail Trade Survey (MARTS) and Monthly Retail Trade Survey (MRTS), we propose Bayesian hierarchical multiple imputation-dependent data models with the goals of automating imputation for the MARTS by using historic MRTS data and providing geographically granular (state-level) estimates for the MRTS via mass imputation using third-party data and spatial dependence. As a natural byproduct of this approach, measures of uncertainty are provided. This article illustrates the advantages of applying established Bayesian hierarchical modeling techniques with multiple source data to address practical problems in official statistics and is, therefore, of independent interest. The motivating empirical studies are unified by their hierarchical modeling framework, which ultimately results in a more principled approach for estimation for the MARTS and a more geographically granular data product for the MRTS. [ABSTRACT FROM AUTHOR]

Imputation procedures are frequently used to treat nonresponse. With random hot deck imputation, missing values are replaced by valid observed values from other units in the same dataset. The recently developed balanced nearest neighbor imputation method, implemented in the SwissCheese R package, generates random hot deck imputation under certain balancing constraints to decrease the variance of the total estimator, in the presence of multivariate nonresponse. The method relies on a notion of neighborhood between units, utilizing a distance measure that becomes difficult to define in high dimensions. In contrast to hot deck imputation methods, many imputation procedures obtain replacement values from prediction models fit from observed data. The missForest method, which uses random forests as prediction models, is an example of this approach. In this article, we propose a new approach that uses the two methods in a complementary manner. We refine the distance measure in the SwissCheese method using missForest predictions. Through a simulation study on empirical data from the Swiss Survey on Income and Living Conditions, we demonstrate reductions in Monte Carlo variance, bias, and mean squared error of the totals obtained by our proposed imputed estimator compared to those obtained using SwissCheese alone. [ABSTRACT FROM AUTHOR]

Multiple imputation is considered as a superior data analysis practice to deal with missing values. This is, in particular, also true in the important case of categorical variables. However, imputation techniques could be a daunting task for the less technical data practitioner. In the case of categorical variables, imputation can be avoided by recoding the indicator matrix of the data set before analysis. New category levels are created for each variable to which missing values are assigned. The recoded indicator matrix enables the separation of missing and observed responses. Subset multiple correspondence analysis (sMCA) allows a focused analysis on either of the missing or non-missing subsets of the recoded indicator matrix while preserving the original variation of the data. The main purpose of this article is to evaluate the sMCA approach using simulated data which incorporate various parameters. In addition, the visualization of multivariate categorical data containing missing values that follows from the sMCA approach is presented. This article could aid as a guide when deciding whether the sMCA approach is suitable for the analysis of incomplete multivariate categorical data. To ensure effortless application of the presented methodology, a complete R script with commented functions is available as . [ABSTRACT FROM AUTHOR]

Observational data with massive sample sizes are often distributed on many local machines. From an experimental design perspective, investigators often desire to identify the effect of new treatments (even ML algorithms) on many blocks of experimental data. With time requirements or budget constraints, assigning all treatments to each block is not always feasible. This creates incomplete responses with respect to a randomized complete block design (RCBD). These incomplete responses are missing by design. However, whether they can be estimated with missing imputation methods is not well understood. Thus, it is challenging to correctly identify the treatment effects with missing data. To this end, this paper provides a method for imputation and analysis of the responses with missing data. The proposed method consists of three steps: Reconstruction, Imputation, and 'Complete'-data Analysis (RICA). The incomplete responses are imputed with the expectation-maximization (EM) algorithm. The RCBD model is then fitted by the resulting dataset. The identifiability result suggests that the missing may be nonignorable for each block, but the whole data of an incomplete design are missing by design when the design is balanced. Theoretical results on relative efficiency also inform us when the missingness should be imputed for incomplete designs with the role of balanced variance. Applications on real-world data verify the efficacy of this method. [ABSTRACT FROM AUTHOR]

Trilinearity is a property of some chemical data that leads to unique decompositions when curve resolution or multiway decomposition methods are used. Curve resolution algorithms, such as Multivariate Curve Resolution–Alternating Least Squares (MCR‐ALS), can provide trilinear models by implementing the trilinearity condition as a constraint. However, some trilinear analytical measurements, such as excitation–emission matrix (EEM) measurements, usually exhibit systematic patterns of missing data due to the nature of the technique, which imply a challenge to the classical implementation of the trilinearity constraint. In this instance, extrapolation or imputation methodologies may not provide optimal results. Recently, a novel algorithmic strategy to constrain trilinearity in MCR‐ALS in the presence of missing data was developed. This strategy relies on the sequential imposition of a classical trilinearity restriction on different submatrices of the original investigated dataset, but, although effective, was found to be particularly slow and requires a proper submatrix selection criterion. In this paper, a much simpler implementation of the trilinearity constraint in MCR‐ALS capable of handling systematic patterns of missing data and based on the principles of the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm is proposed. This novel approach preserves the trilinearity of the retrieved component profiles without requiring data imputation or subset selection steps and, as with all other constraints designed for MCR‐ALS, offers the flexibility to be applied component‐wise or data block‐wise, providing hybrid bilinear/trilinear models. Furthermore, it can be easily extended to cope with any trilinear or higher‐order dataset with whatever pattern of missing values. [ABSTRACT FROM AUTHOR]

Reliable traffic flow data is not only crucial for traffic management and planning, but also the foundation for many intelligent applications. However, the phenomenon of missing traffic flow data often occurs, so we propose an imputation model for missing traffic flow data to overcome the randomness and instability bands of traffic flow. First, k-means clustering is used to classify road segments with traffic flow belonging to the same pattern into a group to utilize the spatial characteristics of roads fully. Then, the LSTM networks optimized with an attention mechanism are used as the base learner to extract the temporal dependence of the traffic flow. Finally, the AdaBoost algorithm is used to integrate all the LSTM-attention networks into a reinforced learner to impute the missing data. To validate the effectiveness of the proposed model, we use the PeMS dataset for validation, we impute the data with missing data rate from 10 to 60% under three missing modes, and we use multiple baseline models for comparison, which confirms that our proposed model improves the stability and accuracy of imputing the missing data of the traffic flow with different scenarios. [ABSTRACT FROM AUTHOR]

The proportion of explained variance is an important statistic in multiple regression for determining how well the outcome variable is predicted by the predictors. Earlier research on 20 different estimators for the proportion of explained variance, including the exact Olkin–Pratt estimator and the Ezekiel estimator, showed that the exact Olkin–Pratt estimator produced unbiased estimates, and was recommended as a default estimator. In the current study, the same 20 estimators were studied in incomplete data, with missing data being treated using multiple imputation. In earlier research on the proportion of explained variance in multiply imputed data sets, an estimator called R̂SP2 was shown to be the preferred pooled estimator for regular R2. For each of the 20 estimators in the current study, two pooled estimators were proposed: one where the estimator was the average across imputed data sets, and one where R̂SP2 was used as input for the calculation of the specific estimator. Simulations showed that estimates based on R̂SP2 performed best regarding bias and accuracy, and that the Ezekiel estimator was generally the least biased. However, none of the estimators were unbiased at all times, including the exact Olkin–Pratt estimator based on R̂SP2. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) is a widely embraced approach for delivering personalized educational assessments, tailoring each test to the real‐time performance of individual examinees. Despite its potential advantages, CAT�s application in small‐scale assessments has been limited due to the complexities associated with calibrating the item bank using sparse response data and small sample sizes. This study addresses these challenges by developing a two‐step item bank calibration strategy that leverages the 1‐bit matrix completion method in conjunction with two distinct incomplete pretesting designs. We introduce two novel 1‐bit matrix completion‐based imputation methods specifically designed to tackle the issues associated with item calibration in the presence of sparse response data and limited sample sizes. To demonstrate the effectiveness of these approaches, we conduct a comparative assessment against several established item parameter estimation methods capable of handling missing data. This evaluation is carried out through two sets of simulation studies, each featuring different pretesting designs, item bank structures, and sample sizes. Furthermore, we illustrate the practical application of the methods investigated, using empirical data collected from small‐scale assessments. [ABSTRACT FROM AUTHOR]

Rubin’s Rules are commonly used to pool the results of statistical analyses across imputed samples when using multiple imputation. Rubin’s Rules cannot be used when the result of an analysis in an imputed dataset is not a statistic and its associated standard error, but a test statistic (e.g. Student’s t-test). While complex methods have been proposed for pooling test statistics across imputed samples, these methods have not been implemented in many popular statistical software packages. The median p-value method has been proposed for pooling test statistics. The statistical significance level of the pooled test statistic is the median of the associated p-values across the imputed samples. We evaluated the performance of this method with nine statistical tests: Student’s t-test, Wilcoxon Rank Sum test, Analysis of Variance, Kruskal-Wallis test, the test of significance for Pearson’s and Spearman’s correlation coefficient, the Chi-squared test, the test of significance for a regression coefficient from a linear regression and from a logistic regression. For each test, the empirical type I error rate was higher than the advertised rate. The magnitude of inflation increased as the prevalence of missing data increased. The median p-value method should not be used to assess statistical significance across imputed datasets. [ABSTRACT FROM AUTHOR]

Good data quality is vital for personalising plans in rehabilitation. Machine learning (ML) improves prognostics but integrating it with Multiple Imputation (MImp) for dealing missingness is an unexplored field. This work aims to provide post-stroke ambulation prognosis, integrating MImp with ML, and identify the prognostic influential factors. Stroke survivors in intensive rehabilitation were enrolled. Data on demographics, events, clinical, physiotherapy, and psycho-social assessment were collected. An independent ambulation at discharge, using the Functional Ambulation Category scale, was the outcome. After handling missingness using MImp, ML models were optimised, cross-validated, and tested. Interpretability techniques analysed predictor contributions. Pre-MImp, the dataset included 54.1% women, 79.2% ischaemic patients, median age 80.0 (interquartile range: 15.0). Post-MImp, 368 non-ambulatory patients on 10 imputed datasets were used for training, 80 for testing. The random forest (the validation best-performing algorithm) obtained 75.5% aggregated balanced accuracy on the test set. The main predictors included modified Barthel index, Fugl-Meyer assessment/motricity index, short physical performance battery, age, Charlson comorbidity index/cumulative illness rating scale, and trunk control test. This is among the first studies applying ML, together with MImp, to predict ambulation recovery in post-stroke rehabilitation. This pipeline reliably exploits the potential of incomplete datasets for healthcare prognosis, identifying relevant predictors. [ABSTRACT FROM AUTHOR]

AbstractPattern-mixture models have received increasing attention as they are commonly used to assess treatment effects in primary or sensitivity analyses for clinical trials with nonignorable missing data. Pattern-mixture models have traditionally been implemented using multiple imputation, where the variance estimation may be a challenge because Rubin’s approach of combining between- and within-imputation variance may not provide consistent variance estimation while bootstrap methods may be time-consuming. Direct likelihood-based approaches have been proposed in the literature and implemented for some pattern-mixture models, but the assumptions are sometimes restrictive and the theoretical framework is fragile. In this article, we propose an analytical framework for an efficient direct likelihood estimation method for commonly used pattern-mixture models corresponding to return-to-baseline, jump-to-reference, placebo washout, and retrieved dropout imputations. A parsimonious tipping point analysis is also discussed and implemented. Results from simulation studies demonstrate that the proposed methods provide consistent estimators. We further illustrate the utility of the proposed methods using data from a clinical trial evaluating a treatment for type 2 diabetes. [ABSTRACT FROM AUTHOR]

This paper studies the autoregressive integrated moving average (ARIMA) state space model combined with Kalman smoothing to impute missing values in a univariate time series before detecting change points. We estimate a scale-dependent time-average variance constant that depends on the length of the data section and is robust to mean shifts under serial dependence. The consistency of the proposed estimator is shown under the assumption allowing heavy tailedness. Integrating the proposed estimator with the moving sum and wild binary segmentation procedures to determine the number and locations of change points is discussed. Furthermore, the performance of the proposed methods is evaluated through extensive simulation studies and applied to the Beijing multi-site air quality dataset to impute missing values and detect mean changes in the data. [ABSTRACT FROM AUTHOR]

In many engineering applications, missing data during system identification can hinder the performance of the identified model. In this paper, a novel two‐stage nonparametric framework is proposed for missing data imputation, uncertainty quantification, and its integration in system identification with reduced computational complexity. The framework does not require functional forms for both the imputation model and the identified mathematical model. Moreover, through the construction of a single imputation model, analytical expressions of predictive distributions can be given for missing entries across all missingness patterns. Furthermore, analytical expressions of the expectation and variance of distribution are provided to impute missing values and quantify uncertainty, respectively. This uncertainty is incorporated into a single mathematical model by mitigating the influence of samples with imputations during training and testing. The framework is applied to three applications, including a simulated example and two real applications on structural health monitoring and seismic attenuation modeling. Results reveal a minimum reduction of 21% in root mean squared error values, compared to those achieved by directly removing incomplete samples. [ABSTRACT FROM AUTHOR]

Missing data occurs frequently in sample surveys. Numerous imputation techniques have been proposed to tackle the problem of missing data, and, in fact, limited works are available in the literature to deal with the issue of missingness while the data are ridden with measurement errors (ME). In addition, no work has been done on the robustness to handle the issue of missing data when it is contaminated with correlated measurement errors (CME). This article proposes some robust imputation methods (RIM) to impute the missing data in the presence of CME. The mean square error (MSE) of the proposed RIM is derived from the first-order approximation and examined with the MSE of the conventional imputation methods. The results are theoretically established and explained using a vast simulation study. Two applications of real data sets are presented to illustrate the efficiency and superiority of the suggested estimators relative to some estimators considered in this study. [ABSTRACT FROM AUTHOR]

Missing values are a common problem in data science and machine learning. Removing instances with missing values is a straightforward workaround, but this can significantly hinder subsequent data analysis, particularly when features outnumber instances. There are a variety of methodologies proposed in the literature for imputing missing values. Denoising Autoencoders, for example, have been leveraged efficiently for imputation. However, neural network approaches have been relatively less effective on smaller datasets. In this work, we propose Autoreplicative Random Forests (ARF) as a multi-output learning approach, which we introduce in the context of a framework that may impute via either an iterative or procedural process. Experiments on several low- and high-dimensional datasets show that ARF is computationally efficient and exhibits better imputation performance than its competitors, including neural network approaches. In order to provide statistical analysis and mathematical background to the proposed missing value imputation framework, we also propose probabilistic ARFs, where the confidence values are provided over different imputation hypotheses, therefore maximizing the utility of such a framework in a machine-learning pipeline targeting predictive performance. [ABSTRACT FROM AUTHOR]

Missing data problems are common and difficult to handle in data analysis. Ad hoc methods, such as simply removing cases with missing values, can lead to invalid analysis results. In this article, we consider a functional linear regression model with partially observed covariate and missing values in the response. We use a reconstruction operator that aims at recovering the missing parts of the explanatory curves, then we are interested in regression imputation method of missing data on the response variable, using functional principal component regression to estimate the functional coefficient of the model. We study the asymptotic behavior of the prediction error when missing values in an original dataset are imputed by multiple sets of plausible values. The method behavior is also evaluated in practice. [ABSTRACT FROM AUTHOR]

AbstractMultiple imputation (MI) is effectively used to deal with missing data when the missing mechanism is missing at random. However, MI may not be effective when the missing mechanism is not missing at random (NMAR). In such cases, additional information is required to obtain an appropriate imputation. Pham et al. (2019) proposed the calibrated-δ adjustment method, which is a multiple imputation method using population information. It provides appropriate imputation in two NMAR settings. However, the calibrated-δ adjustment method has two problems. First, it can be used only when one variable has missing values. Second, the theoretical properties of the variance estimator have not been provided. This article proposes a multiple imputation method using population information that can be applied when several variables have missing values. The proposed method is proven to include the calibrated-δ adjustment method. It is shown that the proposed method provides a consistent estimator for the parameter of the imputation model in an NMAR situation. The asymptotic variance of the estimator obtained by the proposed method and its estimator are also given. [ABSTRACT FROM AUTHOR]

In randomized controlled trials, adjusting for baseline covariates is commonly used to improve the precision of treatment effect estimation. However, covariates often have missing values. Recently, Zhao and Ding studied two simple strategies, the single imputation method and missingness-indicator method (MIM), to handle missing covariates and showed that both methods can provide an efficiency gain compared to not adjusting for covariates. To better understand and compare these two strategies, we propose and investigate a novel theoretical imputation framework termed cross-world imputation (CWI). This framework includes both single imputation and MIM as special cases, facilitating the comparison of their efficiency. Through the lens of CWI, we show that MIM implicitly searches for the optimal CWI values and thus achieves optimal efficiency. We also derive conditions under which the single imputation method, by searching for the optimal single imputation values, can achieve the same efficiency as the MIM. We illustrate our findings through simulation studies and a real data analysis based on the Childhood Adenotonsillectomy Trial. We conclude by discussing the practical implications of our findings. [ABSTRACT FROM AUTHOR]

Imputation of missing values is a strategy for handling non-responses in surveys or data loss in measurement processes, which may be more effective than ignoring the losses and omitting them. The characteristics of variables presenting missing values must be considered when choosing the imputation method to be used; in particular when the variable is a count the literature dealing with this issue is scarce. If the variable has an excess of zeros it is necessary to consider models including parameters for handling zero-inflation. Likewise, if problems of over- or under-dispersion are observed, generalizations of the Poisson, such as the Hermite or Conway–Maxwell Poisson distributions are recommended for carrying out imputation. The aim of this study was to assess the performance of various regression models in the imputation of a discrete variable based on Poisson generalizations, in comparison with classical counting models, through a comprehensive simulation study considering a variety of scenarios and a real data example. To do so we compared the results of estimations using only complete data, and using imputations based on the most common count models. The COMPoisson distribution provides in general better results in any dispersion scenario, especially when the amount of missing information is large. [ABSTRACT FROM AUTHOR]

In this article, I describe the miesize command for the calculation of effect sizes in imputed data. There may be situations where an effect size needs to be estimated for an intervention, an exposure, or a group membership variable but data on the independent or dependent variable are missing. Such missing data are commonly dealt with by multiply imputing plausible values. However, in this circumstance, the estimated effect size and associated standard errors will need to be pooled and estimated from the imputed dataset. The miesize command automates this process and calculates effect sizes for a binary variable from multiply imputed data in wide format. The estimates and standard errors (used to calculate the confidence intervals) are recombined using Rubin's (1987, Multiple Imputation for Nonresponse in Surveys [Wiley]) rules. These rules are applied such that the average point estimate for the effect size is calculated from the imputed datasets. The pooled standard error, and hence confidence intervals, is calculated to account for both the variance between the imputed datasets and the variance within them. Pooled effect sizes and confidence intervals for Cohen's (1988, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. [Lawrence Erlbaum]) d, Hedges's (1981, Journal of Educational Statistics 6: 107-128) g, and Glass's (Smith and Glass, 1977 , American Psychologist 32: 752-760) delta are provided by miesize. [ABSTRACT FROM AUTHOR]

In clinical trials of longitudinal continuous outcomes, reference based imputation (RBI) has commonly been applied to handle missing outcome data in settings where the estimand incorporates the effects of intercurrent events, e.g. treatment discontinuation. RBI was originally developed in the multiple imputation framework, however recently conditional mean imputation (CMI) combined with the jackknife estimator of the standard error was proposed as a way to obtain deterministic treatment effect estimates and correct frequentist inference. For both multiple and CMI, a mixed model for repeated measures (MMRM) is often used for the imputation model, but this can be computationally intensive to fit to multiple data sets (e.g. the jackknife samples) and lead to convergence issues with complex MMRM models with many parameters. Therefore, a step-wise approach based on sequential linear regression (SLR) of the outcomes at each visit was developed for the imputation model in the multiple imputation framework, but similar developments in the CMI framework are lacking. In this article, we fill this gap in the literature by proposing a SLR approach to implement RBI in the CMI framework, and justify its validity using theoretical results and simulations. We also illustrate our proposal on a real data application. [ABSTRACT FROM AUTHOR]

Increasing advancements in building digitization, smart sensing, and metering technologies have allowed large amounts of timeseries data to be collected for monitoring, analyzing, and controlling building systems. However, due to sensor or communication failures, the data collected are often incomplete and poor in quality. Data imputation approaches to replace the missing values, specifically based on either statistical or predictive models have been widely adopted for multivariate datasets in other domains. It is hence of interest to find an effective way to impute timeseries data collected from a building system. In this paper, we evaluate multiple data imputation approaches using data collected from a medium sized building situated in Stockholm, Sweden and a small commercial building from the ASHRAE RP-1312 research project. Sensors with widely varying characteristics from the case study buildings were selected to evaluate the imputation methods. The imputation accuracy and the impact of each chosen imputation method on information entropy, short-term building forecasting model performance, and fault detection strategy were evaluated. Results demonstrate that incorporating time-lagged cross correlations within a k -nearest neighbor ( k NN) model provide the most accurate imputations without affecting the quality of subsequent data analysis. [ABSTRACT FROM AUTHOR]

The purpose of this study is to examine the effect of missing data imputation methods, namely regression imputation (RI), multiple imputation (MI) and k-nearest neighbor (kNN) on differential item functioning (DIF). In this regard, the datasets used in the research were created by deleting some of the data via the missing completely at random mechanism from the complete datasets obtained from 600 students in Türkiye, the United Kingdom, the USA, New Zealand and Australia, who answered booklets 14 and 15 from the PISA 2018 science literacy test. Data imputation was applied to the datasets through missing data using RI, MI and kNN methods and DIF analysis was performed on all datasets in terms of language and gender variables via Lord's χ2 method, Raju's area measurement method and item response theory likelihood ratio method. DIF results from the complete datasets were taken as a reference and they were compared with the results from other datasets. As a result of the research, values close to 10% of accurate imputation were achieved in the RI method depending on language and gen-der variables. In MI and kNN methods, results closest to the complete datasets were obtained at a rate of 5% depending on the language variable. In the MI method, inaccurate results were obtained in all proportions in terms of the gender variable. For the gender variable, the kNN method gave accurate results at rates of 5% and 10%. [ABSTRACT FROM AUTHOR]

Data imputation strategies are necessary to address the prevalent difficulty of missing values in data observation and recording operations. This work utilizes diverse imputation methods to forecast and complete absent values inside a financial time-series dataset, specifically the daily prices of gold. The predictive accuracy of imputed data is assessed in comparison to the original entire dataset to ensure its robustness. The imputation methods are validated using actual closing price data obtained from a daily gold price website. The examined approaches include mean imputation, k-nearest neighbor (KNN), hot deck, random forest, support vector machine (SVM), and spline imputation. Their performance is evaluated based on several metrics, including mean error (ME), mean absolute error (MAE), root mean square error (RMSE), mean percentage error (MPE), and mean absolute percentage error (MAPE). The results indicate that the KNN approach consistently performs better than other methods in terms of all accuracy measures. Nevertheless, the precision of all techniques decreases as the proportion of missing data rises. Therefore, the KNN approach is suggested because to its exceptional performance and dependability in imputation tasks. [ABSTRACT FROM AUTHOR]

Missing data imputation is a key topic in healthcare that covers the issues and strategies involved in dealing with partial data in medical records, clinical trials, and health surveys. Data in healthcare might be missing for a variety of reasons, including non-response in surveys, data entry problems, or unrecorded information during therapeutic appointments. This paper introduces a novel approach to impute missing data utilizing a hybrid model that integrates denoising autoencoders with generative adversarial networks (GANs). We begin by highlighting the prevalence of missing data in health care datasets and the potential impact on analytical outcomes. The proposed methodology leverages the denoising autoencoder's ability to reconstruct data from noisy inputs, coupled with the GAN's proficiency in generating synthetic data that is indistinguishable from real data. By combining these two neural network architectures, our model demonstrates an enhanced capability to predict and fill in missing data points effectively. To validate our approach, we conducted experiments on several large-scale health care datasets with varying degrees of artificially introduced missingness. The performance of our model was benchmarked against traditional imputation methods such as mean imputation and k-nearest neighbors, as well as against standalone denoising autoencoders and GANs. Our results indicate a significant improvement in imputation accuracy, as measured by root mean square error (RMSE) and mean absolute error (MAE), confirming the efficacy of the hybrid model in handling missing data in a robust manner. [ABSTRACT FROM AUTHOR]

Increasing advancements in building digitization, smart sensing, and metering technologies have allowed large amounts of timeseries data to be collected for monitoring, analyzing, and controlling building systems. However, due to sensor or communication failures, the data collected are often incomplete and poor in quality. Data imputation approaches to replace the missing values, specifically based on either statistical or predictive models have been widely adopted for multivariate datasets in other domains. It is hence of interest to find an effective way to impute timeseries data collected from a building system. In this paper, we evaluate multiple data imputation approaches using data collected from a medium sized building situated in Stockholm, Sweden and a small commercial building from the ASHRAE RP-1312 research project. Sensors with widely varying characteristics from the case study buildings were selected to evaluate the imputation methods. The imputation accuracy and the impact of each chosen imputation method on information entropy, short-term building forecasting model performance, and fault detection strategy were evaluated. Results demonstrate that incorporating time-lagged cross correlations within a k -nearest neighbor ( k NN) model provide the most accurate imputations without affecting the quality of subsequent data analysis. [ABSTRACT FROM AUTHOR]

Distributed machine learning has received much attention for more than two decades. Yet, it is still a challenge to achieve acceptable performance in practical scenarios when some features of data samples are missing. Although some imputation methods have been proposed for handling missing data, their performance deteriorates significantly when data are heterogeneously distributed over different nodes in the network. Considering this, in this article, we first propose a general Gaussian mixture model (GMM) consisting of both public and personalized components for modeling the homogeneous and the heterogeneous parts of data distribution, respectively. Then, we develop a distributed personalized expectation–maximization method based on knowledge transfer (KT-dpEM) to estimate the parameters of the proposed general GMM. After that, based on the estimated general GMM, missing data are imputed using the posterior conditional mean. Experimental results show that the proposed KT-dpEM algorithm has better imputation accuracy, higher robustness against different missing probabilities and better classification performance in the downstream classification tasks, compared with state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Ranked set sampling (RSS) is known to increase the efficiency of the estimators while comparing it with simple random sampling. The problem of missingness creates a gap in the information that needs to be addressed before proceeding for estimation. Negligible amount of work has been carried out to deal with missingness utilizing RSS. This paper proposes some logarithmic type methods of imputation for the estimation of population mean under RSS using auxiliary information. The properties of the suggested imputation procedures are examined. A simulation study is accomplished to show that the proposed imputation procedures exhibit better results in comparison to some of the existing imputation procedures. Few real applications of the proposed imputation procedures is also provided to generalize the simulation study. [ABSTRACT FROM AUTHOR]

In clinical datasets, missing data often occur due to various reasons including non-response, data corruption, and errors in data collection or processing. Such missing values can lead to biased statistical analyses, reduced statistical power, and potentially misleading findings, making effective imputation critical. Traditional imputation methods, such as Zero Imputation, Mean Imputation, and k-Nearest Neighbors (KNN) Imputation, attempt to address these gaps. However, these methods often fall short of accurately capturing the underlying data complexity, leading to oversimplified assumptions and errors in prediction. This study introduces a novel Imputation model employing transformer-based architectures to address these challenges. Notably, the model distinguishes between complete EEG signal amplitude data and incomplete data in two datasets: PhysioNet and CHB-MIT. By training exclusively on complete amplitude data, the TabTransformer accurately learns and predicts missing values, capturing intricate patterns and relationships inherent in EEG amplitude data. Evaluation using various error metrics and R2 score demonstrates significant enhancements over traditional methods such as Zero, Mean, and KNN imputation. The Proposed Model achieves impressive R2 scores of 0.993 for PhysioNet and 0.97 for CHB-MIT, highlighting its efficacy in handling complex clinical data patterns and improving dataset integrity. This underscores the transformative potential of transformer models in advancing the utility and reliability of clinical datasets. [ABSTRACT FROM AUTHOR]

Missing data are a pervasive problem in data analysis. Three common methods for addressing the problem are (a) complete-case analysis, where only units that are complete on the variables in an analysis are included; (b) weighting, where the complete cases are weighted by the inverse of an estimate of the probability of being complete; and (c) multiple imputation (MI), where missing values of the variables in the analysis are imputed as draws from their predictive distribution under an implicit or explicit statistical model, the imputation process is repeated to create multiple filled-in data sets, and analysis is carried out using simple MI combining rules. This article provides a non-technical discussion of the strengths and weakness of these approaches, and when each of the methods might be adopted over the others. The methods are illustrated on data from the Youth Cohort (Time) Series (YCS) for England, Wales and Scotland, 1984–2002. [ABSTRACT FROM AUTHOR]

Data-driven methods are increasingly popular for building energy performance assessment. For these to be useful, it is required that good quality data is created through the filtering of outliers and imputation of missing information. In the field of energy use in buildings, there is a clear sensitivity of heat load to outdoor climate, which needs to be considered when identifying outliers and developing imputation methods. We propose to use a well-known changepoint model to define the sensitivity of the data to climate, further segmented by time of the week. Then we use the residuals of the model to identify outliers, where those observation with residuals substantially out of the normality expectations are identified as outliers. Then missing data is repaired by means of linear imputation techniques, considering the patterns for same times of the week in the dataset. As a result of the full process, we were able to identify 5% of outliers, which resulted in the improvement of model metrics in the range of 20% mean absolute error (MAE) and slightly better R2 values. [ABSTRACT FROM AUTHOR]

Background: Electronic Health Records (EHR) are widely used to develop clinical prediction models (CPMs). However, one of the challenges is that there is often a degree of informative missing data. For example, laboratory measures are typically taken when a clinician is concerned that there is a need. When data are the so-called Not Missing at Random (NMAR), analytic strategies based on other missingness mechanisms are inappropriate. In this work, we seek to compare the impact of different strategies for handling missing data on CPMs performance. Methods: We considered a predictive model for rapid inpatient deterioration as an exemplar implementation. This model incorporated twelve laboratory measures with varying levels of missingness. Five labs had missingness rate levels around 50%, and the other seven had missingness levels around 90%. We included them based on the belief that their missingness status can be highly informational for the prediction. In our study, we explicitly compared the various missing data strategies: mean imputation, normal-value imputation, conditional imputation, categorical encoding, and missingness embeddings. Some of these were also combined with the last observation carried forward (LOCF). We implemented logistic LASSO regression, multilayer perceptron (MLP), and long short-term memory (LSTM) models as the downstream classifiers. We compared the AUROC of testing data and used bootstrapping to construct 95% confidence intervals. Results: We had 105,198 inpatient encounters, with 4.7% having experienced the deterioration outcome of interest. LSTM models generally outperformed other cross-sectional models, where embedding approaches and categorical encoding yielded the best results. For the cross-sectional models, normal-value imputation with LOCF generated the best results. Conclusion: Strategies that accounted for the possibility of NMAR missing data yielded better model performance than those did not. The embedding method had an advantage as it did not require prior clinical knowledge. Using LOCF could enhance the performance of cross-sectional models but have countereffects in LSTM models. [ABSTRACT FROM AUTHOR]

Abstract.In this research article, we present novel imputation methods designed to address missing data challenges in sample surveys. We then introduce innovative estimation procedures for calculating population means based on these methods. Our study thoroughly examines the properties of these new estimation procedures, assessing their biases and mean square errors. Through the use of both real and simulated data sets, we demonstrate the superior performance of our proposed estimators compared to existing methods in similar scenarios. In conclusion, we offer practical recommendations for survey practitioners. [ABSTRACT FROM AUTHOR]

Data missing is a ubiquitous problem in real-world systems that adversely affects the performance of machine learning algorithms. Although many useful imputation methods are available to address this issue, they often fail to consider the information provided by both features and labels. As a result, the performance of these methods might be constrained. Furthermore, feature selection as a data quality improvement technique has been widely used and has demonstrated its efficiency. To overcome the limitation of imputation methods, we propose a novel algorithm that combines data imputation and feature selection to tackle classification problems for mixed data. Based on the mean and standard deviation of quantitative features and the selecting probabilities of unique values of categorical features, our algorithm constructs different imputation models for quantitative and categorical features. Particle swarm optimization is used to optimize the parameters of the imputation models and select feature subsets simultaneously. Additionally, we introduce a legacy learning mechanism to enhance the optimization capability of our method. To evaluate the performance of the proposed method, seven algorithms and twelve datasets are used for comparison. The results show that our algorithm outperforms other algorithms in terms of accuracy and F 1 score and has reasonable time overhead. [ABSTRACT FROM AUTHOR]

Diabetes is a chronic disease, which is characterized by abnormally high blood sugar levels. It may affect various organs and tissues, and even lead to life-threatening complications. Accurate prediction of diabetes can significantly reduce its incidence. However, the current prediction methods struggle to accurately capture the essential characteristics of nonlinear data, and the black-box nature of these methods hampers its clinical application. To address these challenges, we propose KCCAM_DNN, a diabetes prediction method that integrates Kendall's correlation coefficient and an attention mechanism within a deep neural network. In the KCCAM_DNN, Kendall's correlation coefficient is initially employed for feature selection, which effectively filters out key features influencing diabetes prediction. For missing values in the data, polynomial regression is utilized for imputation, ensuring data completeness. Subsequently, we construct a deep neural network (KCCAM_DNN) based on the self-attention mechanism, which assigns greater weight to crucial features affecting diabetes and enhances the model's predictive performance. Finally, we employ the SHAP model to analyze the impact of each feature on diabetes prediction, augmenting the model's interpretability. Experimental results show that KCCAM_DNN exhibits superior performance on both PIMA Indian and LMCH diabetes datasets, achieving test accuracies of 99.090% and 99.333%, respectively, approximately 2% higher than the best existing method. These results suggest that KCCAM_DNN is proficient in diabetes prediction, providing a foundation for informed decision-making in the diagnosis and prevention of diabetes. [ABSTRACT FROM AUTHOR]

Haplotype estimation, or phasing, has gained significant traction in large-scale projects due to its valuable contributions to population genetics, variant analysis, and the creation of reference panels for imputation and phasing of new samples. To scale with the growing number of samples, haplotype estimation methods designed for population scale rely on highly optimized statistical models to phase genotype data, and usually ignore read-level information. Statistical methods excel in resolving common variants, however, they still struggle at rare variants due to the lack of statistical information. In this study we introduce SAPPHIRE, a new method that leverages whole-genome sequencing data to enhance the precision of haplotype calls produced by statistical phasing. SAPPHIRE achieves this by refining haplotype estimates through the realignment of sequencing reads, particularly targeting low-confidence phase calls. Our findings demonstrate that SAPPHIRE significantly enhances the accuracy of haplotypes obtained from state of the art methods and also provides the subset of phase calls that are validated by sequencing reads. Finally, we show that our method scales to large data sets by its successful application to the extensive 3.6 Petabytes of sequencing data of the last UK Biobank 200,031 sample release. Author summary: Haplotype estimation, also known as phasing, is now applied to population scale projects, typically of hundreds of thousands of samples to millions of samples. Generally phasing relies on statistical methods as they provide very accurate results for common variations. However, for rare and very rare variants the lack of statistical power often results in poor phasing. The large amount of rare variations discovered with whole-genome sequencing as well as the number of samples makes it expensive to process. We have developed the SAPPHIRE method that leverages whole-genome sequencing data to verify and correct the phase at poorly phased variant loci. It does so by finding sequencing reads that contain both the poorly phased variant and an accurately phased common variant. SAPPHIRE scales with large data sets by specifically targeting variation where statistical phasing performed poorly, therefore it reduces the quantity of sequencing data to be processed and combines the advantages of both read-based and statistical approaches. We show the efficiency of SAPPHIRE by improving the estimated haplotypes for 200,031 samples in the UK Biobank. SAPPHIRE is free and available as open-source software. [ABSTRACT FROM AUTHOR]

In real-life scenarios, encountering data with missing values is common, and if not managed carefully from the outset of a study, it can lead to significant biases in survey estimates. Various methods exist for imputing missing values in sampling procedures. Ranked set sampling (RSS) is widely recognized for its superior efficiency compared to simple random sampling. However, limited research has been conducted on ranked set sampling in the presence of missing data. This article introduces novel imputation methods designed to estimate population means in the context of missing data under RSS. These innovative estimators are developed by integrating ratio, exponential, and logarithmic estimators judiciously. Expressions for the bias and mean squared error of the proposed estimators are derived up to the first-order approximation. Through simulation studies and an application to stunting and its determinants among children in Uttar Pradesh, India's most populous state, the effectiveness of the suggested estimators in handling missing data is demonstrated. Numerical examples involving stunting in Uttar Pradesh, as well as simulated data generated using R software, confirm the superior performance of the proposed estimators over existing methods, as evidenced by comparisons of percentage relative efficiency and mean squared error. The results are promising, indicating improvement over all existing imputation methods. Additionally, pertinent recommendations are provided for survey professionals regarding future research. [ABSTRACT FROM AUTHOR]

When combining imputed and sequenced data in a single gene-based association analysis, the problem of reconstructing genetic correlation matrices arises. It is related to the fact that the correlations between genotypes of all imputed variants and the correlations between genotypes of all sequenced variants are known for a gene but we do not know the correlations between genotypes of variants, one of which is imputed, and the other is sequenced. To recover these correlations, we propose an efficient method based on maximising the determinant of the matrix. This method has a number of useful properties and an analytical solution for our task. Approbation of the proposed method was performed by comparing reconstructed and real correlation matrices constructed on individual genotypes from the UK Biobank. Comparison of the results of gene-based association analysis performed by the SKAT, BT, and PCA methods on reconstructed and real matrices using modelled summary statistics and calculated summary statistics on real phenotypes showed high quality of reconstruction and robustness of the method to different gene structures. [ABSTRACT FROM AUTHOR]

A frequent problem in building initial software effort estimation (SEE) models is the existence of many missing values in historical software engineering datasets. Due to human intervention, this is caused by frequent damage to software project data. Loss of information and bias in data analysis due to missing data are serious problems. This study proposes a method to estimate missing data using a masked-denoising autoencoder (Masked- DAE) with L2-norm regularization, which can handle various types of data, missing patterns, proportions, and distributions. In this study, Cocomo81 and ISBSG-IFPUG datasets from open-source repositories were used. This experiment involved five missing data techniques, eight missing data rates (from 10% to 80%), and two missingness mechanisms (MCAR: missing completely at random and MNAR: missing not at random). The results show that the proposed Mask-DAE method has the best imputation performance in terms of imputation errors by outperforming DAE, k-nearest neighbor imputation (kNNI), random forest (RF) imputation, multiple imputations by chained equation (MICE), mean imputation and mode imputation. We find that the prediction error rate increases with the rate of missing data. Furthermore, prediction errors generated by MCAR mechanisms are lower than those generated by MNAR. Nevertheless, our method can reduce the model variance, which results in lower generalization error. [ABSTRACT FROM AUTHOR]

Numerous real-world data contain missing values, while in contrast, most Machine Learning (ML) algorithms assume complete datasets. For this reason, several imputation algorithms have been proposed to predict and fill in the missing values. Given the advances in predictive modeling algorithms tuned in an Automated Machine Learning context (AutoML) setting, a question that naturally arises is to what extent sophisticated imputation algorithms (e.g., Neural Network based) are really needed, or we can obtain a descent performance using simple methods like Mean/Mode (MM). In this article, we experimentally compare six state-of-the-art representatives of different imputation algorithmic families from an AutoML predictive modeling perspective, including a feature selection step and combined algorithm and hyper-parameter selection. We used a commercial AutoML tool for our experiments, in which we included the selected imputation methods. Experiments ran on 25 binary classification real-world incomplete datasets with missing values and 10 binary classification complete datasets in which synthetic missing values are introduced according to different missingness mechanisms, at varying missing frequencies. The main conclusion drawn from our experiments is that the best method on average is the Denoise AutoEncoder on real-world datasets and the MissForest in simulated datasets, followed closely by MM. In addition, binary indicator variables encoding missingness patterns actually improve predictive performance, on average. Last, although there are cases where Neural-Network-based imputation significantly improves predictive performance, this comes at a great computational cost and requires measuring all feature values to impute new samples. [ABSTRACT FROM AUTHOR]

Copyright of Journal of South China Normal University (Natural Science Edition) / Huanan Shifan Daxue Xuebao (Ziran Kexue Ban) is the property of Journal of South China Normal University (Natural Science Edition) Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)