Search

This study addresses the problem of parameter estimation in spatial autoregressive models with missing data and measurement errors in covariates. Specifically, a corrected likelihood estimation approach is employed to rectify the bias in the log-maximum likelihood function induced by measurement errors. Additionally, a combination of inverse probability weighting (IPW) and mean imputation is utilized to mitigate the bias caused by missing data. Under several mild conditions, it is demonstrated that the proposed estimators are consistent and possess oracle properties. The efficacy of the proposed parameter estimation process is assessed through Monte Carlo simulation studies. Finally, the applicability of the proposed method is further substantiated using the Boston Housing Dataset. [ABSTRACT FROM AUTHOR]

Soft sensors have been extensively utilized to approximate real-time power prediction in wind power generation, which is challenging to measure instantaneously. The short-term forecast of wind power aims at providing a reference for the dispatch of the intraday power grid. This study proposes a soft sensor model based on the Long Short-Term Memory (LSTM) network by combining data preprocessing with Variational Modal Decomposition (VMD) to improve wind power prediction accuracy. It does so by adopting the isolation forest algorithm for anomaly detection of the original wind power series and processing the missing data by multiple imputation. Based on the process data samples, VMD technology is used to achieve power data decomposition and noise reduction. The LSTM network is introduced to predict each modal component separately, and further sum reconstructs the prediction results of each component to complete the wind power prediction. From the experimental results, it can be seen that the LSTM network which uses an Adam optimizing algorithm has better convergence accuracy. The VMD method exhibited superior decomposition outcomes due to its inherent Wiener filter capabilities, which effectively mitigate noise and forestall modal aliasing. The Mean Absolute Percentage Error (MAPE) was reduced by 9.3508%, which indicates that the LSTM network combined with the VMD method has better prediction accuracy. [ABSTRACT FROM AUTHOR]

When integrating data from multiple sources, a common challenge is block-wise missing. Most existing methods address this issue only in cross-sectional studies. In this paper, we propose a method for variable selection when combining datasets from multiple sources in longitudinal studies. To account for block-wise missing in covariates, we impute the missing values multiple times based on combinations of samples from different missing pattern and predictors from different data sources. We then use these imputed data to construct estimating equations, and aggregate the information across subjects and sources with the generalized method of moments. We employ the smoothly clipped absolute deviation penalty in variable selection and use the extended Bayesian Information Criterion criteria for tuning parameter selection. We establish the asymptotic properties of the proposed estimator, and demonstrate the superior performance of the proposed method through numerical experiments. Furthermore, we apply the proposed method in the Alzheimer's Disease Neuroimaging Initiative study to identify sensitive early-stage biomarkers of Alzheimer's Disease, which is crucial for early disease detection and personalized treatment. [ABSTRACT FROM AUTHOR]

The sustainable operation of ambient air quality monitoring stations in developing countries is not always possible. Intermittent failures and breakdowns at air quality monitoring stations often affect the continuous measurement of data as required. These failures and breakdowns result in missing data. This study aimed to impute NO2, SO2, O3, and PM 10 to produce complete data sets of daily average exposures from 2010 to 2017. Models were built for (a) an individual pollutant at a monitoring station, (b) a combined model for the same pollutant from different stations, and (c) a data set with all the pollutants from all the monitoring stations. This study sought to evaluate the efficacy of the Multiple Imputation by Chain Equations (MICE) algorithm in successfully imputing air quality data that are missing at random. The application of classification and regression trees (CART) analysis using the MICE package in the R statistical programming language was compared with the predictive mean matching (PMM) method. The CART method performed better, with the pooled R-squared statistics of the imputed data ranging from 0.3 to 0.7, compared to a range of 0.02 to 0.25 for PMM. The MICE algorithm successfully resolved the incompleteness of the data. It was concluded that the CART method produced better reliable data than the PMM method. However, in this study, the pooled R2 values were accurate for NO2, but not so much for other pollutants. [ABSTRACT FROM AUTHOR]

The ability to translate Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) into different modalities and data types is essential to improve Deep Learning (DL) for predictive medicine. This work presents DACMVA, a novel framework to conduct data augmentation in a cross-modal dataset by translating between modalities and oversampling imputations of missing data. DACMVA was inspired by previous work on the alignment of latent spaces in Autoencoders. DACMVA is a DL data augmentation pipeline that improves the performance in a downstream prediction task. The unique DACMVA framework leverages a cross-modal loss to improve the imputation quality and employs training strategies to enable regularized latent spaces. Oversampling of augmented data is integrated into the prediction training. It is empirically demonstrated that the new DACMVA framework is effective in the often-neglected scenario of DL training on tabular data with continuous labels. Specifically, DACMVA is applied towards cancer survival prediction on tabular gene expression data where there is a portion of missing data in a given modality. DACMVA significantly (p << 0.001, one-sided Wilcoxon signed-rank test) outperformed the non-augmented baseline and competing augmentation methods with varying percentages of missing data (4%, 90%, 95% missing). As such, DACMVA provides significant performance improvements, even in very-low-data regimes, over existing state-of-the-art methods, including TDImpute and oversampling alone. [ABSTRACT FROM AUTHOR]

Simple Summary: By integrating prior biological information into genomic selection methods using appropriate models, it is possible to improve prediction accuracy for complex traits. In this context, we conducted a comparative assessment of two genomic prediction models, namely, genomic best linear unbiased prediction and genomic feature best linear unbiased prediction. The accuracy of these models in predicting the growth traits of backfat thickness and loin muscle area was evaluated. Our results revealed that the genomic feature best linear unbiased prediction model can effectively integrate prior information into the model, which is superior to the genomic best linear unbiased prediction model in some cases. These findings provide valuable ideas for enhancing the genomic prediction accuracy of growth traits in pigs. Enhancing the accuracy of genomic prediction is a key goal in genomic selection (GS) research. Integrating prior biological information into GS methods using appropriate models can improve prediction accuracy for complex traits. Genome-wide association study (GWAS) is widely utilized to identify potential candidate loci associated with complex traits in livestock and poultry, offering essential genomic insights. In this study, a GWAS was conducted on 685 Duroc × Landrace × Yorkshire (DLY) pigs to extract significant single-nucleotide polymorphisms (SNPs) as genomic features. We compared two GS models, genomic best linear unbiased prediction (GBLUP) and genomic feature BLUP (GFBLUP), by using imputed whole-genome sequencing (WGS) data on 651 Yorkshire pigs. The results revealed that the GBLUP model achieved prediction accuracies of 0.499 for backfat thickness (BFT) and 0.423 for loin muscle area (LMA). By applying the GFBLUP model with GWAS-based SNP preselection, the average prediction accuracies for BFT and LMA traits reached 0.491 and 0.440, respectively. Specifically, the GFBLUP model displayed a 4.8% enhancement in predicting LMA compared to the GBLUP model. These findings suggest that, in certain scenarios, the GFBLUP model may offer superior genomic prediction accuracy when compared to the GBLUP model, underscoring the potential value of incorporating genomic features to refine GS models. [ABSTRACT FROM AUTHOR]

With recent advancements in data technologies, particularly machine learning, research focusing on the enhancement of energy efficiency in residential, commercial, and industrial settings through the collection of load data, such as heat, electricity, and gas, has gained significant attention. Nevertheless, issues arising from hardware- or network-related problems can result in missing data, necessitating the development of management techniques to mitigate these challenges. Traditional methods for missing imputation face difficulties when operating in constrained environments characterized by short data collection periods and frequent consecutive missing. In this paper, we introduce the denoising masked autoencoder (DMAE) model as a solution to improve the handling of missing data, even in such restrictive settings. The proposed DMAE model capitalizes on the advantages of the denoising autoencoder (DAE), enabling effective learning of the missing imputation process, even with relatively small datasets, and the masked autoencoder (MAE), allowing for learning in environments with a high missing ratio. By integrating these strengths, the DMAE model achieves an enhanced performance in terms of missing imputation. The simulation results demonstrate that the proposed DMAE model outperforms the DAE or MAE significantly in a constrained environment where the duration of the training data is short, less than a year, and missing values occur frequently with durations ranging from 3 h to 12 h. [ABSTRACT FROM AUTHOR]

Supervisory control and data acquisition (SCADA) systems are widely utilized in power equipment for condition monitoring. For the collected data, there generally exists a problem—missing data of different types and patterns. This leads to the poor quality and utilization difficulties of the collected data. To address this problem, this paper customizes methodology that combines an asymmetric denoising autoencoder (ADAE) and moving average filter (MAF) to perform accurate missing data imputation. First, convolution and gated recurrent unit (GRU) are applied to the encoder of the ADAE, while the decoder still utilizes the fully connected layers to form an asymmetric network structure. The ADAE extracts the local periodic and temporal features from monitoring data and then decodes the features to realize the imputation of the multi-type missing. On this basis, according to the continuity of power data in the time domain, the MAF is utilized to fuse the prior knowledge of the neighborhood of missing data to secondarily optimize the imputed data. Case studies reveal that the developed method achieves greater accuracy compared to existing models. This paper adopts experiments under different scenarios to justify that the MAF-ADAE method applies to actual power equipment monitoring data imputation. [ABSTRACT FROM AUTHOR]

Traffic state data are key to the proper operation of intelligent transportation systems (ITS). However, traffic detectors often receive environmental factors that cause missing values in the collected traffic state data. Therefore, aiming at the above problem, a method for imputing missing traffic state data based on a Diffusion Convolutional Neural Network–Generative Adversarial Network (DCNN-GAN) is proposed in this paper. The proposed method uses a graph embedding algorithm to construct a road network structure based on spatial correlation instead of the original road network structure; through the use of a GAN for confrontation training, it is possible to generate missing traffic state data based on the known data of the road network. In the generator, the spatiotemporal features of the reconstructed road network are extracted by the DCNN to realize the imputation. Two real traffic datasets were used to verify the effectiveness of this method, with the results of the proposed model proving better than those of the other models used for comparison. [ABSTRACT FROM AUTHOR]

In estimating logistic regression models, convergence of the maximization algorithm is critical; however, this may fail. Numerous bias correction methods for maximum likelihood estimates of parameters have been conducted for cases of complete data sets, and also for longitudinal models. Balanced data sets yield consistent estimates from conditional logit estimators for binary response panel data models. When faced with a missing covariates problem, researchers adopt various imputation techniques to complete the data and without loss of generality; consistent estimates still suffice asymptotically. For maximum likelihood estimates of the parameters for logistic regression in cases of imputed covariates, the optimal choice of an imputation technique that yields the best estimates with minimum variance is still elusive. This paper aims to examine the behaviour of the Hessian matrix with optimal values of the imputed covariates vector, which will make the Newton–Raphson algorithm converge faster through a reduced absolute value of the product of the score function and the inverse fisher information component. We focus on a method used to modify the conditional likelihood function through the partitioning of the covariate matrix. We also confirm that the positive moduli of the Hessian for conditional estimators are sufficient for the concavity of the log-likelihood function, resulting in optimum parameter estimates. An increased Hessian modulus ensures the faster convergence of the parameter estimates. Simulation results reveal that model-based imputations perform better than classical imputation techniques, yielding estimates with smaller bias and higher precision for the conditional maximum likelihood estimation of nonlinear panel models. [ABSTRACT FROM AUTHOR]

Missing or unavailable data (NA) in multivariate data analysis is often treated with imputation methods and, in some cases, records containing NA are eliminated, leading to the loss of information. This paper addresses the problem of NA in multiple factor analysis (MFA) without resorting to eliminating records or using imputation techniques. For this purpose, the nonlinear iterative partial least squares (NIPALS) algorithm is proposed based on the principle of available data. NIPALS presents a good alternative when data imputation is not feasible. Our proposed method is called MFA-NIPALS and, based on simulation scenarios, we recommend its use until 15% of NAs of total observations. A case of groups of quantitative variables is studied and the proposed NIPALS algorithm is compared with the regularized iterative MFA algorithm for several percentages of NA. [ABSTRACT FROM AUTHOR]

Modern science is frequently based on the exploitation of large volumes of information storage in datasets and involving complex computational architectures. The statistical analyses of these datasets have to cope with specific challenges and frequently involve making informed but arbitrary decisions. Epidemiological papers have to be concise and focused on the underlying clinical or epidemiological results, not reporting the details behind relevant methodological decisions. In this work, we used an analysis of the cardiovascular-related measures tracked in 4–8-year-old children, using data from the INMA-Asturias cohort for illustrating how the decision-making process was performed and its potential impact on the obtained results. We focused on two particular aspects of the problem: how to deal with missing data and which regression model to use to evaluate tracking when there are no defined thresholds to categorize variables into risk groups. As a spoiler, we analyzed the impact on our results of using multiple imputation and the advantage of using quantile regression models in this context. [ABSTRACT FROM AUTHOR]

Ground meteorological observation data (GMOD) are the core of research on earth-related disciplines and an important reference for societal production and life. Unfortunately, due to operational issues or equipment failures, missing values may occur in GMOD. Hence, the imputation of missing data is a prevalent issue during the pre-processing of GMOD. Although a large number of machine-learning methods have been applied to the field of meteorological missing value imputation and have achieved good results, they are usually aimed at specific meteorological elements, and few studies discuss imputation when multiple elements are randomly missing in the dataset. This paper designed a machine-learning-based multidimensional meteorological data imputation framework (MMDIF), which can use the predictions of machine-learning methods to impute the GMOD with random missing values in multiple attributes, and tested the effectiveness of 20 machine-learning methods on imputing missing values within 124 meteorological stations across six different climatic regions based on the MMDIF. The results show that MMDIF-RF was the most effective missing value imputation method; it is better than other methods for imputing 11 types of hourly meteorological elements. Although this paper applied MMDIF to the imputation of missing values in meteorological data, the method can also provide guidance for dataset reconstruction in other industries. [ABSTRACT FROM AUTHOR]

Dunnett's procedure has been frequently used for multiple comparisons of group means of several treatments with a control, in drug development and other areas. However, in practice, researchers usually face missing observations when performing Dunnett's procedure. This paper presents an improved Dunnett's procedure that can construct unique ensemble confidence intervals for comparing group means of several treatments with a control, in the presence of missing observations, using a derived multivariate t distribution under the framework of Rubin's rule. This procedure fills the current research gap that Rubin's repeated-imputation inferences cannot adjust for multiplicity and, thereby, cannot give a unified confidence interval to control the family-wise error rate (FWER) when dealing with this problem. Simulation results show that the constructed pooled confidence intervals archive nominal joint coverage and the interval estimations preserve comparable precision to Rubin's repeated-imputation inference as the missing rate increases. The proposed procedure with propensity-score imputation method is shown to produce more accurate interval estimations and control the FWER well. [ABSTRACT FROM AUTHOR]

In intelligent traffic control systems, the features extracted by loop detectors are insufficient to accurately impute missing data. Most of the existing imputation methods use only these extracted features, which leads to the construction of data models that cannot fulfill the required accuracy. This deficiency is the main motivation to propose an enrichment imputation method for loop detectors namely EIM-LD, in which the imputation accuracy is increased for different missing patterns and ratios by introducing a data enrichment technique using statistical multi-class labeling. It first enriches the clean data by adding a statistical multi-class label, including C1...Cn classes. Then, the class of samples in the missed-volume data is labeled using the best data model constructed from the labeled clean data by five different classifiers. Experts of the traffic control department in Isfahan city determined classes of the statistical multi-class label for n = 5 (class labels), and we also developed subclass labels (n = 20) since the number of samples in the subclass labels was sufficient. Next, the enriched data are divided into n datasets, each of them is imputed independently using various imputation methods, and their results are finally merged. To evaluate the impact of using the proposed method, the original data, including missing volumes, are first imputed without our enrichment method. Then, the proposed method's accuracy is evaluated by considering two class labels and subclass labels. The experimental and statistical results prove that the proposed EIM-LD method can enrich the real data collected by loop detectors, by which the comparative imputation methods construct a more accurate data model. In addition, using subclass labels further enhances the imputation method's accuracy. [ABSTRACT FROM AUTHOR]

Navigating the intricate world of data analytics, one method has emerged as a key tool in confronting missing data: multiple imputation. Its strength is further fortified by its powerful variant, robust imputation, which enhances the precision and reliability of its results. In the challenging landscape of data analysis, non-robust methods can be swayed by a few extreme outliers, leading to skewed imputations and biased estimates. This can apply to both representative outliers—those true yet unusual values of your population—and non-representative outliers, which are mere measurement errors. Detecting these outliers in large or high-dimensional data sets often becomes as complex as unraveling a Gordian knot. The solution? Turn to robust imputation methods. Robust (imputation) methods effectively manage outliers and exhibit remarkable resistance to their influence, providing a more reliable approach to dealing with missing data. Moreover, these robust methods offer flexibility, accommodating even if the imputation model used is not a perfect fit. They are akin to a well-designed buffer system, absorbing slight deviations without compromising overall stability. In the latest advancement of statistical methodology, a new robust imputation algorithm has been introduced. This innovative solution addresses three significant challenges with robustness. It utilizes robust bootstrapping to manage model uncertainty during the imputation of a random sample; it incorporates robust fitting to reinforce accuracy; and it takes into account imputation uncertainty in a resilient manner. Furthermore, any complex regression or classification model for any variable with missing data can be run through the algorithm. With this new algorithm, we move one step closer to optimizing the accuracy and reliability of handling missing data. Using a realistic data set and a simulation study including a sensitivity analysis, the new alogorithm imputeRobust shows excellent performance compared with other common methods. Effectiveness was demonstrated by measures of precision for the prediction error, the coverage rates, and the mean square errors of the estimators, as well as by visual comparisons. [ABSTRACT FROM AUTHOR]

The Air Quality Index (AQI) dataset contains information on measurements of pollutants and ambient air quality conditions at certain location that can be used to predict air quality. Unfortunately, this dataset often has many missing observations and imbalanced classes. Both of these problems can affect the performance of the prediction model. In particular, predictions for the minority class are very important because inaccurate predictions can be fatal or cause big losses. Moreover, the missing data may lead to biased results. This paper proposes the single imputation of the median and the multiple imputations of the k -Nearest Neighbor (KNN) regressor to handle missing values of less than or equal to 10% and more than 10%, respectively. At the same time, the SMOTE-Tomek Links address the imbalanced class. These proposed approaches to handle both issues are then used to assess the air quality prediction of the India AQI dataset using Naive Bayes (NB), KNN, and C4.5. The five treatments show that the proposed method of the Median-KNN regressor-SMOTE-Tomek Links is able to improve the performance of the India air quality prediction model. In other words, the proposed method succeeds in overcoming the problems of missing values and class imbalance. [ABSTRACT FROM AUTHOR]

Many datasets in statistical analyses contain missing values. As omitting observations containing missing entries may lead to information loss or greatly reduce the sample size, imputation is usually preferable. However, imputation can also introduce bias and impact the quality and validity of subsequent analysis. Focusing on binary classification problems, we analyzed how missing value imputation under MCAR as well as MAR missingness with different missing patterns affects the predictive performance of subsequent classification. To this end, we compared imputation methods such as several MICE variants, missForest, Hot Deck as well as mean imputation with regard to the classification performance achieved with commonly used classifiers such as Random Forest, Extreme Gradient Boosting, Support Vector Machine and regularized logistic regression. Our simulation results showed that Random Forest based imputation (i.e., MICE Random Forest and missForest) performed particularly well in most scenarios studied. In addition to these two methods, simple mean imputation also proved to be useful, especially when many features (covariates) contained missing values. [ABSTRACT FROM AUTHOR]

In a multivariate database, the missing data can be obtained through several imputation techniques, which are particularly useful for data that are difficult to obtain, for any reason, or have high uncertainties or scarce variables. A Self-Organizing Maps (SOM) neural network is an effective tool for the analysis of multidimensional data applied for the imputation of data. In this paper, data from drilling were used for training, testing, and validation using the variables: total Au recovery (%), which means gold recovery from a gravity concentration plus hydrometallurgical process, Au (g/t), As (ppm), S (%), Al2O3 (%), CaO (%), K2O (%), and MgO (%). After training, the partial omission of Au content and recovery was carried out, from 10% to 50%, to evaluate the data imputation performance for those variables. The results imputed by the SOM were compared with the original data values and evaluated according to descriptive statistics; the results indicated a determination coefficient of 85% when 50% of the data were omitted and 93% when 10% of the data were omitted. Once demonstrated, the correlation between the original data and SOM imputation analysis can help geologists and metallurgists to obtain results with a high degree of reliability of metallurgical recovery through related chemical variables, making it possible to implement SOM analysis as a powerful tool to input analytical data. One of the practical applications of the proposed model is to produce a pattern of imputed data that can be a good alternative in the construction or generation of a synthetic geometallurgical database with missing data. [ABSTRACT FROM AUTHOR]

In this paper, we study the problem of extracting trends from time series data involving missing values. In particular, we investigate a general class of procedures that impute the missing data and then extract trends using seasonal-trend decomposition based on loess (STL), where loess stands for locally weighted smoothing, a popular tool for describing the regression relationship between two variables by a smooth curve. We refer to them as the imputation-STL procedures. Two results are obtained in this paper. First, we settle a theoretical issue, namely the connection between imputation error and the overall error from estimating the trend. Specifically, we derive the bounds for the overall error in terms of the imputation error. This subsequently facilitates the error analysis of any imputation-STL procedure and justifies its use in practice. Second, we investigate loess-STL, a particular imputation-STL procedure with the imputation also being performed using loess. Through both theoretical arguments and simulation results, we show that loess-STL has the capacity of handling a high proportion of missing data and providing reliable trend estimates if the underlying trend is smooth and the missing data are dispersed over the time series. In addition to mathematical derivations and simulation study, we apply our loess-STL procedure to profile radiosonde records of upper air temperature at 22 Antarctic research stations covering the past 50 years. For purpose of illustration, we present in this paper only the results for Novolazaravskaja station which has temperature records with more than 8.4% dispersed missing values at 8 pressure levels from October/1969 to March/2011. [ABSTRACT FROM AUTHOR]

Ground temperature (GT) or soil temperature (ST) is simply the measurement of the warmness of the soil. Even though GT plays a meaningful role in agricultural production, the direct method of measuring the GT is time-consuming, expensive, and requires human effort. The foremost objective of this study is to build machine learning (ML) models for hourly GT prediction at different depths (5, 10, 20, and 30 cm) with the optimum hyperparameter tuning with less complexity. The present study utilizes a statistical model (multiple linear regression (MLR)) and four different ML models (support vector regression (SVR), random forest regression (RFR), multi-layered perceptron (MLP), and XGBoost (XGB)) for predicting GT. Overall, 13 independent variables and 5 GTs with different depths as response variables were collected from a meteorological station at an interval of 1 h between 1 January 2017 and 1 July 2021. In addition, two different input datasets named M1 (selected number of parameters) and M2 (collected dataset with all variables) were used to assess the model. The current study employed the Spearman rank correlation coefficient approach to extract the best features and used it as the M1 dataset; in addition, the present study adopted regression imputation for solving the missing data issues. From the results, the XGB model outperformed the other standard ML-based models in any depth GT prediction (MAE = 1.063; RMSE = 1.679; R2 = 0.978 for GT; MAE = 0.887; RMSE = 1.263; R2 = 0.979 for GT_5; MAE = 0.741; RMSE = 1.025; R2 = 0.985 for GT_10; MAE = 0.416; RMSE = 0.551; R2 = 0.995 for GT_20; MAE = 0.280; RMSE = 0.367; R2 = 0.997 for GT_20). Therefore, the present study developed a simpler, less-complex, faster, and more versatile model to predict the GT at different depths for a short-term prediction with a minimum number of predictor attributes. [ABSTRACT FROM AUTHOR]

Satellite data is of high importance for ocean environment monitoring and protection. However, due to the missing values in satellite data, caused by various force majeure factors such as cloud cover, bad weather and sensor failure, the quality of satellite data is reduced greatly, which hinders the applications of satellite data in practice. Therefore, a variety of methods have been proposed to conduct missing data imputation for satellite data to improve its quality. However, these methods cannot well learn the short-term temporal dependence and dynamic spatial dependence in satellite data, resulting in bad imputation performance when the data missing rate is large. To address this issue, we propose the Spatio-Temporal Attention Generative Adversarial Network (STA-GAN) for missing value imputation in satellite data. First, we develop the Spatio-Temporal Attention (STA) mechanism based on Graph Attention Network (GAT) to learn features for capturing both short-term temporal dependence and dynamic spatial dependence in satellite data. Then, the learned features from STA are fused to enrich the spatio-temporal information for training the generator and discriminator of STA-GAN. Finally, we use the generated imputation data by the trained generator of STA-GAN to fill the missing values in satellite data. Experimental results on real datasets show that STA-GAN largely outperforms the baseline data imputation methods, especially for filling satellite data with large missing rates. [ABSTRACT FROM AUTHOR]

In recent years, the development of computer technology has promoted the informatization and intelligentization of hospital management systems and thus produced a large amount of medical data. These medical data are valuable resources for research. We can obtain inducers and unknown symptoms that can help discover diseases and make earlier diagnoses. Hypertensive disorder in pregnancy (HDP) is a common obstetric complication in pregnant women, which has severe adverse effects on the life safety of pregnant women and fetuses. However, the early and mid-term symptoms of HDP are not obvious, and there is no effective solution for it except for terminating the pregnancy. Therefore, detecting and preventing HDP is of great importance. This study aims at the preprocessing of pregnancy examination data, which serves as a part of HDP prediction. We found that the problem of missing data has a large impact on HDP prediction. Unlike general data, pregnancy examination data have high dimension and a high missing rate, are in a time series, and often have many non-linear relations. Current methods are not able to process the data effectively. To this end, we propose an improved bi-LSTM-based missing value imputation approach. It combines traditional machine learning and bidirectional LSTM to deal with missing data of pregnancy examination data. Our missing value imputation method obtains a good effect and improves the accuracy of the later prediction of HDP using examination data. [ABSTRACT FROM AUTHOR]

Missing data is a common and inevitable phenomenon. In practical applications, the datasets usually contain noises for various reasons. Most of the existing missing data imputing algorithms are affected by noises which reduce the accuracy of the imputation. This paper proposes a noise-aware missing data multiple imputation algorithm NPMI in static data. Different multiple imputation models are proposed according to the missing mechanism of data. Secondly, the method to determine the imputation order of multivariablesmissing is given. A random sampling consistency algorithm is proposed to estimate the initial values of the parameters of the multiple imputation model to reduce the influence of noise data and improve the algorithm's robustness. Experiments on two real datasets and two synthetic datasets verify the accuracy and efficiency of the proposed NPMI algorithm, and the results are analyzed. [ABSTRACT FROM AUTHOR]

This paper extends the conditional autoregressive range (CARR) model to the multivariate CARR (MCARR) model and further to the two-stage MCARR-return model to model and forecast volatilities, correlations and returns of multiple financial assets. The first stage model fits the scaled realised Parkinson volatility measures using individual series and their pairwise sums of indices to the MCARR model to obtain the fitted volatilities. Then covariances are calculated to construct the fitted variance–covariance matrix of returns which are imputed into the stage-two return model to capture the heteroskedasticity of assets' returns. We investigate different choices of mean functions to describe the volatility dynamics. Empirical applications are based on the Standard and Poor 500, Dow Jones Industrial Average and Dow Jones United States Financial Service Indices. Results show that the stage-one MCARR models using asymmetric mean functions give better in-sample model fits than those based on symmetric mean functions. They also provide better out-of-sample volatility forecasts than those using CARR models based on two robust loss functions. We also find that the stage-two return models with constant means and multivariate Student-t errors give better in-sample fits than the Baba–Engle–Kraft–Kroner generalised autoregressive conditional heteroskedasticity models. The estimates and forecasts of value-at-risk (VaR) and conditional VaR based on the best MCARR-return models for each asset are provided and tested using backtests to confirm the accuracy of the VaR forecasts. [ABSTRACT FROM AUTHOR]

Hourly traffic volumes, collected by automatic traffic recorders (ATRs), are of paramount importance since they are used to calculate average annual daily traffic (AADT) and design hourly volume (DHV). Hence, it is necessary to ensure the quality of the collected data. Unfortunately, ATRs malfunction occasionally, resulting in missing data, as well as unreliable counts. This naturally has an impact on the accuracy of the key parameters derived from the hourly counts. This study aims to solve this problem. ATR data from New South Wales, Australia was screened for irregularities and invalid entries. A total of 25% of the reliable data was randomly selected to test thirteen different imputation methods. Two scenarios for data omission, i.e., 25% and 100%, were analyzed. Results indicated that missForest outperformed other imputation methods; hence, it was used to impute the actual missing data to complete the dataset. AADT values were calculated from both original counts before imputation and completed counts after imputation. AADT values from imputed data were slightly higher. The average daily volumes when plotted validated the quality of imputed data, as the annual trends demonstrated a relatively better fit. [ABSTRACT FROM AUTHOR]

Insights and analysis are only as good as the available data. Data cleaning is one of the most important steps to create quality data decision making. Machine learning (ML) helps deal with data quickly, and to create error-free or limited-error datasets. One of the quality standards for cleaning the data includes handling the missing data, also known as data imputation. This research focuses on the use of machine learning methods to deal with missing data. In particular, we propose a generative adversarial network (GAN) based model called DEGAIN to estimate the missing values in the dataset. We evaluate the performance of the presented method and compare the results with some of the existing methods on publicly available Letter Recognition and SPAM datasets. The Letter dataset consists of 20,000 samples and 16 input features and the SPAM dataset consists of 4601 samples and 57 input features. The results show that the proposed DEGAIN outperforms the existing ones in terms of root mean square error and Frechet inception distance metrics. [ABSTRACT FROM AUTHOR]

The purpose of this study is to characterize the impact of the timing and duration of missing actigraphy data on interdaily stability (IS) and intradaily variability (IV) calculation. The performance of three missing data imputation methods (linear interpolation, mean time of day (ToD), and median ToD imputation) for estimating IV and IS was also tested. Week-long actigraphy records with no non-wear or missing timeseries data were masked with zeros or 'Not a Number' (NaN) across a range of timings and durations for single and multiple missing data bouts. IV and IS were calculated for true, masked, and imputed (i.e., linear interpolation, mean ToD and, median ToD imputation) timeseries data and used to generate Bland–Alman plots for each condition. Heatmaps were used to analyze the impact of timings and durations of and between bouts. Simulated missing data produced deviations in IV and IS for longer durations, midday crossings, and during similar timing on consecutive days. Median ToD imputation produced the least deviation among the imputation methods. Median ToD imputation is recommended to recapitulate IV and IS under missing data conditions for less than 24 h. [ABSTRACT FROM AUTHOR]

Addressing data anomalies (e.g., garbage data, outliers, redundant data, and missing data) plays a vital role in performing accurate analytics (billing, forecasting, load profiling, etc.) on smart homes' energy consumption data. From the literature, it has been identified that the data imputation with machine learning (ML)-based single-classifier approaches are used to address data quality issues. However, these approaches are not effective to address the hidden issues of smart home energy consumption data due to the presence of a variety of anomalies. Hence, this paper proposes ML-based ensemble classifiers using random forest (RF), support vector machine (SVM), decision tree (DT), naive Bayes, K-nearest neighbor, and neural networks to handle all the possible anomalies in smart home energy consumption data. The proposed approach initially identifies all anomalies and removes them, and then imputes this removed/missing information. The entire implementation consists of four parts. Part 1 presents anomaly detection and removal, part 2 presents data imputation, part 3 presents single-classifier approaches, and part 4 presents ensemble classifiers approaches. To assess the classifiers' performance, various metrics, namely, accuracy, precision, recall/sensitivity, specificity, and F1 score are computed. From these metrics, it is identified that the ensemble classifier "RF+SVM+DT" has shown superior performance over the conventional single classifiers as well the other ensemble classifiers for anomaly handling. [ABSTRACT FROM AUTHOR]

Sustainable groundwater management requires an accurate characterization of aquifer-storage change over time. This process begins with an analysis of historical water levels at observation wells. However, water-level records can be sparse, particularly in developing areas. To address this problem, we developed an imputation method to approximate missing monthly averaged groundwater-level observations at individual wells since 1948. To impute missing groundwater levels at individual wells, we used two global data sources: Palmer Drought Severity Index (PDSI), and the Global Land Data Assimilation System (GLDAS) for regression. In addition to the meteorological datasets, we engineered four additional features and encoded the temporal data as 13 parameters that represent the month and year of an observation. This extends previous similar work by using inductive bias to inform our models on groundwater trends and structure from existing groundwater observations, using prior estimates of groundwater behavior. We formed an initial prior by estimating the long-term ground trends and developed four additional priors by using smoothing. These prior features represent the expected behavior over the long term of the missing data and allow the regression approach to perform well, even over large gaps of up to 50 years. We demonstrated our method on the Beryl-Enterprise aquifer in Utah and found the imputed results follow trends in the observed data and hydrogeological principles, even over long periods with no observed data. [ABSTRACT FROM AUTHOR]

Background: Type 1 diabetes (T1D) is a devastating disease with serious health complications. Early T1D biomarkers that could enable timely detection and prevention before the onset of clinical symptoms are paramount but currently unavailable. Despite their promise, omics approaches have so far failed to deliver such biomarkers, likely due to the fragmented nature of information obtained through the single omics approach. We recently demonstrated the utility of parallel multi-omics for the identification of T1D biomarker signatures. Our studies also identified challenges. Methods: Here, we evaluated a novel computational approach of data imputation and amplification as one way to overcome challenges associated with the relatively small number of subjects in these studies. Results: Using proprietary algorithms, we amplified our quadra-omics (proteomics, metabolomics, lipidomics, and transcriptomics) dataset from nine subjects a thousand-fold and analyzed the data using Ingenuity Pathway Analysis (IPA) software to assess the change in its analytical capabilities and biomarker prediction power in the amplified datasets compared to the original. These studies showed the ability to identify an increased number of T1D-relevant pathways and biomarkers in such computationally amplified datasets, especially, at imputation ratios close to the "golden ratio" of 38.2%:61.8%. Specifically, the Canonical Pathway and Diseases and Functions modules identified higher numbers of inflammatory pathways and functions relevant to autoimmune T1D, including novel ones not identified in the original data. The Biomarker Prediction module also predicted in the amplified data several unique biomarker candidates with direct links to T1D pathogenesis. Conclusions: These preliminary findings indicate that such large-scale data imputation and amplification approaches are useful in facilitating the discovery of candidate integrated biomarker signatures of T1D or other diseases by increasing the predictive range of existing data mining tools, especially when the size of the input data is inherently limited. [ABSTRACT FROM AUTHOR]

The problem addressed by dictionary learning (DL) is the representation of data as a sparse linear combination of columns of a matrix called dictionary. Both the dictionary and the sparse representations are learned from the data. We show how DL can be employed in the imputation of multivariate time series. We use a structured dictionary, which is comprised of one block for each time series and a common block for all the time series. The size of each block and the sparsity level of the representation are selected by using information theoretic criteria. The objective function used in learning is designed to minimize either the sum of the squared errors or the sum of the magnitudes of the errors. We propose dimensionality reduction techniques for the case of high-dimensional time series. For demonstrating how the new algorithms can be used in practical applications, we conduct a large set of experiments on five real-life data sets. The missing data (MD) are simulated according to various scenarios where both the percentage of MD and the length of the sequences of MD are considered. This allows us to identify the situations in which the novel DL-based methods are superior to the existing methods. [ABSTRACT FROM AUTHOR]

A great challenge for intelligent transportation systems (ITS) is missing traffic data. Traffic data are input from various transportation applications. In the past few decades, several methods for traffic temporal data imputation have been proposed. A key issue is that temporal information collected by neighbor detectors can make traffic missing data imputation more accurate. This review analyzes traffic temporal data imputation methods. Research methods, missing patterns, assumptions, imputation styles, application conditions, limitations, and public datasets are reviewed. Then, five representative methods are tested under different missing patterns and missing ratios. California performance measurement system (PeMS) data including traffic volume and speed are selected to conduct the test. Probabilistic principal component analysis performs the best under the most conditions. [ABSTRACT FROM AUTHOR]

Accurate, timely air quality index (AQI) forecasting helps industries in selecting the most suitable air pollution control measures and the public in reducing harmful exposure to pollution. This article proposes a comprehensive method to forecast AQIs. Initially, the work focused on predicting hourly ambient concentrations of PM2.5 and PM10 using artificial neural networks. Once the method was developed, the work was extended to the prediction of other criteria pollutants, i.e., O3, SO2, NO2, and CO, which fed into the process of estimating AQI. The prediction of the AQI not only requires the selection of a robust forecasting model, it also heavily relies on a sequence of pre-processing steps to select predictors and handle different issues in data, including gaps. The presented method dealt with this by imputing missing entries using missForest, a machine learning-based imputation technique which employed the random forest (RF) algorithm. Unlike the usual practice of using RF at the final forecasting stage, we utilized RF at the data pre-processing stage, i.e., missing data imputation and feature selection, and we obtained promising results. The effectiveness of this imputation method was examined against a linear imputation method for the six criteria pollutants and the AQI. The proposed approach was validated against ambient air quality observations for Al-Jahra, a major city in Kuwait. Results obtained showed that models trained using missForest-imputed data could generalize AQI forecasting and with a prediction accuracy of 92.41% when tested on new unseen data, which is better than earlier findings. [ABSTRACT FROM AUTHOR]

Missing observations in time series will distort the data characteristics, change the dataset expectations, high-order distances, and other statistics, and increase the difficulty of data analysis. Therefore, data imputation needs to be performed first. Generally, data imputation methods include statistical imputation, regression imputation, multiple imputation, and imputation based on machine learning methods. However, these methods currently have problems such as insufficient utilization of time characteristics, low imputation efficiency, and poor performance under high missing rates. In response to these problems, we propose the informer-WGAN, a network model based on adversarial training and a self-attention mechanism. With the help of the discriminator network and the random missing rate training method, the informer-WGAN can efficiently solve the problem of multidimensional time series imputation. According to the experimental results under different missing rates, the informer-WGAN model achieves better imputation results than the original informer on two datasets. Our model also shows excellent performance on time series imputation of the key parameters of a spacecraft control moment gyroscope (CMG). [ABSTRACT FROM AUTHOR]

The problem of missing data is a common feature in any study, and a single imputation method is often applied to deal with this problem. The first contribution of this paper is to analyse the empirical performance of some traditional single imputation methods when they are applied to the estimation of the Gini index, a popular measure of inequality used in many studies. Various methods for constructing confidence intervals for the Gini index are also empirically evaluated. We consider several empirical measures to analyse the performance of estimators and confidence intervals, allowing us to quantify the magnitude of the non-response bias problem. We find extremely large biases under certain non-response mechanisms, and this problem gets noticeably worse as the proportion of missing data increases. For a large correlation coefficient between the target and auxiliary variables, the regression imputation method may notably mitigate this bias problem, yielding appropriate mean square errors. We also find that confidence intervals have poor coverage rates when the probability of data being missing is not uniform, and that the regression imputation method substantially improves the handling of this problem as the correlation coefficient increases. [ABSTRACT FROM AUTHOR]

Information criteria such as the Akaike information criterion (AIC) and Bayesian information criterion (BIC) are commonly used for model selection. However, the current theory does not support unconventional data, so naive use of these criteria is not suitable for data with missing values. Imputation, at the core of most alternative methods, is both distorted as well as computationally demanding. We propose a new approach that enables the use of classic well-known information criteria for model selection when there are missing data. We adapt the current theory of information criteria through normalization, accounting for the different sample sizes used for each candidate model (focusing on AIC and BIC). Interestingly, when the sample sizes are different, our theoretical analysis finds that A I C j / n j is the proper correction for A I C j that we need to optimize (where n j is the sample size available to the j th model) while − (B I C j − B I C i) / (n j − n i) is the correction of BIC. Furthermore, we find that the computational complexity of normalized information criteria methods is exponentially better than that of imputation methods. In a series of simulation studies, we find that normalized-AIC and normalized-BIC outperform previous methods (i.e., normalized-AIC is more efficient, and normalized BIC includes only important variables, although it tends to exclude some of them in cases of large correlation). We propose three additional methods aimed at increasing the statistical efficiency of normalized-AIC: post-selection imputation, Akaike sub-model averaging, and minimum-variance averaging. The latter succeeds in increasing efficiency further. [ABSTRACT FROM AUTHOR]

Real-life data are bounded and heavy-tailed variables. Zero-one-inflated beta (ZOIB) regression is used for modelling them. There are no appropriate methods to address the problem of missing data in repeated bounded outcomes. We developed an imputation method using ZOIB (i-ZOIB) and compared its performance with those of the naïve and machine-learning methods, using different distribution shapes and settings designed in the simulation study. The performance was measured employing the absolute error (MAE), root-mean-square-error (RMSE) and the unscaled mean bounded relative absolute error (UMBRAE) methods. The results varied depending on the missingness rate and mechanism. The i-ZOIB and the machine-learning ANN, SVR and RF methods showed the best performance. [ABSTRACT FROM AUTHOR]

The aim of this paper is the imputation of missing data of COVID-19 hospitalized and intensive care curves in several Spanish regions. Taking into account that the curves of cases, deceases and recovered people are completely observed, a function-on-function regression model is proposed to estimate the missing values of the functional responses associated with hospitalized and intensive care curves. The estimation of the functional coefficient model in terms of principal components' regression with the completely observed data provides a prediction equation for the imputation of the unobserved data for the response. An application with data from the first wave of COVID-19 in Spain is developed after properly homogenizing, registering and smoothing the data in a common interval so that the observed curves become comparable. Finally, Canonical Correlation Analysis is performed on the functional principal components to interpret the relationship between hospital occupancy rate and illness response variables. [ABSTRACT FROM AUTHOR]

Smart cities leverage large amounts of data acquired in the urban environment in the context of decision support tools. These tools enable monitoring the environment to improve the quality of services offered to citizens. The increasing diffusion of personal Internet of things devices capable of sensing the physical environment allows for low-cost solutions to acquire a large amount of information within the urban environment. On the one hand, the use of mobile and intermittent sensors implies new scenarios of large-scale data analysis; on the other hand, it involves different challenges such as intermittent sensors and integrity of acquired data. To this effect, edge computing emerges as a methodology to distribute computation among different IoT devices to analyze data locally. We present here a new methodology for imputing environmental information during the acquisition step, due to missing or otherwise out of order sensors, by distributing the computation among a variety of fixed and mobile devices. Numerous experiments have been carried out on real data to confirm the validity of the proposed method. [ABSTRACT FROM AUTHOR]

Wind tunnel testing techniques are the main research tools for evaluating the wind loadings of buildings. They are significant in designing structurally safe and comfortable buildings. The wind tunnel pressure measurement technique using pressure sensors is significant for assessing the cladding pressures of buildings. However, some pressure sensors usually fail and cause loss of data, which are difficult to restore. In the literature, numerous techniques are implemented for imputing the single instance data values and data imputation for multiple instantaneous time intervals with accurate predictions needs to be addressed. Thus, the data imputation capacity of machine learning models is used to predict the missing wind pressure data for tall buildings in this study. A generative adversarial imputation network (GAIN) is proposed to predict the pressure coefficients at various instantaneous time intervals on tall buildings. The proposed model is validated by comparing the performance of GAIN with that of the K-nearest neighbor and multiple imputations by chained equation models. The experimental results show that the GAIN model provides the best fit, achieving more accurate predictions with the minimum average variance and minimum average standard deviation. The average mean-squared error for all four sides of the building was the minimum (0.016), and the average R-squared error was the maximum (0.961). The proposed model can ensure the health and prolonged existence of a structure based on wind environment. [ABSTRACT FROM AUTHOR]

Greenhouses require accurate and reliable data to interpret the microclimate and maximize resource use efficiency. However, greenhouse conditions are harsh for electrical sensors collecting environmental data. Convolutional neural networks (ConvNets) enable complex interpretation by multiplying the input data. The objective of this study was to impute missing tabular data collected from several greenhouses using a ConvNet architecture called U-Net. Various data-loss conditions with errors in individual sensors and in all sensors were assumed. The U-Net with a screen size of 50 exhibited the highest coefficient of determination values and the lowest root-mean-square errors for all environmental factors used in this study. U-Net50 correctly learned the changing patterns of the greenhouse environment from the training dataset. Therefore, the U-Net architecture can be used for the imputation of tabular data in greenhouses if the model is correctly trained. Growers can secure data integrity with imputed data, which could increase crop productivity and quality in greenhouses. [ABSTRACT FROM AUTHOR]

Independent component analysis (ICA) is one of the most effective approaches in extracting independent signals from a global navigation satellite system (GNSS) regional station network. However, ICA requires the involved time series to be complete, thereby the missing data of incomplete time series should be interpolated beforehand. In this contribution, a modified ICA is proposed, by which the missing data are first recovered based on the reversible property between the original time series and decomposed principal components, then the complete time series are further processed with FastICA. To evaluate the performance of the modified ICA for extracting independent components, 24 regional GNSS network stations located in North China from 2011 to 2019 were selected. After the trend, annual and semiannual terms were removed from the GNSS time series, the first two independent components captured 17.42, 18.44 and 17.38% of the total energy for the North, East and Up coordinate components, more than those derived by the iterative ICA that accounted for 16.21%, 17.72% and 16.93%, respectively. Therefore, modified ICA can extract more independent signals than iterative ICA. Subsequently, selecting the 7 stations with less missing data from the network, we repeatedly process the time series after randomly deleting parts of the data and compute the root mean square error (RMSE) from the differences of reconstructed signals before and after deleting data. All RMSEs of modified ICA are smaller than those of iterative ICA, indicating that modified ICA can extract more exact signals than iterative ICA. [ABSTRACT FROM AUTHOR]

Cost–benefit analysis is widely used to elucidate the association between foraging group size and resource size. Despite advances in the development of theoretical frameworks, however, the empirical systems used for testing are hindered by the vagaries of field surveys and incomplete data. This study developed the three approaches to data imputation based on machine learning (ML) algorithms with the aim of rescuing valuable field data. Using 163 host spider webs (132 complete data and 31 incomplete data), our results indicated that the data imputation based on random forest algorithm outperformed classification and regression trees, the k-nearest neighbor, and other conventional approaches (Wilcoxon signed-rank test and correlation difference have p-value from < 0.001–0.030). We then used rescued data based on a natural system involving kleptoparasitic spiders from Taiwan and Vietnam (Argyrodes miniaceus, Theridiidae) to test the occurrence and group size of kleptoparasites in natural populations. Our partial least-squares path modelling (PLS-PM) results demonstrated that the size of the host web (T = 6.890, p = 0.000) is a significant feature affecting group size. The resource size (T = 2.590, p = 0.010) and the microclimate (T = 3.230, p = 0.001) are significant features affecting the presence of kleptoparasites. The test of conformation of group size distribution to the ideal free distribution (IFD) model revealed that predictions pertaining to per-capita resource size were underestimated (bootstrap resampling mean slopes

Medical data usually have missing values; hence, imputation methods have become an important issue. In previous studies, many imputation methods based on variable data had a multivariate normal distribution, such as expectation-maximization and regression-based imputation. These assumptions may lead to deviations in the results, which sometimes create a bottleneck. In addition, directly deleting instances with missing values may have several problems, such as losing important data, producing invalid research samples, and leading to research deviations. Therefore, this study proposed a safe-region imputation method for handling medical data with missing values; we also built a medical prediction model and compared the removed missing values with imputation methods in terms of the generated rules, accuracy, and AUC. First, this study used the kNN imputation, multiple imputation, and the proposed imputation to impute the missing data and then applied four attribute selection methods to select the important attributes. Then, we used the decision tree (C4.5), random forest, REP tree, and LMT classifier to generate the rules, accuracy, and AUC for comparison. Because there were four datasets with imbalanced classes (asymmetric classes), the AUC was an important criterion. In the experiment, we collected four open medical datasets from UCI and one international stroke trial dataset. The results show that the proposed safe-region imputation is better than the listing imputation methods and after imputing offers better results than directly deleting instances with missing values in the number of rules, accuracy, and AUC. These results will provide a reference for medical stakeholders. [ABSTRACT FROM AUTHOR]

Building operation data are important for monitoring, analysis, modeling, and control of building energy systems. However, missing data is one of the major data quality issues, making data imputation techniques become increasingly important. There are two key research gaps for missing sensor data imputation in buildings: the lack of customized and automated imputation methodology, and the difficulty of the validation of data imputation methods. In this paper, a framework is developed to address these two gaps. First, a validation data generation module is developed based on pattern recognition to create a validation dataset to quantify the performance of data imputation methods. Second, a pool of data imputation methods is tested under the validation dataset to find an optimal single imputation method for each sensor, which is termed as an ensemble method. The method can reflect the specific mechanism and randomness of missing data from each sensor. The effectiveness of the framework is demonstrated by 18 sensors from a real campus building. The overall accuracy of data imputation for those sensors improves by 18.2% on average compared with the best single data imputation method. [ABSTRACT FROM AUTHOR]

The immense problem of missing satellite aerosol retrievals (Aerosol Optical Depth, (AOD)) detrimentally affects the prediction ability of ground-level PM2.5 concentrations and may lead to unavoidable biases. An appropriate missing-imputation method has not been well developed to date. This study developed a two-stage approach (AOD-imputation stage and PM2.5-prediction stage) to predict short-term PM2.5 exposure in mainland China from 2013–2018. At the AOD-imputation stage, geostatistical methods and machine learning (ML) algorithms were examined to interpolate 1 km satellite aerosol retrievals. At the PM2.5-prediction stage, the daily levels of PM2.5 were predicted at a resolution of 1 km, based on interpolated AOD and meteorological data. The statistical performances of the different interpolation methods were comprehensively compared at each stage. The original coverage of retrieved AOD was 15.46% on average. For the AOD-imputation stage, ML methods produced a higher coverage (98.64%) of AOD than geostatistical methods (21.43–87.31%). Among ML algorithms, random forest (RF) or extreme gradient boosted (XG-interpolated) AOD produced better interpolated quality (CV R2 = 0.89 and 0.85) than other algorithms (0.49–0.78), but XGBoost required only 15% of the computing time of RF. For the PM2.5 predicted stage, neither RF-AOD nor XG-AOD could guarantee higher accuracy in PM2.5 estimations (CV R2 = 0.88 (RF or XG-AOD) compared to 0.85 (original)), or more stable spatial and temporal extrapolation (spatial, (temporal) CV R2 = 0.83 (0.83), 0.82 (0.82), and 0.65 (0.61) for RF, XG, and original). For the AOD-imputation stage, the missing-filled efficiency depended more on external information, while the missing-filled accuracy relied more on model structure. For the PM2.5 predicted stage, efficient AOD interpolation (or the ability to eliminate the missing data) was a precondition for the stable spatial and temporal extrapolation, while the quality of interpolated AOD showed less significant improvements. It was found that XG-AOD is a better choice to estimate daily PM2.5 exposure in health assessments. [ABSTRACT FROM AUTHOR]

Granular computing techniques are a huge discipline in which the basic component is to operate on groups of similar objects according to a fixed similarity measure. The first references to the granular computing can be seen in the works of Zadeh in fuzzy set theory. Granular computing allows for a very natural modelling of the world. It is very likely that the human brain, while solving problems, performs granular calculations on data collected from the senses. The researchers of this paradigm have proven the unlimited possibilities of granular computing. Among other things, they are used in the processes of classification, regression, missing values handling, for feature selection, and as mechanisms of data approximation. It is impossible to quote all methods based on granular computing—we can only discuss a selected group of techniques. In the article, we have presented a review of recently developed granulation techniques belonging to the family of approximation algorithms founded by Polkowski—in the framework of rough set theory. Starting from the basic Polkowski's standard granulation, we have described further developed by us concept dependent, layered, and epsilon variants, and our recent homogeneous granulation. We are presenting simple numerical examples and samples of research results. The effectiveness of these methods in terms of decision system size reduction and maintenance of the internal knowledge from the original data are presented. The reduction in the number of objects in our techniques while maintaining classification efficiency reaches 90 percent—for standard granulation with usage of a kNN classifier (we achieve similar efficiency for the concept-dependent technique for the Naive Bayes classifier). The largest reduction achieved in the number of exhaustive set of rules at the efficiency level to the original data are 99 percent—it is for concept-dependent granulation. In homogeneous variants, the reduction is less than 60 percent, but the advantage of these techniques is that it is not necessary to look for optimal granulation parameters, which are selected dynamically. We also describe potential directions of development of granular computing techniques by prism of described methods. [ABSTRACT FROM AUTHOR]

Various traffic-sensing technologies have been employed to facilitate traffic control. Due to certain factors, e.g., malfunctioning devices and artificial mistakes, missing values typically occur in the Intelligent Transportation System (ITS) sensing datasets, resulting in a decrease in the data quality. In this study, an integrated imputation algorithm based on fuzzy C-means (FCM) and the genetic algorithm (GA) is proposed to improve the accuracy of the estimated values. The GA is applied to optimize the parameter of the membership degree and the number of cluster centroids in the FCM model. An experimental test of the taxi global positioning system (GPS) data in Manhattan, New York City, is employed to demonstrate the effectiveness of the integrated imputation approach. Three evaluation criteria, the root mean squared error (RMSE), correlation coefficient (R), and relative accuracy (RA), are used to verify the experimental results. Under the ±5% and ±10% thresholds, the average RAs obtained by the integrated imputation method are 0.576 and 0.785, which remain the highest among different methods, indicating that the integrated imputation method outperforms the history imputation method and the conventional FCM method. On the other hand, the clustering imputation performance with the Euclidean distance is better than that with the Manhattan distance. Thus, our proposed integrated imputation method can be employed to estimate the missing values in the daily traffic management. [ABSTRACT FROM AUTHOR]

Despite a rapid increase of public interest for electric mobility, several factors still impede Battery Electric Vehicles' (BEVs) acceptance. These factors include their limited range and inconvenient charging. For mitigating these limitations to users, certain BEV-specific services are required. Therefore, such services provide a reliable range prediction and routing, including charging-stop planning. The basis of these services is a precise and reliable Energy Demand (ED) prediction. For that matter, aggregated fleet-vehicle data combined with map-specific data (e.g., road slope) form an energetic map, which can serve for precise ED predictions. However, data coverage is paramount for these predictions, more specifically regarding gapless energetic maps. This work aims to eliminate the energetic map's gaps using two Machine Learning (ML) approaches: regression and classification. The proposed ML solution builds upon the synergy between map-information and crowdsourced driving profiles of 4.6 million kilometres of training and test traces. For evaluation, two test-scenarios capture the models' performance for the analysed problem in two perspectives. First, we evaluate our ML models, followed by the problem-specific energetic evaluation perspective for better interpretability. From the latter, the results indicate energetic map data imputation performs promisingly better when using the regression instead of the classification model. [ABSTRACT FROM AUTHOR]