Search

Most of the medical datasets suffer from missing data, due to the expense of some tests or human faults while recording these tests. This issue affects the performance of the machine learning models because the values of some features will be missing. Therefore, there is a need for a specific type of methods for imputing these missing data. In this research, the salp swarm algorithm (SSA) is used for generating and imputing the missing values in the pain in my ass (also known Pima) Indian diabetes disease (PIDD) dataset, the proposed algorithm is called (ISSA). The obtained results showed that the classification performance of three different classifiers which are support vector machine (SVM), K-nearest neighbour (KNN), and Naïve Bayesian classifier (NBC) have been enhanced as compared to the dataset before applying the proposed method. Moreover, the results indicated that issa was performed better than the statistical imputation techniques such as deleting the samples with missing values, replacing the missing values with zeros, mean, or random values.

Clustering method is a technique used for comparisons reduction between the candidates records in the duplicate detection process. The process of clustering records is affected by the quality of data. The more error-free the data, the more efficient the clustering algorithm, as data errors cause data to be placed in incorrect groups. Window algorithms suffer from the window size. The larger the window, the greater the number of unnecessary comparisons, and the smaller the window size may prevent the detection of duplicates that are supposed to be within the window. In this paper, we propose a data pre-processing method that increases the efficiency of window algorithms in grouping similar records together. In addition, the proposed method also deal s with the window size problem. In the proposed method, high-rank attributes are selected and then preparators are applied to the selected traits. A compensation algorithm is implemented to reduce the problem of missing and distorted sort keys. Two datasets (compact disc database (CDDB) and MusicBrainz) were used to test duplicates detection algorithms. The duplicates detection toolkit(DuDe) was used as a benchmark for the proposed method. Experiments showed that the proposed method achieved a high rate of accuracy in detecting duplicates. In addition, the proposed method.

[EN] Nowadays, there is an increasing concern in machine learning about the causes underlying unfair decision making, that is, algorithmic decisions discriminating some groups over others, especially with groups that are defined over protected attributes, such as gender, race and nationality. Missing values are one frequent manifestation of all these latent causes: protected groups are more reluctant to give information that could be used against them, sensitive information for some groups can be erased by human operators, or data acquisition may simply be less complete and systematic for minority groups. However, most recent techniques, libraries and experimental results dealing with fairness in machine learning have simply ignored missing data. In this paper, we present the first comprehensive analysis of the relation between missing values and algorithmic fairness for machine learning: (1) we analyse the sources of missing data and bias, mapping the common causes, (2) we find that rows containing missing values are usually fairer than the rest, which should discourage the consideration of missing values as the uncomfortable ugly data that different techniques and libraries for handling algorithmic bias get rid of at the first occasion, (3) we study the trade-off between performance and fairness when the rows with missing values are used (either because the technique deals with them directly or by imputation methods), and (4) we show that the sensitivity of six different machine-learning techniques to missing values is usually low, which reinforces the view that the rows with missing data contribute more to fairness through the other, nonmissing, attributes. We end the paper with a series of recommended procedures about what to do with missing data when aiming for fair decision making., Ministerio de Economia, Industria y Competitividad, Gobierno de Espana (ES), Grant/Award Number: RTI2018-094403-B-C3; Generalitat Valenciana, Grant/Award Number: PROMETEO/2019/09; Future of Life Institute, Grant/Award Number: RFP2-15; European Commission, Grant/Award Number: DG JRC - HUMAINT project

Missing values imputation is often evaluated on some similarity measure between actual and imputed data. However, it may be more meaningful to evaluate downstream algorithm performance after imputation than the imputation itself. We describe a straightforward unsupervised imputation algorithm, a minimax approach based on optimal recovery, and derive probabilistic error bounds on downstream non-negative matrix factorization (NMF). Under certain geometric conditions, we prove upper bounds on NMF relative error, which is the first bound of this type for missing values. We also give probabilistic bounds for the same geometric assumptions. Experiments on image data and biological data show that this theoretically-grounded technique performs as well as or better than other imputation techniques that account for local structure. We also comment on imputation fairness.

We consider the problem of building the relationship of high-energy electron flux between Geostationary Earth Orbit (GEO) and Medium Earth Orbit (MEO). A time-series decomposition technique is first applied to the original data, resulting in trend and detrended part for both GEO and MEO data. Then we predict MEO trend with GEO data using three machine learning models: Linear Regression (LR), Random Forest (RF), and Multi-Layer Perceptron (MLP). Experiment shows that RF gains best performance in all scenarios. Feature extraction analysis demonstrates that the inclusion of lagged features and (possible) ahead features is substantially helpful to the prediction. At last, an application of imputing missing values for MEO data is presented, in which RF model with selected features is used to handle the trend part while a moving block method is for the detrended part.

Pristine and trustworthy data are required for efficient computer modelling for medical decision-making, yet data in medical care is frequently missing. As a result, missing values may occur not just in training data but also in testing data that might contain a single undiagnosed episode or a participant. This study evaluates different imputation and regression procedures identified based on regressor performance and computational expense to fix the issues of missing values in both training and testing datasets. In the context of healthcare, several procedures are introduced for dealing with missing values. However, there is still a discussion concerning which imputation strategies are better in specific cases. This research proposes an ensemble imputation model that is educated to use a combination of simple mean imputation, k-nearest neighbour imputation, and iterative imputation methods, and then leverages them in a manner where the ideal imputation strategy is opted among them based on attribute correlations on missing value features. We introduce a unique Ensemble Strategy for Missing Value to analyse healthcare data with considerable missing values to identify unbiased and accurate prediction statistical modelling. The performance metrics have been generated using the eXtreme gradient boosting regressor, random forest regressor, and support vector regressor. The current study uses real-world healthcare data to conduct experiments and simulations of data with varying feature-wise missing frequencies indicating that the proposed technique surpasses standard missing value imputation approaches as well as the approach of dropping records holding missing values in terms of accuracy.

In many real-world applications scientists are often confronted with the problem of incomplete datasets due to several reasons. The direct analysis of datasets with missing values in attributes inevitably results in inaccurate learning models and erroneous results. Facing effectively the challenge of missing values is an essential step of the data mining process. Imputation is often employed to overcome the shortcomings incurred by missing data during the pre-process stage of data analysis. Therefore, a plethora of statistical and machine learning methods have been proposed and employed with a view to imputing the missing values in incomplete data with their potential or actual values. In this context, the main objective of this paper is to put forward an iterative stepwise imputation method based on the semi-supervised learning approach, called IRSSI. Semi-supervised methods have proved to be particularly effective for exploiting incomplete or partially labeled data with regard to the values of the target attribute. The proposed algorithm was experimentally evaluated on real-world benchmark datasets and artificially generated datasets using different high ratios of missing data. The experimental results demonstrate the efficiency of IRSSI algorithm compared to typical imputation methods.

Rapid expansion of the world’s population has negatively impacted the environment, notably water quality. As a result, water-quality prediction has arisen as a hot issue during the last decade. Existing techniques fall short in terms of good accuracy. Furthermore, presently, the dataset available for analysis contains missing values; these missing values have a significant effect on the performance of the classifiers. An automated system for water-quality prediction that deals with the missing values efficiently and achieves good accuracy for water-quality prediction is proposed in this study. To handle the accuracy problem, this study makes use of the stacked ensemble H2O AutoML model; to handle the missing values, this study makes use of the KNN imputer. Moreover, the performance of the proposed system is compared to that of seven machine learning algorithms. Experiments are performed in two scenarios: removing missing values and using the KNN imputer. The contribution of each feature regarding prediction is explained using SHAP (SHapley Additive exPlanations). Results reveal that the proposed stacked model outperforms other models with 97% accuracy, 96% precision, 99% recall, and 98% F1-score for water-quality prediction.

High-dimensional omics datasets frequently contain missing data points, which typically occur due to concentrations below the limit of detection (LOD) of the profiling platform. The presence of such missing values significantly limits downstream statistical analysis and result interpretation. Two common techniques to deal with this issue include the removal of samples with missing values and imputation approaches that substitute the missing measurements with reasonable estimates. Both approaches, however, suffer from various shortcomings and pitfalls. In this paper, we present “rox”, a novel statistical model for the analysis of omics data with missing values without the need for imputation. The model directly incorporates missing values as “low” concentrations into the calculation. We show the superiority of rox over common approaches on simulated data and on six metabolomics datasets. Fully leveraging the information contained in LOD-based missing values, rox provides a powerful tool for the statistical analysis of omics data.

As the application of information technology in the medical field is resulting in a large amount of medical data. As early withdrawal and refusal of participants, there are a lot of missing values in medical data. Although various processing methods for missing values have been proposed, few methods for those medical data with characteristics of imbalance and mixed-type data. In this work, we proposed an adaptive Laplacian weight random forest, called ALWRF. In ALWRF, feature weights were adjusted dynamically when model constructing, which increases selection probabilities of features with low Laplacian score and high importance. Meanwhile, a random operator is introduced to increase the diversity of trees. Furthermore, we proposed an imputation method based on SMOTE-NC oversampling technology and the ALWRF method for imbalanced and mixed-type data, called SncALWRFI. Meanwhile, Bayesian optimization and cross-validation were employed to search optimal parameters. The experimental results showed that the ALWRF method outperforms random forest and Bayesian optimized random forest in terms of classification and regression accuracy. Further, in the experiment for missing values, the SncALWRFI showed the best imputation accuracy, and it performed high imputation effectiveness in public datasets with characteristics of imbalanced and mixed-type. 2022 Elsevier Ltd Author Lijuan REN is grateful for the support of China Scholarship Council (CSC). In addition, this research is supported by the Sichuan Science and Technology Program of China (No. 2021YFH0107 ). All the authors are thankful to their respective universities for their support: University of Lyon, Chengdu University of Information Technology and Qatar University. Scopus

Missing value (MV) is one form of data completeness problem in massive datasets. To deal with missing values, data imputation methods were proposed with the aim to improve the completeness of the datasets concerned. Data imputation's accuracy is a common indicator of a data imputation technique's efficiency. However, the efficiency of data imputation can be affected by the nature of the language in which the dataset is written. To overcome this problem, it is necessary to normalize the data, especially in non-Latin languages such as the Arabic language. This paper proposes a method that will address the challenge inherent in Arabic datasets by extending the enhanced robust association rules (ERAR) method with Arabic detection and correction functions. Iterative and Decision Tree methods were used to evaluate the proposed method in an experiment. Experiment results show that the proposed method offers a higher data imputation accuracy than the Iterative and Decision Tree methods.

Remaining useful life (RUL) prediction is recently a hot spot in industrial big data analysis research. It aims at obtaining the health status of the equipment in advance and making intelligent maintenance decisions. However, values missing is a common problem in real industrial applications which severely restricts the performance and application scope of RUL prediction. To deal with this problem, a novel prediction model called self-attention-based multi-task network (SMTN)is proposed. The spatiotemporal feature fusion module utilizes the self-attention mechanism and long short-term memory to fully exploit the information in space and time dimensions, multi-task learning module tries to learn a complete representation from incomplete data by performing the missing values imputation task, and the representation is simultaneously used for RUL prediction. Comparison experiments conducted on the C-MAPSS dataset verified the effectiveness of the proposed SMTN.

Complete hydrological time series are necessary for water resources management and modeling. This can be challenging in data scarce environments where data gaps are ubiquitous. In many applications, repetitive gaps can have unfortunate consequences including ineffective model calibration, unreliable timing of peak flows, and biased statistics. Here, Direct Sampling (DS) is used as a non-parametric stochastic method for infilling gaps in daily streamflow records. A thorough gap-filling framework including the selection of predictor stations and the optimization of the DS parameters is developed and applied to data collected in the Volta River basin, West Africa. Various synthetic missing data scenarios are developed to assess the performance of the method, followed by a real-case application to the existing gaps in the flow records. The contribution of this study includes the assessment of the method for different climatic zones and hydrological regimes and for different upstream-downstream relations among the gauging stations used for gap filling. Tested in various missing data conditions, the method allows a precise and reliable simulation of the missing data by using the data patterns available in other stations as predictor variables. The developed gap-filling framework is transferable to other hydrological applications, and it is promising for environmental modeling.

Real-time underwater monitoring has been widely applied in many applications of underwater wireless sensor networks (UWSNs). Due to the long acoustic communication delays, the real-time data collection in UWSNs is challenging. Moreover, the underwater acoustic transmission faces the problem of high data loss rate, which causes a longer delay time due to the need for packet retransmissions. To address these problems, we propose a recurrent neural network (RNN)-based underwater monitoring framework with the consideration of delay, energy, and data quality. We drop the automatic retransmission mechanism applied in the MAC protocols to reduce the long end-to-end delay and energy cost. Facing high data loss, we propose an efficient RNN learning model, LSTM-Decay, to analyze the raw data with the time-related decay weights features and predict the missing values. The experiments with the real-world underwater sensing datasets show that our learning model can achieve an accurate estimation with different degrees of missing rates and can provide better performance compared with the non-RNN and RNN baselines.

BackgroundMachine learning (ML) models can improve prediction of major adverse cardiovascular events (MACE), but in clinical practice some values may be missing. We evaluated the influence of missing values in ML models for patient-specific prediction of MACE risk.MethodsWe included 20,179 patients from the multicenter REFINE SPECT registry with MACE follow-up data. We evaluated seven methods for handling missing values: 1) removal of variables with missing values (ML-Remove), 2) imputation with median and unique category for continuous and categorical variables, respectively (ML-Traditional), 3) unique category for missing variables (ML-Unique), 4) cluster-based imputation (ML-Cluster), 5) regression-based imputation (ML-Regression), 6) missRanger imputation (ML-MR), and 7) multiple imputation (ML-MICE). We trained ML models with full data and simulated missing values in testing patients. Prediction performance was evaluated using area under the receiver-operating characteristic curve (AUC) and compared with a model without missing values (ML-All), expert visual diagnosis and total perfusion deficit (TPD).ResultsDuring mean follow-up of 4.7±1.5 years, 3,541 patients experienced at least one MACE (3.7% annualized risk). ML-All (reference model-no missing values) had AUC 0.799 for MACE risk prediction. All seven models with missing values had lower AUC (ML-Remove: 0.778, ML-MICE: 0.774, ML-Cluster: 0.771, ML-Traditional: 0.771, ML-Regression: 0.770, ML-MR: 0.766, and ML-Unique: 0.766; p&nbsp

In high-stakes applications of data-driven decision making like healthcare, it is of paramount importance to learn a policy that maximizes the reward while avoiding potentially dangerous actions when there is uncertainty. There are two main challenges usually associated with this problem. Firstly, learning through online exploration is not possible due to the critical nature of such applications. Therefore, we need to resort to observational datasets with no counterfactuals. Secondly, such datasets are usually imperfect, additionally cursed with missing values in the attributes of features. In this paper, we consider the problem of constructing personalized policies using logged data when there are missing values in the attributes of features in both training and test data. The goal is to recommend an action (treatment) when $\Xt$, a degraded version of $\Xb$ with missing values, is observed. We consider three strategies for dealing with missingness. In particular, we introduce the \textit{conservative strategy} where the policy is designed to safely handle the uncertainty due to missingness. In order to implement this strategy we need to estimate posterior distribution $p(\Xb|\Xt)$, we use variational autoencoder to achieve this. In particular, our method is based on partial variational autoencoders (PVAE) which are designed to capture the underlying structure of features with missing values.

Missing data or missing values are a common phenomenon in applied panel data research and of great interest for panel data unit root testing. The standard approach in the literature is to balance the panel by removing units and/or trimming a common time period for all units. However, this approach can be costly in terms of lost information. Instead, existing panel unit root tests could be extended to the case of unbalanced panels, but this is often difficult because the missing observations affect the bias correction which is usually involved. This paper contributes to the literature in two ways; it extends two popular panel unit root tests to allow for missing values, and secondly, it employs asymptotic local power functions to analytically study the impact of various missing-value methods on power. We find that zeroing-out the missing observations is the method that results in the greater test power, and that this result holds for all deterministic component specifications, such as intercepts, trends and structural breaks.

Isobaric labeling has the promise of combining high sample multiplexing with precise quantification. However, normalization issues and the missing value problem of complete n-plexes hamper quantification across more than one n-plex. Here, we introduce two novel algorithms implemented in MaxQuant that substantially improve the data analysis with multiple n-plexes. First, isobaric matching between runs makes use of the three-dimensional MS1 features to transfer identifications from identified to unidentified MS/MS spectra between liquid chromatography-mass spectrometry runs in order to utilize reporter ion intensities in unidentified spectra for quantification. On typical datasets, we observe a significant gain in MS/MS spectra that can be used for quantification. Second, we introduce a novel PSM-level normalization, applicable to data with and without the common reference channel. It is a weighted median-based method, in which the weights reflect the number of ions that were used for fragmentation. On a typical dataset, we observe complete removal of batch effects and dominance of the biological sample grouping after normalization. Furthermore, we provide many novel processing and normalization options in Perseus, the companion software for the downstream analysis of quantitative proteomics results. All novel tools and algorithms are available with the regular MaxQuant and Perseus releases, which are downloadable at http://maxquant.org. Notes The MS proteomics data have been deposited to the ProteomeXchange consortium (http://proteomecentral.proteomexchange.org) via the PRIDE partner repository with the dataset identifier PXD019880 and PXD019881.

Missing values are a huge constraint in microarray technologies towards improving and identifying disease-causing genes. Estimating missing values is an undeniable scenario faced by field experts. The imputation method is an effective way to impute the proper values to proceed with the next process in microarray technology. Missing value imputation methods may increase the classification accuracy. Although these methods might predict the values, classification accuracy rates prove the ability of the methods to identify the missing values in gene expression data. In this study, a novel method, Optimised Hybrid of Fuzzy C-Means and Majority Vote (opt-FCMMV), was proposed to identify the missing values in the data. Using the Majority Vote (MV) and optimisation through Particle Swarm Optimisation (PSO), this study predicted missing values in the data to form more informative and solid data. In order to verify the effectiveness of opt-FCMMV, several experiments were carried out on two publicly available microarray datasets (i.e. Ovary and Lung Cancer) under three missing value mechanisms with five different percentage values in the biomedical domain using Support Vector Machine (SVM) classifier. The experimental results showed that the proposed method functioned efficiently by showcasing the highest accuracy rate as compared to the one without imputations, with imputation by Fuzzy C-Means (FCM), and imputation by Fuzzy C-Means with Majority Vote (FCMMV). For example, the accuracy rates for Ovary Cancer data with 5% missing values were 64.0% for no imputation, 81.8% (FCM), 90.0% (FCMMV), and 93.7% (opt-FCMMV). Such an outcome indicates that the opt-FCMMV may also be applied in different domains in order to prepare the dataset for various data mining tasks.

Proceedings of: International Work conference on the Interplay between Natural and Artificial Computation (IWINAC 2015) Missing values in a dataset is one of the most common difficulties in real applications. Many different techniques based on machine learning have been proposed in the literature to face this problem. In this work, the great representation capability of the stacked denoising auto-encoders is used to obtain a new method of imputating missing values based on two ideas: deletion and compensation. This method improves imputation performance by artificially deleting values in the input features and using them as targets in the training process. Nevertheless, although the deletion of samples is demonstrated to be really efficient, it may cause an imbalance between the distributions of the training and the test sets. In order to solve this issue, a compensation mechanism is proposed based on a slight modification of the error function to be optimized. Experiments over several datasets show that the deletion and compensation not only involve improvements in imputation but also in classification in comparison with other classical techniques. The work of A. R. Figueiras-Vidal has been partly supported by Grant Macro-ADOBE (TEC 2015-67719-P, MINECO/FEDER&FSE). The work of J.L. Sancho-Gómez has been partly supported by Grant AES 2017 (PI17/00771, MINECO/FEDER).

This paper is aimed at deriving some specific-oriented bootstrap confidence intervals for missing sequences of observations in multivariate time series. The procedure is based on a spatial-dynamic model and imputes the missing values using a linear combination of the neighbor contemporary observations and their lagged values. The resampling procedure implements a residual bootstrap approach which is then used to approximate the sampling distribution of the estimators of the missing values. The normal based and the percentile bootstrap confidence intervals have been computed. A Monte Carlo simulation study shows the good empirical coverage performance of the proposal, even in the case of long sequences of missing values.

En el camp de l'aprenentatge automàtic, sovint ens trobem amb conjunts de dades incomplets i necessitem l'ús de mètodes de tractament dades que falten per superar aquest problema. Tanmateix, algunes tècniques d'aprenentatge automàtic només requereixen el càlcul de distàncies o funcions de kernel. Per tant, cal imputar primer els valors perduts per calcular posteriorment les distàncies o és millor estimar directament les distàncies sense haver d'imputar? En aquest treball, realitzem un estudi comparatiu de diversos mètodes de tractament de dades perdudes aplicats a problemes de classificació mitjançant l'anomenada Radial Basis Function Neural Network per veure si s'ha d'utilitzar sempre un pas previ d'imputació o no. L'estudi s'ha dut a terme mitjançant dades sintètiques generant valors perduts artificialment, i dades del món real amb valors perduts de forma natural i induïts per nosaltres. A més, hem realitzat una anàlisi de la complexitat dels mètodes usats en termes de temps i espai. Els resultats experimentals mostren que predir les distàncies sense haver d'imputar dóna millors resultats que imputar primer i després calcular la distància. Per tant, només s'haurien d'utilitzar els mètodes d'imputació quan fos estrictament necessari, i no per resoldre qualsevol problema. Per concloure, també introduïm la necessitat de calcular una cota superior pels diversos mètodes, i exposem dos casos diferents en els quals calculem aquesta cota superior. In the machine learning field, we are frequently faced with incomplete datasets, and require the use of missing data treatment methods to overcome this problem. However, some machine learning techniques only require the computation of distances or kernel functions. Hence, is it necessary to impute the missing entries first for later computing the distances or is it better to estimate the distances directly without imputing? In this work, we perform a comparative study of several missing data treatment methods applied to classification problems using Radial Basis Function Neural Networks to see if it should always be used as a preprocessing imputation step or not. The study has been conducted using synthetic data with artificial missing values and realworld classification datasets with both natural and induced missing values. Moreover, we have done a complexity analysis in terms of time and space complexities for the methods used. Experimental results show that predicting distances without having to impute gives better results than imputing first and then calculating the distance. Therefore, imputation methods should be used only when strictly necessary, and not to solve any problem you are facing. To conclude, we also introduce the need to compute an upper bound for the missing data treatment methods, and expose two different case scenarios in which we compute this upper bound. En el campo del aprendizaje automático, a menudo nos encontramos con conjuntos de datos incompletos y necesitamos el uso de métodos de tratamiento de los valores perdidos para superar este problema. No obstante, algunas técnicas de aprendizaje automático solo requieren el cálculo de distancias o funciones de kernel. Por lo tanto, ¿es necesario imputar primero los valores perdidos para calcular posteriormente las distancias o es mejor predecir directamente las distancias sin tener que imputar primero? En este trabajo, realizamos un estudio comparativo de varios métodos de tratamiento de datos perdidos aplicados a problemas de clasificación mediante la llamada Radial Basis Function Neural Network para ver si se debe utilizar siempre un paso previo de imputación o no. El estudio se ha llevado a cabo mediante el uso de datos sintéticos generando valores perdidos artificialmente, y datos del mundo real con valores perdidos de forma natural e inducidos por nosotros. Además, hemos realizado un análisis de la complejidad de los métodos usados en términos de tiempo y espacio. Los resultados experimentales muestran que predecir las distancias sin tener que imputar da mejores resultados que imputar primero para después calcular la distancia. Por lo tanto, solo deberían utilizarse los métodos de imputación cuando fuera estrictamente necesario, y no para solventar cualquier problema. Para concluir, también introducimos la necesidad de calcular una cota superior para los diversos métodos, y exponemos dos casos diferentes en los que calculamos esta cota superior.

Examinees differ in how they interact with assessments. In low-stakes large-scale assessments (LSAs), missing responses pose an obvious example of such differences. Understanding the underlying mechanisms is paramount for making appropriate decisions on how to deal with missing responses in data analysis and drawing valid inferences on examinee competencies. Against this background, the present work aims at providing approaches for a nuanced modeling and understanding of test-taking behavior associated with the occurrence of missing responses in LSAs. These approaches are aimed at a) improving the treatment of missing responses in LSAs, b) supporting a better understanding of missingness mechanisms in particular and examinee test-taking behavior in general, and c) considering differences in test-taking behavior underlying missing responses when drawing inferences about examinee competencies. To that end, the present work leverages the additional information contained in response times and integrates research on modeling missing responses with research on modeling response times associated with observed responses. By documenting lengths of interactions, response times contain valuable information on how examinees interact with assessments and may as such critically contribute to understanding the processes underlying both observed and missing responses. This work presents four modeling approaches that focus on different aspects and mechanisms of missing responses. The first two approaches focus on modeling not-reached items. The second two approaches aim at modeling omitted items. The first approach employs the framework for the joint modeling of speed and ability by van der Linden (2007) for modeling the mechanism underlying not-reached items due to lack of working speed. On the basis of both theoretical considerations as well as a comprehensive simulation study, it is argued that by accounting for differences in speed this framework is well suited for modeling the mechanism underlying not-reached items due to lack thereof. In assessing empirical test-level response times, it is, however, also illustrated that some examinees quit the assessment before reaching the end of the test or being forced to stop working due to a time limit. Building on these results, the second approach of this work aims at disentangling and jointly modeling multiple mechanisms underlying not-reached items. Employing information on response times, not-reached items due to lack of speed are distinguished from not-reached items due to quitting. The former is modeled by considering examinee speed. Quitting behavior - defined as stopping to work before the time limit is reached while there are still unanswered items - is modeled as a survival process, with the item position at which examinees are most likely to quit being governed by their test endurance, conceptualized as a third latent variable besides speed and ability. The third approach presented in this work focuses on jointly modeling omission behavior and response behavior, thus providing a better understanding of how these two types of behavior differ. For doing so, the approach extends the framework for jointly modeling speed and ability by a model component for the omission process and introduces the concept of different speed levels examinees operate on when generating responses and omitting items. This approach supports a more nuanced understanding of both the missingness mechanism underlying omissions and examinee pacing behavior through assessment of whether examinees employ different pacing strategies when generating responses or omitting items The fourth approach builds on previous theoretical work relating omitted responses to examinee disengagement and provides a model-based approach that allows for identifying and modeling examinee disengagement in terms of both omission and guessing behavior. Disengagement is identified at the item-by-examinee level by employing a mixture modeling approach that allows for different data-generating processes underlying item responses and omissions as well as different distributions of response times associated with engaged and disengaged behavior. Item-by-examinee mixing proportions themselves are modeled as a function of additional person and item parameters. This allows relating disengagement to ability and speed as well as identifying items that are likely to evoke disengaged test-taking behavior. The approaches presented in this work are tested and illustrated by a) evaluating their statistical performance under conditions typically encountered in LSAs by means of comprehensive simulation studies, b) illustrating their advances over previously developed approaches, and c) applying them to real data from major LSAs, thereby illustrating their potential for understanding examinee test-taking behavior in general and missingness mechanisms in particular. The potential of the approaches developed in this work for deepening the understanding of results from LSAs is discussed and implications for the improvement of assessment procedures - ranging from construction and administration to analysis, interpretation and reporting - are derived. Limitations of the proposed approaches are discussed and suggestions for future research are provided.

With the steady growth of energy demands and resource depletion in today’s world, energy prediction models have gained more and more attention recently. Reducing energy consumption and carbon footprint are critical factors for achieving efficiency in sustainable cities. Unfortunately, traditional energy prediction models focus only on prediction performance. However, explainable models are essential to building trust and engaging users to accept AI-based systems. In this paper, we propose an explainable deep learning model, called Expect, to forecast energy consumption from time series effectively. Our results demonstrate our proposal’s robustness and accuracy when compared to the baseline methods.

High density air monitoring campaigns provide spatial patterns of pollutant concentrations which are integral in exposure assessment. Such analysis can assist with the determination of links between air quality and health outcomes, however, problems due to missing data can threaten to compromise these studies. This research evaluates four methods; mean value imputation, inverse distance weighting (IDW), inter-species ratios, and regression, to address missing spatial concentration data ranging from one missing data point up to 50% missing data. BTEX (benzene, toluene, ethylbenzene, and xylenes) concentrations were measured in Windsor and Sarnia, Ontario in the fall of 2005. Concentrations and inter-species ratios were generally similar between the two cities. Benzene (B) was observed to be higher in Sarnia, whereas toluene (T) and the T/B ratios were higher in Windsor. Using these urban, industrialized cities as case studies, this research demonstrates that using inter-species ratios or regression of the data for which there is complete information, along with one measured concentration (i.e. benzene) to predict for missing concentrations (i.e. TEX) results in good agreement between predicted and measured values. In both cities, the general trend remains that best agreement is observed for the leave-one-out scenario, followed by 10% and 25% missing, and the least agreement for the 50% missing cases. In the absence of any known concentrations IDW can provide reasonable agreement between observed and estimated concentrations for the BTEX species, and was superior over mean value imputation which was not able to preserve the spatial trend. The proposed methods can be used to fill in missing data, while preserving the general characteristics and rank order of the data which are sufficient for epidemiologic studies.

Pairwise comparison matrix as a crucial component of AHP, presents the prefer- ence relations among alternatives. However, in many cases, the pairwise comparison matrix is difficult to complete, which obstructs the subsequent operations of the clas- sical AHP. In this paper, based on DEMATEL which has ability to derive the total relation matrix from direct relation matrix, a new completion method for incomplete pairwise comparison matrix is proposed. The proposed method provides a new per- spective to estimate the missing values with explicit physical meaning. Besides, the proposed method has low computational cost. This promising method has a wide application in multi-criteria decision-making., Comment: 29 pages

BACKGROUND: The number of contingent valuation (CV) studies in dental medicine using willingness-to-pay (WTP) methodology has substantially increased in recent years. Missing values due to absent information (i.e., missingness) or false information (i.e., protest zeros) are a common problem in WTP studies. The objective of this study is to evaluate the prevalence of missing values in CV studies in dental medicine, to assess how these have been dealt with, and to suggest recommendations for future research. METHODS: We systematically searched electronic databases (MEDLINE, Web of Science, Cochrane Library, PROSPERO) on 8 June 2021, and hand-searched references of selected reviews. CV studies in clinical dentistry using WTP for valuing a good or service were included. RESULTS: We included 49 WTP studies in our review. Out of these, 19 (38.8%) reported missing values due to absent information, and 28 (57.1%) reported zero values (i.e., WTP valued at zero). Zero values were further classified into true zeros (i.e., representing the underlying preference of the respondent) or protest zeros (i.e., false information as a protest behavior) in only 9 studies. Most studies used a complete case analysis to address missingness while only one study used multiple imputation. CONCLUSIONS: There is uncertainty in the dental literature on how to address missing values and zero values in CV studies. Zero values need to be classified as true zeros versus protest zeros with follow-up questions after the WTP elicitation procedure, and then need to be handled differently. Advanced statistical methods are available to address both missing values due to missingness and due to protest zeros but these are currently underused in dental medicine. Failing to appropriately address missing values in CV studies may lead to biased WTP estimates of dental interventions. ispartof: INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH vol:18 issue:14 ispartof: location:Switzerland status: published

In this study, it is aimed to investigate the impact of different missing data handling methods on the detection of Differential Item Functioning methods (Mantel Haenszel and Standardization methods based on Classical Test Theory and Likelihood Ratio Test method based on Item Response Theory). In this regard, on the data acquired from 1046 candidates who entered to Foreign National Student Exam (FNSE) held in year 2016 by Mersin University (MEU) and answered Basic Skills subtest, using different missing data handling methods, differential item functioning analyses with Mantel Haenszel, Standardization and Likelihood Ratio Test methods are performed. Basic Skills test consists of 80 multiple choice items. The items are all binary scored (1-0) items. Among the participants 523 are female and 523 are male. The findings showed that the number of items flagged as DIF has changed with the used missing data handling methods. The DIF detection methods based on Classical Test Theory are more consistent within themselves compared to DIF detection method based on Item Response Theory, whereas the used missing data handling methods differentiate the DIF detected items and this difference reaches a significant level for Mantel Haenszel method

Missing values in data are common in real world applications. There are several methods that deal with this problem. In this paper we present lookahead selective sampling (LSS) algorithms for datasets with missing values. We developed two versions of selective sampling. The first one integrates a distance function that can measure the similarity between pairs of incomplete points within the framework of the LSS algorithm. The second algorithm uses ensemble clustering in order to represent the data in a cluster matrix without missing values and then run the LSS algorithm based on the ensemble clustering instance space (LSS-EC). To construct the cluster matrix, we use the k-means and mean shift clustering algorithms especially modified to deal with incomplete datasets. We tested our algorithms on six standard numerical datasets from different fields. On these datasets we simulated missing values and compared the performance of the LSS and LSS-EC algorithms for incomplete data to two other basic methods. Our experiments show that the suggested selective sampling algorithms outperform the other methods.

© 2019 Association for Computing Machinery. The goal of this work is to investigate the impact of missing values in clustering joint categorical social sequences. Identifying patterns in sociodemographic longitudinal data is important in a number of social science settings. However, performing analytical operations, such as clustering on life course trajectories, is challenging due to the categorical and multidimensional nature of the data, their mixed data types, and corruption by missing and inconsistent values. Data quality issues were investigated previously on single variable sequences. To understand their effects on multivariate sequence analysis, we employ a dataset of mixed data types and missing values, a dissimilarity measure designed for joint categorical sequence data, together with dimensionality reduction methodologies in a systematic design of sequence clustering experiments. Given the categorical nature of our data, we employ an “edit” distance using optimal matching. Because each data record has multiple variables of different types, we investigate the impact of mixing these variables in a single dissimilarity measure. Between variables with binary values and those with multiple nominal values, we find that the ability to overcome missing data problems is more difficult in the nominal domain than in the binary domain. Additionally, alignment of leading missing values can result in systematic biases in dissimilarity matrices and subsequently introduce both artificial clusters and unrealistic interpretations of associated data domains. We demonstrate the usage of t-distributed stochastic neighborhood embedding to visually guide mitigation of such biases by tuning the missing value substitution cost parameter or determining an optimal sequence span.

BACKGROUND: Self-rated health (SRH) is a health measure used in studies of older adults. The objective of this study is to analyze SRH as a predictor of mortality in the institutionalized older population and the characteristics of those who do not provide information about their SRH on health questionnaires. METHODS: This is a 15-year follow-up study of older adult residents in nursing or care homes in of Madrid, Spain. SRH was measured on a 5-point Likert type scale. The association between answering the SRH question and socio-demographic and health characteristics was evaluated through prevalence ratio (PR), estimated by Poisson regression models. Survival rates associated with SRH were studied through a multivariate Cox regression. RESULTS: The sample has a mean age of 83.4 (standard deviation, SD = 7.3), with 75.7% women. Twelve percent did not answer the SRH item. Those who did not answer showed a higher probability of disability (Barthel index, PR = 0.76, 95% confidence interval = 0.67-0.86) and/or dementia (PR = 8.03, 3.38-19.03). A trend for higher mortality was observed in those persons who did not respond (adjusted hazard ratio HR = 1.26, 0.75-2.11). The mortality rate was 32% higher for those who declared poor SRH in comparison with those who reported good SRH (adjusted HR = 1.32, 1.08-1.6). CONCLUSIONS: There is an elevated number of people who do not respond to the SRH item, mainly those with disabilities and cognitive deterioration. Lack of response to SRH is a good indicator of 15-year mortality for persons institutionalized in care or nursing homes. This work was supported by the Institute of Health Carlos III (grant number, PI15CIII00037) and partially supported by the ENCAGE-CM programme (ref: S2015/HUM-3367, co-funded by Madrid Community and FEDER). The funding bodies did not have any role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript. Sí

Predicting the progression of Alzheimer’s Disease (AD) has been held back for decades due to the lack of sufficient longitudinal data required for the development of novel machine learning algorithms. This study proposes a novel machine learning algorithm for predicting the progression of Alzheimer’s disease using a distributed multimodal, multitask learning method. More specifically, each individual task is defined as a regression model, which predicts cognitive scores at a single time point. Since the prediction tasks for multiple intervals are related to each other in chronological order, multitask regression models have been developed to track the relationship between subsequent tasks. Furthermore, since subjects have various combinations of recording modalities together with other genetic, neuropsychological and demographic risk factors, special attention is given to the fact that each modality may experience a specific sparsity pattern. The model is hence generalized by exploiting multiple individual multitask regression coefficient matrices for each modality. The outcome for each independent modality-specific learner is then integrated with complementary information, known as risk factor parameters, revealing the most prevalent trends of the multimodal data. This new feature space is then used as input to the gradient boosting kernel in search for a more accurate prediction. This proposed model not only captures the complex relationships between the different feature representations, but it also ignores any unrelated information which might skew the regression coefficients. Comparative assessments are made between the performance of the proposed method with several other well-established methods using different multimodal platforms. The results indicate that by capturing the interrelatedness between the different modalities and extracting only relevant information in the data, even in an incomplete longitudinal dataset, will yield minimized prediction errors.

Nowadays, the veracity related with data quality such as incomplete, inconsistent, vague or noisy data creates a major challenge to data mining and data analysis. Rough set theory presents a special tool for handling the incomplete and imprecise data in information systems. In this paper, rough set based matrix-represented approximations are presented to compute lower and upper approximations. The induced approximations are conducted as inputs for data analysis method, LERS (Learning from Examples based on Rough Set) used with LEM2 (Learning from Examples Module, Version2) rule induction algorithm. Analyzes are performed on missing datasets with “do not care” conditions and missing datasets with lost values. In addition, experiments on missing datasets with different missing percent by using different thresholds are also provided. The experimental results show that the system outperforms when missing data are characterized as “do not care” conditions than represented as lost values.

In this paper we recast the problem of missing values in the covariates of a regression model as a latent Gaussian Markov random field (GMRF) model in a fully Bayesian framework. Our proposed approach is based on the definition of the covariate imputation sub-model as a latent effect with a GMRF structure. We show how this formulation works for continuous covariates and provide some insight on how this could be extended to categorical covariates. The resulting Bayesian hierarchical model naturally fits within the integrated nested Laplace approximation (INLA) framework, which we use for model fitting. Hence, our work fills an important gap in the INLA methodology as it allows to treat models with missing values in the covariates. As in any other fully Bayesian framework, by relying on INLA for model fitting it is possible to formulate a joint model for the data, the imputed covariates and their missingness mechanism. In this way, we are able to tackle the more general problem of assessing the missingness mechanism by conducting a sensitivity analysis on the different alternatives to model the non-observed covariates. Finally, we illustrate the proposed approach with two examples on modeling health risk factors and disease mapping. Here, we rely on two different imputation mechanisms based on a typical multiple linear regression and a spatial model, respectively. Given the speed of model fitting with INLA we are able to fit joint models in a short time, and to easily conduct sensitivity analyses.

Holistic human proteome maps are expected to complement comprehensive profile assessment of health and disease phenotypes. However, methodologies to analyze proteomes in human tissue or body fluid samples at relevant scale and performance are still limited in clinical research. Their deployment and demonstration in large enough human populations are even sparser. In the present study, we have characterized and compared the plasma proteomes of two large independent cohorts of obese and overweight individuals using shotgun mass spectrometry (MS)-based proteomics. Herein, we showed, in both populations from different continents of about 500 individuals each, the concordance of plasma protein MS measurements in terms of variability, gender-specificity, and age-relationship. Additionally, we replicated several known and new associations between proteins, clinical and molecular variables, such as insulin and glucose concentrations. In conclusion, our MS-based analyses of plasma samples from independent human cohorts proved the practical feasibility and efficiency of a large and unified discovery/replication approach in proteomics, which was also recently coined “rectangular” design.

Summary We present an integration based procedure for predicting the distribution f of an indicator of interest in situations where, in addition to the sample data, one has access to covariates that are available for the entire population. The proposed method, based on similar ideas that have been used in the literature on policy evaluation, provides an alternative to existing simulation and imputation methods. It is very simple to apply, flexible, requires no additional assumptions, and does not involve the inclusion of artificial random terms. It therefore yields reproducible estimates and allows for valid inference. It also provides a tool for future predictions, scenarios and ex-ante impact evaluation. We illustrate our procedure by predicting income distributions in a case with sample selection, and both current and future doctor visits. We find our approach outperforms other commonly used procedures substantially.

Novel classifier for the diagnosis of Aldosterone-Producing Adenoma.Improved accuracy (as measured by EUC metric) with respect to state of art.Feature correlation, missing values and class imbalance all effectively managed.Validation through a novel, robust technique over a large dataset of patients. We develop a novel classifier for the diagnosis of Aldosterone-Producing Adenoma (APA), which induces Primary Aldosteronism, the most common endocrine cause of curable hypertension. The classifier considerably improves upon the state of the art, and it is devised and tested on a large dataset of patients, each described by several demographic and biochemical features. As customary in real-world datasets, ours is affected by feature correlation, missing values, and class imbalance. We make explicit provisions for dealing with all of these issues through an ensemble of ensembles, that is, a multilevel fusion of different component classifiers. Using the Wilcoxon signed-rank test at 0.05 significance level, we show that our ensemble significantly outperforms the state-of-the-art classifier and any individual component in the ensemble. Our experiments employ a "leave-one-out-clinical" cross validation as patients were treated in 15 different specialized centers for hypertension; in each fold, 14 centers are used for training and 1 as the test set. Our classifier is available at http://www.dei.unipd.it/node/2357 (MATLAB code).

Medical data usually have missing values, hence, imputation methods have become an important issue. In previous studies, many imputation methods based on variable data had a multivariate normal distribution, such as expectation-maximization and regression-based imputation. These assumptions may lead to deviations in the results, which sometimes create a bottleneck. In addition, directly deleting instances with missing values may have several problems, such as losing important data, producing invalid research samples, and leading to research deviations. Therefore, this study proposed a safe-region imputation method for handling medical data with missing values, we also built a medical prediction model and compared the removed missing values with imputation methods in terms of the generated rules, accuracy, and AUC. First, this study used the kNN imputation, multiple imputation, and the proposed imputation to impute the missing data and then applied four attribute selection methods to select the important attributes. Then, we used the decision tree (C4.5), random forest, REP tree, and LMT classifier to generate the rules, accuracy, and AUC for comparison. Because there were four datasets with imbalanced classes (asymmetric classes), the AUC was an important criterion. In the experiment, we collected four open medical datasets from UCI and one international stroke trial dataset. The results show that the proposed safe-region imputation is better than the listing imputation methods and after imputing offers better results than directly deleting instances with missing values in the number of rules, accuracy, and AUC. These results will provide a reference for medical stakeholders.

As the penetration of electric vehicles (EVs) accelerates according to eco-friendly policies, the impact of electric vehicle charging demand on a power distribution network is becoming significant for reliable power system operation. In this regard, accurate power demand or load forecasting is of great help not only for unit commitment problem considering demand response but also for long-term power system operation and planning. In this paper, we present a forecasting model of EV charging station load based on long short-term memory (LSTM). Besides, to improve the forecasting accuracy, we devise an imputation method for handling missing values in EV charging data. For the verification of the forecasting model and our imputation approach, performance comparison with several imputation techniques is conducted. The experimental results show that our imputation approach achieves significant improvements in forecasting accuracy on data with a high missing rate. In particular, compared to a strategy without applying imputation, the proposed imputation method results in reduced forecasting errors of up to 9.8%.

Diabetes mellitus is a group of metabolic diseases in which blood sugar levels are too high. About 8.8% of the world was diabetic in 2017. It is projected that this will reach nearly 10% by 2045. The major challenge is that when machine learning-based classifiers are applied to such data sets for risk stratification, leads to lower performance. Thus, our objective is to develop an optimized and robust machine learning (ML) system under the assumption that missing values or outliers if replaced by a median configuration will yield higher risk stratification accuracy. This ML-based risk stratification is designed, optimized and evaluated, where: (i) the features are extracted and optimized from the six feature selection techniques (random forest, logistic regression, mutual information, principal component analysis, analysis of variance, and Fisher discriminant ratio) and combined with ten different types of classifiers (linear discriminant analysis, quadratic discriminant analysis, naïve Bayes, Gaussian process classification, support vector machine, artificial neural network, Adaboost, logistic regression, decision tree, and random forest) under the hypothesis that both missing values and outliers when replaced by computed medians will improve the risk stratification accuracy. Pima Indian diabetic dataset (768 patients: 268 diabetic and 500 controls) was used. Our results demonstrate that on replacing the missing values and outliers by group median and median values, respectively and further using the combination of random forest feature selection and random forest classification technique yields an accuracy, sensitivity, specificity, positive predictive value, negative predictive value and area under the curve as: 92.26%, 95.96%, 79.72%, 91.14%, 91.20%, and 0.93, respectively. This is an improvement of 10% over previously developed techniques published in literature. The system was validated for its stability and reliability. RF-based model showed the best performance when outliers are replaced by median values. Electronic supplementary material The online version of this article (10.1007/s10916-018-0940-7) contains supplementary material, which is available to authorized users.

In many applications of high- and low-stakes ability tests, a non-negligible amount of respondents may fail to reach the end of the test within the specified time limit. Since for respondents that ran out of time some item responses will be missing, this raises the question of how to best deal with these missing responses for the purpose of obtaining an optimal assessment of ability. Commonly, researchers consider three general solutions: ignore the missing responses, treat them as being incorrect, or treat the responses as missing but model the missingness mechanism. This paper approaches the issue of dealing with not reached items from a measurement perspective, and considers the question what the operationalization of ability should be in maximum performance tests that work with effective time limits. We argue that the target ability that the test attempts to measure is maximum performance when operating at the test-indicated speed, and that the test instructions should be taken to imply that respondents should operate at this target speed. The phenomenon of the speed-ability trade-off informs us that the ability that is measured by the test will depend on this target speed, as different speed levels will result in different levels of performance on the same set of items. Crucially, since respondents with not reached items worked at a speed level lower than this target speed, the level of ability that they have been able to display on the items that they did reach is higher than the level of ability that they would have displayed if they had worked at the target speed (i.e., higher than their level on the target ability). Thus, statistical methods that attempt to obtain unbiased estimates of the ability as displayed on the items that were reached will result in biased estimates of the target ability. The practical implications are studied in a simulation study where different methods of dealing with not reached items are contrasted, which shows that current methods result in biased estimates of target ability when a speed-ability trade-off is present. The paper concludes with a discussion of ways in which the issue can be resolved.

Liquid chromatography-mass spectrometry (LC-MS) untargeted experiments require complex chemometrics strategies to extract information from the experimental data. Here we discuss "data preprocessing", the set of procedures performed on the raw data to produce a data matrix which will be the starting point for the subsequent statistical analysis. Data preprocessing is a crucial step on the path to knowledge extraction, which should be carefully controlled and optimized in order to maximize the output of any untargeted metabolomics investigation.

Supervised learning methods with missing data have been extensively studied not just due to the techniques related to low-rank matrix completion. Also in unsupervised learning one often relies on imputation methods. As a matter of fact, missing values induce a bias in various estimators such as the sample covariance matrix. In the present paper, a convex method for sparse subspace estimation is extended to the case of missing and corrupted measurements. This is done by correcting the bias instead of imputing the missing values. The estimator is then used as an initial value for a nonconvex procedure to improve the overall statistical performance. The methodological as well as theoretical frameworks are applied to a wide range of statistical problems. These include sparse Principal Component Analysis with different types of randomly missing data and the estimation of eigenvectors of low-rank matrices with missing values. Finally, the statistical performance is demonstrated on synthetic data., Comment: 32 pages, 4 figures

The goal of this paper is to investigate the impact of missing values in categorical time series sequences on common data analysis tasks. Being able to more effectively identify patterns in socio-demographic longitudinal data is an important component in a number of social science settings. However, performing fundamental analytical operations, such as clustering for grouping these data based on similarity patterns, is challenging due to the categorical and multi-dimensional nature of the data, and their corruption by missing and inconsistent values. To study these data quality issues, we employ longitudinal sequence data representations, a similarity measure designed for categorical and longitudinal data, together with state-of-the art clustering methodologies reliant on hierarchical algorithms. The key to quantifying the similarity and difference among data records is a distance metric. Given the categorical nature of our data, we employ an 'edit' type distance using Optimal Matching (OM). Because each data record has multiple variables of different types, we investigate the impact of mixing these variables in a single similarity measure. Between variables with binary values and those with multiple nominal values, we find that the ability to overcome missing data problems is harder in the nominal domain versus the binary domain. Additionally, artificial clusters introduced by the alignment of leading missing values can be resolved by tuning the missing value substitution cost parameter.

The goal of this paper is to investigate the impact of missing values in categorical time series sequences on common data analysis tasks. Being able to more effectively identify patterns in socio-demographic longitudinal data is an important component in a number of social science settings. However, performing fundamental analytical operations, such as clustering for grouping these data based on similarity patterns, is challenging due to the categorical and multi-dimensional nature of the data, and their corruption by missing and inconsistent values. To study these data quality issues, we employ longitudinal sequence data representations, a similarity measure designed for categorical and longitudinal data, together with state-of-the art clustering methodologies reliant on hierarchical algorithms. The key to quantifying the similarity and difference among data records is a distance metric. Given the categorical nature of our data, we employ an 'edit' type distance using Optimal Matching (OM). Because each data record has multiple variables of different types, we investigate the impact of mixing these variables in a single similarity measure. Between variables with binary values and those with multiple nominal values, we find that the ability to overcome missing data problems is harder in the nominal domain versus the binary domain. Additionally, artificial clusters introduced by the alignment of leading missing values can be resolved by tuning the missing value substitution cost parameter.

Contains fulltext : 152606.pdf (Publisher’s version ) (Open Access) Objective The aim of this study was to assess which characteristics and results of vaginal examination are predictive for delivery within 7 days, in women with threatened preterm labor after initial treatment. Study Design A secondary analysis of a randomized controlled trial on maintenance nifedipine includes women who remained undelivered after threatened preterm labor for 48 hours. We developed one model for women with premature prelabor rupture of membranes (PPROM) and one without PPROM. The predictors were identified by backward selection. We assessed calibration and discrimination and used bootstrapping techniques to correct for potential overfitting. Results For women with PPROM (model 1), nulliparity, history of preterm birth, and vaginal bleeding were included in the multivariable analysis. For women without PPROM (model 2), maternal age, vaginal bleeding, cervical length, and fetal fibronectin (fFN) status were in the multivariable analysis. Discriminative capability was moderate to good (c-statistic 0.68; 95% confidence interval [CI] 0.60-0.77 for model 1 and 0.89; 95% CI, 0.84-0.93 for model 2). Conclusion PPROM and vaginal bleeding in the current pregnancy are relevant predictive factors in all women, as are maternal age, cervical length, and fFN in women without PPROM and nulliparity, history of preterm birth in women with PPROM.

Can we learn from the unknown? Logical data sets of the ternary kind are often found in information systems. They contain unknown as well as true/false values. An unknown value may represent a missing entry (lost or indeterminable) or have meaning, like a Don't Know response in a questionnaire. In this paper, we introduce algorithms for reducing the dimensionality of logical data (categorical data in general) in the context of a new data mining challenge: Ternary Matrix Factorization (TMF). For a ternary data matrix, TMF exploits ternary logic to produce a basis matrix (which holds the major patterns in the data) and a usage matrix (which maps patterns to original observations). Both matrices are interpretable, and their ternary matrix product approximates the original matrix. TMF has applications in (1) finding targeted structure in ternary data, (2) imputing values through pattern discovery in highly incomplete categorical data sets, and (3) solving instances of its encapsulated Binary Matrix Factorization problem. Our elegant algorithm FasTer (FASt TERnary Matrix Factorization) has linear run-time complexity with respect to the dimensions of the data set and is parameter-robust. A variant of FasTer that exploits useful results from combinatorics provides accuracy bounds for a core part of the algorithm in certain situations. Experiments on synthetic and real-world data sets show that our algorithms are able to outperform state-of-the-art techniques in all three TMF applications with respect to run-time and effectiveness. Finally, convincing speedup and efficiency results on a parallel version of FasTer demonstrate its suitability for weak-and strong-scaling scenarios.

ObjectiveStandard portions or substitution of missing portion sizes with medians may generate bias when quantifying the dietary intake from FFQ. The present study compared four different methods to include portion sizes in FFQ.DesignWe evaluated three stochastic methods for imputation of portion sizes based on information about anthropometry, sex, physical activity and age. Energy intakes computed with standard portion sizes, defined as sex-specific medians (median), or with portion sizes estimated with multinomial logistic regression (MLR), ‘comparable categories’ (Coca) ork-nearest neighbours (KNN) were compared with a reference based on self-reported portion sizes (quantified by a photographic food atlas embedded in the FFQ).SettingThe Danish Health Examination Survey 2007–2008.SubjectsThe study included 3728 adults with complete portion size data.ResultsCompared with the reference, the root-mean-square errors of the mean daily total energy intake (in kJ) computed with portion sizes estimated by the four methods were (men; women): median (1118; 1061), MLR (1060; 1051), Coca (1230; 1146), KNN (1281; 1181). The equivalent biases (mean error) were (in kJ): median (579; 469), MLR (248; 178), Coca (234; 188), KNN (−340; 218).ConclusionsThe methods MLR and Coca provided the best agreement with the reference. The stochastic methods allowed for estimation of meaningful portion sizes by conditioning on information about physiology and they were suitable for multiple imputation. We propose to use MLR or Coca to substitute missing portion size values or when portion sizes needs to be included in FFQ without portion size data.

Background: Paper questionnaires have traditionally been the first choice for data collection in research. However, declining response rates over the past decade have increased the risk of selection bias in cross-sectional studies. The growing use of the Internet offers new ways of collecting data, but trials using Web-based questionnaires have so far seen mixed results. A secure, online digital mailbox (e-Boks) linked to a civil registration number became mandatory for all Danish citizens in 2014 (exemption granted only in extraordinary cases). Approximately 89% of the Danish population have a digital mailbox, which is used for correspondence with public authorities. Objective: We aimed to compare response rates, completeness of data, and financial costs for different invitation methods: traditional surface mail and digital mail. Methods: We designed a cross-sectional comparative study. An invitation to participate in a survey on help-seeking behavior in out-of-hours care was sent to two groups of randomly selected citizens from age groups 30-39 and 50-59 years and parents to those aged 0-4 years using either traditional surface mail (paper group) or digital mail sent to a secure online mailbox (digital group). Costs per respondent were measured by adding up all costs for handling, dispatch, printing, and work salary and then dividing the total figure by the number of respondents. Data completeness was assessed by comparing the number of missing values between the two methods. Socioeconomic variables (age, gender, family income, education duration, immigrant status, and job status) were compared both between respondents and nonrespondents and within these groups to evaluate the degree of selection bias. Results: A total 3600 citizens were invited in each group; 1303 (36.29%) responded to the digital invitation and 1653 (45.99%) to the paper invitation (difference 9.66%, 95% CI 7.40-11.92). The costs were €1.51 per respondent for the digital group and €15.67 for paper group respondents. Paper questionnaires generally had more missing values; this was significant in five of 17 variables (P

51 Pags.- 12 Figs.- 3 Tabls. The definitive version is available at: http://www.int-res.com/journals/cr/cr-home/, This work presents a method for the reconstruction of fragmentary daily precipitation datasets. The method aims to preserve the local and temporal variability characteristic of high-frequency precipitation data, and does not use the time-structure of the data. Based on the precipitation values recorded at closest neighbours during a target day, 2 reference values (RVs) are computed: a binomial prediction (BP) expressing the probability of occurrence of a wet day; and a magnitude prediction (MP), referring to the amount of precipitation. Generalised linear models (GLMs) are used to compute the RVs using the precipitation data (occurrence and magnitude) of the 10 nearest neighbours as the dependent variable, and the geographic information of each station (latitude, longitude, and altitude) as the independent variables. The RVs are then used to (1) apply quality control to the data, flagging suspect records according to 5 predefined criteria; (2) obtain serially complete time series by imputing RVs to missing observations in the original dataset; and (3) create new time series at locations where there were no observations or gridded datasets with even spatial coverage over the study area. The routines used were compiled into an R-package called ‘reddPrec’ (reconstruction of daily data - Precipitation) available to any user. We applied these methods to the complete daily precipitation dataset of the island of Majorca in Spain, spanning the period from 1971-2014., This study was supported by research projects CGL2012-31668, CGL2015-69985-R, CGL2011-24185 and CGL2014-52135-C3-1-R, financed by the Spanish Ministerio de Economía y Competitividad (MINECO) and FEDER-ERDF funds. The researchers were supported by the Government of Aragón through the ‘Programme of research groups’ (groups H38, ‘Clima, Cambio Global y Sistemas Naturales’ and ‘E68, Geomorfología y Cambio Global’).