Search

Clinical rating scales can be used to assess the severity of bipolar disorder; however, their use involves clinician–patient interactions, which is labor-intensive. Therefore, this study proposes a digital-phenotyping-based system that provides clinical ratings of bipolar disorder severity using global positioning system, self-scale, daily mood, user emotion, sleep time, and multimedia data; these ratings are given on Hamilton Depression Rating Scale (HAM-D) and Young Mania Rating Scale (YMRS). A K-nearest-neighbor-based imputation method was used to handle missing data. In this method, missing data points are filled in with the multiple correlations between different features. Furthermore, the Least Absolute Shrinkage and Selection Operator (Lasso)-regression-based multilayer perceptron (Lasso-MLP) method was adopted to predict the total and factor scores on the HAM-D and YMRS. Five-fold cross-validation were used in evaluation experiments. When the designed data imputation method was used with Lasso-MLP, the mean square errors of the total score and average factor score on HAM-D (the YMRS) were 0.56 (0.38) and 1.88 (0.98), respectively, which were smaller than the corresponding values obtained through Lasso regression (by 0.12 and 0.05, respectively, for HAM-D and by 0.12 and 0.10, respectively, for the YMRS). The experimental results also indicated that the models trained with the imputed data outperformed those trained without imputed data. Thus, the developed approaches can eliminate the missing data problem and provide accurate clinical ratings. [ABSTRACT FROM AUTHOR]

Missing data is a critical challenge in industrial data analysis, particularly during anomaly incidents caused by system equipment malfunctions or, more critically, by cyberattacks in industrial systems. It impedes effective imputation and compromises data integrity. Existing statistical and machine learning techniques struggle with heavily missing data, often failing to restore original data characteristics. To address this, we propose Anomaly Signal Imputation Using Latent Coordination Relations, a framework employing a variational autoencoder (VAE) to learn from complete data and establish a robust imputation model based on latent space coordination points. Experimental results from a water treatment testbed show significant improvements in output signal fidelity despite substantial data loss, outperforming conventional techniques.

This paper addresses the critical issue of missing data in power demand time series by emphasizing the relevance of imputation-based approaches in data-driven technologies. A comparative analysis of imputation methods is performed, where the reference from the state of the art is selected as K-Nearest Neighbors (KNN) applied in the time domain. Two innovative methods are proposed. The former method is defined as Historical Data Informed Regression Technique (H-DIRT) and is based on incorporating historical data for setting up a multivariate linear regression and then imputing through the estimated relation between the missing power demand measurement and the historical data. When the available historical data are insufficient, the algorithm proceeds by averaging or by a linear interpolation between the first available measurement before and after the missing value. The latter proposed method is defined as Seasonal KNN (SKNN) and it is based on enriching the data set with features related to yearly, seasonal, weekly and daily trends and then proceeding by baseline KNN. Experiments are set up with random and continuous data clipping, even with rather extreme pruning (up 70% of the data). The results in general demonstrate a significant improvement in imputation accuracy compared to the state of the art. The average error metrics (like Mean Absolute Error and Root Mean Square Error) for the SKNN method are in the order of respectively one third and one half those of the baseline KNN, in the cases of random and continuous data clipping. In general, the SKNN method provides more accurate results and better captures the statistical features of the data set to impute. Anyway, if the share of data to impute is not too large, the H-DIRT method provides comparable accuracy at a much lower computational cost. Hence, this study presents an easily implementable and computationally affordable approach for improving, in various contexts, the state of the art in power demand data imputation. It establishes a foundation for future exploration into trends, seasonal factors, and external variables influencing power load parameters.

The emergence of single-cell RNA sequencing (scRNA-seq) has brought to light the critical need for scrutinizing transcriptomes at the individual cellular level with unparalleled precision. A pivotal aspect of scRNA-seq data analysis involves cell identification, commonly accomplished through diverse clustering methodologies. However, scRNA-seq datasets frequently encounter missing values due to technical limitations, posing a significant challenge that can compromise the accuracy of clustering outcomes. In response, we present a novel approach that seamlessly integrates missing value estimation with scRNA-seq clustering. Our method harnesses the power of an imputation autoencoder network to predict missing values, coupled with the deployment of a deep clustering network for efficient cell categorization. To mitigate the risk of deep clustering networks converging towards suboptimal local minima, we have devised a self-guided learning strategy. This approach exploits shared parameters between the imputation and clustering networks, fostering a symbiotic relationship that enhances overall performance. Through rigorous empirical evaluations, we substantiate the effectiveness of our methodology, demonstrating its comparability to, or surpassing, several established single-cell clustering techniques. Furthermore, our analysis of cellular trajectories underscores the proficiency of the proposed method in accurately deducing cellular trajectories by leveraging the clustering results to discern biologically meaningful cell types.

Health-related data has a decisive role in disease diagnosis. Collecting relevant information from health-related data in medical records has been facilitated by evaluating the features of the data. Relevant research has shown that outcomes are significantly impacted by the use of feature selection (FS) in different medical domain data. FS provides an analysis of the most significant features to improve classification accuracy. The FS technique aims at minimizing the number of input variables and computational overload to maximize classification performance results. However, identifying the optimal features poses issues due to the high dimensionality of large features and the small sample size of health-related data. The metaheuristics optimization algorithm (MOA) plays an important role in generating the best subset features with exploration and exploitation phases. This study experiments with well-known MOAs and supervised learning from the UC Irvine Machine Learning Repository, PhysioNet, Kent Ridge Bio-Medical Dataset, and MIMIC-III v1.4 Repository with varying feature dimensions. To increase the quality of health-related data, this study proposes missing data imputation based on a deep learning approach, an autoencoder (AE). With AE imputation, the performance results obtain 0.0167 mean squared error (MSE) and 0.129 root mean squared error (RMSE). As a result, MOA shows its excellence in achieving minimal features, but still outstanding performance in low- and high-dimensional data. MOA is successfully applied to varying diverse health-related datasets with low- and high-dimensional data.

Missing data imputation is a technique to deal with incomplete datasets. Since many models and algorithms cannot be applied to data containing missing values, a pre-processing step needs to be performed to remove incomplete data or to estimate the missing values. This is a well-known problem referred to as the data imputation problem. Several approaches have been designed for data imputation. These algorithms can be divided into two main categories: statistical and machine learning-based algorithms. As machine learning algorithms are optimized, they usually have better performance compared with statistical ones. In this paper, we review the most recent literature related to missing data imputation based on generative adversarial networks (GANs) that have gained tremendous attention in dealing with missing values. We examine the structures of GANs for missing data imputation and discuss the commonly used datasets and metrics for evaluation. We also cover the influence of the missing datatype, the effect of the missing data fraction, and the algorithm-related problems on data imputation performance. We conduct experiments on two publicly available datasets and evaluate the performance of GAIN, a missing data imputation algorithm to that of existing state-of-the-art approaches, demonstrating that the GAN-based algorithm outperforms the others in terms of RMSE and FID.

The electricity sector has added plenty of new technologies in recent years. Smart Grids are characterized by the use of monitoring and communication technologies almost in whole system. The application and use of such new technologies triggers a significant growth in the data number, increasing the amount of errors and missing data, thus hindering the analysis. In this context, this paper performs the modeling, implementation, validation and comparative analysis of four data imputation techniques: K-Nearest Neighbor, Median Imputation, Last Observation Carried Forward, and Makima. The aim is to verify if they could be applied to the electric segment - more specifically to the Smart Grids environment. The database used in the research is obtained from the electricity utility CEEE and its underground substations, located in southern Brazil. Following this, five simulation scenarios are created and one data set is removed, based on pre-established criteria. Finally, the techniques are applied and the new database is compared with the original one. From the simulation results, the technique which presented the best results is Makima, it is validated as robust to be applied in the Smart Grids environment, especially in electrical data missing from an electric power substation.

Missing data are a universal data quality problem in many domains, leading to misleading analysis and inaccurate decisions. Much research has been done to investigate the different mechanisms of missing data and the proper techniques in handling various data types. In the last decade, machine learning has been utilized to replace conventional methods to address the problem of missing values more efficiently. By studying and analyzing recently proposed methods using machine learning approaches, vital adoptions in accuracy, performance, and time consumed can be highlighted. This study aimed to help data analysts and researchers address the limitations of machine learning imputation methods by conducting a systematic literature review to provide a comprehensive overview of using such methods to impute missing values. Novel proposed machine learning approaches used for data imputation are analyzed and summarized to assist researchers in selecting a proper machine learning method based on several factors and settings. The review was performed on research studies published between 2016 and 2021 on adopting machine learning to impute missing values, focusing on their strengths and limitations. A total of 684 research articles from various scientific databases were analyzed using search engines, and 94 of them were selected as primary studies. Finally, several recommendations were given to guide future researchers in applying machine learning to impute missing values.

As the importance of utility condition is increasingly acknowledged, the use of asset management technologies in the electric power industry has rapidly grown. The global trend of asset management follows risk management that accounts for the probability and consequences of failures. Because asset management systems tend to be composed of various legacy systems, each of which is installed and designed to collect data according to a certain data type and acquisition purpose, it is necessary to develop a system that cleans and integrates data acquired from each legacy system. This study explores the development of an asset management system for a transmission system as a representative linear asset consisting of different segments in a sequence. First, the configurations and characteristics of linear asset datasets are analyzed. Second, an automatic data cleaning system, equipped with six types of data cleaning functions for extracting dirty data from entire datasets, is proposed. An algorithm for data imputation, which is essential for estimating the remaining useful life, is developed based on principal component analysis–iterative algorithm (PCA–IA). Afterward, the performance of the proposed system is verified using actual data with the help of the Korea Electric Power Corporation (KEPCO). Specifically, to evaluate the performance of the proposed system, an automatic cleaning process is demonstrated using actual legacy datasets.

Complete data are required for the operation, maintenance, and detection of faults in semiconductor equipment. Missing data occur frequently because of defects such as sensor, data storage, and communication faults, leading to reductions in yield, quality, and productivity. Although many attempts have been made to solve this problem in other fields, few studies have specifically addressed data imputation in the semiconductor industry. In this study, an improved generative adversarial network (GAN)-based missing data imputation for the semiconductor industry called Semi-GAN is proposed. This study introduces a machine learning approach for dealing with data imputation in the semiconductor industry. The proposed method was applied to real data and evaluated using traditional techniques. In particular, the proposed method showed excellent results compared to traditional attribution methods when all missing data ratios in the experiments were less than 20%. It was also observed to be superior when simple and repetitive patterns were omitted rather than repetitive but not simple patterns.

Neonatal seizures are sudden events in brain activity with detrimental effects in neurological functions usually related to epileptic fits. Though neonatal seizures can be identified from electroencephalography (EEG), this is a challenging endeavour since expert visual inspection of EEG recordings is time consuming and prone to errors due the data’s nonstationarity and low signal-to-noise ratio. Towards the greater aim of automatic clinical decision making and monitoring, we propose a multi-output Gaussian process (MOGP) framework for neonatal EEG modelling. In particular, our work builds on the multi-output spectral mixture (MOSM) covariance kernel and shows that MOSM outperforms other commonly-used covariance functions in the literature when it comes to data imputation and hyperparameter-based seizure detection. To the best of our knowledge, our work is the first attempt at modelling and classifying neonatal EEG using MOGPs. Our main contributions are: i) the development of an MOGP-based framework for neonatal EEG analysis; ii) the experimental validation of the MOSM covariance kernel on real-world neonatal EEG for data imputation; and iii) the design of features for EEG based on MOSM hyperparameters and their validation for seizure detection (classification) in a patient specific approach.

Missing data is quite common in the industrial field, resulting in problems in downstream applications, as most data driven methods used in these applications rely on complete and high-quality dataset to build a high-quality model. Existing methods deal with missing data individually regardless of its downstream application, treating all variables equally without considering their different roles in the downstream application. This would affect imputation performance for key variables, thus deteriorating the accuracy of the downstream model. A considerable challenge is how to refine the missing data imputation task. In this paper, a new method termed fine-tuned imputation GAN (FIGAN) is designed to achieve customized data imputation for industrial soft sensor. The major contribution of the paper lies in two aspects: 1) different from the original imputation GAN (GAIN) which treats all variables equally, FIGAN is guided by a soft sensor module so as to achieve customized data imputation by performing improved data imputation on quality-related variables. Enhanced accuracy for the final industrial soft sensor would be possible; 2) in addition, since labels of the soft sensor might also have missing data, a soft sensor with pseudo labeling is designed to conquer the problem with data imputation and label prediction being optimized interactively. Case studies on a converter steelmaking process and a penicillin fermentation process show the feasibility of the proposed FIGAN. It is noted that such customized imputation could be readily transferred to other downstream applications with missing data. Note to Practitioners—Industrial data is often incomplete and needs proper treatment. Meanwhile, downstream applications with preprocessed data vary under different industrial scenes. The focus of this study is to develop a customized data imputation method for specific downstream applications such as soft sensing. A fine-tuned imputation GAN is designed with a soft sensor module so as to guide the data imputation for good imputation on key variables of the soft sensor. Considering that missing data exists not only in measurement variables but also in soft sensor labels, a semi-supervised soft sensor is designed to handle missing data in the labels, optimized together with the imputation model. The customized data imputation can thus improve the final performance of the downstream model which is a soft sensor in this work. The customization could be transferred to other applications such as an anomaly detection model as well. [ABSTRACT FROM AUTHOR]

With the rapid development of sensor technologies, time series data collected by multiple and spatially distributed sensors have been widely used in different research fields. Examples of such data include geo-tagged temperature data collected by temperature sensors, air pollutant monitoring data, and traffic data collected by road traffic sensors. Due to sensor failure, communication errors and storage loss, etc., data collected by sensors inevitably includes missing data. However, models commonly used in the analysis of such large-scale data often rely on complete data sets. This paper proposes a model for the imputation of missing data of traffic flow, which combines a self-attention mechanism, an auto-encoder, and a generative adversarial network, into a self-attention generative adversarial imputation net (SA-GAIN). The introduction of the self-attention mechanism can help the proposed model to effectively capture correlations between spatially-distributed sensors at different time points. Adversarial training through two neural networks, called generators and discriminators, allows the proposed model to generate imputed data close to the real data. In comparison with different imputation models, the proposed model shows the best performance in imputing missing data. [ABSTRACT FROM AUTHOR]

Imputation of missing sensor-collected data is often an important step prior to machine learning and statistical data analysis. One particular data imputation challenge is filling large data gaps when the only related data comes from the same sensor station. In this paper, we propose a framework to improve the popular multivariate imputation by chained equations (MICE) method for dealing with missing data. One key strategy we use to improve model accuracy is to reshape the original sensor data to leverage the correlation between the missing data and the observed data. We demonstrate our framework using data from continuous water quality monitoring stations in Vermont. Because of possible irregularly spaced peaks throughout the time series, the reshaped data is split into extreme and normal values and two MICE models are built. We also recommend that sensor-collected data should be transformed to meet the machine learning model assumptions. According to our experimental results, these strategies can improve MICE data imputation model accuracy at least 23% for large data gaps based on $\text {R}^{{2}}$ values and are promising to be applied for other data imputation algorithms. [ABSTRACT FROM AUTHOR]

With the successful application of single-cell sequencing technology, a large number of single-cell multi-omics sequencing (scMO-seq)data have been generated, which enables researchers to study heterogeneity between individual cells. One prominent problem in single-cell data analysis is the prevalence of dropouts, caused by failures in amplification during the experiments. It is necessary to develop effective approaches for imputing the missing values. Different with general methods imputing single type of single-cell data, we propose an imputation method called scLRTD, using low-rank tensor decomposition based on nuclear norm to impute scMO-seq data and single-cell RNA-sequencing (scRNA-seq)data with different stages, tissues or conditions. Furthermore, four sets of simulated and two sets of real scRNA-seq data from mouse embryonic stem cells and hepatocellular carcinoma, respectively, are used to carry out numerical experiments and compared with other six published methods. Error accuracy and clustering results demonstrate the effectiveness of proposed method. Moreover, we clearly identify two cell subpopulations after imputing the real scMO-seq data from hepatocellular carcinoma. Further, Gene Ontology identifies 7 genes in Bile secretion pathway, which is related to metabolism in hepatocellular carcinoma. The survival analysis using the database TCGA also show that two cell subpopulations after imputing have distinguished survival rates. [ABSTRACT FROM AUTHOR]

Accurate computational models for clinical decision support systems require clean and reliable data but, in clinical practice, data are often incomplete. Hence, missing data could arise not only from training datasets but also test datasets which could consist of a single undiagnosed case, an individual. This work addresses the problem of extreme missingness in both training and test data by evaluating multiple imputation and classification workflows based on both diagnostic classification accuracy and computational cost. Extreme missingness is defined as having ∼50% of the total data missing in more than half the data features. In particular, we focus on dementia diagnosis due to long time delays, high variability, high attrition rates and lack of practical data imputation strategies in its diagnostic pathway. We identified and replicated the extreme missingness structure of data from a real-world memory clinic on a larger open dataset, with the original complete data acting as ground truth. Overall, we found that computational cost, but not accuracy, varies widely for various imputation and classification approaches. Particularly, we found that iterative imputation on the training dataset combined with a reduced-feature classification model provides the best approach, in terms of speed and accuracy. Taken together, this work has elucidated important factors to be considered when developing a predictive model for a dementia diagnostic support system. [ABSTRACT FROM AUTHOR]

We introduce a new method for online Phasor Measurement Unit (PMU) missing value replacement. Our approach allows us to decompose PMU event responses into a non-dynamic component (denoted the participation factor) that can be inferred directly from the past and a dynamic component that can be inferred directly from all other PMUs (denoted the event strength). When missing values occur, we can use these two components, which do not rely on the missing index, to estimate the correct value. The method is extremely fast and can easily be used for online applications. Furthermore, extensive testing on real power system event data reveals that our approach achieves state-of-the-art performance in terms of Mean Absolute Percent Errors (MAPEs) for PMU data dropped during event periods. The method also yields an interpretable and simplified view of events for further analysis and applications. The method relies only on PMU data and does not take outside information such as network topology. [ABSTRACT FROM AUTHOR]

Due to the limitation of data collection techniques and privacy issues, the problem of missing spatial origin-destination flows frequently occurs. Data imputation provides great support for the acquisition of complete flow data, which enables us to better understand regional connections and mobility patterns. However, existing models or approaches neglect the network structure of spatial flows, thus resulting in inappropriate estimates and a low performance. The development of graph neural networks offers a powerful tool to deal with graph-structured data. In this article, we proposed a spatial interaction graph convolutional network model, which combines graph convolution and a mapping function to predict flow data from the perspective of network learning. This model utilizes geographical unit embedding in local spatial networks to improve prediction accuracy. A negative sampling technique is adopted to reduce misestimation. Experiments on Beijing taxi trip data verified the usefulness of our model in spatial flow prediction. We also demonstrated that a biased training sample had a negative impact on the model’s performance. More attributes of geographical units, a more proper negative sampling rate and a larger training set can increase the prediction accuracy of flow data. [ABSTRACT FROM AUTHOR]

Autonomous driving is the main application of Internet of Things (IoT) technology in the field of intelligent transportation. In autonomous driving, self-driving cars will avoid changing lanes in a short distance. When the self-driving car executes the follow-up instruction, the road smoothness in front of the car will affect the driving safety and comfort of the car. The real-time acquisition of road information in front of driving will help the self-driving car adjust driving behavior. However, other vehicles on the road will lead to the failure of Light Detection And Ranging (LiDAR) detectors to obtain complete road point cloud data. The incomplete road point cloud data need to be imputated to avoid potential misjudgements of the road conditions. Currently, little research work specifically focuses on imputating the incomplete road point cloud data that are caused by obstacle vehicles. In this paper, we propose a fast method to imputate the incomplete road point cloud data using a Graphics Processing Unit (GPU)-based parallel Inverse Distance Weighted (IDW) interpolation algorithm to enhance the safety of autonomous driving. To evaluate the performance of the proposed method, two groups of experiments are conducted. The experimental results indicate the following: (1) the known point cloud data within 5 meters around the obstacle vehicle are sufficient to guarantee the imputation accuracy; (2) when the weight parameter of the IDW interpolation is 4, the efficiency and accuracy of the imputation can be optimally balanced; and (3) it takes approximately 0.6 seconds to imputate the incomplete dataset consisting of 15 million points, while the imputation error is approximately 5 millimeters. The proposed method is capable of efficiently and effectively imputating the incomplete road point cloud data that are induced by obstacle vehicles and outperforms other interpolation algorithms and machine learning algorithms.

Supply and demand increase in response to healthcare trends. Moreover, personal health records (PHRs) are being managed by individuals. Such records are collected using different avenues and vary considerably in terms of their type and scope depending on the particular circumstances. As a result, some data may be missing, which has a negative effect on the data analysis, and such data should, therefore, be replaced with appropriate values. In this study, a method for estimating missing data using a multi-modal autoencoder applied to the field of healthcare big data is proposed. The proposed method uses a stacked denoising autoencoder to estimate the missing data that occur during the data collection and processing stages. Autoencoders are neural networks that output value of x^ similar to an input value of x. In the present study, data from the Korean National Health Nutrition Examination Survey (KNHNES), conducted by the Korea Centers for Disease Control and Prevention (KCDC), are used. As representative healthcare data from South Korea, they contain a large number of parameters identical to those used in the PHRs. Based on this, models can be generated to estimate missing data occurring in PHRs. Furthermore, PHRs involve a multi-modality that allows the data to be collected from multiple sources for a single object. Therefore, the stacked denoising autoencoder applied is configured under a multi-modal setting. Through pre-processing, a set of data without missing value in KNHNES is designed. In the data set based learning, a label is set as original data, and an autoencoder input is set as noised input that additionally has as many random zero numbers as noise factor. In this way, the autoencoder learns in the way of making the zero-based noise value similar to the original label value. When the amount of missing data in a dataset reaches approximately 25%, the accuracy of the proposed method using a multi-modal stacked denoising autoencoder is 0.9217, which is higher than that achieved by other ordinary methods. For a single-modal denoising autoencoder, the accuracy is 0.932, with a slight difference of approximately 0.01, which falls within the allowable limits in data analysis. In terms of computational performance, a single-modal autoencoder has 10,384 parameters, which is 5,594 more than those used in a multi-modal stacked autoencoder. These parameters affect the speed of the model. Both models exhibit a significant difference in the number of parameters but demonstrate a relatively small difference in accuracy, suggesting that the proposed multi-modal stacked denoising autoencoder is advantageous over a single-modal model when used on a personal device. Moreover, a multi-modal model can save additional time when processing large amounts of data in locations such as hospitals and institutions.

We propose a statistical learning-based traffic speed estimation method that uses sparse vehicle trajectory information. Using a convolutional encoder-decoder based architecture, we show that a well trained neural network can learn spatio-temporal traffic speed dynamics from time-space diagrams. We demonstrate this for a homogeneous road section using simulated vehicle trajectories and then validate it using real-world data from the Next Generation Simulation (NGSIM) program. Our results show that with probe vehicle penetration levels as low as 5%, the proposed estimation method can provide a sound reconstruction of macroscopic traffic speeds and reproduce realistic shockwave patterns, implying applicability in a variety of traffic conditions. We further discuss the model's reconstruction mechanisms and confirm its ability to differentiate various traffic behaviors such as congested and free-flow traffic states, transition dynamics, and shockwave propagation. We also provide a comparison against a widely used adaptive smoothing technique used for the same purpose and demonstrate the superiority of the proposed approach, even with probe vehicle lower penetration levels.

A health model based on data has various missing values depending on the user situation, and the accuracy of a health model requiring variables that the user cannot collect appears low. A deep learning health model is fitted by learning weights to increase accuracy. In the process of applying a deep-learning-based health model to the user situation, accuracy may be degraded if learning is omitted. In this paper, we propose hybrid multimodal deep learning using a collaborative concat layer in health big data. The proposed method uses a machine learning technique to alleviate the issue caused by the change in the data observation range according to a change in the user situation, and occurring in multimodal health deep learning. It is a layer composed of the connection, input, and output of the model of the collaborative node (CN). A CN is a node that predicts absent variables through filtering using the similarity of input values. With CN, a collaborative concat layer (CCL) that handles missing values from the input of the health model can be configured, and the issue related to missing values occurring in the health model can be resolved. With the proposed CCL, it is possible to reuse existing models or construct new models through the concatenation of several single-modal deep learning models. By evaluating the effect on the input and output of the model according to the structural position of the CCL, various networks can be configured, and the performance of the single-modal model can be maintained. In particular, the accuracy of a deep learning model is more stable when the CCL is used, suggesting the experiment progress based on the assumption that a specific variable is absent depending on the user situation.

Missing values are a common problem of time series such as sensor-generated data due to issues of data collection, transmission, and storage. To deal with this problem, various data imputation methods have been proposed. Among these methods, variants of recurrent neural networks (RNNs) have attracted a lot of attention as they provide a natural representation of temporal dynamics. Though such methods have been shown to achieve higher imputation accuracy, they mostly focus on the modeling of temporal dynamics from a short-term perspective while ignoring the long-term features such as trend and seasonal information embedded in time series data. In this paper, we investigate different paradigms of time series imputation methods to capture temporal dynamics from both long-term and short-term perspectives. In particular, we propose a new learning paradigm called MuSDRI which combines the time series decomposition method Seasonal Trend decomposition using Loess (STL) with RNNs. Specifically, STL is used to decompose long sequences of time series into trend, seasonal, and remainder components such that long-term dynamics can be revealed; RNNs are used to capture local patterns from the remainder component extracted by STL such that short-term dynamics can be refined. Moreover, MuSDRI is able to handle time series with multiple seasonal patterns by learning their contributions with respect to the imputation task. We evaluate the proposed imputation methods on three real-world datasets. Experimental results show that the imputation methods considering both the long-term and short-term dynamics achieve higher imputation accuracy, and among them MuSDRI in general achieves the best performance. [ABSTRACT FROM AUTHOR]

This study proposes a data recovery model where substituted values can be further limited by nonlinear and inequality constraints to approximate the ground truth. The objective is to generate substituted values for multifactors while considering their lower/upper bounds, data means, and nonlinearity at the same time. This is critical when data need to fall inside a nonlinear range, e.g., a partial hypersphere centered at a given mean. In view of such, this study proposes collaborative filtering with nonlinear inequality constraints to tackle the problem. The proposed method consists of three steps. First, the system finds class-dependent and box-bounded imputation basis factors for an incomplete data set. Class-dependent bases can reflect data domains well. Second, class-dependent imputation coefficients are located by the proposed nonnegative coefficient discovery with nonlinear inequality constraints. This step limits searching space and avoids generating substituted values out of range. Finally, constrained iterative projection pursuit is proposed for measuring the quality of recovered data by examining reconstruction residuals. By using both the nonlinear inequality constraints and the constrained iterative projection pursuit, the system can recover data while satisfying multifactor nonlinear coeffects required by manufacturers. Experimental results showed that the proposed method was capable of generating substituted values with lower root-mean-squared errors. In addition, errors were reduced by at least 10.06% on average, better than those of the baselines. Furthermore, the classification accuracy of the proposed method after data imputation was higher than that of the baselines. Such findings indicated that the proposed method could approximate the characteristics of data when missing values appeared. Note to Practitioners—This work was motivated by the problem of missing values in industrial heterogeneous sensor readings. When data recovery is performed, the process should consider multifactor nonlinearity, lower/upper bounds, historical references, and the divergence between substituted values and references at the same time in order to reconstruct original sensor readings as many as possible. Existing approaches generally have solutions to linear or nonlinear equality constraints but not the aforementioned nonlinear inequality ones. This work designs a self-dictionary method—class-dependent and box-bounded imputation basis factors along with constrained iterative projection pursuit—for finding substituted values. Real industrial experiments were conducted based on $\mathcal {L}_{2}$ -norms. Future research will address the design for other norms. [ABSTRACT FROM AUTHOR]

Recent advances in acquisition equipment are providing experiments with growing amounts of precise, yet affordable sensors. At the same time, an improved computational power, coming from new hardware resources [GPU, field-programmable gate array (FPGA), adaptive compute acceleration platform (ACAP)] has been made available at relatively low costs. This led us to explore the possibility of completely renewing the chain of acquisition for a fusion experiment, where many high-rate sources of data, coming from different diagnostics, can be combined in a wide framework of algorithms. If, on the one hand, adding new data sources with different diagnostics enriches our knowledge about physical aspects, on the other hand, the dimensions of the overall model grow, making relations among variables more and more opaque. A new approach for integrating such heterogeneous diagnostics, based on the composition of deep variational autoencoders, could ease this problem, acting as a structural sparse regularizer. This has been applied to RFX-mod experimental data, integrating the soft X-ray linear images of plasma temperature with the magnetic state. However, to ensure a real-time signal analysis, these algorithmic techniques must be adapted to run in well-suited hardware. In particular, it is shown that, attempting a quantization of neuron transfer functions, such models can be adapted to run in an embedded programmable logic device. The resulting firmware, approximating the deep inference model to a set of simple operations, fits well with the simple logic units that are largely abundant in FPGAs. This is the key factor that permits the use of affordable hardware with complex deep neural topology and operates them in real-time. [ABSTRACT FROM AUTHOR]

Data incompleteness is a common data quality problem in databases. Recent work proposes to retrieve missing string values from the World Wide Web for higher imputation recall, but on the other hand, takes the risk of introducing web noises into the imputation results. So far there lacks an effective way to control the quality of web-based data imputation, given the complexity of the quality model and lacking of enough ground truth data. In this article, an EM-based quality model is first built for web-based data imputation which investigates three key factors jointly, i.e., precision of web sources, correlation among web sources, and precision and recall of the employed extractors. However, the accuracy of the EM-based quality model could be harmed when the EM (Expectation Maximization) assumption that “the majority agree on the truth” does not hold in some cases. To solve this problem, we introduce crowd intervention to help improve the quality model. While a straightforward but expensive way is to let the crowd to identify all these undesirable cases and provide the right imputation values for these blanks, a most crowd-economic way is to select a small set of blanks for crowd-based imputation, whose results could help to adjust the EM-based quality model towards a better one. To achieve this, an adaptive blank selection strategy is proposed to select a sequence of blanks for crowd-based imputation. Also, we work on finding a proper time to stop further crowd intervention for the balance of crowd efficiency and quality improvement. Our experiments performed on three real world and one simulated data collections prove that the proposed quality model can effectively help improve the quality of the web-based imputation results by more than 15 percent, while our crowd cost saving strategy saves more than 75 percent crowd cost. [ABSTRACT FROM AUTHOR]

The RFX-mod2 Nuclear Fusion experiment is an upgrade of RFX-mod, which was shutdown in 2016. Among the other improvements in the machine structure and diagnostics, a larger number of electromagnetic probes (EMs) is foreseen to provide more information about plasma instabilities and to allow an improved real-time plasma control. An analog-to-digital converter (ADC) architecture able to provide, at the same time, both transient recording and real-time streaming, as well as field-programmable gate array (FPGA)-based time integration of the inputs, is foreseen in RFX-mod2. Transient recording provides full-speed data acquisition (up to 1 MSample/s) by recording data in local memory and reading memory content after the plasma discharge. Real-time streaming of the subsampled data is required for active control. The chosen technology is based on the XILINX Zynq architecture that provides, in the same chip, a multicore Advanced RISC Machines (ARM) processor tightly coupled to an FPGA. Time-critical functions are carried out by the FPGA, such as the management of the circular data buffer, low-pass filtering for subsampling of the samples to be streamed, and digital integration. Other functions are carried out by the processor, such as the management of the configuration setting, received via Transmission Control Protocol (TCP)/IP or Hypertext Transfer Protocol (HTTP), the data readout of acquired samples in transient recording buffers, and network data streaming of data collected for active real-time plasma control. [ABSTRACT FROM AUTHOR]

The presence of missing data is problematic in most quantitative research studies. Water distribution systems (WDSs) are not immune to this problem. In fact, missing data is an inherent feature of a WDS. There are various techniques and methods to address missing data ranging from simply deleting the data to using complex algorithms to impute missing data. This paper reviews the different imputation options available from traditional methods (such as deletion and single imputation) to more modern and advanced methods (such as multiple imputation, model-based procedures, and machine learning techniques). The concept, application, and qualitative advantages and disadvantages of these methods are discussed. In addition, a novel approach for selecting an applicable technique is presented. The approach is a “top-down bottom-up”two-prong approach for the selection of a data analysis and missing data technique. The bottom-up approach facilitates the top-down selection of a suitable technique by analyzing the data and narrowing down the selection options. As a use case, this paper also reviews techniques that are used to impute missing data in WDSs.

Due to the development of sensor networks and information technology, data-driven fault detection and diagnosis (FDD) has been made possible with real-time multiple sensor measurements. However, due to inevitable sensor errors or communication failures, the raw data are usually incomplete with corrupted values, lost values, or undetected missing values. In practice, the incomplete data are usually dealt with by directly excluding incomplete measurements and abnormal spikes. In addition, some preprocessing methods, which naively impute data though averaging or smoothing, have also been widely applied. In this article, we address the building FDD problem with incomplete data by proposing a new approach, the adjacent information recovery (AIR) filter. The AIR filter is utilized to deal with the FDD for a typical air handling unit (AHU) system with incomplete data based on the ASHRAE Research Project 1312. Experimental results show that the proposed method improves FDD performance by recovering missing sensor measurements and outperforms the state-of-the-art methods. Note to Practitioners—Fault detection and diagnosis (FDD) for smart buildings by addressing the fact that FDD systems are of great importance for saving energy and improving occupancy comfort levels and building safety levels. Existing FDD methods are mainly based on the assumption that sensor data are complete and reliable, which are rarely true in real practice. To solve the building FDD problem with incomplete data, in this article, the adjacent information recovery (AIR) filter is proposed to recover the missing data before applying FDD methods. The AIR filter takes the time series adjacency information into consideration via hidden Markov model (HMM) and includes the channel adjacency information with the collaborating filtering technique. [ABSTRACT FROM AUTHOR]

Incomplete information often occurs along with many database applications, e.g., in data integration, data cleaning, or data exchange. The idea of data imputation is often to fill the missing data with the values of its neighbors who share the same/similar information. Such neighbors could either be identified certainly by editing rules or extensively by similarity relationships. Owing to data sparsity, the number of neighbors identified by editing rules w.r.t. value equality is rather limited, especially in the presence of data values with variances. To enrich the imputation candidates, a natural idea is to extensively consider the neighbors with similarity relationship. However, the candidates suggested by these (heterogenous) similarity neighbors may conflict with each other. In this paper, we propose to utilize the similarity rules with tolerance to small variations (instead of the aforesaid editing rules with strict equality constraints) to rule out the invalid candidates provided by similarity neighbors. To enrich the data imputation, i.e., imputing the missing values more, we study the problem of maximizing the missing data imputation. Our major contributions include (1) the np-hardness analysis on solving as well as approximating the problem, (2) exact algorithms for tackling the problem, and (3) efficient approximation with performance guarantees. Experiments on real and synthetic data sets demonstrate the superiority of our proposal in filling accuracy. We also demonstrate that the record matching application is indeed improved, after applying the proposed imputation. [ABSTRACT FROM AUTHOR]

Manufacturing semiconductor wafers involves many sequential processes, and each process has various equipment-related variables or factors, which results in high-dimensional data. However, measuring the quality of all wafers is time and cost intensive, and only a small proportion of the wafers is labeled. Further, equipment factors are not always measured by sensors due to the complicated process. Variable selection, which is performed to reduce the dimensionality of the input variable space while improving or preserving regression performance by selecting important input factors, plays an important role in regression problems. We propose a variable selection procedure to find the main equipment factors that affect in-process wafer quality in consideration of the following issues: 1) imputation for missing values; 2) semi-supervised regression for unlabeled data; and 3) redundancy among variables. In the proposed procedure, partial least squares and least absolute shrinkage and selection operator regression are utilized as prediction models. Experiments using two semiconductor equipment datasets were conducted to evaluate the performance of the proposed procedure. [ABSTRACT FROM AUTHOR]

This paper presents a low-rank tensor model for vehicular traffic volume data. Contrarily to previous works, we capitalize on a definition of rank, called the tensor train, that is as effective as possible; so that it exploits all the correlation between local structures that are present in the multiple modes, but practical enough that efficient optimization algorithms still hold. From our model, a formulation to find balanced (higher order) tensors is derived. The resulting optimally-balanced tensor improves the imputation accuracy of the tensor train rank. Then, we design specific experiments, which are numerically evaluated using real-world traffic data from Tampere city, Finland. The experimental results are promising, our proposed approach outperforms existing algorithms in both imputation accuracy and, in some instances, computation time. [ABSTRACT FROM AUTHOR]

Data imputation aims at filling in missing attribute values in databases. Most existing imputation methods to string attribute values are inferring-based approaches, which usually fail to reach a high imputation recall by just inferring missing values from the complete part of the data set. Recently, some retrieving-based methods are proposed to retrieve missing values from external resources such as the World Wide Web, which tend to reach a much higher imputation recall, but inevitably bring a large overhead by issuing a large number of search queries. In this paper, we investigate the interaction between the inferring-based methods and the retrieving-based methods. We show that retrieving a small number of selected missing values can greatly improve the imputation recall of the inferring-based methods. With this intuition, we propose an inTeractive Retrieving-Inferring data imPutation approach (TRIP), which performs retrieving and inferring alternately in filling in missing attribute values in a data set. To ensure the high recall at the minimum cost, TRIP faces a challenge of selecting the least number of missing values for retrieving to maximize the number of inferable values. Our proposed solution is able to identify an optimal retrieving-inferring scheduling scheme in deterministic data imputation, and the optimality of the generated scheme is theoretically analyzed with proofs. We also analyze with an example that the optimal scheme is not feasible to be achieved in $\tau$ -constrained stochastic data imputation ($\tau$ -SDI), but still, our proposed solution identifies an expected-optimal scheme in $\tau$ -SDI. Extensive experiments on four data collections show that TRIP retrieves on average 20 percent missing values and achieves the same high recall that was reached by the retrieving-based approach. [ABSTRACT FROM AUTHOR]

Some traffic data from loop detectors settled in urban arterial streets are incomplete. The importance of effectively imputing the missing values emerges. The letter introduces least squares support vector machines (LS-SVMs) to missing traffic flow prediction based on spatio-temporal analysis. It is the first time to apply the technique to missing data imputation. A baseline imputation technique, expectation maximization/data augmentation (EM/DA), is selected for comparison because of its proved effectiveness. Experimental results demonstrate that our method is more applicable and performs better at relatively high missing data rates. This reveals that it is a promising approach in the field. [ABSTRACT FROM AUTHOR]