Search

This paper addresses the critical issue of missing data in power demand time series by emphasizing the relevance of imputation-based approaches in data-driven technologies. A comparative analysis of imputation methods is performed, where the reference from the state of the art is selected as K-Nearest Neighbors (KNN) applied in the time domain. Two innovative methods are proposed. The former method is defined as Historical Data Informed Regression Technique (H-DIRT) and is based on incorporating historical data for setting up a multivariate linear regression and then imputing through the estimated relation between the missing power demand measurement and the historical data. When the available historical data are insufficient, the algorithm proceeds by averaging or by a linear interpolation between the first available measurement before and after the missing value. The latter proposed method is defined as Seasonal KNN (SKNN) and it is based on enriching the data set with features related to yearly, seasonal, weekly and daily trends and then proceeding by baseline KNN. Experiments are set up with random and continuous data clipping, even with rather extreme pruning (up 70% of the data). The results in general demonstrate a significant improvement in imputation accuracy compared to the state of the art. The average error metrics (like Mean Absolute Error and Root Mean Square Error) for the SKNN method are in the order of respectively one third and one half those of the baseline KNN, in the cases of random and continuous data clipping. In general, the SKNN method provides more accurate results and better captures the statistical features of the data set to impute. Anyway, if the share of data to impute is not too large, the H-DIRT method provides comparable accuracy at a much lower computational cost. Hence, this study presents an easily implementable and computationally affordable approach for improving, in various contexts, the state of the art in power demand data imputation. It establishes a foundation for future exploration into trends, seasonal factors, and external variables influencing power load parameters.

The emergence of single-cell RNA sequencing (scRNA-seq) has brought to light the critical need for scrutinizing transcriptomes at the individual cellular level with unparalleled precision. A pivotal aspect of scRNA-seq data analysis involves cell identification, commonly accomplished through diverse clustering methodologies. However, scRNA-seq datasets frequently encounter missing values due to technical limitations, posing a significant challenge that can compromise the accuracy of clustering outcomes. In response, we present a novel approach that seamlessly integrates missing value estimation with scRNA-seq clustering. Our method harnesses the power of an imputation autoencoder network to predict missing values, coupled with the deployment of a deep clustering network for efficient cell categorization. To mitigate the risk of deep clustering networks converging towards suboptimal local minima, we have devised a self-guided learning strategy. This approach exploits shared parameters between the imputation and clustering networks, fostering a symbiotic relationship that enhances overall performance. Through rigorous empirical evaluations, we substantiate the effectiveness of our methodology, demonstrating its comparability to, or surpassing, several established single-cell clustering techniques. Furthermore, our analysis of cellular trajectories underscores the proficiency of the proposed method in accurately deducing cellular trajectories by leveraging the clustering results to discern biologically meaningful cell types.

Health-related data has a decisive role in disease diagnosis. Collecting relevant information from health-related data in medical records has been facilitated by evaluating the features of the data. Relevant research has shown that outcomes are significantly impacted by the use of feature selection (FS) in different medical domain data. FS provides an analysis of the most significant features to improve classification accuracy. The FS technique aims at minimizing the number of input variables and computational overload to maximize classification performance results. However, identifying the optimal features poses issues due to the high dimensionality of large features and the small sample size of health-related data. The metaheuristics optimization algorithm (MOA) plays an important role in generating the best subset features with exploration and exploitation phases. This study experiments with well-known MOAs and supervised learning from the UC Irvine Machine Learning Repository, PhysioNet, Kent Ridge Bio-Medical Dataset, and MIMIC-III v1.4 Repository with varying feature dimensions. To increase the quality of health-related data, this study proposes missing data imputation based on a deep learning approach, an autoencoder (AE). With AE imputation, the performance results obtain 0.0167 mean squared error (MSE) and 0.129 root mean squared error (RMSE). As a result, MOA shows its excellence in achieving minimal features, but still outstanding performance in low- and high-dimensional data. MOA is successfully applied to varying diverse health-related datasets with low- and high-dimensional data.

Abstract Spatial omics technologies can help identify spatially organized biological processes, but existing computational approaches often overlook structural dependencies in the data. Here, we introduce Smoother, a unified framework that integrates positional information into non-spatial models via modular priors and losses. In simulated and real datasets, Smoother enables accurate data imputation, cell-type deconvolution, and dimensionality reduction with remarkable efficiency. In colorectal cancer, Smoother-guided deconvolution reveals plasma cell and fibroblast subtype localizations linked to tumor microenvironment restructuring. Additionally, joint modeling of spatial and single-cell human prostate data with Smoother allows for spatial mapping of reference populations with significantly reduced ambiguity.

In this paper, we propose a novel methodology for estimating missing data in energy consumption datasets. Conventional data imputation methods are not suitable for these datasets, because they are time series with special characteristics and because, for some applications, it is quite important to preserve the shape of the daily energy profile. Our answer to this need is the use of autoencoders. First, we split the problem into two subproblems: how to estimate the total amount of daily energy, and how to estimate the shape of the daily energy profile. We encode the shape as a new feature that can be modeled and predicted using autoencoders. In this way, the problem of imputation of profile data are reduced to two relatively simple problems on which conventional methods can be applied. However, the two predictions are related, so special care should be taken when reconstructing the profile. We show that, as a result, our data imputation methodology produces plausible profiles where other methods fail. We tested it on a highly corrupted dataset, outperforming conventional methods by a factor of 3.7.

Accurate load forecasting, especially in the short term, is crucial for the safe and stable operation of power systems and their market participants. However, as modern power systems become increasingly complex, the challenges of short-term load forecasting are also intensifying. To address this challenge, data-driven deep learning techniques and load aggregation technologies have gradually been introduced into the field of load forecasting. However, data quality issues persist due to various factors such as sensor failures, unstable communication, and susceptibility to network attacks, leading to data gaps. Furthermore, in the domain of aggregated load forecasting, considering the potential interactions among aggregated loads can help market participants engage in cross-market transactions. However, aggregated loads often lack clear geographical locations, making it difficult to predefine graph structures. To address the issue of data quality, this study proposes a model named adversarial graph convolutional imputation network (AGCIN), combined with local and global correlations for imputation. To tackle the problem of the difficulty in predefining graph structures for aggregated loads, this study proposes a learnable adjacency matrix, which generates an adaptive adjacency matrix based on the relationships between different sequences without the need for geographical information. The experimental results demonstrate that the proposed imputation method outperforms other imputation methods in scenarios with random and continuous missing data. Additionally, the prediction accuracy of the proposed method exceeds that of several baseline methods, affirming the effectiveness of our approach in imputation and prediction, ultimately enhancing the accuracy of aggregated load forecasting.

The Internet of Things (IoT) is leading the physical and digital world of technology to converge. Real-time and massive scale connections produce a large amount of versatile data, where Big Data comes into the picture. Big Data refers to large, diverse sets of information with dimensions that go beyond the capabilities of widely used database management systems, or standard data processing software tools to manage within a given limit. Almost every big dataset is dirty and may contain missing data, mistyping, inaccuracies, and many more issues that impact Big Data analytics performances. One of the biggest challenges in Big Data analytics is to discover and repair dirty data; failure to do this can lead to inaccurate analytics results and unpredictable conclusions. Different imputation methods were employed in the experimentation with various missing value imputation techniques, and the performances of machine learning (ML) models were compared. A hybrid model that integrates ML and sample-based statistical techniques for missing value imputation is being proposed. Furthermore, the continuation involved the dataset with the best missing value imputation, chosen based on ML model performance for subsequent feature engineering and hyperparameter tuning. K-means clustering and principal component analysis were applied in our study. Accuracy, the evaluated outcome, improved dramatically and proved that the XGBoost model gives very high accuracy at around 0.125 root mean squared logarithmic error (RMSLE). To overcome overfitting, K-fold cross-validation was implemented.

A continuous and multi-decadal surface water extent (SWE) record is vital for water resources management, flood risk assessment, and comprehensive climate change impact studies. The advancements in remote sensing technologies offer a valuable tool for monitoring surface water with high temporal and spatial resolution. However, challenges persist due to image gaps resulting from sensor issues and adverse weather conditions during data collection. To address this issue, one way to fill the gaps is by leveraging in situ measurements such as streamflow discharges (SFDs). We investigate the relationship between SFDs and Landsat-derived SWE in the New England region watersheds (eight-digit hydrological unit code (HUC)) on a monthly scale. While previous studies indicate the relationship exists, it remains elusive for larger domains. Recent research suggests using monthly average SFD data from a single stream gage to fill the gaps in SWE. However, as SWE represents a monthly maximum value, relying on a single gage with average values may not capture the complex dynamics of surface water. Our study introduces a novel approach by replacing the monthly average SFD with the maximum day streamflow discharge anomaly (SFDA) within a month. This adjustment aims to better reflect extreme scenarios, and we explore the relationship using ridge regression, incorporating data from all stream gages in the study domain. The SWE and SFDA are both transformed to stabilize the variance. We found that there is no discernible correlation between the magnitude of the correlation and the size of the basins. The correlations vary based on HUC and display a wide range, indicating the variances of the importance of stream gages to each HUC. The maximum correlation is found when the stream gage is located outside of the target HUC, further verifying the complex relationship between SWE and SFDA. Covering over 30 years of data across 45 HUCs, the imputing technique using ridge regression shows satisfactory performance for most of the HUCs analyzed. The results show that 41 out of 45 HUCs achieve a root-mean-square error (RMSE) of less than 10, and 44 out of 45 HUCs exhibit a normalized root-mean-square error (NRMSE) of less than 0.1. Of 45 HUCs, 42 have an R-squared (R2) score higher than 0.7. The Nash–Sutcliffe efficiency index (Ef) shows consistent results with R2, with the relative bias ranging from –0.02 to 0.03. The established relationship serves as an effective imputing technique, filling gaps in the time series of SWE. Moreover, our approach facilitates the identification and visualization of the most significant gages for each HUC, contributing to a more refined understanding of surface water dynamics. [ABSTRACT FROM AUTHOR]

Length, completeness, and quality of hydrological time-series can affect considerably the efficiency of decisions in water resources engineering. Regrettably, obtaining short, incomplete, and low-quality data is not rare. In this study, different machine learning techniques have been implemented and applied to fill in missed data of streamflow at Coxs River, in Australia. The implemented techniques are Support Vector Regression improved by Equilibrium Optimizer (SVR-EO) and Particle Swarm Optimizer (SVR-PSO), alongside Artificial Neural Networks trained by EO (ANN-EO) and PSO (ANN-PSO). Multivariate Adaptive Regression Splines (MARS) and Multiple Linear Regression (MLR) have been used for comparison purposes. Rainfall data provided by five climatic stations located near Coxs River along with Kowmung River streamflow records have been used to fill the gaps in the Coxs River time-series. The gamma test has been used to select the convenient data combination that reduces errors in prediction models. According to the findings, SVR-PSO and SVR-EO outperformed the other techniques with R 2 ≈ 0.94 for training, and R 2 ≈ 0.85 for testing part. The imputation process and the developed SVR-EO and SVR-PSO could be applied to other rivers in different countries to ensure whether these methods could be generalized. [ABSTRACT FROM AUTHOR]

Multi-dimensional data are common in many applications, such as videos and multi-variate time series. While tensor decomposition (TD) provides promising tools for analyzing such data, there still remains several limitations. First, traditional TDs assume multi-linear structures of the latent embeddings, which greatly limits their expressive power. Second, TDs cannot be straightforwardly applied to datasets with massive samples. To address these issues, we propose a nonparametric TD with amortized inference networks. Specifically, we establish a non-linear extension of tensor ring decomposition, using neural networks, to model complex latent structures. To jointly model the cross-sample correlations and physical structures, a matrix Gaussian process (GP) prior is imposed over the core tensors. From learning perspective, we develop a VAE-like amortized inference network to infer the posterior of core tensors corresponding to new tensor data, which enables TDs to be applied to large datasets. Our model can be also viewed as a kind of decomposition of VAE, which can additionally capture hidden tensor structure and enhance the expressiveness power. Finally, we derive an evidence lower bound such that a scalable optimization algorithm is developed. The advantages of our method have been evaluated extensively by data imputation on the Healing MNIST dataset and four multi-variate time series data. [ABSTRACT FROM AUTHOR]

Ratings are frequently used to evaluate and compare subjects in various applications, from education to healthcare, because ratings provide succinct yet credible measures for comparing subjects. However, when multiple rating lists are combined or considered together, subjects often have missing ratings, because most rating lists do not rate every subject in the combined list. In this study, we propose analyses on missing value patterns using six real-world data sets in various applications, as well as the conditions for applicability of imputation algorithms. Based on the special structures and properties derived from the analyses, we propose optimization models and algorithms that minimize the total rating discordance across rating providers to impute missing ratings in the combined rating lists, using only the known rating information. The total rating discordance is defined as the sum of the pairwise discordance metric, which can be written as a quadratic function. Computational experiments based on real-world and synthetic rating data sets show that the proposed methods outperform the state-of-the-art general imputation methods in the literature in terms of imputation accuracy. [ABSTRACT FROM AUTHOR]

Data imputation is prevalent in crowdsensing, especially for Internet of Things (IoT) devices. On the one hand, data collected from sensors will inevitably be affected or damaged by unpredictability. On the other hand, extending the active time of sensor networks has urgently aspired environmental monitoring. Using neural networks to design a data imputation algorithm can take advantage of the prior information stored in the models. This paper proposes a preprocessing algorithm to extract a subset for training a neural network on an IoT dataset, including time window determination, sensor aggregation, sensor exclusion and data frame shape selection. Moreover, we propose a data imputation algorithm using deep compressed sensing with generative models. It explores novel representation matrices and can impute data in the case of a high missing ratio situation. Finally, we test our subset extraction algorithm and data imputation algorithm on the EPFL SensorScope dataset, respectively, and they effectively improve the accuracy and robustness even with extreme data loss. [ABSTRACT FROM AUTHOR]

Temperature prediction is an important and significant step for monitoring global warming and the environment to save and protect human lives. The climatology parameters such as temperature, pressure, and wind speed are time-series data and are well predicted with data driven models. However, data-driven models have certain constraints, due to which these models are unable to predict the missing values and erroneous data caused by factors like sensor failure and natural disasters. In order to solve this issue, an efficient hybrid model, i.e., attention-based bidirectional long short term memory temporal convolution network (ABTCN) architecture is proposed. ABTCN uses k-nearest neighbor (KNN) imputation method for handling the missing data. A bidirectional long short term memory (Bi-LSTM) network with self-attention mechanism and temporal convolutional network (TCN) model that aids in the extraction of features from complex data and prediction of long data sequence. The performance of the proposed model is evaluated in comparison to various state-of-the-art deep learning models using error metrics such as MAE, MSE, RMSE, and R2 score. It is observed that our proposed model is superior over other models with high accuracy. [ABSTRACT FROM AUTHOR]

Supervisory control and data acquisition (SCADA) systems are widely utilized in power equipment for condition monitoring. For the collected data, there generally exists a problem—missing data of different types and patterns. This leads to the poor quality and utilization difficulties of the collected data. To address this problem, this paper customizes methodology that combines an asymmetric denoising autoencoder (ADAE) and moving average filter (MAF) to perform accurate missing data imputation. First, convolution and gated recurrent unit (GRU) are applied to the encoder of the ADAE, while the decoder still utilizes the fully connected layers to form an asymmetric network structure. The ADAE extracts the local periodic and temporal features from monitoring data and then decodes the features to realize the imputation of the multi-type missing. On this basis, according to the continuity of power data in the time domain, the MAF is utilized to fuse the prior knowledge of the neighborhood of missing data to secondarily optimize the imputed data. Case studies reveal that the developed method achieves greater accuracy compared to existing models. This paper adopts experiments under different scenarios to justify that the MAF-ADAE method applies to actual power equipment monitoring data imputation. [ABSTRACT FROM AUTHOR]

Traffic state data are key to the proper operation of intelligent transportation systems (ITS). However, traffic detectors often receive environmental factors that cause missing values in the collected traffic state data. Therefore, aiming at the above problem, a method for imputing missing traffic state data based on a Diffusion Convolutional Neural Network–Generative Adversarial Network (DCNN-GAN) is proposed in this paper. The proposed method uses a graph embedding algorithm to construct a road network structure based on spatial correlation instead of the original road network structure; through the use of a GAN for confrontation training, it is possible to generate missing traffic state data based on the known data of the road network. In the generator, the spatiotemporal features of the reconstructed road network are extracted by the DCNN to realize the imputation. Two real traffic datasets were used to verify the effectiveness of this method, with the results of the proposed model proving better than those of the other models used for comparison. [ABSTRACT FROM AUTHOR]

Due to the overall declining costs of photovoltaic systems, market players in the operation and maintenance sector are under increasing price pressure when offering their services. The automation and standardization of maintenance and failure tickets as well as their statistical and economical evaluation are key to ensure optimal yield and long lifetime. A thorough understanding of typical faults, classified through a standardized taxonomy, can be a pathway of developing location and technology specific decision support, offering cost‐time efficient solutions to reduce component downtime and costs in case of failure appearance. A useful method for this approach is the Cost Priority Number, a methodology to assess technical failures and their economic impact in energy systems. In this work, this method is further improved to be applied to individual use cases to make it useful in the actual operation of PV plants. A standardized ticket taxonomy for the operational phase of PV systems has been developed where more than 35,000 PV systems' tickets have been statistically evaluated, and a fully automated methodology to calculate the cost of individual maintenance tickets has been developed. [ABSTRACT FROM AUTHOR]

In the realm of Internet of Things (IoT) sensor data, missing patterns often occur due to sensor glitches and communication problems. Conventional missing data imputation methods struggle to handle multiple missing patterns, as they fail to fully leverage the available data as well as partially imputed data. To address this challenge, we propose a novel approach called Univariate data Imputation using Fast Similarity Search (UIFSS). The proposed method solved the missing data problem of IoT data using fast similarity search that can suits different patterns of missingness. Exploring similarities between data elements, a problem known as all-pairs-similarity-search, has been extensively studied in fields like text analysis. Surprisingly, applying this concept to time series subsequences hasn’t seen much progress, likely due to the complexity of the task. Even for moderately sized datasets, the traditional approach can take a long time, and common techniques to speed it up only help a bit. Notably, for very large datasets, our algorithm can be easily adapted to produce high-quality approximate results quickly. UIFSS consists of two core components:Sensor sorting with Similar Node Clustering (SSNC) and Imputation Estimator using Fast Similarity Search(IEFSS). The SSNC, encompassing missing sensor sorting depending on their entropy to guide the imputation process. Subsequently, IEFSS uses global similar sensors and captures local region volatility, prioritizing data preservation while improving accuracy through z-normalized query based similarity search. Through experiments on simulated and bench mark datasets, UIFSS outperforms existing methods across various missing patterns. This approach offers a promising solution for handling missing IoT sensor data and with improved imputation accuracy. [ABSTRACT FROM AUTHOR]

Retinal blood vessel segmentation is a valuable tool for clinicians to diagnose conditions such as atherosclerosis, glaucoma, and age-related macular degeneration. This paper presents a new framework for segmenting blood vessels in retinal images. The framework has two stages: a multi-layer preprocessing stage and a subsequent segmentation stage employing a U-Net with a multi-residual attention block. The multi-layer preprocessing stage has three steps. The first step is noise reduction, employing a U-shaped convolutional neural network with matrix factorization (CNN with MF) and detailed U-shaped U-Net (D_U-Net) to minimize image noise, culminating in the selection of the most suitable image based on the PSNR and SSIM values. The second step is dynamic data imputation, utilizing multiple models for the purpose of filling in missing data. The third step is data augmentation through the utilization of a latent diffusion model (LDM) to expand the training dataset size. The second stage of the framework is segmentation, where the U-Nets with a multi-residual attention block are used to segment the retinal images after they have been preprocessed and noise has been removed. The experiments show that the framework is effective at segmenting retinal blood vessels. It achieved Dice scores of 95.32, accuracy of 93.56, precision of 95.68, and recall of 95.45. It also achieved efficient results in removing noise using CNN with matrix factorization (MF) and D-U-NET according to values of PSNR and SSIM for (0.1, 0.25, 0.5, and 0.75) levels of noise. The LDM achieved an inception score of 13.6 and an FID of 46.2 in the augmentation step. [ABSTRACT FROM AUTHOR]

Imputation methods were developed to define estimates for missing data and hence solve possible problems generated by the loss of this information. This study aims to assess whether data variability influences the results obtained after applying an imputation method. Incomplete databases were generated from complete real databases of experiments of tomato plants conducted using the randomized block design with three replications and 12 treatments by removing different amounts of data. The evaluated variables consisted of fruit weight per plant, number of fruits per plant, and average fruit length and width, forming eight balanced databases. Subsequently, the distribution-free multiple imputation method was applied, generating complete databases from imputation. The number of missing information influenced the accuracy measures for the data in this study. Data imputation was inadequate when there was high variability but more precise and accurate in cases of low variability. It confirmed the importance of assessing data variability before choosing to apply the imputation method.

Data missing is a ubiquitous problem in real-world systems that adversely affects the performance of machine learning algorithms. Although many useful imputation methods are available to address this issue, they often fail to consider the information provided by both features and labels. As a result, the performance of these methods might be constrained. Furthermore, feature selection as a data quality improvement technique has been widely used and has demonstrated its efficiency. To overcome the limitation of imputation methods, we propose a novel algorithm that combines data imputation and feature selection to tackle classification problems for mixed data. Based on the mean and standard deviation of quantitative features and the selecting probabilities of unique values of categorical features, our algorithm constructs different imputation models for quantitative and categorical features. Particle swarm optimization is used to optimize the parameters of the imputation models and select feature subsets simultaneously. Additionally, we introduce a legacy learning mechanism to enhance the optimization capability of our method. To evaluate the performance of the proposed method, seven algorithms and twelve datasets are used for comparison. The results show that our algorithm outperforms other algorithms in terms of accuracy and F1 score and has reasonable time overhead.

In this current era, the identification of both known and novel cell types, the representation of cells, predicting cell fates, classifying various tumor types, and studying heterogeneity in various cells are the key areas of interest in the analysis of single-cell RNA sequencing (scRNA-seq) data. Due to the nature of the data, cluster identification in single-cell sequencing data with high dimensions presents several difficulties. In this paper, we introduce a new framework that combines various strategies such as imputed matrix, minimum redundancy maximum relevance (MRMR) feature selection, and shrinkage clustering to discover gene signatures from scRNA-seq data. Firstly, we conducted the pre-filtering of the "drop-out" value in the data focusing solely on imputing the identified "drop-out" values. Next, we applied the MRMR feature selection method to the imputed data and obtained the top 100 features based on the MRMR feature selection optimization scores for further downstream analysis. Thereafter, we employed shrinkage clustering on the selected feature matrix to identify the cell clusters using a global optimization approach. Finally, we applied the Limma-Voom R tool employing voom normalization and an empirical Bayes test to detect differentially expressed features with a false discovery rate (FDR) < 0.001. In addition, we performed the KEGG pathway and gene ontology enrichment analysis of the identified biomarkers using David 6.8 software. Furthermore, we conducted miRNA target detection for the top gene markers and performed miRNA target gene interaction network analysis using the Cytoscape online tool. Subsequently, we compared our detected 100 markers with our previously detected top 100 cluster-specified markers ranked by FDR of the latest published article and discovered three common markers; namely, C y p 2 b 10 , M t 1 , A l p i , along with 97 novel markers. In addition, the Gene Set Enrichment Analysis (GSEA) of both marker sets also yields similar outcomes. Apart from this, we performed another comparative study with another published method, demonstrating that our model detects more significant markers than that model. To assess the efficiency of our framework, we apply it to another dataset and identify 20 strongly significant up-regulated markers. Additionally, we perform a comparative study of different imputation methods and include an ablation study to prove that every key phase of our framework is essential and strongly recommended. In summary, our proposed integrated framework efficiently discovers differentially expressed stronger gene signatures as well as up-regulated markers in single-cell RNA sequencing data. [ABSTRACT FROM AUTHOR]

To tackle the issue of traffic data loss due to various software/hardware failures in urban road network environments, this paper proposed a traffic data imputation method based on spatial-temporal residual tensor learning (ST-RTL）. This method constructed a 3D traffic tensor with missing value to characterize original spatiotemporal attributes of road network maxi-mally. Then it adopted Gibbs sampling to perform a CANDECOMP/PARAFAC (CP) tensor decomposition and low-rank reconstruction of missing traffic data based on the assumption of Gaussian distribution. Considering the residual value produced by the tensor repair process, the study designed a bidirectional residual optimization structure with dynamic iterations to capture the residual spatiotemporal dependencies to enable the accurate repair of the missing traffic data. The experiments took a publicly available Hangzhou metro passenger flow for model construction and validation. The results indicate that when the missing rates are 10%～80%, the three missing scenarios (random, cluster and hybrid missing) have large differences on tensor structure damage, among which cluster missing has the greatest destruction and the evaluation indexes MAPE, RMSE and MAE of ST-RTL lied in 3.1071～7.0371, 16.3779～58.4286 and 3.7434～8.0135; and each indicator of ST-RTL model shows an accelerated increasing trend as the missing rate rises. Compared with the representative baseline models such as HaLRTC, GAIN and BGCP, the ST-RTL exhibits lower performance metrics and stronger stability in the acceptable computational costs, which can provide high-quality basic data for intelligent transportation systems. [ABSTRACT FROM AUTHOR]

Evaluating data imputation and augmentation performance is a critical issue in data science. In statistics, methods like Kolmogorov-Smirnov K-S test, Cramér-von Mises W 2 , Anderson-Darling A 2 , Pearson's χ 2 and Watson's U 2 exists for decades to compare the distribution of two datasets. In the context of data generation, typical evaluation metrics have the same flaw: They calculate the feature's error and the global error on the generated data without weighting the error with the feature's importance. In most cases, the importance of the features is imbalanced, and it can induce a bias on the features and global errors. This paper proposes a novel metric named "Explainable Global Error Weighted on Feature Importance" (xGEWFI). This new metric is tested in a whole preprocessing method that 1. Process the outliers, 2. impute the missing data, and 3. augments the data. At the end of the process, the xGEWFI error is calculated. The distribution error between the original and generated data is calculated using a Kolmogorov-Smirnov test (K-S test) for each feature. Those results are multiplied by the importance of the respective features and calculated using a Random Forest (RF) algorithm. The metric result is expressed in an explainable format, aiming for an ethical AI. This novel method provides a more precise evaluation of a data generation process than if only a K-S test were used. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) is an important component in a number of health applications, including rehabilitation, Parkinson's disease, daily activity monitoring, and fitness monitoring. State-of-the-art HAR approaches use multiple sensors on the body to accurately identify activities at runtime. These approaches typically assume that data from all sensors are available for runtime activity recognition. However, data from one or more sensors may be unavailable due to malfunction, energy constraints, or communication challenges between the sensors. Missing data can lead to significant degradation in the accuracy, thus affecting quality of service to users. A common approach for handling missing data is to train classifiers or sensor data recovery algorithms for each combination of missing sensors. However, this results in significant memory and energy overhead on resource-constrained wearable devices. In strong contrast to prior approaches, this paper presents a clustering-based approach (CIM) to impute missing data at runtime. We first define a set of possible clusters and representative data patterns for each sensor in HAR. Then, we create and store a mapping between clusters across sensors. At runtime, when data from a sensor are missing, we utilize the stored mapping table to obtain most likely cluster for the missing sensor. The representative window for the identified cluster is then used as imputation to perform activity classification. We also provide a method to obtain imputation-aware activity prediction sets to handle uncertainty in data when using imputation. Experiments on three HAR datasets show that CIM achieves accuracy within 10% of a baseline without missing data for one missing sensor when providing single activity labels. The accuracy gap drops to less than 1% with imputation-aware classification. Measurements on a low-power processor show that CIM achieves close to 100% energy savings compared to state-of-the-art generative approaches. [ABSTRACT FROM AUTHOR]

A key problem in systems biology is the discovery of regulatory mechanisms that drive phenotypic behaviour of complex biological systems in the form of multi-level networks. Modern multi-omics profiling techniques probe these fundamental regulatory networks but are often hampered by experimental restrictions leading to missing data or partially measured omics types for subsets of individuals due to cost restrictions. In such scenarios, in which missing data is present, classical computational approaches to infer regulatory networks are limited. In recent years, approaches have been proposed to infer sparse regression models in the presence of missing information. Nevertheless, these methods have not been adopted for regulatory network inference yet. In this study, we integrated regression-based methods that can handle missingness into KiMONo, a Knowledge guided Multi-Omics Network inference approach, and benchmarked their performance on commonly encountered missing data scenarios in single- and multi-omics studies. Overall, two-step approaches that explicitly handle missingness performed best for a wide range of random- and block-missingness scenarios on imbalanced omics-layers dimensions, while methods implicitly handling missingness performed best on balanced omics-layers dimensions. Our results show that robust multi-omics network inference in the presence of missing data with KiMONo is feasible and thus allows users to leverage available multi-omics data to its full extent. [ABSTRACT FROM AUTHOR]