Search

Missing data imputation is a technique to deal with incomplete datasets. Since many models and algorithms cannot be applied to data containing missing values, a pre-processing step needs to be performed to remove incomplete data or to estimate the missing values. This is a well-known problem referred to as the data imputation problem. Several approaches have been designed for data imputation. These algorithms can be divided into two main categories: statistical and machine learning-based algorithms. As machine learning algorithms are optimized, they usually have better performance compared with statistical ones. In this paper, we review the most recent literature related to missing data imputation based on generative adversarial networks (GANs) that have gained tremendous attention in dealing with missing values. We examine the structures of GANs for missing data imputation and discuss the commonly used datasets and metrics for evaluation. We also cover the influence of the missing datatype, the effect of the missing data fraction, and the algorithm-related problems on data imputation performance. We conduct experiments on two publicly available datasets and evaluate the performance of GAIN, a missing data imputation algorithm to that of existing state-of-the-art approaches, demonstrating that the GAN-based algorithm outperforms the others in terms of RMSE and FID.

Images are increasingly used for AI-based diagnosis and analysis of many diseases like cervical cancer, mouth cancer, glucose analysis from retina etc. In many cases, data collection is done by specialised camera modules which capture images of affected areas. As with any other sources of data, this process is also error-prone and may contain unwanted objects and regions that may require cleaning by removing them. Outliers in these kinds of dataset may adversely affect the performance of machine learning models. Manually cleaning would be a tedious task, especially when the data is collated from different sources. Hence, cleaning the data before training the model is of utmost importance. In this paper, we propose a Few-Shot learning based model pre-trained in supervised contrastive learning settings to automate the process of data cleaning. Our model learns the dataset distribution and distinguishes the accurate data points from noisy data points. We also show that scaling up the model can greatly improve the Few-Shot performance. On the noisy MobileODT cervical data, which was collected from Kaggle, our model obtained 52% accuracy without cleaning data using an EfficientNet architecture for the classification task. Whereas the same architecture with ROI cropping achieved an accuracy of 76.56% after cleaning through the proposed Deep Cleaner approach that requires only 100 clean images. The proposed approach performs 2.74% better than a denoising auto-encoder, which is considered a powerful anomaly detection technique.

Data analysis is increasingly performed over data assembled from uncontrolled sources, facing inconsistency in knowledge-representation conventions. The typical practice is to create “clean” data for analysis, matching entities and merging variants to overcome differences in knowledge representation. Despite progress in data management techniques to automate this process, it still needs labor-intensive supervision from the analyst. In this paper, we evaluate the benefit of advanced statistical tools to address directly many analytic tasks across data sources without such entity-matching cleaning. Reframing analytical questions as machine-learning tasks enables to replace exact matching of entities by continuous descriptions–vectorial embeddings– that expose similarities between entries. But are analyses with less cleaning trustworthy? We answer this question with a thorough benchmark on questions typical of socio-economic studies across 14 employee databases: we compare the approaches based on machine learning to manual data cleaning (entity matching). It reveals that using embeddings and machine learning improves results validity (smaller estimation error) more than manual cleaning, with considerably less human labor. While machine learning is often combined with data management for the purpose of cleaning, our study suggests that using it directly for analysis is beneficial because it captures ambiguities hard to represent during curation.

As the importance of utility condition is increasingly acknowledged, the use of asset management technologies in the electric power industry has rapidly grown. The global trend of asset management follows risk management that accounts for the probability and consequences of failures. Because asset management systems tend to be composed of various legacy systems, each of which is installed and designed to collect data according to a certain data type and acquisition purpose, it is necessary to develop a system that cleans and integrates data acquired from each legacy system. This study explores the development of an asset management system for a transmission system as a representative linear asset consisting of different segments in a sequence. First, the configurations and characteristics of linear asset datasets are analyzed. Second, an automatic data cleaning system, equipped with six types of data cleaning functions for extracting dirty data from entire datasets, is proposed. An algorithm for data imputation, which is essential for estimating the remaining useful life, is developed based on principal component analysis–iterative algorithm (PCA–IA). Afterward, the performance of the proposed system is verified using actual data with the help of the Korea Electric Power Corporation (KEPCO). Specifically, to evaluate the performance of the proposed system, an automatic cleaning process is demonstrated using actual legacy datasets.

A growing attention is given to exploiting Photoplethysmography (PPG) signals in non-invasively measuring of many physiological vital signs. Many machine deep learning models were trained for predicting the continuous arterial blood pressure (ABP) or just the systolic and diastolic blood pressure (BP) values based on a public database. However, jointly cleaning the PPG-ABP dataset that is the most critical pre-processing step for training quality is still in need for more investigations. There is a considerable amount of anomaly data that has to be excluded before any training stage. This paper introduces a two-level joint PPG-ABP cleaning technique conducted in a signal level and per-beat level. Many quality metrics have been checked successively for excluding improper data. These metrics achieve a coarse cleaning step. Finally, principal component analysis (PCA) is exploited for fine cleaning the remaining data from the former stage. The cleaning efficiency is evaluated by measuring its impact on the deep-learning-based BP estimation models. The trained model based on our cleaned data shows performance enhancement in terms of prediction error and the correlation between the predicted and ground-truth BP. Segmented and cleaned PPG/ABP dataset will be publicly available in both signal level and beat level. Based on the simulation results, the proposed cleaning technique enhances standard deviation of the prediction error of systolic and diastolic blood pressure by 11.68 % and 10.81 %, respectively. Also, it enhances mean absolute error of the prediction of systolic and diastolic blood pressure by 14.79 % and 11.70 %, respectively.

Classical data cleaning performs a minimal set of operations on the data to satisfy the given integrity constraints. Often, this minimization is equivalent to vertex cover, for example when tuples can be removed due to the violation of functional dependencies. Classically, the uncertainty of tuples and constraints is ignored. We propose not to view data as dirty but the uncertainty information about data. Since probabilities are often unavailable and their treatment is limited due to correlations in the data, we investigate a qualitative approach to uncertainty. Tuples are assigned degrees of possibility with which they occur, and constraints are assigned degrees of certainty that say to which tuples they apply. Our approach is non-invasive to the data as we lower the possibility degree of tuples as little as possible. The new resulting qualitative version of vertex cover remains NP-hard. We establish an algorithm that is fixed-parameter tractable in the size of the qualitative vertex cover. Experiments with synthetic and real-world data show that our algorithm outperforms the classical algorithm proportionally to the available number of uncertainty degrees. By mining the certainty degrees with which constraints hold, our framework becomes applicable even when uncertainty information is unavailable. [ABSTRACT FROM AUTHOR]

With the increase of dirty data, data cleaning turns into a crux of data analysis. The accuracy limitation of the existing integrity constraints-based cleaning approaches results from insufficient rules. In this paper, we present a novel hybrid data cleaning framework on top of Markov logic networks (MLNs), termed as ${\sf MLNClean}$ MLNClean , which is capable of learning instantiated rules to supplement the insufficient integrity constraints. ${\sf MLNClean}$ MLNClean consists of two steps, i.e., pre-processing and two-stage data cleaning. In the pre-processing step, ${\sf MLNClean}$ MLNClean first infers a set of probable instantiated rules according to MLNs and then builds a two-layer MLN index structure to generate multiple data versions and facilitate the cleaning process. In the two-stage data cleaning step, ${\sf MLNClean}$ MLNClean first presents a concept of reliability score to clean errors within each data version separately, and afterward eliminates the conflict values among different data version using a novel concept of fusion score. Considerable experimental results on both real and synthetic scenarios demonstrate the effectiveness of ${\sf MLNClean}$ MLNClean in practice. [ABSTRACT FROM AUTHOR]

Bridge condition assessment (BCA) plays an important role in modern bridge management. Existing assessment methods are time-consuming, labor-intensive and error-prone. The use of machine learning for BCA can effectively solve the above problems. However, the large amount of label noise in the dataset severely affected the performance of the BCA model. In this paper, we present an effective label ranking approach for BCA (LR-BCA). Our proposed LR-BCA method considers the natural order relationship between bridge condition ratings. Moreover, a heuristic data cleaning (HDC) approach is proposed for cleaning bridge condition dataset. The HDC method firstly identifies all the label conflict examples, then iteratively filters out the noise. Experimental results on real-world dataset confirm the effectiveness of the HDC method and demonstrate that our proposed LR-BCA method achieves 99% Top-2 accuracy, which is highly competitive compared to baseline methods.

Data quality plays a key role in big data management today. With the explosive growth of data from a variety of sources, the quality of data is faced with multiple problems. Motivated by this, we study the multiple data cleaning on incompleteness and inconsistency with currency reasoning and determination in this paper. We introduce a 4-step framework, named ${\sf Imp3C}$ Imp 3 C , for errors detection and quality improvement in incomplete and inconsistent data without timestamps. We achieve an integrated currency determining method to compute the currency orders among tuples, according to currency constraints. Thus, the inconsistent data and missing values are repaired effectively considering the temporal impact. For both effectiveness and efficiency consideration, we carry out inconsistency repair ahead of incompleteness repair. A currency-related consistency distance metric is defined to measure the similarity between dirty tuples and clean ones more accurately. In addition, currency orders are treated as an important feature in the missing imputation training process. The solution algorithms are introduced in detail with case studies. A thorough experiment on three real-life datasets verifies our method ${\sf Imp3C}$ Imp 3 C improves the performance of data repairing with multiple quality problems. ${\sf Imp3C}$ Imp 3 C outperforms the existing advanced methods, especially in the datasets with complex currency orders. [ABSTRACT FROM AUTHOR]

Errors are prevalent in time series data, which is particularly common in the industrial field. Data with errors could not be stored in the database, which results in the loss of data assets. At present, to deal with these time series containing errors, besides keeping original erroneous data, discarding erroneous data and manually checking erroneous data, we can also use the cleaning algorithm widely used in the database to automatically clean the time series data. This survey provides a classification of time series data cleaning techniques and comprehensively reviews the state-of-the-art methods of each type. Besides we summarize data cleaning tools, systems and evaluation criteria from research and industry. Finally, we highlight possible directions time series data cleaning.

Agriculture is the backbone of a country’s economic system, considering that it not only provides food and raw materials but also employment opportunities for a large percentage of the population. In this way, determining the degree of agricultural vulnerability represents a guide for sustainability and adaptability focused on changing future conditions. In many cases, vulnerability analysis data is restricted to use by authorized personnel only, leaving open data policies aside. Furthermore, data in its native format (raw data) by nature tend to be diverse in structure, storage formats, and access protocols. In addition, having a large amount of open data is important (though not sufficient) to obtain accurate results in data-driven analysis. These data require a strict preparation process and having guides that facilitate this process is becoming increasingly necessary. In this study, we present the step by step processing of several open data sources in order to obtain quality information for feedback on different agricultural vulnerability analysis. The data preparation process is applied to a case study corresponding to the upper Cauca river basin in Colombia. All data sources in this study are public, official and are available from different web platforms where they were collected. In the same way, a ranking with the importance of variables for each dataset was obtained through automatic methods and validated through expert knowledge. Experimental validation showed an acceptable agreement between the ranking of automatic methods and the ranking of raters. The result of this study corresponds to 16 processed data sources ready to feed data-driven systems, as well as agricultural vulnerability methodologies.

Learning from minority class has been a significant and challenging task which has many potential applications. Weather classification is such a case of imbalanced label distribution. This is because in places like Beijing, some types of weather, such as rain and snow, are relatively rare compared to sunny and haze days. Existing methods are primary to classify the weather conditions relying on expensive sensors or human assistance, which however usually are expensive and time-consuming. In this paper, we propose a new ensemble framework based on the advanced generative adversarial network and an effective data cleaning way to address the class imbalance problem for weather classification. The proposed method not only generates new and reliable samples for the minority class to restore balance, but also filters those generated samples which are unreliable. Experiments show that our approach outperforms the state-of-the-art methods by a huge margin for imbalanced weather classification on several benchmark data sets.

A rough environment or unexpected accident of data acquisition instrument can introduce some anomalies in monitoring data. Those anomalies reduce data quality and lead to the incorrect recognition of machine health status. However, the research on anomaly detection of machine monitoring data (MMD) is very scarce. Moreover, anomaly detection methods in other fields cannot be directly applied to MMD. Therefore, a robust anomaly detection method called similarity-measured isolation forest (SM-iForest) is proposed to detect abnormal segments and the data therein. The inadaptability and instability of iForest were reduced while processing MMD benefiting from the characteristics of sliding-window processing. Moreover, an anomaly identification stage measuring the relative similarity of possible abnormal segments further improved the robustness of iForest. The effectiveness of the proposed method was verified with a vibration simulation signal and three sets of milling force signals. The results demonstrate that SM-iForest can detect the missing, shifting, and swelling segments robustly. Detection results of comparing seven methods suggest that SM-iForest is a promising method to detect MMD anomaly with a high detection rate and low false alarm rate. [ABSTRACT FROM AUTHOR]

Oversampling is an efficient technique in dealing with class-imbalance problem. It addresses the problem by reduplicating or generating the minority class samples to balance the distribution between the samples of the majority and the minority class. Synthetic minority oversampling technique (SMOTE) is one of the typical representatives. During the past decade, researchers have proposed many variants of SMOTE. However, the existing oversampling methods may generate wrong minority class samples in some scenarios. Furthermore, how to effectively mine the inherent complex characteristics of imbalanced data remains a challenge. To this end, this paper proposes a parameter-free data cleaning method to improve SMOTE based on constructive covering algorithm. The dataset generated by SMOTE is first partitioned into a group of covers, then the hard-to-learn samples can be detected based on the characteristics of sample space distribution. Finally, a pair-wise deletion strategy is proposed to remove the hard-to-learn samples. The experimental results on 25 imbalanced datasets show that our proposed method is superior to the comparison methods in terms of various metrics, such as F-measure, G-mean, and Recall. Our method not only can reduce the complexity of the dataset but also can improve the performance of the classification model.

With the rapid development of wind farm worldwide, monitoring the status of numerous wind turbines becomes the essential work. Abnormal data in wind power curve (WPC) are quite important for wind farm operations and maintenances because they usually reveal wind turbine failures or some extreme conditions. This paper proposes a new algorithm of WPC abnormal data detection and cleaning by image thresholding based on minimization of dissimilarity-and-uncertainty-based energy (MDUE). The basic idea is to transform the scattered data into a digital image and the problem of data cleaning is turned into an image segmentation problem. For all data pixels, the confidences of being classified as normal class are computed and make up a grey level feature image. Then the optimum threshold is determined by searching through the energy space based on intensity-based class uncertainty and shape dissimilarity. Finally, the normal and three types of abnormal data are marked after applying image thresholding to the feature image. The algorithm is compared with several data-based algorithms and a recently published image-based algorithm. A large number of experiments conducted on real-world WPC data collected from 37 wind turbines in two wind farms verified the superior performance of the proposed method. [ABSTRACT FROM AUTHOR]

A good solar power photovoltaic generation forecast depends on good quality time series data from measurements of Global Horizontal Irradiance and Solar Power Generation. However, measurement system failures and errors in data handling can corrupt data records with gaps and outliers that undermine forecasting accuracy. Therefore, it is important that the fitting of solar energy prediction models must be preceded by a data analysis in order to detect and correct measurement errors. Given that Global Horizontal Irradiance and Solar Power Generation are correlated variables, this paper aims to present the main characteristics of an offline approach developed for the joint treatment of hourly data values of both variables in a photovoltaic plant. In the proposed methodology, the measurements of Global Horizontal Irradiance and Solar Power Generation are analyzed by using reanalysis data and statistical and data mining techniques for the correction of outliers and the filling of data gaps. The application of the approach is illustrated by the analysis of measurements from a real solar PV system. [ABSTRACT FROM AUTHOR]

Methods for cleaning dirty data typically employ additional information about the data such as user-provided constraints specifying when data is dirty, e.g., domain restrictions, illegal value combinations, or logical rules. However, real-world scenarios usually only have dirty data available, without known constraints. In such settings, constraints are automatically discovered on dirty data and discovered constraints are used to detect and repair errors. Typical repairing processes stop there. Yet, when constraint discovery algorithms are re-run on the repaired data (assumed to be clean), new constraints and thus errors are often found. The repairing process then introduces new constraint violations. We present a different type of repairing method, which prevents introducing new constraint violations, according to a discovery algorithm. Summarily, our repairs guarantee that all errors identified by constraints discovered on the dirty data are fixed; and the constraint discovery process cannot identify new constraint violations. We do this for a new kind of constraints, called forbidden itemsets (FBIs), capturing unlikely value co-occurrences. We show that FBIs detect errors with high precision. Evaluation on real-world data shows that our repair method obtains high-quality repairs without introducing new FBIs. Optional user interaction is readily integrated, with users deciding how much effort to invest. [ABSTRACT FROM AUTHOR]

The presence of incorrect data leads to the decrease of condition-monitoring big data quality. As a result, unreliable or misleading results are probably obtained by analyzing these poor-quality data. In this paper, to improve the data quality, an incorrect data detection method based on an improved local outlier factor (LOF) is proposed for data cleaning. First, a sliding window technique is used to divide data into different segments. These segments are considered as different objects and their attributes consist of time-domain statistical features extracted from each segment, such as mean, maximum and peak-to-peak value. Second, a kernel-based LOF (KLOF) is calculated using these attributes to evaluate the degree of each segment being incorrect data. Third, according to these KLOF values and a threshold value, incorrect data are detected. Finally, a simulation of vibration data generated by a defective rolling element bearing and three real cases concerning a fixed-axle gearbox, a wind turbine, and a planetary gearbox are used to verify the effectiveness of the proposed method, respectively. The results demonstrate that the proposed method is able to detect both missing segments and abnormal segments, which are two typical incorrect data, effectively, and thus is helpful for big data cleaning of machinery condition monitoring. [ABSTRACT FROM AUTHOR]

With the accumulation of data on personal behavior and the development of machine learning models and algorithms, it is becoming possible to use social media data for personal credit scoring. In this article, we use the systematic sampling method to obtain Douban's social media data. Because there are many abnormal users in these data, they are "real but false data" for personal credit evaluation. In order to better carry out personal credit scoring, we propose three criteria, power exponents of time interval distribution of individual user $\gamma _i$γi, user activity $A_i$Ai, and the ratio of out-degree and in-degree $R_i$Ri of user $i$i, which are used to systematically clean the data. And then, we used the logistic regression method to score the individual credits of users before and after data cleaning, and found that the rank order of personal credit scoring has changed significantly. This change is largely attributed to the changes of network structure after data cleaning. We believe that our work is very important to use the social media data to establish a credible personal credit evaluation system to reduce the credit risk of the current Internet financial industry. [ABSTRACT FROM AUTHOR]

With the advent of 5G, the industrial Internet of Things has developed rapidly. The industrial sensor-cloud system (SCS) has also received widespread attention. In the future, a large number of integrated sensors that simultaneously collect multifeature data will be added to industrial SCS. However, the collected big data are not trustworthy due to the harsh environment of the sensor. If the data collected at the bottom networks are directly uploaded to the cloud for processing, the query and data mining results will be inaccurate, which will seriously affect the judgment and feedback of the cloud. The traditional method of relying on sensor nodes for data cleaning is insufficient to deal with big data, whereas edge computing provides a good solution. In this article, a new data cleaning method is proposed based on the mobile edge node during data collection. An angle-based outlier detection method is applied at the edge node to obtain the training data of the cleaning model, which is then established through support vector machine. Besides, online learning is adopted for model optimization. Experimental results show that multidimensional data cleaning based on mobile edge nodes improves the efficiency of data cleaning while maintaining data reliability and integrity, and greatly reduces the bandwidth and energy consumption of the industrial SCS. [ABSTRACT FROM AUTHOR]

The performance of a classifier might greatly deteriorate due to missing data. Many different techniques to handle this problem have been developed. In this paper, we solve the problem of missing data using a novel transfer learning perspective and show that when an additive least squares support vector machine (LS-SVM) is adopted, model transfer learning can be used to enhance the classification performance on incomplete training datasets. A novel transfer-based additive LS-SVM classifier is accordingly proposed. This method also simultaneously determines the influence of classification errors caused by each incomplete sample using a fast leave-one-out cross validation strategy, as an alternative way to clean the training data to further improve the data quality. The proposed method has been applied to seven public datasets. The experimental results indicate that the proposed method achieves at least comparable, if not better, performance than case deletion, mean imputation, and ${k}$ -nearest neighbor imputation methods, followed by the standard LS-SVM and support vector machine classifiers. Moreover, a case study on a community healthcare dataset using the proposed method is presented in detail, which particularly highlights the contributions and benefits of the proposed method to this real-world application. [ABSTRACT FROM AUTHOR]

Currently, the cleaning process for power equipment monitoring data is cumbersome and often leads to loss of information. To address these problems, a data cleaning method based on stacked denoising autoencoder (SDAE) networks is proposed in this paper. SDAE networks have a strong ability to denoise and restore corrupted data and have a strong feature extraction capability. The status monitoring data of equipment under normal conditions are trained by SDAE to obtain the cleaning parameters and the reconstruction errors. An upper threshold of the reconstruction errors obtained from training samples is determined through Kernel density estimation. A tolerance window width is added to achieve rapid anomaly detection. The abnormal data are classified as outliers, missing data, or fault status data according to the relationship between the reconstruction error and the threshold and between the duration of abnormal data and the tolerance window. To verify the effectiveness of the proposed method, the SDAE model is used to process the data for the dissolved gas concentration in transformer oil and the temperature of the transmission line. The results show that the proposed method can effectively identify and repair outliers and missing information. The model can perform rapid anomaly detection when the equipment is running abnormally.

Entity Resolution (ER) is the task of finding entity profiles that correspond to the same real-world entity. Progressive ER aims to efficiently resolve large datasets when limited time and/or computational resources are available. In practice, its goal is to provide the best possible partial solution by approximating the optimal comparison order of the entity profiles. So far, Progressive ER has only been examined in the context of structured (relational) data sources, as the existing methods rely on schema knowledge to save unnecessary comparisons: they restrict their search space to similar entities with the help of schema-based blocking keys (i.e., signatures that represent the entity profiles). As a result, these solutions are not applicable in Big Data integration applications, which involve large and heterogeneous datasets, such as relational and RDF databases, JSON files, Web corpus etc. To cover this gap, we propose a family of schema-agnostic Progressive ER methods, which do not require schema information, thus applying to heterogeneous data sources of any schema variety. First, we introduce two naïve schema-agnostic methods, showing that straightforward solutions exhibit a poor performance that does not scale well to large volumes of data. Then, we propose four different advanced methods. Through an extensive experimental evaluation over 7 real-world, established datasets, we show that all the advanced methods outperform to a significant extent both the naïve and the state-of-the-art schema-based ones. We also investigate the relative performance of the advanced methods, providing guidelines on the method selection. [ABSTRACT FROM AUTHOR]

Quality of experience (QoE) serves as a direct evaluation of users’ experiences in mobile video transmission and thus essential for network management, such as network optimization. In this paper, we propose a deep learning-based QoE prediction approach with a large-scale QoE dataset for mobile video transmission. Specifically, we develop a mobile phone application for collecting user QoE data when viewing videos transmitted over the mobile internet in a practical environment. Then, we construct a large-scale dataset by collecting over 80000 piece of data with four kinds of subjective scores and 89 network parameters. Each QoE metric is related to only some of the 89 network parameters. Therefore, we apply the feature selection method to find the feature parameters related to user scores. Additionally, the boxplot method is used to clean the raw data by removing outliers. Finally, a deep neural network (DNN) is developed to learn the relationships between the network parameters and the subjective QoE scores. The proposed DNN can also be seen as a data-driven objective QoE prediction approach for mobile video transmission, which can be used to predict the user QoE scores. The experimental results show that the proposed approach can effectively remove most features irrelevant to QoE prediction. Moreover, the performance of QoE prediction by the proposed model outperforms other state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

With energy scarcity and environmental pollution becoming increasingly serious, the accurate estimation of fuel consumption of vehicles has been important in vehicle management and transportation planning toward a sustainable green transition. Fuel consumption is calculated by fuel-level data collected from high-precision fuel-level sensors. However, in the vehicle monitor platform, there are many types of error in the data collection and transmission processes, such as the noise, interference, and collision errors that are common in the high speed and dynamic vehicle environment. In this paper, an effective method for cleaning and repairing the fuel-level data is proposed, which adopts the threshold to acquire abnormal fuel data, the time quantum to identify abnormal data, and linear interpolation based algorithm to correct data errors. Specifically, a modified Gaussian mixture model (GMM) based on the synchronous iteration method is proposed to acquire the thresholds, which uses the particle swarm optimization algorithm and the steepest descent algorithm to optimize the parameters of GMM. The experiment results based on the fuel-level data of vehicles collected over one month prove that the modified GMM is superior to GMM-expectation maximization on fuel-level data, and the proposed method is effective for cleaning and repairing outliers of fuel-level data. [ABSTRACT FROM AUTHOR]

There exist plenty of outliers in power curve of wind turbines, which is not conducive to the follow-up information mining. Improving the wind power curve data quality of wind turbines has great engineering value. According to the spatial distribution characteristics and shapes, the outliers of wind power curve are divided into four types: the bottom curve, the mid curve, and the top curve stacked outliers, as well as scattered outliers around the curve. Based on the outlier distribution characteristics, this paper proposes a method and process of cleaning the outliers based on the change point grouping algorithm and the quartile algorithm, followed by a theoretical analysis of its feasibility. Case studies and its comparisons with the quartile–change point grouping algorithm and the local outlier factor algorithm show that the proposed change point grouping–quartile algorithm can effectively identify the four types of outliers, with good cleaning effect, high efficiency, and strong versatility. [ABSTRACT FROM AUTHOR]

Map matching is the process of assigning observed geographic positions of vehicles and their trajectories to the actual road links in a road network. In this paper, we present Visual Interactive Map Matching, a visual analytics approach to fine-tune the data preprocessing and matching process. It is based on ST-matching, a state-of-the-art and easy-to-understand map matching algorithm. Parameters of the preprocessing step and algorithm can be optimized with immediate visual feedback. Visualizations show current matching issues and performance metrics on a map and in diagrams. Manual and computer-supported editing of the road network model leads to a refined alignment of trajectories and roads. We demonstrate our approach with large-scale taxi trajectory data. We show that optimizing the matching on a subsample results in considerably improved matching quality, also when later scaled to the full dataset. An optimized matching ensures data faithfulness and prevents misinterpretation when the matched data might be investigated in follow-up analysis. [ABSTRACT FROM PUBLISHER]

Nowadays, mobile devices have become a ubiquitous medium supporting various forms of functionality and are widely accepted for commons. In this study, we investigate using Wi-Fi logs from a mobile device to discover user preferences. The core ideas are two folds. First, every Wi-Fi access point is with a network name, normally a human-readable string, called SSID (Service Set Identifier). Since SSIDs are often with semantics, from which we can infer the place where the user stayed. Second, a Wi-Fi log is produced when the user is near a Wi-Fi access point. A high frequency of a consecutively observed SSID implies a long stay duration at a place. To the best of our knowledge, our work is the first attempting to understand users from the collected Wi-Fi logs from mobile devices. However, Wi-Fi logs are essentially of various information types and with noises. How to assess the information types, eliminate irrelevant information, and clean up the noises within partial-informative SSIDs are therefore keys for profiling user preferences over Wi-Fi logs. In this paper, we propose a data cleaning and information enrichment framework for enabling the user preference understanding through collected Wi-Fi logs, and introduce a data clean framework for cleaning, correcting, and refining Wi-Fi logs. In addition, a comprehensive experiment with data collected from users is made to verify the effectiveness of the proposed techniques for cleaning noisy Wi-Fi data for user preferences profiling. The experiment results demonstrate the effectiveness of the proposed framework for profiling user preferences through Wi-Fi logs. [ABSTRACT FROM PUBLISHER]

The growing volumes of time-stamped data available from sensors, social media sources, Web logs, and medical histories present remarkable opportunities for researchers and policy analysts. Although big data resources can provide valuable insights to help us understand complex systems and lead to better decisions for business, national security, cybersecurity, and healthcare, there are many challenges to dealing with the volume and variety of data. Data cleaning and data wrangling has received some attention with the development of application tools, but data focusing to sharpen the analytic focus remains a challenge. To address this challenge, this article provides a taxonomy of analytic focusing strategies for temporal event sequences. [ABSTRACT FROM AUTHOR]

Duplicate detection is the process of identifying multiple representations of same real world entities. Today, duplicate detection methods need to process ever larger datasets in ever shorter time: maintaining the quality of a dataset becomes increasingly difficult. We present two novel, progressive duplicate detection algorithms that significantly increase the efficiency of finding duplicates if the execution time is limited: They maximize the gain of the overall process within the time available by reporting most results much earlier than traditional approaches. Comprehensive experiments show that our progressive algorithms can double the efficiency over time of traditional duplicate detection and significantly improve upon related work. [ABSTRACT FROM AUTHOR]

Today's manufacturing environments are very dynamic and turbulent. Traditional enterprise information systems (EISs) have mostly been implemented upon hierarchical architectures, which are inflexible to adapt changes and uncertainties promptly. Next-generation EISs must be agile and adaptable to accommodate changes without significant time delays. It is essential for an EIS to obtain real-time data from the distributed and dynamic manufacturing environment for decision making. Wireless sensor networks (WSNs) and radio-frequency identification (RFID) systems provide an excellent infrastructure for data acquisition, distribution, and processing. In this paper, some key challenges related to the integration of WSN and RFID technologies are discussed. A five-layer system architecture has been proposed to achieve synergistic performance. For the integration of WSN and RFID, one of the critical issues is the low efficiency of communication due to redundant data as redundant data increases energy consumption and causes time delay. To address it, an improved data cleaning algorithm has been proposed; its feasibility and effectiveness have been verified via simulation and a comparison with a published algorithm. To illustrate the capacity of the developed architecture and new data cleaning algorithm, their application in relief supplies storage management has been discussed. [ABSTRACT FROM PUBLISHER]

The past few years have witnessed the emergence of an increasing number of applications for tracking and tracing based on radio frequency identification (RFID) technologies. However, raw RFID readings are usually of low quality and may contain numerous anomalies. An ideal solution for RFID data cleansing should address the following issues. First, in many applications, duplicate readings of the same object are very common. The solution should take advantage of the resulting data redundancy for data cleaning. Second, prior knowledge about the environment may help improve data quality, and a desired solution must be able to take into account such knowledge. Third, the solution should take advantage of physical constraints in target applications to elevate the accuracy of data cleansing. There are several existing RFID data cleansing techniques. However, none of them support all the aforementioned features. In this paper, we propose a Bayesian inference-based framework for cleaning RFID raw data. We first design an $(n)$-state detection model and formally prove that the three-state model can maximize the system performance. Then, we extend the $(n)$-state model to support two-dimensional RFID reader arrays and compute the likelihood efficiently. In addition, we devise a Metropolis-Hastings sampler with constraints, which incorporates constraint management to clean RFID data with high efficiency and accuracy. Moreover, to support real-time object monitoring, we present the streaming Bayesian inference method to cope with real-time RFID data streams. Finally, we evaluate the performance of our solutions through extensive experiments. [ABSTRACT FROM AUTHOR]

Although there is a long line of work on identifying duplicates in relational data, only a few solutions focus on duplicate detection in more complex hierarchical structures, like XML data. In this paper, we present a novel method for XML duplicate detection, called XMLDup. XMLDup uses a Bayesian network to determine the probability of two XML elements being duplicates, considering not only the information within the elements, but also the way that information is structured. In addition, to improve the efficiency of the network evaluation, a novel pruning strategy, capable of significant gains over the unoptimized version of the algorithm, is presented. Through experiments, we show that our algorithm is able to achieve high precision and recall scores in several data sets. XMLDup is also able to outperform another state-of-the-art duplicate detection solution, both in terms of efficiency and of effectiveness. [ABSTRACT FROM AUTHOR]

Entity resolution (ER) is the problem of identifying which records in a database refer to the same entity. In practice, many applications need to resolve large data sets efficiently, but do not require the ER result to be exact. For example, people data from the web may simply be too large to completely resolve with a reasonable amount of work. As another example, real-time applications may not be able to tolerate any ER processing that takes longer than a certain amount of time. This paper investigates how we can maximize the progress of ER with a limited amount of work using “hints,” which give information on records that are likely to refer to the same real-world entity. A hint can be represented in various formats (e.g., a grouping of records based on their likelihood of matching), and ER can use this information as a guideline for which records to compare first. We introduce a family of techniques for constructing hints efficiently and techniques for using the hints to maximize the number of matching records identified using a limited amount of work. Using real data sets, we illustrate the potential gains of our pay-as-you-go approach compared to running ER without using hints. [ABSTRACT FROM AUTHOR]