Search

The combined complexity of wind turbine systems and harsh operating conditions pose significant challenges to the accuracy of operational data in Supervisory Control and Data Acquisition (SCADA) systems. Improving the precision of data cleaning for high proportions of stacked abnormalities remains an urgent problem. This paper deeply analyzes the distribution characteristics of abnormal data and proposes a novel method for abnormal data cleaning based on a classification processing framework. Firstly, the first type of abnormal data is cleaned based on operational criteria; secondly, the quartile method is used to eliminate sparse abnormal data to obtain a clearer boundary line; on this basis, the Random Sample Consensus (RANSAC) algorithm is employed to eliminate stacked abnormal data; finally, the effectiveness of the proposed algorithm in cleaning abnormal data with a high proportion of stacked abnormalities is verified through case studies, and evaluation indicators are introduced through comparative experiments to quantitatively assess the cleaning effect. The research results indicate that the algorithm excels in cleaning effectiveness, efficiency, accuracy, and rationality of data deletion. The cleaning accuracy improvement is particularly significant when dealing with a high proportion of stacked anomaly data, thereby bringing significant value to wind power applications such as wind power prediction, condition assessment, and fault detection.

Nowadays, ubiquitous technology makes life easier, especially devices that use the internet (IoT). IoT devices have been used to generate data in various domains, including healthcare, industry, and education. However, there are often problems with this generated data such as missing values, duplication, and data errors, which can significantly affect data analysis results and lead to inaccurate decision making. Enhancing the quality of real-time data streams has become a challenging task as it is crucial for better decisions. In this paper, we propose a framework to improve the quality of a real-time data stream by considering different aspects, including context-awareness. The proposed framework tackles several issues in the data stream, including duplicated data, missing values, and outliers to improve data quality. The proposed framework also provides recommendations on appropriate data cleaning techniques to the user to help improve data quality in real time. Also, the data quality assessment is included in the proposed framework to provide insight to the user about the data stream quality for better decisions. We present a prototype to examine the concept of the proposed framework. We use a dataset that is collected in healthcare and process these data using a case study. The effectiveness of the proposed framework is verified by the ability to detect and repair stream data quality issues in selected context and to provide a recommended context and data cleaning techniques to the expert for better decision making in providing healthcare advice to the patient. We evaluate our proposed framework by comparing the proposed framework against previous works.

Photovoltaic (PV) power generation has attracted widespread interest as a clean and sustainable energy source, with increasing global attention given to renewable energy. However, the operation and monitoring of PV power generation systems often result in large amounts of data containing missing values, outliers, and noise, posing challenges for data analysis and application. Therefore, PV data cleaning plays a crucial role in ensuring data quality, enhancing data availability and reliability. This study proposes a PV data cleaning method based on Rasterized Data Image Processing (RDIP) technology, which integrates rasterization and image processing techniques to select optimal contours and extract essential data. To validate the effectiveness of our method, we conducted comparative experiments using three data cleaning methods, including our RDIP algorithm, the Pearson correlation coefficient interpolation method, and cubic spline interpolation method. Subsequently, the cleaned datasets from these methods were utilized for power prediction using two linear regression models and two neural network models. The experimental results demonstrated that data cleaned using the RDIP algorithm improved the short-term forecast accuracy by approximately 1.0% and 3.7%, respectively, compared to the other two methods, indicating the feasibility and effectiveness of the RDIP approach. However, it is worth noting that the RDIP technique has limitations due to its reliance on integer parameters for grid division, potentially leading to coarse grid divisions. Future research efforts could focus on optimizing the selection of binarization thresholds to achieve better cleaning results and exploring other potential applications of RDIP in PV data analysis.

Over the past few decades, rapid economic development has led to the establishment of numerous monitoring systems, resulting in the accumulation of vast amounts of monitoring data. Among these data, dynamic acceleration data stand out prominently. However, the quality of collected acceleration data is often compromised due to factors such as challenging operational environments and sensor malfunctions. This severely hampers the value extracted from the data. Although manual identification and classification of data anomalies are more reliable, they are time consuming and labor intensive. To address the challenge of identifying and classifying anomalies in massive acceleration data, this paper proposes a two-stage model for intelligent data cleaning. Firstly, raw acceleration data are transformed into IPDF and PSD features, and a one-dimensional convolutional neural network is trained to preliminarily identify and classify acceleration data anomalies. Subsequently, the RPV indicator is extracted from the original data of the normal and outlier categories to achieve precise classification based on threshold values. The proposed method is successfully validated using acceleration monitoring data from a large-span arch bridge, achieving an accuracy of over 99%. Furthermore, compared to directly employing a one-dimensional CNN classification model, the approach significantly enhances the model’s perception of local significant disturbances.

The application of data science (DS) techniques has become increasingly essential in various fields, including epidemiology and climatology in agricultural production systems. In this sector, traditionally large amounts of data are acquired, but not well-managed and -analyzed as a basis for evidence-based decision-making processes. Here, we present a comprehensive step-by-step guide that explores the use of DS in managing epidemiological and climatological data within rice production systems under tropical conditions. Our work focuses on using the multi-temporal dataset associated with the monitoring of diseases and climate variables in rice in Colombia during eight years (2012–2019). The study comprises four main phases: (I) data cleaning and organization to ensure the integrity and consistency of the dataset; (II) data management involving web-scraping techniques to acquire climate information from free databases, like WordClim and Chelsa, validation against in situ weather stations, and bias removal to enrich the dataset; (III) data visualization techniques to effectively represent the gathered information, and (IV) a basic analysis related to the clustering and climatic characterization of rice-producing areas in Colombia. In our work, a process of evaluation and the validation of climate data are conducted based on errors (r, R2, MAE, RSME) and bias evaluation metrics. In addition, in phase II, climate clustering was conducted based on a PCA and K-means algorithm. Understanding the association of climatic and epidemiological data is pivotal in predicting and mitigating disease outbreaks in rice production areas. Our research underscores the significance of DS in managing epidemiological and climatological data for rice production systems. By applying a protocol responsible for DS tools, our study provides a solid foundation for further research into disease dynamics and climate interactions in rice-producing regions and other crops, ultimately contributing to more informed decision-making processes in agriculture.

Due to the rapid development of the mobile Internet and the Internet of Things, the volume of generated data keeps growing. The topic of data quality has gained increasing attention recently. Numerous studies have explored various data quality (DQ) problems across several fields, with corresponding effective data-cleaning strategies being researched. This paper begins with a comprehensive and systematic review of studies related to DQ. On the one hand, we classify these DQ-related studies into six types: redundant data, missing data, noisy data, erroneous data, conflicting data, and sparse data. On the other hand, we discuss the corresponding data-cleaning strategies for each DQ type. Secondly, we examine DQ issues and potential solutions for a public bus transportation system, utilizing a real-world traffic big data platform. Finally, we provide two representative examples, noise filtering and filling missing values, to demonstrate the DQ improvement practice. The experimental results show that: (1) The GPS noise filtering solution we proposed surpasses the baseline and achieves an accuracy of 97%; (2) The multi-source data fusion method can achieve a 100% missing repair rate (MRR) for bus arrival and departure. The average relative error (ARE) of bus arrival and departure times at stations is less than 1%, and the correlation coefficient (R) is also close to 1. Our research can offer guidance and lessons for enhancing data governance and quality improvement in the bus transportation system.

Changes in the deflection of cable-stayed bridges due to thermal effects may adversely affect the bridge structure and reflect the degradation of bridge performance. Therefore, complete deflection field data are important for bridge health monitoring. A strong linear correlation has been found between temperature-induced deflections in different positions of the same span of a cable-stayed bridge in many studies, which make the deflection data matrix/tensor have a low-rank structure. Therefore, it is appropriate to use a low-rank matrix/tensor learning to model the temperature–deflection field of a cable-stayed bridge. Moreover, to avoid disturbing the recovery results via abnormal data (e.g., baseline shift and outliers), a Bayesian robust tensor learning method is proposed to extract the spatio-temporal characteristics of the bridge temperature–deflection field. The missing data recovery and abnormal data cleaning are achieved simultaneously in the process of reconstructing the temperature-induced field via tensor learning. The performance of the method is verified with actual continuous monitoring data from a cable-stayed bridge. The experimental results show that low-order tensor (i.e., matrix) learning has a good recovery and cleaning performance. The extension to higher-order tensor learning is proposed to extract the spatial symmetry of the sensor locations, which is experimentally proven to have better missing recovery and abnormal data cleaning performance.

The Internet of things (IoT) combines different sources of collected data which are processed and analyzed to support smart city applications. Machine learning and deep learning algorithms play a vital role in edge intelligence by minimizing the amount of irrelevant data collected from multiple sources to facilitate these smart city applications. However, the data collected by IoT sensors can often be noisy, redundant, and even empty, which can negatively impact the performance of these algorithms. To address this issue, it is essential to develop effective methods for detecting and eliminating irrelevant data to improve the performance of intelligent IoT applications. One approach to achieving this goal is using data cleaning techniques, which can help identify and remove noisy, redundant, or empty data from the collected sensor data. This paper proposes a deep reinforcement learning (deep RL) framework for IoT sensor data cleaning. The proposed system utilizes a deep Q-network (DQN) agent to classify sensor data into three categories: empty, garbage, and normal. The DQN agent receives input from three received signal strength (RSS) values, indicating the current and two previous sensor data points, and receives reward feedback based on its predicted actions. Our experiments demonstrate that the proposed system outperforms a common time-series-based fully connected neural network (FCDQN) solution, with an accuracy of around 96% after the exploration mode. The use of deep RL for IoT sensor data cleaning is significant because it has the potential to improve the performance of intelligent IoT applications by eliminating irrelevant and harmful data.

Insights and analysis are only as good as the available data. Data cleaning is one of the most important steps to create quality data decision making. Machine learning (ML) helps deal with data quickly, and to create error-free or limited-error datasets. One of the quality standards for cleaning the data includes handling the missing data, also known as data imputation. This research focuses on the use of machine learning methods to deal with missing data. In particular, we propose a generative adversarial network (GAN) based model called DEGAIN to estimate the missing values in the dataset. We evaluate the performance of the presented method and compare the results with some of the existing methods on publicly available Letter Recognition and SPAM datasets. The Letter dataset consists of 20,000 samples and 16 input features and the SPAM dataset consists of 4601 samples and 57 input features. The results show that the proposed DEGAIN outperforms the existing ones in terms of root mean square error and Frechet inception distance metrics.

In the industrial domain, maintenance is essential to guarantee the correct operations, availability, and efficiency of machinery and systems. With the advent of Industry 4.0, solutions based on machine learning can be used for the prediction of future failures by exploiting historical failure data. Most of the time, these historical data have been collected by companies without a specific structure, schema, or even best practices, resulting in a potential loss of knowledge. In this paper, we analyze the historical data on maintenance alerts of the components of a revamping topping plant (referred to as RT2) belonging to the SARAS group. This analysis is done in collaboration with the ITALTELECO company, a partner of SARAS, that provided the necessary data. The pre-processing methodology to clean and fill these data and extract features useful for a prediction task will be shown. More in detail, we show the process to fill missing fields of these data to provide (i) a category for each fault by using simple natural language processing techniques and performing a clustering, and (ii) a data structure that can enable machine learning models and statistical approaches to perform reliable failure predictions. The data domain in which this methodology is applied is oil and gas, but it may be generalized and reformulated in various industrial and/or academic fields. The ultimate goal of our work is to obtain a procedure that is simple and can be applied to provide strategic support for the definition of an adequate maintenance plan.

How can one best transform a paper-based publication into a living online resource? This is the theme of a project at The National Museum of Art, Architecture and Design in Norway, supported by the Arts Council Norway. The National Museum aims to create, publish and maintain an authority list of Norwegian artists, architects, designers and craftsmen. The objective is to ease the digitisation process for other museums, scholars and the public in general and contribute to better data quality in Norwegian online collections. The list will in part be based on the Norsk Kunstnerleksikon (Encyclopaedia of Norwegian Artists in English), published in 1982⁻1986 and subsequently digitised in 2013. With the help of other public collections in Norway, the purpose is to make the new resource as complete as possible and available in both human- and machine-readable formats. Although the original paper publication contains biographical texts as well as lists of exhibitions, education, travels, publications and more, the data in the new authority list will be constrained to a set of core biographical data. It will however carry references to online biographical resources such as Norsk Kunstnerleksikon (NKL), Wikidata, Union List of Artist Names (ULAN) and Virtual International Authority File (VIAF). This article discusses the process of defining the scope of and setting constraints for the list, how to enrich and reconcile existing data, as well as strategies to ensure that other institutions contribute both as content publishers and end users. It will also shed light on issues concerning keeping such a resource updated and maintained.

Data on wildfire growth are useful for multiple research purposes but are frequently unavailable and often have data quality problems. For these reasons, we developed a protocol for collecting daily burned area time series from the InciWeb website, Incident Management Situation Reports (IMSRs), and other sources. We apply this protocol to create the Warehouse of Multiple Burned Area Time Series (WoMBATS) data, which are a collection of burned area time series with cross-check data for 514 wildfires in the United States for the years 2018–2020. We compare WoMBATS-derived distributions of wildfire occurrence and size to those derived from MTBS data to identify potential biases. We also use WoMBATS data to cross tabulate the frequency of missing data in InciWeb and IMSRs and calculate differences in size estimates. We identify multiple instances where WoMBATS data fails to reproduce wildfire occurrence and size statistics derived from MTBS data. We show that WoMBATS data are typically much more complete than either of the two constituent data sources, and that the data collection protocol allows for the identification of otherwise undetectable errors. We find that although disagreements between InciWeb and IMSRs are common, the magnitude of these differences are usually small. We illustrate how WoMBATS data can be used in practice by validating two simple wildfire growth forecasting models.

Measurement data cleaning is a key step of edge computing in a distribution network; it is beneficial to improve the state perception and regional autonomy level of a distribution network. According to the temporal and spatial correlation of measurement data in the distribution network, a joint cleaning method of measurement data in a distribution network is proposed based on the correntropy criterion with variable center unscented Kalman filter (CC-VC-UKF). Initially, the mean square error (MSE) in the original unscented Kalman filter (UKF) is replaced by the correntropy criterion with variable center (CC-VC) to improve the accuracy of filtering the measurement data in the distribution network with a non-Gaussian non-zero mean measurement deviation. Then, the measured data of different measuring devices located on the same section of the line are filtered based on the CC-VC-UKF algorithm according to their respective reference time series to improve the signal-to-noise ratio of the measured data. Then, the filtered measured data are filtered and cleaned based on the CC-VC-UKF algorithm according to the space–time joint filtering and cleaning technology. Finally, the method is used to test the measurement data of the distribution network obtained by a power supply company in a city in north China to solve the problem of measurement deviation caused by the existence of space distance. Results show that this method can obtain FTU measurement data with higher precision from network topology based on the filtered TTU measurement data through the media of filtered spatial measurement deviation.

With the increase of the scale of wind turbines, the problem of data quality of wind turbines has become increasingly prominent, which seriously affects the follow-up research. A large number of abnormal data exist in the historical data recorded by the wind turbine Supervisory Control And Data Acquisition (SCADA) system. In order to improve data quality, it is necessary to clean a large number of abnormal data in the original data. Aiming at the problem that the cleaning effect is not good in the presence of a large number of abnormal data, a method for cleaning abnormal data of wind turbines based on constrained curve fitting is proposed. According to the wind speed-power characteristics of wind turbines, the constrained wind speed-power curve is fit with the least square method, and the constrained optimization problem is transformed into an unconstrained optimization problem by using the external penalty function method. Data cleaning was performed on the fitted curve using an improved 3-σ standard deviation. Experiments show that, compared with the existing methods, this method can still perform data cleaning well when the historical wind turbine data contains many abnormal data, and the method is insensitive to parameters, simple in the calculation, and easy to automate.

Ship hull and propeller fouling has a severe impact both on fuel costs and on greenhouse gas emission, thus, it is an important subject to consider when developing a maintenance plan to enhance energy efficiency and reduce gas emissions from ships. This paper presents a two-step approach for evaluating the actual propulsive performance of a ship through analyzing continuous onboard measurement. First, the onboard monitored data are corrected for the wind and wave effects using fast and transparent empirical methods. Second, the corrected data are filtered based on hydrodynamic criteria. A case study is presented to demonstrate the effectiveness of the methodology. Subsequently, the processed data are analyzed by mathematical tools to derive an engine power–rpm curve to represent the actual propulsive performance of the ship. By comparing the derived curve against the nominal curve for engine design, it is possible to identify the deterioration of the ship’s condition, which is the key task in developing the identification capability that is needed in a proactive maintenance procedure. This methodology is expected to support the maritime sector in combating climate change by reducing the Greenhouse Gas (GHG) emissions from ship operations.

Asset management technology is rapidly growing in the electric power industry because utilities are paying attention to which of their aged assets should be replaced first. The global trend of asset management follows risk management that comprehensively considers the probability and consequences of failures. In the asset management system, the risk assessment algorithm operates by interfacing digital datasets from various legacy systems. In this study, among the various electric power assets, we consider transmission cable systems as a representative linear asset consisting of different segments. First, the configurations and characteristics of linear asset datasets are analyzed. Second, six types of data cleaning functions are proposed for extracting dirty data from the entire dataset. Third, three types of data integration functions are developed to simulate the risk assessment algorithm. This technique supports the integration of distributed asset data in various legacy systems into one dataset. Finally, an automatic data cleaning and integration system is developed and the algorithm could repeat the cleaning and integration process until data quality is satisfied. To evaluate the performance of the proposed system, an automatic cleaning process is demonstrated using actual legacy datasets.

To improve the recognition rate of chip appearance defects, an algorithm based on a convolution neural network is proposed to identify chip appearance defects of various shapes and features. Furthermore, to address the problems of long training time and low accuracy caused by redundant input samples, an automatic data sample cleaning algorithm based on prior knowledge is proposed to reduce training and classification time, as well as improve the recognition rate. First, defect positions are determined by performing image processing and region-of-interest extraction. Subsequently, interference samples between chip defects are analyzed for data cleaning. Finally, a chip appearance defect classification model based on a convolutional neural network is constructed. The experimental results show that the recognition miss detection rate of this algorithm is zero, and the accuracy rate exceeds 99.5%, thereby fulfilling industry requirements.

The aim of this paper is to provide an extended analysis of the outlier detection, using probabilistic and AI techniques, applied in a demo pilot demand response in blocks of buildings project, based on real experiments and energy data collection with detected anomalies. A numerical algorithm was created to differentiate between natural energy peaks and outliers, so as to first apply a data cleaning. Then, a calculation of the impact in the energy baseline for the demand response computation was implemented, with improved precision, as related to other referenced methods and to the original data processing. For the demo pilot project implemented in the Technical University of Cluj-Napoca block of buildings, without the energy baseline data cleaning, in some cases it was impossible to compute the established key performance indicators (peak power reduction, energy savings, cost savings, CO2 emissions reduction) or the resulted values were far much higher (>50%) and not realistic. Therefore, in real case business models, it is crucial to use outlier’s removal. In the past years, both companies and academic communities pulled their efforts in generating input that consist in new abstractions, interfaces, approaches for scalability, and crowdsourcing techniques. Quantitative and qualitative methods were created with the scope of error reduction and were covered in multiple surveys and overviews to cope with outlier detection.

It is recognized that the performance of any prediction model is a function of several factors. One of the most significant factors is the adopted preprocessing techniques. In other words, preprocessing is an essential process to generate an effective and efficient classification model. This paper investigates the impact of the most widely used preprocessing techniques, with respect to numerical features, on the performance of classification algorithms. The effect of combining various normalization techniques and handling missing values strategies is assessed on eighteen benchmark datasets using two well-known classification algorithms and adopting different performance evaluation metrics and statistical significance tests. According to the reported experimental results, the impact of the adopted preprocessing techniques varies from one classification algorithm to another. In addition, a statistically significant difference between the considered data preprocessing techniques is demonstrated.

Preprocessing data in a reproducible and robust way is one of the current challenges in untargeted metabolomics workflows. Data curation in liquid chromatography–mass spectrometry (LC–MS) involves the removal of biologically non-relevant features (retention time, m/z pairs) to retain only high-quality data for subsequent analysis and interpretation. The present work introduces TidyMS, a package for the Python programming language for preprocessing LC–MS data for quality control (QC) procedures in untargeted metabolomics workflows. It is a versatile strategy that can be customized or fit for purpose according to the specific metabolomics application. It allows performing quality control procedures to ensure accuracy and reliability in LC–MS measurements, and it allows preprocessing metabolomics data to obtain cleaned matrices for subsequent statistical analysis. The capabilities of the package are shown with pipelines for an LC–MS system suitability check, system conditioning, signal drift evaluation, and data curation. These applications were implemented to preprocess data corresponding to a new suite of candidate plasma reference materials developed by the National Institute of Standards and Technology (NIST; hypertriglyceridemic, diabetic, and African-American plasma pools) to be used in untargeted metabolomics studies in addition to NIST SRM 1950 Metabolites in Frozen Human Plasma. The package offers a rapid and reproducible workflow that can be used in an automated or semi-automated fashion, and it is an open and free tool available to all users.

Radio Frequency Identification (RFID) technology has been widely used in many fields. However, previous studies have mainly focused on product life cycle tracking, and there are few studies on real-time status monitoring of workpieces in manufacturing workshops. In this paper, a wisdom manufacturing model is introduced, a sensing-aware environment for a wisdom manufacturing workshop is constructed, and RFID event models are defined. A synthetic data cleaning method is applied to clean the raw RFID data. The Complex Event Processing (CEP) technology is adopted to monitor abnormal conditions of workpieces in real time. The RFID data cleaning method and data mining technology are examined by simulation and physical experiments. The results show that the synthetic data cleaning method preprocesses data well. The CEP based on the Rifidi® Edge Server technology completed abnormal condition monitoring of workpieces in real time. This paper reveals the importance of RFID spatial and temporal data analysis in real-time status monitoring of workpieces in wisdom manufacturing workshops.

It is an undeniable fact that Internet of Things (IoT) technologies have become a milestone advancement in the digital healthcare domain, since the number of IoT medical devices is grown exponentially, and it is now anticipated that by 2020 there will be over 161 million of them connected worldwide. Therefore, in an era of continuous growth, IoT healthcare faces various challenges, such as the collection, the quality estimation, as well as the interpretation and the harmonization of the data that derive from the existing huge amounts of heterogeneous IoT medical devices. Even though various approaches have been developed so far for solving each one of these challenges, none of these proposes a holistic approach for successfully achieving data interoperability between high-quality data that derive from heterogeneous devices. For that reason, in this manuscript a mechanism is produced for effectively addressing the intersection of these challenges. Through this mechanism, initially, the collection of the different devices’ datasets occurs, followed by the cleaning of them. In sequel, the produced cleaning results are used in order to capture the levels of the overall data quality of each dataset, in combination with the measurements of the availability of each device that produced each dataset, and the reliability of it. Consequently, only the high-quality data is kept and translated into a common format, being able to be used for further utilization. The proposed mechanism is evaluated through a specific scenario, producing reliable results, achieving data interoperability of 100% accuracy, and data quality of more than 90% accuracy.

Radio Frequency IDentification (RFID) technology promises to revolutionize the way we track items and assets, but in RFID systems, missreading is a common phenomenon and it poses an enormous challenge to RFID data management, so accurate data cleaning becomes an essential task for the successful deployment of systems. In this paper, we present the design and development of a RFID data cleaning system, the first declarative, behavior-based unreliable RFID data smoothing system. We take advantage of kinematic characteristics of tags to assist in RFID data cleaning. In order to establish the conversion relationship between RFID data and kinematic parameters of the tags, we propose a movement behavior detection model. Moreover, a Reverse Order Filling Mechanism is proposed to ensure a more complete access to get the movement behavior characteristics of tag. Finally, we validate our solution with a common RFID application and demonstrate the advantages of our approach through extensive simulations.

Unreliability of the data streams generated by RFID readers is among the primary factors which limit the widespread adoption of the RFID technology. RFID data cleaning is, therefore, an essential task in the RFID middleware systems in order to reduce reading errors, and to allow these data streams to be used to make a correct interpretation and analysis of the physical world they are representing. In this paper we propose an adaptive sliding-window based approach called WSTD which is capable of efficiently coping with both environmental variation and tag dynamics. Our experimental results demonstrate the efficacy of the proposed approach.

In this study an algorithm for missing data imputation is presented. The algorithm uses measurements from neighboring sensors to estimate the missing values. Data-driven approach is used and methodology chooses the optimal available combination of modeling algorithm and available measurements to produce an estimate from the model with lowest error. The methodology was tested on Ljubljana polje aquifer data and has produced close to perfect results.

Given the popularization of GPS technologies, the massive amount of spatiotemporal GPS traces collected by vehicles are becoming a new kind of big data source for urban geographic information extraction. The growing volume of the dataset, however, creates processing and management difficulties, while the low quality generates uncertainties when investigating human activities. Based on the conception of the error distribution law and position accuracy of the GPS data, we propose in this paper a data cleaning method for this kind of spatial big data using movement consistency. First, a trajectory is partitioned into a set of sub-trajectories using the movement characteristic points. In this process, GPS points indicate that the motion status of the vehicle has transformed from one state into another, and are regarded as the movement characteristic points. Then, GPS data are cleaned based on the similarities of GPS points and the movement consistency model of the sub-trajectory. The movement consistency model is built using the random sample consensus algorithm based on the high spatial consistency of high-quality GPS data. The proposed method is evaluated based on extensive experiments, using GPS trajectories generated by a sample of vehicles over a 7-day period in Wuhan city, China. The results show the effectiveness and efficiency of the proposed method.