Search

While many instructors are aware of the Literary Digest 1936 poll as an example of biased sampling methods, this article details potential further explorations for the Digest’s 1924–1936 quadrennial U.S. presidential election polls. Potential activities range from lessons in data acquisition, cleaning, and validation, to basic data literacy and visualization skills, to exploring one or more methods of adjustment to account for bias based on information collected at that time. Students can also compare how those methods would have performed. One option could be to give introductory students a first look at the idea of “sampling adjustment” and how this principle can be used to account for difficulties in modern polling, but the context is rich in other opportunities that can be discussed at various times in the course or in more advanced sampling courses. Supplementary materials for this article are available online.

Data cleaning is a core process when it comes to using data from dairy sensor technologies. This article presents guidelines for sensor data cleaning with a specific focus on dairy herd management and breeding applications. Prior to any data cleaning steps, context and purpose of the data use must be considered. Recommendations for data cleaning are provided in five distinct steps: 1) validate the data merging process, 2) get to know the data, 3) check completeness of the data, 4) evaluate the plausibility of sensor measures and detect outliers, and 5) check for technology related noise. Whenever necessary, the recommendations are supported by examples of different sensor types (bolus, accelerometer) collected in an international project (D4Dairy) or supported by relevant literature. To ensure quality and reproducibility, data users are required to document their approach throughout the process. The target group for these guidelines are professionals involved in the process of collecting, managing, and analyzing sensor data from dairy herds. Providing guidelines for data cleaning could help to ensure that the data used for analysis is accurate, consistent, and reliable, ultimately leading to more informed management decisions and better breeding outcomes for dairy herds.

Abstract Premise The Flora e Funga do Brasil project is the most comprehensive effort to reliably document Brazilian plant and fungal diversity. It involves the collaborative work of hundreds of taxonomists, integrating detailed and standardized morphological descriptions, nomenclatural status, and geographic distribution information of plants, algae, and fungi collected throughout Brazil. Despite the extensive information available, managing the information from the Flora e Funga do Brasil website poses certain challenges. Methods and Results florabr is an R package developed to facilitate the exploration and geographical analysis of species information derived from the Flora e Funga do Brasil. Unique to florabr is its ability to interact with the latest, or any other version of the dataset, which undergoes weekly updates. I illustrate the practical application of florabr in common tasks in biogeography and conservation studies. Conclusions florabr is anticipated to be of significant interest to biogeographers, ecologists, curators of biological collections, and taxonomists actively contributing to the Flora e Funga do Brasil.

Urban water demand prediction is not only the foundation of water resource planning and management, but also an important component of water supply system optimization and scheduling. Therefore, predicting future water demand is of great significance. For univariate time series data, the issue of outliers can be solved through data preprocessing. Then, the data input dimension is increased through feature engineering, and finally, the LightGBM (Light Gradient Boosting Machine) model is used to predict future water demand. The results demonstrate that cubic polynomial interpolation outperforms the Prophet model and the linear method in the context of missing value interpolation tasks. In terms of predicting water demand, the LightGBM model demonstrates excellent forecasting performance and can effectively predict future water demand trends. The evaluation indicators MAPE (mean absolute percentage error) and NSE (Nash–Sutcliffe efficiency coefficient) on the test dataset are 4.28% and 0.94, respectively. These indicators can provide a scientific basis for short-term prediction of water supply enterprises. HIGHLIGHTS Interpolation of raw training data may not necessarily improve the performance of predictive models.; Accurate prediction of univariate data can be achieved through feature engineering and machine learning.;

Abstract Background Understanding biodiversity patterns is a central topic in biogeography and ecology, and it is essential for conservation planning and policy development. Diversity estimates that consider the evolutionary relationships among species, such as phylogenetic diversity and phylogenetic endemicity indices, provide valuable insights into the functional diversity and evolutionary uniqueness of biological communities. These estimates are crucial for informed decision-making and effective global biodiversity management. However, the current methodologies used to generate these metrics encounter challenges in terms of efficiency, accuracy, and data integration. Results We introduce PhyloNext, a flexible and data-intensive computational pipeline designed for phylogenetic diversity and endemicity analysis. The pipeline integrates GBIF occurrence data and OpenTree phylogenies with the Biodiverse software. PhyloNext is free, open-source, and provided as Docker and Singularity containers for effortless setup. To enhance user accessibility, a user-friendly, web-based graphical user interface has been developed, facilitating easy and efficient navigation for exploring and executing the pipeline. PhyloNext streamlines the process of conducting phylogenetic diversity analyses, improving efficiency, accuracy, and reproducibility. The automated workflow allows for periodic reanalysis using updated input data, ensuring that conservation strategies remain relevant and informed by the latest available data. Conclusions PhyloNext provides researchers, conservationists, and policymakers with a powerful tool to facilitate a broader understanding of biodiversity patterns, supporting more effective conservation planning and policy development. This new pipeline simplifies the creation of reproducible and easily updatable phylogenetic diversity analyses. Additionally, it promotes increased interoperability and integration with other biodiversity databases and analytical tools.

Abstract Ecology increasingly relies on a massive volume of biodiversity occurrence records to draw insights into large‐scale biogeographical, ecological and evolutionary phenomena. This often involves defining a set of criteria that guides the collection, filtering and standardising of available records. These curation processes are often neither described in detail nor well documented. This hampers the comparability and reproducibility of studies and thus undermines the robustness of any ecological result. Yet, to date, there is no guide providing a friendly way to reason and document a complete data curation process. We reviewed the available literature on data curation, including tools such as R packages and workflows. From this assessment, we created a complete guide organised into five modules that allows users to consciously select and validate occurrence records based on these curation criteria. This is presented in the user‐friendly Shiny R OCCUR application, available at https://ecoinformatic.shinyapps.io/OCCUR/. OCCUR application provides a guide for researchers to deal with the trade‐off between data certainty and coverage generated in each data curation step. An interactive graph of these changes is provided within each module. OCCUR also produces a custom‐made final report including all steps used for data filtering. This report helps to streamline the writing of the methods section of manuscripts and technical reports, thus promoting the reproducibility of data curation processes. This Shiny application provides an interactive overview of the data curation methods and their use for handling occurrence records from public repositories. It brings together the taxonomic, temporal and spatial dimensions of data and also the identification of duplicates of the records. OCCUR can be applied for multiple purposes, such as teaching R to ecologists or enhancing reproducibility of macroecology and biogeography.

In this contribution, we present clickR, an R package intended for data cleaning following a semi-automatic and supervised procedure. Few packages and commercial software with cleaning capacities are available. In all cases, their functionalities just cover part of the overall data pre-processing and do not follow an integral approach to cleaning up the data. In contrast, clickR brings together all functions needed for correcting the main structural, variable-assignment and typographical errors found in databases and allows researchers to have a strict control on the suggested changes. This is possible because the package creates a data frame that keeps track of all the implemented data modifications. To prove its capacity for detecting and fixing errors, we clean a messy database that exhibits multiple types of errors within date, numeric and factor variables.

Abstract Mitigating the impact of class imbalance datasets on classifiers poses a challenge to the machine learning community. Conventional classifiers do not perform well as they are habitually biased toward the majority class. Among existing solutions, the synthetic minority oversampling technique (SMOTE) has shown great potential, aiming to improve the dataset rather than the classifier. However, SMOTE still needs improvement because of its equal oversampling to each minority instance. Based on the consensus that instances far from the borderline contribute less to classification, a refined method for oversampling borderline minority instances (OBMI) is proposed in this paper using a two-stage Tomek link-finding procedure. In the oversampling stage, the pairs of between-class instances nearest to each other are first found to form Tomek links. Then, these minority instances in Tomek links are extracted as base instances. Finally, new minority instances are generated, each of which is linearly interpolated between a base instance and one minority neighbor of the base instance. To address the overlap caused by oversampling, in the cleaning stage, Tomek links are employed again to remove the borderline instances from both classes. The OBMI is compared with ten baseline methods on 17 benchmark datasets. The results show that it performs better on most of the selected datasets in terms of the F1-score and G-mean. Statistical analysis also indicates its higher-level Friedman ranking.

The combined complexity of wind turbine systems and harsh operating conditions pose significant challenges to the accuracy of operational data in Supervisory Control and Data Acquisition (SCADA) systems. Improving the precision of data cleaning for high proportions of stacked abnormalities remains an urgent problem. This paper deeply analyzes the distribution characteristics of abnormal data and proposes a novel method for abnormal data cleaning based on a classification processing framework. Firstly, the first type of abnormal data is cleaned based on operational criteria; secondly, the quartile method is used to eliminate sparse abnormal data to obtain a clearer boundary line; on this basis, the Random Sample Consensus (RANSAC) algorithm is employed to eliminate stacked abnormal data; finally, the effectiveness of the proposed algorithm in cleaning abnormal data with a high proportion of stacked abnormalities is verified through case studies, and evaluation indicators are introduced through comparative experiments to quantitatively assess the cleaning effect. The research results indicate that the algorithm excels in cleaning effectiveness, efficiency, accuracy, and rationality of data deletion. The cleaning accuracy improvement is particularly significant when dealing with a high proportion of stacked anomaly data, thereby bringing significant value to wind power applications such as wind power prediction, condition assessment, and fault detection.

The Rehabilitation Services Administration (RSA)-911 is a potent dataset. The purpose of this study is to improve the veracity of research using the RSA-911 dataset. To achieve this, 43 research studies that were published between 2018 and 2022 that used Rehabilitation Services Administration (RSA)-911 data were evaluated in a scoping review. The articles were assessed on several characteristics–reporting of data cleaning strategies utilized, discussion of variable definitions, and the methodological rigor of the statistical analyses that were reported. Opportunities for improvement in data cleaning strategies, reporting accurate definitions of the variables that were selected for the studies, improving the rigor of study methodologies, and recommendations for practice in State Vocational Rehabilitation (VR) agencies were found. Recommendations are provided that may improve research studies that are conducted with the RSA-911 data which may ultimately improve services for participants with disabilities served in State VR agencies.

Abstract Plant trait data are used to quantify how plants respond to environmental factors and can act as indicators of ecosystem function. Measured trait values are influenced by genetics, trade‐offs, competition, environmental conditions, and phenology. These interacting effects on traits are poorly characterized across taxa, and for many traits, measurement protocols are not standardized. As a result, ancillary information about growth and measurement conditions can be highly variable, requiring a flexible data structure. In 2007, the TRY initiative was founded as an integrated database of plant trait data, including ancillary attributes relevant to understanding and interpreting the trait values. The TRY database now integrates around 700 original and collective datasets and has become a central resource of plant trait data. These data are provided in a generic long‐table format, where a unique identifier links different trait records and ancillary data measured on the same entity. Due to the high number of trait records, plant taxa, and types of traits and ancillary data released from the TRY database, data preprocessing is necessary but not straightforward. Here, we present the ‘rtry’ R package, specifically designed to support plant trait data exploration and filtering. By integrating a subset of existing R functions essential for preprocessing, ‘rtry’ avoids the need for users to navigate the extensive R ecosystem and provides the functions under a consistent syntax. ‘rtry’ is therefore easy to use even for beginners in R. Notably, ‘rtry’ does not support data retrieval or analysis; rather, it focuses on the preprocessing tasks to optimize data quality. While ‘rtry’ primarily targets TRY data, its utility extends to data from other sources, such as the National Ecological Observatory Network (NEON). The ‘rtry’ package is available on the Comprehensive R Archive Network (CRAN; https://cran.r‐project.org/package=rtry) and the GitHub Wiki (https://github.com/MPI‐BGC‐Functional‐Biogeography/rtry/wiki) along with comprehensive documentation and vignettes describing detailed data preprocessing workflows.

In order to improve the data quality, the big data cleaning method for distribution networks is studied in this paper. First, the Local Outlier Factor (LOF) algorithm based on DBSCAN clustering is used to detect outliers. However, due to the difficulty in determining the LOF threshold, a method of dynamically calculating the threshold based on the transformer districts and time is proposed. In addition, the LOF algorithm combines the statistical distribution method to reduce the misjudgment rate. Aiming at the diversity and complexity of data missing forms in power big data, this paper has improved the Random Forest imputation algorithm, which can be applied to various forms of missing data, especially the blocked missing data and even some completely missing horizontal or vertical data. The data in this paper are from real data of 44 transformer districts of a certain 10 kV line in a distribution network. Experimental results show that outlier detection is accurate and suitable for any shape and multidimensional power big data. The improved Random Forest imputation algorithm is suitable for all missing forms, with higher imputation accuracy and better model stability. By comparing the network loss prediction between the data using this data cleaning method and the data removing outliers and missing values, it can be found that the accuracy of network loss prediction has improved by nearly 4 % using the data cleaning method identified in this paper. Additionally, as the proportion of bad data increased, the difference between the prediction accuracy of cleaned data and that of uncleaned data is more significant.

Aiming at some "pain points" in data access of the wargame system, a bus-based real-time data collection and management platform for wargame is designed by absorbing the experience and lessons of domestic and foreign wargame systems. The distributed storage platform is introduced as the underlying storage foundation, and the in-memory database is introduced as the external service interface. The bus-based data collection is used as the collection source, and data cleaning and branch management based on data segments are performed. The system as a whole is divided into three modules, namely collection module, management module, and service module. It realizes real-time collection, cleaning, storage and management of wargame data. The practical application shows that the platform provides high-speed and reliable data access support for other modules in the wargame system.

Abstract The rapid dissemination of hot information on the Internet has become a common phenomenon in today's society. Traditional methods of information capture and preprocessing often require a lot of manpower and material resources, and the captured information has low timeliness and accuracy. The purpose of this paper was to use sensor technology to find and locate network hotspots in time. By collecting user generated content, social media data, news reports, etc., the data is analyzed and mined to identify popular topics and events. In terms of information capture, sensor technology can monitor and understand user activities, the popularity of posts, emotional tendencies, user attention, user interaction, etc., through information monitoring. Network data was collected, such as network latency, data loss rate, and bandwidth utilization. Sensor technology was used to collect social media data to understand the level of public attention to hot events. In information preprocessing, sensor technology was used to remove noise and redundant information in data to ensure data quality. The data were labeled and classified, and the information dissemination rules of network hotspot were analyzed in depth. The average capture accuracy of Method 1 for Hotspot 1, Hotspot 2, and Hotspot 3 was 72.11%, 71.81%, and 72.54%, respectively. The average capture accuracy of Method 2 for Hotspot 1, Hotspot 2, and Hotspot 3 was 82.55%, 83.14%, and 82.91%, respectively. When the data was 40, 80, and 120, the preprocessing times of Method 1 for Post 1 were 8.81 s, 15.47 s, and 18.77 s, respectively. The preprocessing times of Method 2 for Post 1 were 5.97 s, 7.80 s, and 9.25 s, respectively. The application of sensor technology in the capture and preprocessing of network hot information dissemination has brought a variety of innovations, including multi-modal data acquisition, real-time monitoring and analysis, user behavior analysis, data cleaning and integration, anomaly detection and early warning, intelligent recommendation and personalized service, etc., thus improving the accuracy, real-time and personalized degree of information acquisition.

Abstract Nowadays, big data is an emerging area of computer science. Data are generated through different sources such as social media, e-commerce, blogs, banking, healthcare, transactions, apps, websites, opinion platforms, etc. It is processed for effective utilization in different industries, including healthcare. These enormous generated data are essential for data analysis and processing for industrial needs. This paper reviews the work of various authors who have contributed to data collection, analyzing, processing, and viewing to explore the importance and possibilities of big data in industrial processing applications and healthcare sectors. It identifies different opportunities and challenges (data cleaning, missing values, and outlier analysis) along with applications and features of big data. This systematic review further proposed dirty data detection and cleaning and outlier detection models that can be used for many applications. The data cleaning and outlier detection models use the optimizations concept to solve the optimal centroid selection problem and suspected data.

As high-throughput methods become more common, training undergraduates to analyze data must include having them generate informative summaries of large datasets. This flexible case study provides an opportunity for undergraduate students to become familiar with the capabilities of R programming in the context of high-throughput evolutionary data collected using macroarrays. The story line introduces a recent graduate hired at a biotech firm and tasked with analysis and visualization of changes in gene expression from 20,000 generations of the Lenski Lab’s Long-Term Evolution Experiment (LTEE). Our main character is not familiar with R and is guided by a coworker to learn about this platform. Initially this involves a step-by-step analysis of the small Iris dataset built into R which includes sepal and petal length of three species of irises. Practice calculating summary statistics and correlations, and making histograms and scatter plots, prepares the protagonist to perform similar analyses with the LTEE dataset. In the LTEE module, students analyze gene expression data from the long-term evolutionary experiments, developing their skills in manipulating and interpreting large scientific datasets through visualizations and statistical analysis. Prerequisite knowledge is basic statistics, the Central Dogma, and basic evolutionary principles. The Iris module provides hands-on experience using R programming to explore and visualize a simple dataset; it can be used independently as an introduction to R for biological data or skipped if students already have some experience with R. Both modules emphasize understanding the utility of R, rather than creation of original code. Pilot testing showed the case study was well-received by students and faculty, who described it as a clear introduction to R and appreciated the value of R for visualizing and analyzing large datasets.

The Internet of Things (IoT) is leading the physical and digital world of technology to converge. Real-time and massive scale connections produce a large amount of versatile data, where Big Data comes into the picture. Big Data refers to large, diverse sets of information with dimensions that go beyond the capabilities of widely used database management systems, or standard data processing software tools to manage within a given limit. Almost every big dataset is dirty and may contain missing data, mistyping, inaccuracies, and many more issues that impact Big Data analytics performances. One of the biggest challenges in Big Data analytics is to discover and repair dirty data; failure to do this can lead to inaccurate analytics results and unpredictable conclusions. Different imputation methods were employed in the experimentation with various missing value imputation techniques, and the performances of machine learning (ML) models were compared. A hybrid model that integrates ML and sample-based statistical techniques for missing value imputation is being proposed. Furthermore, the continuation involved the dataset with the best missing value imputation, chosen based on ML model performance for subsequent feature engineering and hyperparameter tuning. K-means clustering and principal component analysis were applied in our study. Accuracy, the evaluated outcome, improved dramatically and proved that the XGBoost model gives very high accuracy at around 0.125 root mean squared logarithmic error (RMSLE). To overcome overfitting, K-fold cross-validation was implemented.

The advancement of data collection technology has led to a rapid increase in the size of datasets.Due to the big scale and high complexity of the data volume, serious data quality issues arise.Therefore, data cleaning is a necessary and important step in data activities.To effectively reduce human annotation costs while ensuring the accuracy of cleaning, an iterative data cleaning method (IDCHI) with human participation was proposed.This method proposed a data selection optimization method in the detection module, which enables the classifier to have high accuracy in the initial stage; and further proposed a method for selecting data to be manually annotated, effectively reducing the amount of data to be manually annotated.The experimental results show that the proposed method is effective and efficient in cleaning erroneous data.

With the advent of Industry 4.0 (I4.0), predictive maintenance (PdM) methods have been widely adopted by businesses to deal with the condition of their machinery. With the help of I4.0, digital transformation, information techniques, computerised control, and communication networks, large amounts of data on operational and process conditions can be collected from multiple pieces of equipment and used to make an automated fault detection and diagnosis, all with the goal of reducing unscheduled maintenance, improving component utilisation, and lengthening the lifespan of the equipment. In this paper, we use smart approaches to create a PdM planning model. The five key steps of the created approach are as follows: (1) cleaning the data, (2) normalising the data, (3) selecting the best features, (4) making a decision about the prediction network, and (5) producing a prediction. At the outset, PdM-related data undergo data cleaning and normalisation to get everything in order and within some kind of bounds. The next step is to execute optimal feature selection in order to eliminate unnecessary data. This research presents the golden search optimization (GSO) algorithm, a powerful population-based optimization technique for efficient feature selection. The first phase of GSO is to produce a set of possible solutions or objects at random. These objects will then interact with one another using a straightforward mathematical model to find the best feasible answer. Due to the wide range over which the prediction values fall, machine learning and deep learning confront challenges in providing reliable predictions. This is why we recommend a multilayer hybrid convolution neural network (MLH-CNN). While conceptually similar to VGGNet, this approach uses fewer parameters while maintaining or improving classification correctness by adjusting the amount of network modules and channels. The projected perfect is evaluated on two datasets to show that it can accurately predict the future state of components for upkeep preparation.

Nowadays, ubiquitous technology makes life easier, especially devices that use the internet (IoT). IoT devices have been used to generate data in various domains, including healthcare, industry, and education. However, there are often problems with this generated data such as missing values, duplication, and data errors, which can significantly affect data analysis results and lead to inaccurate decision making. Enhancing the quality of real-time data streams has become a challenging task as it is crucial for better decisions. In this paper, we propose a framework to improve the quality of a real-time data stream by considering different aspects, including context-awareness. The proposed framework tackles several issues in the data stream, including duplicated data, missing values, and outliers to improve data quality. The proposed framework also provides recommendations on appropriate data cleaning techniques to the user to help improve data quality in real time. Also, the data quality assessment is included in the proposed framework to provide insight to the user about the data stream quality for better decisions. We present a prototype to examine the concept of the proposed framework. We use a dataset that is collected in healthcare and process these data using a case study. The effectiveness of the proposed framework is verified by the ability to detect and repair stream data quality issues in selected context and to provide a recommended context and data cleaning techniques to the expert for better decision making in providing healthcare advice to the patient. We evaluate our proposed framework by comparing the proposed framework against previous works.

Photovoltaic (PV) power generation has attracted widespread interest as a clean and sustainable energy source, with increasing global attention given to renewable energy. However, the operation and monitoring of PV power generation systems often result in large amounts of data containing missing values, outliers, and noise, posing challenges for data analysis and application. Therefore, PV data cleaning plays a crucial role in ensuring data quality, enhancing data availability and reliability. This study proposes a PV data cleaning method based on Rasterized Data Image Processing (RDIP) technology, which integrates rasterization and image processing techniques to select optimal contours and extract essential data. To validate the effectiveness of our method, we conducted comparative experiments using three data cleaning methods, including our RDIP algorithm, the Pearson correlation coefficient interpolation method, and cubic spline interpolation method. Subsequently, the cleaned datasets from these methods were utilized for power prediction using two linear regression models and two neural network models. The experimental results demonstrated that data cleaned using the RDIP algorithm improved the short-term forecast accuracy by approximately 1.0% and 3.7%, respectively, compared to the other two methods, indicating the feasibility and effectiveness of the RDIP approach. However, it is worth noting that the RDIP technique has limitations due to its reliance on integer parameters for grid division, potentially leading to coarse grid divisions. Future research efforts could focus on optimizing the selection of binarization thresholds to achieve better cleaning results and exploring other potential applications of RDIP in PV data analysis.

S&T management data has a wide range of sources and types, and the innovation of S&T management data service model is an important way for the efficient utilization of S&T resources in the new era, so this paper creates an innovative model of S&T management data service based on Data-Information-Knowledge-Wisdom model and artificial intelligence technology. Heterogeneous data mining technology based on association rules is used to obtain the connection between S&T management data, Lagrange interpolation is used for heterogeneous data cleaning to predict the missing values of S&T management data, and data lineage resolution technology is used to solve the challenges brought by the complex and diverse S&T management data components. Experimental analyses are conducted from both S&T management data processing and data service cases to verify the effectiveness and scientificity of the S&T management data service innovation model proposed in this paper. The results show that in S&T management data processing, this paper’s method consumes less than 2.45s for associated data rule mining, which has high mining efficiency, and the duplication rate and missing rate of data are below 0.0260 and 0.0222. Through the analysis of the service quality data, it can be seen that the degree of explanation of the service process quality problems of the model proposed in this paper tends to be close to 1, which can reflect the differences in the service process quality problems, and provide accurate, intelligent and personalized services for the main body of science and technology innovation.

This paper constructs the coupling logic of the two systems of vocational education and cultural tourism with the joint action of external and internal driving forces. It contributes to the coupling of the two systems through the synergistic relationship of same-frequency resonance and applies dimensionless processing to data checking and data cleaning of vocational education data. The coupling coordination development level of vocational education and the cultural tourism industry can be measured using the modified coupling coordination degree model. An empirical study of the coupling coordination degree of vocational education and cultural tourism industry in Province G is conducted using the entropy value method. The results show that the coupling and coordination degree of vocational education and cultural tourism system in province G shows a decreasing trend from 2013 to 2019, and the coupling and coordination degree between the development level of the cultural tourism industry and vocational education gradually rises to 0.658388 in 2022 after 2019, and the interaction between the two is gradually obvious. The integration of industry and vocational education in cultural tourism is promoted by this paper through valuable references and drawable ideas.

In this paper, web crawler technology is used to obtain research data on news dissemination of self media and mainstream media, data cleaning is performed on the initial data through Pandas package, and the processed research data is stored in the database. From the acquired research data of self media and mainstream media news dissemination, the independent variables and dependent variables are determined to complete the construction of linear regression model of mainstream media news dissemination, and the influence of self media on mainstream media news dissemination is judged by calculating the correlation coefficients of the linear regression model. The results show that compared with X1, X3 (Bate=0.223, p

This study centers on the English translation of Chinese medicine terms and the construction of knowledge graph, to solve the problem of the accuracy of Chinese medicine terminology translation through scientific and technological means, and to improve the communication efficiency and accuracy of Chinese medicine in the international arena. First, we collected Chinese medicine-related data from websites such as “Seek Medical Help” and “Seek Medical Help” through crawler technology, then performed data cleaning and fusion processing to eliminate data redundancy and inconsistency. Then, we defined 17 types of concepts and 22 conceptual relationships related to TCM ancient books, as well as the corresponding attribute definitions, to ensure the accuracy and completeness of the knowledge graph. The Bi-LSTM model for text disambiguation and Labeling further improved data processing efficiency. Eventually, this study successfully constructed a knowledge graph of English translation of TCM terms containing 50051 nodes and 13521 relations. This knowledge graph improves the accuracy of TCM terminology translation and provides a powerful tool for international dissemination and academic research of TCM terminology.

At present, the daily operation and maintenance of large-scale data centers such as banks in China, due to a variety of reasons, often brings about the problem of unexpected events that are difficult to locate. In order to ensure that the systems running in the data center work efficiently, this paper proposes a method for improving the operation, maintenance, and monitoring of financial data centers based on the cloud-native model. First, we sequentially cleanse and process the financial center data to eliminate any negative impact and generate a time-trending correlation of financial attributes. We then apply association mining to data center operation and maintenance, using stock information as an example to analyze the operational results in stock trading transactions. The result of correlation mining is component B index (up)⇒ component A index (up), support = 12/100, confidence = 12/19, which indicates that in 100 trading days, the number of days that the component B index and the component A index rise together is 12 days, while the number of days that the component B rises alone is 19 days. In the case study examining the impact of association mining in stock trading, on March 15, 2022, the stock price experienced a rise from 11.456 to 11.498 within a mere 0.1s. The financial data operation and maintenance system, using association mining, identified this as “abnormal,” demonstrating the model’s successful detection of abnormal behavior.

With the rapid development of big data technology, the number of economic data grows faster, and it is more challenging for people to grasp and calculate economic data. In this paper, an economic statistical model is proposed to optimize data-cleaning technology. The economic data application framework is constructed using Multi-Agents, and the mining of economic data is achieved using Microsoft time series and clustering algorithm. The GMDH algorithm, which is the core of self-organized data mining, is proposed and improved to complete economic data analysis and prediction by using the fitting error or prediction variance criterion as the identification criterion. The wild value identification method based on the regression model is utilized to eliminate potential wild values and carry out data cleaning. Statistical analysis of the economic data of the printing industry in Shanghai, China, reveals that the total assets, total industrial output value, and total profit of the printing industry in Shanghai in 2023 declined by 2.63%, 4.77%, and 5.68%, respectively. External investment and enterprise R & D investment up to 109,440,100 U.S. dollars, 140,301,000 yuan. The overall number of employees declined, and the profit margin on output value decreased by 5.68%. It is predicted that the number of enterprises, total assets, total industrial output value, and total profit of Shanghai’s printing industry will rebound in 2024, and the external direct investment and R&D investment will be appropriately reduced.

TidyGEO is a Web-based tool for downloading, tidying, and reformatting data series from Gene Expression Omnibus (GEO). As a freely accessible repository with data from over 6 million biological samples across more than 4000 organisms, GEO provides diverse opportunities for secondary research. Although scientists may find assay data relevant to a given research question, most analyses require sample-level annotations. In GEO, such annotations are stored alongside assay data in delimited, text-based files. However, the structure and semantics of the annotations vary widely from one series to another, and many annotations are not useful for analysis purposes. Thus, every GEO series must be tidied before it is analyzed. Manual approaches may be used, but these are error prone and take time away from other research tasks. Custom computer scripts can be written, but many scientists lack the computational expertise to create such scripts. To address these challenges, we created TidyGEO, which supports essential data-cleaning tasks for sample-level annotations, such as selecting informative columns, renaming columns, splitting or merging columns, standardizing data values, and filtering samples. Additionally, users can integrate annotations with assay data, restructure assay data, and generate code that enables others to reproduce these steps.

The wind turbine power curve model is critical to a wind turbine’s power prediction and performance analysis. However, abnormal data in the training set decrease the prediction accuracy of trained models. This paper proposes a sample average approach-based method to construct an interval model of a wind turbine, which increases robustness against abnormal data and further improves the model accuracy. We compare our proposed methods with the traditional neural network-based and Bayesian neural network-based models in experimental data-based validations. Our model shows better performance in both accuracy and computational time.

The stability of NOx concentration in a coal-fired unit affects the operation performance of denitrification system. By constructing the stability evaluation model, the quantitative evaluation of the big data distribution of NOx concentration was realized. The original big data cleaning method combining K-means clustering and 3σ criterion was constructed, and the data of different working conditions were divided by the sliding window method. A NOx concentration reference value model was established based on the least-squares regression method. Based on the deviation degree, the deviation degree function and the stability coefficient function were constructed, and the stability scoring function was obtained after normalization, so as to realize the quantitative evaluation of the stability of full load NOx. The model was used to evaluate NOx data from a 660 MW coal-fired unit. The results show that the overall concentration of NOx concentration distribution is high and positively correlated with load rate. The stability score of variable load condition is about 2.35% lower than that of stable condition. When the load rate is increased from 50% to 100%, the stability score is increased by 21.28%. It shows that the model can effectively evaluate the stability of NOx distribution and provide a basis for benchmarking coal-fired units.

The paper considers the nature of input data used by Data Science algorithms of modern-day application domains. It then proposes three algorithms designed to remove statistical anomalies from datasets as a part of the Data Science pipeline. The main advantages of given algorithms are their relative simplicity and a small number of configurable parameters. Parameters are determined by machine learning with respect to the properties of input data. These algorithms are flexible and have no strict dependency on the nature and origin of data. The efficiency of the proposed approaches is verified with a modeling experiment conducted using algorithms implemented in Python. The results are illustrated with plots built using raw and processed datasets. The algorithms application is analyzed, and results are compared.

Abstract Species records from biological collections are becoming increasingly available online. This unprecedented availability of records has largely supported recent studies in taxonomy, biogeography, macroecology and biodiversity conservation. Biological collections vary in their documentation and notation standards, which have changed through time. For different reasons, neither collections nor data repositories perform the editing, formatting and standardisation of the data, leaving these tasks to the final users of the species records (e.g. taxonomists, ecologists and conservationists). These tasks are challenging, particularly when working with millions of records from hundreds of biological collections. To help collection curators and final users perform those tasks, we introduce plantR, an open‐source package that provides a comprehensive toolbox to manage species records from biological collections. The package is accompanied by the proposal of a reproducible workflow to manage this type of data in taxonomy, ecology and biodiversity conservation. It is implemented in R and designed to handle relatively large datasets as fast as possible. Initially designed to handle plant species records, many of the plantR features also apply to other groups of organisms, given that the data structure is similar. The plantR workflow includes tools to (a) download records from different data repositories, (b) standardise typical fields associated with species records, (c) validate the locality, geographical coordinates, taxonomic nomenclature and species identifications, including the retrieval of duplicates across collections, and (d) summarise and export records, including the construction of species lists with vouchers. Other R packages provide tools to tackle some of the workflow steps described above. But in addition to the new tools and resources related to data standardisation and validation, the greatest strength of plantR is to provide a comprehensive and user‐friendly workflow in one single environment, performing all tasks from data retrieval to export. Thus, plantR can help researchers better assess data quality and avoid data leakage in a wide variety of studies using species records.

Abstract Introduction Surveys are common research tools, and questionnaires revisions are a common occurrence in longitudinal studies. Revisions can, at times, introduce systematic shifts in measures of interest. We formulate that questionnaire revision are a stochastic process with transition matrices. Thus, revision shifts can be reduced by first estimating these transition matrices, which can be utilized in estimation of interested measures. Materials and method An ideal survey response model is defined by mapping between the true value of a participant’s response to an interval in the grouped data type scale. A population completed surveys multiple times, as modeled with multiple stochastic process. This included stochastic processes related to true values and intervals. While multiple factors contribute to changes in survey responses, here, we explored the method that can mitigate the effects of questionnaire revision. We proposed the Version Alignment Method (VAM), a data preprocessing tool, which can separate the transitions according to revisions from all transitions via solving an optimization problem and using the revision-related transitions to remove the revision effect. To verify VAM, we used simulation data to study the estimation error and a real life MJ dataset containing large amounts of long-term questionnaire responses with several questionnaire revisions to study its feasibility. Result We compared the difference of the annual average between consecutive years. Without adjustment, the difference is 0.593 when the revision occurred, while VAM brought it down to 0.115, where difference between years without revision was in the 0.005, 0.125 range. Furthermore, our method rendered the responses to the same set of intervals, thus comparing the relative frequency of items before and after revisions became possible. The average estimation error in L infinity was 0.0044 which occupied the 95% CI which was constructed by bootstrap analysis. Conclusion Questionnaire revisions can induce different response bias and information loss, thus causing inconsistencies in the estimated measures. Conventional methods can only partly remedy this issue. Our proposal, VAM, can estimate the aggregate difference of all revision-related systematic errors and can reduce the differences, thus reducing inconsistencies in the final estimations of longitudinal studies.

Images are increasingly used for AI-based diagnosis and analysis of many diseases like cervical cancer, mouth cancer, glucose analysis from retina etc. In many cases, data collection is done by specialised camera modules which capture images of affected areas. As with any other sources of data, this process is also error-prone and may contain unwanted objects and regions that may require cleaning by removing them. Outliers in these kinds of dataset may adversely affect the performance of machine learning models. Manually cleaning would be a tedious task, especially when the data is collated from different sources. Hence, cleaning the data before training the model is of utmost importance. In this paper, we propose a Few-Shot learning based model pre-trained in supervised contrastive learning settings to automate the process of data cleaning. Our model learns the dataset distribution and distinguishes the accurate data points from noisy data points. We also show that scaling up the model can greatly improve the Few-Shot performance. On the noisy MobileODT cervical data, which was collected from Kaggle, our model obtained 52% accuracy without cleaning data using an EfficientNet architecture for the classification task. Whereas the same architecture with ROI cropping achieved an accuracy of 76.56% after cleaning through the proposed Deep Cleaner approach that requires only 100 clean images. The proposed approach performs 2.74% better than a denoising auto-encoder, which is considered a powerful anomaly detection technique.

Short text classification techniques have been widely studied.When these techniques are applied to domain short text forproduction,as textual data accumulates,people often encounter problems mainly in two aspects:the imperfect labels and mistakenly-labeled training dataset.First,the class label set is generally dynamic in nature.Second,when domain annotators label textual data,it is hard to distinguish some fine-grained class label from others.For the above problems,this paper analyzes the shortcomings of an actual and complex telecom domain label set with numerous classes in depth and proposes a conceptual model for the imperfect multi-classification label system.Based on the conceptual model,for repairing the conflicts and omissions in a labeled dataset,we introduce a semi-automatic method for detecting these problems iteratively with the help of a seed dataset.After repairing the conflicts and omissions caused by a dynamic label set and mistakes of annotators,after about six months of iteration,the F1-score of the BERT-based classification model is above 0.9 after filtering out 10% tickets with low classification confidence.

Missing data imputation is a technique to deal with incomplete datasets. Since many models and algorithms cannot be applied to data containing missing values, a pre-processing step needs to be performed to remove incomplete data or to estimate the missing values. This is a well-known problem referred to as the data imputation problem. Several approaches have been designed for data imputation. These algorithms can be divided into two main categories: statistical and machine learning-based algorithms. As machine learning algorithms are optimized, they usually have better performance compared with statistical ones. In this paper, we review the most recent literature related to missing data imputation based on generative adversarial networks (GANs) that have gained tremendous attention in dealing with missing values. We examine the structures of GANs for missing data imputation and discuss the commonly used datasets and metrics for evaluation. We also cover the influence of the missing datatype, the effect of the missing data fraction, and the algorithm-related problems on data imputation performance. We conduct experiments on two publicly available datasets and evaluate the performance of GAIN, a missing data imputation algorithm to that of existing state-of-the-art approaches, demonstrating that the GAN-based algorithm outperforms the others in terms of RMSE and FID.

Radio frequency identification (RFID) technology is an automatic identification method, which relies on the use of radio repeaters called RFID tags to quickly store and retrieve data. Because RFID tags do not need linear contact when communicating with readers, it is possible to collect a large amount of data in a short time. However, the collected data also produce problems such as false negative readings, false positive readings, duplicated readings, out-of-order readings and so on. In this case, how to efficiently clean the large-scale RFID data in short time has become an important research topic in the field of database. This paper mainly summarizes the existing RFID data cleaning technology. Firstly, the relevant definitions and descriptions of RFID system and RFID data cleaning problem are given, and typical datasets and evaluation criteria are listed. Then the existing RFID data cleaning work is compared and summarized in detail from the classification, subcategory, basic idea, advantages, limitations, application scenarios and other aspects of related technologies. At the same time, the relevant application systems are compared and analyzed. Then, for the key problems of the false negative reading, false positive reading, duplicated reading and out-of-order reading, the existing studies are compared and summarized in detail. Finally, this paper proposes next five research directions worthy of paying attention to in the field of RFID data cleaning, such as the construction of RFID original data and benchmark dataset, the cleaning strategy of encryption and privacy protection data, the accuracy of data collection, the timeliness of cleaning results, and scene self-learning.

Data cleaning is considered as an effective approach of improving data quality in order to help practitioners and researchers be devoted to downstream analysis and decision-making without worrying about data trustworthiness. This paper provides a systematic summary of the two main stages of data cleaning for Internet of Things (IoT) data with time series characteristics, including error data detection and data repairing. In respect to error data detection techniques, it categorizes an overview of quantitative data error detection methods for detecting single-point errors, continuous errors, and multidimensional time series data errors and qualitative data error detection methods for detecting rule-violating errors. Besides, it provides a detailed description of error data repairing techniques, involving statistics-based repairing, rule-based repairing, and human-involved repairing. We review the strengths and the limitations of the current data cleaning techniques under IoT data applications and conclude with an outlook on the future of IoT data cleaning.

This report does some work on big data analysis of EV loads and extends automatic generation control (AGC) to automatic active power control (APC) by combining the current operating conditions of the grid and the spatial and temporal distribution characteristics of the adjustable capacity of EV loads. One year of user data from three EV charging stations provided by Chongqing is processed, cleaned, and clustered to analyze several typical EV load behaviors and the assessment of their adjustable characteristics. The data processing methods and the comparison of the K-means algorithm, the direct clustering, and the hierarchical clustering algorithm DPC clustering algorithm are also shown for this example.

Abstract Background RNA-seq has become a standard technology to quantify mRNA. The measured values usually vary by several orders of magnitude, and while the detection of differences at high values is statistically well grounded, the significance of the differences for rare mRNAs can be weakened by the presence of biological and technical noise. Results We have developed a method for cleaning RNA-seq data, which improves the detection of differentially expressed genes and specifically genes with low to moderate transcription. Using a data modeling approach, parameters of randomly distributed mRNA counts are identified and reads, most probably originating from technical noise, are removed. We demonstrate that the removal of this random component leads to the significant increase in the number of detected differentially expressed genes, more significant pvalues and no bias towards low-count genes. Conclusion Application of RNAdeNoise to our RNA-seq data on polysome profiling and several published RNA-seq datasets reveals its suitability for different organisms and sequencing technologies such as Illumina and BGI, shows improved detection of differentially expressed genes, and excludes the subjective setting of thresholds for minimal RNA counts. The program, RNA-seq data, resulted gene lists and examples of use are in the supplementary data and at https://github.com/Deyneko/RNAdeNoise .

Data Cleaning is a subfield of Data Mining that is thriving in the recent years. Ensuring the reliability of data, either when generated or received, is of vital importance to provide the best services possible to users. Accomplishing the aforementioned task is easier said than done, since data are complex, generated at an extremely high rate and are of enormous size. A variety of techniques and methods that are part of other subfields from the domain of the Computer Science have been invoked to assist in making Data Cleaning the most efficient and effective possible. Those subfields include, among others, Natural Language Processing (NLP), which in essence refers to the interaction among computers and human language, seeking to find a way to program computers to be able to process and analyze huge volumes of human language data. NLP is a concept that exists for a long time, but, as time goes by, it is proposed that it can be applied to a variety of concepts that are not solely NLP-related. In this paper, a rule-based data cleaning mechanism is proposed, which utilizes NLP to ensure data reliability. Making use of NLP enabled the mechanism not only to be extremely effective but also to be a lot more efficient compared to other corresponding mechanisms that do not utilize NLP. The mechanism was evaluated upon diverse healthcare datasets, not however being limited to the healthcare domain, but supporting a generalized data cleaning concept.

T cell receptors (TR) underpin the diversity and specificity of T cell activity. As such, TR repertoire data is valuable both as an adaptive immune biomarker, and as a way to identify candidate therapeutic TR. Analysis of TR repertoires relies heavily on computational analysis, and therefore it is of vital importance that the data is standardized and computer-readable. However in practice, the usage of different abbreviations and non-standard nomenclature in different datasets makes this data pre-processing non-trivial. tidytcells is a lightweight, platform-independent Python package that provides easy-to-use standardization tools specifically designed for TR nomenclature. The software is open-sourced under the MIT license and is available to install from the Python Package Index (PyPI). At the time of publishing, tidytcells is on version 2.0.0.

Aiming at the data preprocessing requirements and label data cost issues arising from the intelligent operation and maintenance of electromechanical equipment, this article mainly studies structured data cleaning methods and fault prediction algorithms for a small number of label samples. First, this article introduces the overall architecture of the intelligent operation and maintenance system for electromechanical equipment. Second, based on the electromechanical equipment operation and maintenance data access service, data cleaning, and fault prediction, this article constructs an electromechanical equipment intelligent operation and maintenance platform based on Kafka message queue, Spark cluster, and other components, and introduces the functional composition of the system in detail. Finally, the article describes the functions of each component of data access service, data cleaning, and fault prediction in detail. To address the cost issue associated with sufficient labeled sample data for data analysis, we propose a semi-supervised learning algorithm, IF-GBDT, based on improved independent forests and Gradient Boosting Decision Tree. The independent forest algorithm supplements labels for unlabeled data based on the learning results of a small number of labeled samples. We also use the gradient lifting tree algorithm to train the model based on the new tag data set for fault prediction, thereby reducing the impact of lack of tags on the accuracy of the prediction model. Experiments show that this method improves classification accuracy and has good adaptability and concurrency performance for a small number of tags.

AbstractTextbook data is essential for teaching statistics and data science methods because it is clean, allowing the instructor to focus on methodology. Ideally textbook datasets are refreshed regularly, especially when they are subsets taken from an ongoing data collection. It is also important to use contemporary data for teaching, to imbue the sense that the methodology is relevant today. This article describes the trials and tribulations of refreshing a textbook dataset on wages, extracted from the National Longitudinal Survey of Youth (NLSY79) in the early 1990s. The data is useful for teaching modeling and exploratory analysis of longitudinal data. Subsets of NLSY79, including the wages data, can be found in supplementary materials from numerous textbooks and research articles. The NLSY79 database has been continually updated through to 2018, so new records are available. Here we describe our journey to refresh the wages data, and document the process so that the data can be regularly updated into the future. Our journey was difficult because the steps and decisions taken to get from the raw data to the wages textbook subset have not been clearly articulated. We have been diligent to provide a reproducible workflow for others to follow, which also hopefully inspires more attempts at refreshing data for teaching. Three new datasets and the code to produce them are provided in the open source R package called yowie. Supplementary materials for this article are available online.

The current real-time and integrity of massive data acquisition in fully mechanized working faces are poor. The abnormal data cleaning takes a long time. The data mining delays are large. This leads to low utilization rate of fully mechanized working data and incapability to assist management in issuing decision-making instructions in real-time. In order to solve the above problems, a massive data mining and analysis platform for fully mechanized working faces is designed. The platform consists of a data source layer, a data acquisition and storage layer, a data mining layer, and a front-end application layer. The data source layer is provided with raw data by various hardware devices on the working surface. The data acquisition and storage layer uses the OPC UA gateway to collect real-time monitoring information from underground sensors, and then stores the data in the InfluxDB storage engine through the MQTT protocol and RESTful interface. The data mining layer uses the Hive data engine and Yarn resource manager to filter out abnormal data caused by workplace interference during the data acquisition process. It solves the problem of local data acquisition order disorder caused by network latency. The Spark distributed mining engine is used to explore the potential value of massive working condition data in the working face device group, improving the running speed of the data mining model. The front-end application layer utilizes visual components to associate with the back-end database. It interacts with the back-end data in real-time through AJAX technology to achieve visual display of model mining results and various monitoring data. The test results show that the platform can fully ensure the real-time and integrity of data acquisition. The cleaning efficiency is 5 times better than a standalone MySQL query engine and the mining efficiency is 4 times better than a standalone Python mining engine.

Over the past few decades, rapid economic development has led to the establishment of numerous monitoring systems, resulting in the accumulation of vast amounts of monitoring data. Among these data, dynamic acceleration data stand out prominently. However, the quality of collected acceleration data is often compromised due to factors such as challenging operational environments and sensor malfunctions. This severely hampers the value extracted from the data. Although manual identification and classification of data anomalies are more reliable, they are time consuming and labor intensive. To address the challenge of identifying and classifying anomalies in massive acceleration data, this paper proposes a two-stage model for intelligent data cleaning. Firstly, raw acceleration data are transformed into IPDF and PSD features, and a one-dimensional convolutional neural network is trained to preliminarily identify and classify acceleration data anomalies. Subsequently, the RPV indicator is extracted from the original data of the normal and outlier categories to achieve precise classification based on threshold values. The proposed method is successfully validated using acceleration monitoring data from a large-span arch bridge, achieving an accuracy of over 99%. Furthermore, compared to directly employing a one-dimensional CNN classification model, the approach significantly enhances the model’s perception of local significant disturbances.

The application of data science (DS) techniques has become increasingly essential in various fields, including epidemiology and climatology in agricultural production systems. In this sector, traditionally large amounts of data are acquired, but not well-managed and -analyzed as a basis for evidence-based decision-making processes. Here, we present a comprehensive step-by-step guide that explores the use of DS in managing epidemiological and climatological data within rice production systems under tropical conditions. Our work focuses on using the multi-temporal dataset associated with the monitoring of diseases and climate variables in rice in Colombia during eight years (2012–2019). The study comprises four main phases: (I) data cleaning and organization to ensure the integrity and consistency of the dataset; (II) data management involving web-scraping techniques to acquire climate information from free databases, like WordClim and Chelsa, validation against in situ weather stations, and bias removal to enrich the dataset; (III) data visualization techniques to effectively represent the gathered information, and (IV) a basic analysis related to the clustering and climatic characterization of rice-producing areas in Colombia. In our work, a process of evaluation and the validation of climate data are conducted based on errors (r, R2, MAE, RSME) and bias evaluation metrics. In addition, in phase II, climate clustering was conducted based on a PCA and K-means algorithm. Understanding the association of climatic and epidemiological data is pivotal in predicting and mitigating disease outbreaks in rice production areas. Our research underscores the significance of DS in managing epidemiological and climatological data for rice production systems. By applying a protocol responsible for DS tools, our study provides a solid foundation for further research into disease dynamics and climate interactions in rice-producing regions and other crops, ultimately contributing to more informed decision-making processes in agriculture.

Developments in big data technology, wireless networks, Geographic information system (GIS) technology, and internet growth has increased the volume of data at an exponential rate. Internet users are generating data with every single click. Geospatial metadata is widely used for urban planning, map making, spatial data analysis, and so on. Scientific databases use metadata for computations and query processing. Cleaning of data is required for improving the quality of geospatial metadata for scientific computations and spatial data analysis. In this paper, we have designed a data cleaning tool named as GeoWebCln to remove useless data from geospatial metadata in a user-friendly environment using the Python console of QGIS Software.

Purpose: Oncological Information Systems (OIS) manage information in radiotherapy (RT) departments. Due to database structure limitations, stored information can rarely be directly used except for vendor-specific purposes. Our aim is to enable the use of such data in various external applications by creating a tool for automatic data extraction, cleaning and formatting. Methods and materials: We used OIS data from a nine-linac RT department in Sweden (70 weeks, 2015–16). Extracted data included patients’ referrals and appointments with details for RT sub-tasks. The data extraction tool to prepare the data for external use was built in C# programming language. It used excel-automation queries to remove unassigned/duplicated values, substitute missing data and perform application-specific calculations. Descriptive statistics were used to verify the output with the manually prepared dataset from the corresponding time period. Results: From the initial raw data, 2030 (51 %)/907 (23 %) patients had known curative and palliative treatment intent for 84 different cancer diagnoses. After removal of incomplete entries, 373 (10 %) patients had unknown treatment intents which were substituted based on the known curative/palliative ratio. Automatically- and manuallyprepared datasets differed

For a long-distance open-channel water transfer project, the correctness of the hydrological state study and hydrodynamic numerical simulation is impacted directly by the spatiotemporal and consistency of the flow monitoring data along the project. Reported here is an inverted data cleaning model for flow monitoring stations based on the principle of water dynamic balance and the longest sequence method of interval flow, targeting the abnormal phenomenon of unequal flow at monitoring stations in the dispatching operation and using the middle route of the South-to-North Water Diversion Project in China as the research object. As an example, a hydrodynamic model is built for verification using flow and water level data after model cleaning as the upper and lower boundaries, respectively, for the middle-route project from the Baihe River to Huangjin River. The findings indicate that the cleaning model enhances the accuracy of flow monitoring data and reduces both the mean absolute error of the water level in front of the gate and the root-mean-square error by 0.0757 and 0.0895 m, respectively. In terms of data spatial consistency and logic, the cleaned flow data are superior to the measured flow data.