Search

Record linkage is an important tool to enhance database integration. This is even more valuable in a scenario with more hefty budget cuts and a growing drop in response rate in traditional surveys. This strategy makes it possible to expand the crossing alternatives with variables not present in the original base. However, there are many different data pairing methods exposed in the literature. In this sense, the objective of this paper is to compare well-known methods of record linkage. The comparison was made in synthetic dataset. To compare the methods, it was adopted a quantitative approach based on the Precision, Recall, and F-Statistics metrics, using two comparison functions: Levenshtein and Jaro-Winkler. Among the six types of classifiers analyzed, the supervised methods had the best results.

Mobile communication providers continuously collect many parameters, statistics, and key performance indicators (KPIs) with the goal of identifying operation scenarios that can affect the quality of Internet-based services. In this regard, anomaly detection and classification in mobile networks have become challenging tasks due to both the huge number of involved variables and the unknown distributions exhibited by input features. This paper introduces an unsupervised methodology based on both a data-cleaning strategy and explainable machine learning models to detect and classify performance anomalies in mobile networks. Specifically, this methodology dubbed explainable machine learning for anomaly detection and classification (XMLAD) aims at identifying features and operation scenarios characterizing performance anomalies without resorting to parameter tuning. To this end, this approach includes a data cleaning stage that extracts and removes outliers from experiments and features to train the anomaly detection engine with the cleanest possible dataset. Moreover, the methodology considers the differences between discretized values of the target KPI and labels predicted by the anomaly detection engine to build the anomaly classification engine which identifies features and thresholds that could cause performance anomalies. The proposed methodology incorporates two decision tree classifiers to build explainable models of anomaly detection and classification engines whose decision structures recognize features and thresholds describing both normal behaviors and performance anomalies. We evaluate the XMLAD methodology on real datasets captured by operational tests in commercial networks. In addition, we present a testbed that generates synthetic data using a known TCP throughput model to assess the accuracy of the proposed approach. Spanish State Research Agency - Spanish Ministry of Science and Innovation Ministry of Economic Affairs and Digital Transformation, European Union NextGeneration-EU Department of Education and Research of the Regional Government of Madrid, through the 2018 R&D technology program for research groups, co-financed by the Operational Programs of the European Social Fund (ESF) and the European Regional Development Fund (ERDF) Nokia Spain TRUE inpress

The New York Times recently published an article on how prisoners in the State of Louisiana often remain incarcerated well after their release dates (https://www.nytimes.com/2022/12/11/us/politics/louisiana-prison-overdetention.html). The article notes many contributing factors to this problem, including issues with the Criminal and Justice Unified Network system that the state uses to track offenders within the corrections system. In fact, this system was audited by the state, which found many issues with the management and accuracy of data in this system: https://app.lla.state.la.us/go.nsf/getSummary?OpenAgent&arlkey=40160023CWIN-AYMATY. In this presentation, I will discuss my own experiences with data from this system, including issues with both the "tidyness" of the data and their accuracy and the challenges of working with them in the context of helping a student work with the data for his dissertation. The presentation will also be in part a case study of the challenges that accompany working with administrative data that are not necessarily collected with researchers in mind.

This is a story about the challenges and opportunities that surfaced while answering a deceptively complex question - where's the data? As faculty and researchers publish articles, datasets, and other research outputs to meet promotion and tenure requirements, address federal funding policies, and institutional open access and data sharing policies, many online locations for publishing these materials have developed over time. How can we capture where all of the research generated on an academic campus is shared and preserved? This presentation will discuss how our multi-institution collaboration, the Reality of Academic Data Sharing (RADS) Initiative, sought to answer this question. We programmatically pulled DOIs from DataCite and CrossRef, making the naive assumption that these platforms, the two predominant DOI registration agencies for US data, would present us with a neutral and unbiased view of where data from our affiliated researchers were shared. However, as we dug into the data, we found inconsistencies in the use and completeness of the necessary metadata fields for our questions, as well as differences in how DOIs were assigned across repositories. Additionally, we recognized the systematic and privileged bias introduced by our choice of data sources. Specifically, while DataCite and CrossRef provide easy discovery of research outputs because they aggregate DOIs, they are also costly commercial services. Many repositories that cannot afford such services or lack local staffing and knowledge required to use these services are left out of the technology that has recently been labeled “global research infrastructure”. Our presentation will identify the challenges we encountered in conducting this research specifically around finding the data, and cleaning and interpreting the data. We will further engage the audience in a discussion around increasing representation in the global research infrastructure to discover and account for more research outputs.

The technical project was a website created to help Uva students with the Off-Grounds Housing search. The objective was to create a central application that makes the search personalized and easy. The website included features to allow students to browse and sort different housing options with accurate information and to communicate with others regarding housing. Most notably, our project enabled students to look for housing through our map search and sorting features. Students were also able to communicate with each other through publicly available reviews and a public forum. This website utilized Django as the web framework and involved the use of UI/UX design, unit testing, databases, and Heroku deployment. Although all of the features of the website are functional, the UI/UX design can be improved to reflect the changing needs of the students in the future. The STS Research Paper focuses on studying the sources of biases in AI hiring systems in the hopes of mitigating them. The bias against underrepresented demographics in AI hiring systems can be examined and potentially mitigated by using the Actor Network Theory to examine the role of different actors in the AI hiring system. The ANT framework can be applied to AI hiring systems by examining the roles of algorithms, datasets, candidates, and hiring managers and how they propagate bias in hiring decisions. By analyzing the effect of these actors on the technological development of AI hiring systems, the source of the bias can be pinpointed and potentially mitigated.

Best student paper award ICRAT 2020; International audience; Moving object analysis is a constantly growing field with numerous concrete applications in terms of traffic understanding, prediction and simulation. While many algorithms and analytic processes exist, there are still areas of investigation with novel trajectory analysis methods. As such, the geometric information analyses data with respect to its statistical distribution along extracted dimensions. This opens new ways of gaining a better understanding of large and complex trajectory data sets while providing flexible data manipulations. In this paper, we report our investigations with the development of an interactive methodology based on the geometric information analytic process where users can analyze trajectories sets, cluster and deform them maintaining the actual statistical properties of the investigated trajectories. As a contribution, this paper shows how geometric information can provide novel support for trajectory analyses taking into account the statistical properties of the investigated clusters. We also provide recommendations of good usage of such techniques with actual examples validated by a a domain expert of air traffic flow analysis

International audience; Live weight (LW) is a key and conventional indicator for monitoring and assessing overall animal performance and welfare, representing the progress through different physiological stages, while providing close indication of individual physical and health status. Measuring LW in practice is still, however, quite rare and infrequent under commercial sheep farming conditions, mainly because sessions are time-consuming, stressful either for the operator or the animals. A Walk-over-Weighing (WoW) system was tested in this experiment lasting 14 weeks (i.e. 3 weeks for acclimation and adaptation and 11 weeks for data collection). We validated its use for routine and frequent monitoring of growth rate in postweaned Merinos d’Arles ewe lambs (n = 100), reared under Mediterranean grazing conditions. The necessity for an initial adaptation period of the animals was confirmed. Also, the importance of conducting an effective data cleaning procedure of the raw database automatically collected by the WoW was corroborated. Adaptation of naive ewe lambs enabled the required voluntary passages across the weighing platform and a high volume of individual and daily data after 2–3 weeks. Close monitoring of individual growth was then possible after performing sound data cleaning. A good agreement was demonstrated between WoW LW and a reference LW value (measured with a standard static scale). At the individual level, even with the lowest number of LW values collected with WoW, it was possible to monitor variations in LW at daily intervals. The establishment of an early warning system to help farmer decision-making could therefore be possible. Our results show interesting prospects for more accurate and frequent monitoring of LW in grazing sheep without human intervention, compared to what is currently carried out on commercial farms.

Data wrangling tasks such as obtaining and linking data from various sources, transforming data formats, and correcting erroneous records, can constitute up to 80% of typical data engineering work. Despite the rise of machine learning and artificial intelligence, data wrangling remains a tedious and manual task. We introduce AI assistants, a class of semi-automatic interactive tools to streamline data wrangling. An AI assistant guides the analyst through a specific data wrangling task by recommending a suitable data transformation that respects the constraints obtained through interaction with the analyst. We formally define the structure of AI assistants and describe how existing tools that treat data cleaning as an optimization problem fit the definition. We implement AI assistants for four common data wrangling tasks and make AI assistants easily accessible to data analysts in an open-source notebook environment for data science, by leveraging the common structure they follow. We evaluate our AI assistants both quantitatively and qualitatively through three example scenarios. We show that the unified and interactive design makes it easy to perform tasks that would be difficult to do manually or with a fully automatic tool., Accepted for publication in IEEE Transactions on Knowledge and Data Engineering

Formation de 2 heures

Background: Wearable actigraphy monitors have seen rapid growth in both clinical and physical activity (PA) research. For research studies that utilize human participants, these portable devices provide a means to objectively quantify symptom burden and therapy effectiveness. The highly detailed data gathered from actigraphy devices allows researchers to identify even the subtlest changes in rest, sleep, and activity patterns that occur with other therapeutics. Compared to participant self-report, the data collection automation by a passive activity monitor improves the quality and reliability of the data. However, there is no consensus in how the data from these monitors are processed, different actigraphy brands and models used, and inconsistent reporting making comparisons difficult. There are several actigraphy data processing methods, but no consensus on definitions for PA values or cleaning guidelines to facilitate comparison across studies. PA research could be advanced with globally acceptable standardization in the processing of PA data. We will conduct a scoping review to identify common cleaning and pre-processing techniques of raw PA counts collected from commercially available actigraphy devices. Methods: We will use the Joanna Briggs Institute’s methodology to guide the conduct of this review and the PRISMA extension for scoping reviews checklist for reporting. A librarian will search PubMed (NLM), Scopus (Elsevier), and Web of Science: Core Collection (Clarivate Analytics) using keywords and controlled vocabulary. The reference lists of included articles and consultation with experts will be used to identify additional records. Two authors will first screen the titles and abstracts and then full-text using Covidence. Studies describing the cleaning or preparation of PA data collected from actigraphy monitors for human movement will be included. Two authors will independently extract from each included article the type of PA analyzed, criteria and descriptions used to categorize PA, accelerometry data collection methods, actigraphy cleaning method, actigraphy algorithms, and PA variables. Results will be presented through a descriptive numerical summary of studies, PRISMA flow diagram, and tables showing the cleaning methods and algorithms used and PA variables. Discussion: The findings of this scoping review will identify common actigraphy data cleaning procedures to fill the existing gap in PA research.

Feature engineering (FE) is one of the most important steps in data science research. FE provides useful features to be used later in the study. Due to climate change, the research focus is moving towards air quality estimation and the impacts of air pollution on health in Malaysia. Malaysia has 66 air quality monitoring (AQM) stations, and the air quality data for research is provided in an excel worksheet format by the Department of Environment, Malaysia. The data generated by the AQM stations is in a raw custom format, and it is virtually impossible to clean and engineer this data manually due to the sheer number of files. Hence, we propose a novel feature engineering algorithm to transform and combine this data into a useable format. The results show that the proposed feature engineering algorithm was able to efficiently extract and combine the hourly and daily values for pollutant and meteorological variables in useful row format. This algorithm will help all the researchers using the data from the AQM station in Malaysia as well as other countries using the same AQM station. The implementation of the feature engineering algorithm is also available to use at GitHub (https://github.com/rajasherafgun/featureengineeringaq) under AFL-3.0 license.

This is a how-to guide for cleaning vaginal photoplethysmography (VPP) signal using AcqKnowledge (v. 5.0.5) and Excel (or equivalent database management software, like OpenOffice).

Repository with R vignettes to support data organization and analysis for the SHARE study

Premise: There has been a great increase in using climatic data in phylogenetic studies over the past decades. However, compiling the high-quality spatial data needed to perform accurate climatic reconstructions is time-consuming and can result in poor geographical coverage. Therefore, researchers often resort to qualitative approximations. Our aim was to evaluate the climatic characterization of the genera of the Asian Palmate Group (AsPG) of Araliaceae as an exemplar lineage of plants showing tropical–temperate transitions. Methods: We compiled a curated worldwide spatial database of the AsPG genera and created five raster layers representing bioclimatic regionalizations of the world. Then, we crossed the database with the layers to climatically characterize the AsPG genera. Results: We found large disagreement in the climatic characterization of genera among regionalizations and little support for the climatic nature of the tropical–temperate distribution of the AsPG. Both results are attributed to the complexity of delimiting tropical, subtropical, and temperate climates in the world and to the distribution of the study group in regions with transitional climatic conditions. Conclusions: The complexity in the climatic classification of this example of the tropical–temperate transitions calls for a general climatic revision of other tropical–temperate lineages. In fact, we argue that, to properly evaluate tropical–temperate transitions across the tree of life, we cannot ignore the complexity of distribution ranges, This study was supported by the Spanish Ministry of Economy, Industry and Competitiveness (CGL2017‐87198‐ P) and the Spanish Ministry of Science an Innovation (PID2019‐106840GA‐C22). M.C.d.l.I. was supported by the Youth Employment Initiative of European Social Fund and Community of Madrid (PEJ‐2017‐AI‐AMB‐6636 and CAM2020PEJD‐2019‐11 PRE/AMB‐15871)

International audience; Conservation efforts in global biodiversity hotspots often face a common predicament: an urgent need for conservation action hampered by a significant lack of knowledge about that biodiversity. In recent decades, the computerisation of primary biodiversity data worldwide has provided the scientific community with raw material to increase our understanding of the shared natural heritage. These datasets, however, suffer from a lot of geographical and taxonomic inaccuracies. Automated tools developed to enhance their reliability have shown that detailed expert examination remains the best way to achieve robust and exhaustive datasets. In New Caledonia, one of the most important biodiversity hotspots worldwide, the plant diversity inventory is still underway, and most taxa awaiting formal description are narrow endemics, hence by definition hard to discern in the datasets. In the meantime, anthropogenic pressures, such as nickel-ore mining, are threatening the unique ultramafic ecosystems at an increasing rate. The conservation challenge is therefore a race against time, as the rarest species must be identified and protected before they vanish. In this study, based on all available datasets and resources, we applied a workflow capable of highlighting the lesser known taxa. The main challenges addressed were to aggregate all data available worldwide, and tackle the geographical and taxonomic biases, avoiding the data loss resulting from automated filtering. Every doubtful specimen went through a careful taxonomic analysis by a local and international taxonomist panel. Geolocation of the whole dataset was achieved through dataset cross-checking, local botanists’ field knowledge, and historical material examination. Field studies were also conducted to clarify the most unresolved taxa. With the help of this method and by analysing over 85,000 data, we were able to double the number of known narrow endemic taxa, elucidate 68 putative new species, and update our knowledge of the rarest species’ distributions so as to promote conservation measures.

The process of preparing potentially large and complex data sets for further analysis or manual examination is often called data wrangling. In classical warehousing environments, the steps in such a process are carried out using Extract-Transform-Load platforms, with significant manual involvement in specifying, configuring or tuning many of them. In typical big data applications, we need to ensure that all wrangling steps, including web extraction, selection, integration and cleaning, benefit from automation wherever possible. Towards this goal, in the paper we: (i) introduce a notion of data context, which associates portions of a target schema with extensional data of types that are commonly available; (ii) define a scalable methodology to bootstrap an end-to-end data wrangling process based on data profiling; (iii) describe how data context is used to inform automation in several steps within wrangling, specifically, matching, value format transformation, data repair, and mapping generation and selection to optimise the accuracy, consistency and relevance of the result; and (iv) we evaluate the approach with real estate data and financial data, showing substantial improvements in the results of automated wrangling.

The goal of data cleaning is to make data fit for purpose, i.e., to improve data quality, through updates and data transformations, such that downstream analyses can be conducted and lead to trustworthy results. A transparent and reusable data cleaning workflow can save time and effort through automation, and make subsequent data cleaning on new data less error-prone. However, the reusability of data cleaning workflows has received little to no attention in the research community. We identify some challenges and opportunities for reusing data cleaning workflows. We present a high-level conceptual model to clarify what we mean by reusability and propose ways to improve reusability along different dimensions. We use the opportunity of presenting at IDCC to invite the community to share their use cases, experiences, and desiderata for the reuse of data cleaning workflows and recipes in order to foster new collaborations and guide future work., This lightning talk is given on 2022-06-14, 17th International Digital Curation Conference (IDCC22)

Data cleaning and preparation are essential parts of data curation lifecycles and scientific workflow. It is also known that exploratory data mining and data cleaning takes 80% of the scientific research pipeline. However, a data cleaning task can be very tedious for a single user, involving lots of exploration and iteration, and prone to error, especially when a curator finds various problems in the dataset. Nevertheless, the single-user data cleaning can also introduce bias where the cleaning quality will only be as good as their knowledge. Therefore, we can assign a data cleaning task to multiple data curators to collaborate on curating datasets. However, when a data cleaning task involves multiple users, it can introduce new problems such as data change disagreement and conflicting process dependency. Understanding this variation in changes and analyzing the merging workflow is important for data curation to evolve the data cleaning workflow and improve the dataset's quality. In line with the reusability theme for IDCC 2022, this approach can help improve the data curation pipeline by improving the data cleaning pipeline through collaboration.&nbsp

Agriculture is the backbone of a country's economic system, considering that it not only provides food and raw materials but also employment opportunities for a large percentage of the population. In this way, determining the degree of agricultural vulnerability represents a guide for sustainability and adaptability focused on changing future conditions. In many cases, vulnerability analysis data is restricted to use by authorized personnel only, leaving open data policies aside. Furthermore, data in its native format (raw data) by nature tend to be diverse in structure, storage formats, and access protocols. In addition, having a large amount of open data is important (though not sufficient) to obtain accurate results in data-driven analysis. These data require a strict preparation process and having guides that facilitate this process is becoming increasingly necessary. In this study, we present the step by step processing of several open data sources in order to obtain quality information for feedback on different agricultural vulnerability analysis. The data preparation process is applied to a case study corresponding to the upper Cauca river basin in Colombia. All data sources in this study are public, official and are available from different web platforms where they were collected. In the same way, a ranking with the importance of variables for each dataset was obtained through automatic methods and validated through expert knowledge. Experimental validation showed an acceptable agreement between the ranking of automatic methods and the ranking of raters. The result of this study corresponds to 16 processed data sources ready to feed data-driven systems, as well as agricultural vulnerability methodologies.

Errors are prevalent in time series data, which is particularly common in the industrial field. Data with errors could not be stored in the database, which results in the loss of data assets. At present, to deal with these time series containing errors, besides keeping original erroneous data, discarding erroneous data and manually checking erroneous data, we can also use the cleaning algorithm widely used in the database to automatically clean the time series data. This survey provides a classification of time series data cleaning techniques and comprehensively reviews the state-of-the-art methods of each type. Besides we summarize data cleaning tools, systems and evaluation criteria from research and industry. Finally, we highlight possible directions time series data cleaning.

Im Beitrag wird die These vertreten, dass die Nutzung von Metadaten für Forschung in den digitalen Geisteswissenschaften, wie andere historische Forschung auch, digitale Quellenkritik voraussetzt. Diese Quellenkritik findet schon in der Vorbereitung von Daten für die Analyse, dem sogenannten data cleaning statt. Untersucht werden Metadaten zu philosophischen Dissertationen des 17. Jahrhunderts, die an katholischen Bildungseinrichtungen entstanden sind und in VD17, der deutschen Nationalbibliographie für Drucke dieser Periode, verzeichnet sind. Im Mittelpunkt steht dabei die Frage, wie viele solcher Dissertationen in VD17 enthalten sind. Zu zeigen ist, warum die Beantwortung dieser Frage schwerer fällt, als dies zunächst zu erwarten war. Ein Beitrag zur 8. Tagung des Verbands "Digital Humanities im deutschsprachigen Raum" - DHd 2022 Kulturen des digitalen Gedächtnisses.

International audience; The field of data repairing is very active and produces a lot of repairing methods. This abundance of options can make choosing a re- pairing method hard. Our research question stems from this problem: Is it always better to repair data? Our work is placed in the context of clas- sification tasks and numerical data. We investigates our research ques- tion through five criteria: C1 the perceived difficulty of using a repairing method according to experts, C2 the impact of the level of degradation of data on accuracies and f1 scores, C3 the effectiveness of the repair- ing tool, C4 the impact of the type of error present in data and C5 the impact of the classification model used. Our main contributions are a method to evaluate the difficulty of using a repairing tool and a study of experimental results on accuracies and f1 scores investigating C2 to C5. We were able to identify two categories of error types: low impact and high impact on accuracy and f1 score. We also observed that repair- ing methods perform similarly both at very low and very high levels of errors.

International audience; Digital data is produced in many data models, ranging from highly structured (typically relational) to semi-structured models (XML, JSON) to various graph formats (RDF, property graphs) or text. Most real-world datasets contain a certain amount of null values, denoting missing, unknown, or inapplicable information. While some data models allow representing nulls by special tokens, so-called disguised missing values (DMVs, in short) are also frequently encountered: these are values that are not syntactically speaking nulls, but which do, nevertheless, denote the absence, unavailability, or inapplicability of the information. In this work, we tackle the detection of a particular kind of DMV: texts freely entered by human users. This problem is not tackled by DMV detection methods focused on numeric or categoric data; further, it also escapes DMV detection methods based on value frequency, since such free texts are often different from each other, thus most DMVs are unique. We encountered this problem within the ConnectionLens [6,7,8,12] project where heterogeneous data is integrated into large graphs. We present two DMV detection methods for our specific problem: (i) leveraging Information Extraction, already applied in ConnectionLens graphs; and (ii) through text embeddings and classification. We detail their performanceprecision trade-offs on real-world datasets.

The Internet of things (IoT) combines different sources of collected data which are processed and analyzed to support smart city applications. Machine learning and deep learning algorithms play a vital role in edge intelligence by minimizing the amount of irrelevant data collected from multiple sources to facilitate these smart city applications. However, the data collected by IoT sensors can often be noisy, redundant, and even empty, which can negatively impact the performance of these algorithms. To address this issue, it is essential to develop effective methods for detecting and eliminating irrelevant data to improve the performance of intelligent IoT applications. One approach to achieving this goal is using data cleaning techniques, which can help identify and remove noisy, redundant, or empty data from the collected sensor data. This paper proposes a deep reinforcement learning (deep RL) framework for IoT sensor data cleaning. The proposed system utilizes a deep Q-network (DQN) agent to classify sensor data into three categories: empty, garbage, and normal. The DQN agent receives input from three received signal strength (RSS) values, indicating the current and two previous sensor data points, and receives reward feedback based on its predicted actions. Our experiments demonstrate that the proposed system outperforms a common time-series-based fully connected neural network (FCDQN) solution, with an accuracy of around 96% after the exploration mode. The use of deep RL for IoT sensor data cleaning is significant because it has the potential to improve the performance of intelligent IoT applications by eliminating irrelevant and harmful data.

Even though data preparation constitutes a large proportion of the total effort involved in electronic health record (EHR) based secondary analysis, guidelines for EHR data preparation are still insufficient to date. This study proposes a data preparation framework that can guide and validate the cleaning of EHRs for secondary analysis. The developed framework consists of three core themes—workflow, assessment and cleaning methods, and cleaning evaluation scheme. To illustrate the viability of the proposed framework, we applied it to a hip-fracture readmission scenario using the underlying data extracted from a large EHR database. The case study demonstrated the effectiveness of the proposed framework in organizing and standardizing phases and processes within an EHR data preparation workflow. Furthermore, the cleaning evaluation scheme was found to be effective in validating EHR cleaning methods, especially for those used to handle complex issues that usually appear in patient demographics, longitudinal attributes of EHRs, and the application of filtering and imputation cleaning methods.

Data about mobility provides information to improve city planning, identify traffic patterns, detect traffic jams, and route vehicles around them. This data often contains proprietary and personal information that companies and individuals do not wish others to know, for competitive and personal reasons. This sets up a paradox: the data needs to be analyzed, but it cannot be without revealing information that must be kept secret. A solution is to sanitize the data—i.e., remove or suppress the sensitive information. The goal of sanitization is to protect sensitive information while enabling analyses of the data that will produce the same results as analyses of the unsanitized data. However, protecting information requires that sanitized data cannot be linked to data from other sources in a manner that leads to desanitization. This project reviews typical strategies used to sanitize datasets, the research on how some of these strategies are unsuccessful, and the questions that must be addressed to better understand the risks of desanitization.

In this paper, we compare the most popular Atom-to-Atom Mapping (AAM) tools: ChemAxon,([1]) Indigo,([2]) RDTool,([3]) NameRXN (NextMove),([4]) and RXNMapper([5]) which implement different AAM algorithms. An open-source RDTool program was optimized, and its modified version ("new RDTool") was considered together with several consensus mapping strategies. The Condensed Graph of Reaction approach was used to calculate chemical distances and develop the "AAM fixer" algorithm for an automatized correction of erroneous mapping. The benchmarking calculations were performed on a Golden dataset containing 1851 manually mapped and curated reactions. The best performing RXNMapper program together with the AMM Fixer was applied to map the USPTO database. The Golden dataset, mapped USPTO and optimized RDTool are available in the GitHub repository https://github.com/Laboratoire-de-Chemoinformatique.

This presentation was given as part of a data management workshop associated with the 2021 Online OBFS Meeting., Funding: NSF DBI-1931143 and NSF DBI-1931174

Insights and analysis are only as good as the available data. Data cleaning is one of the most important steps to create quality data decision making. Machine learning (ML) helps deal with data quickly, and to create error-free or limited-error datasets. One of the quality standards for cleaning the data includes handling the missing data, also known as data imputation. This research focuses on the use of machine learning methods to deal with missing data. In particular, we propose a generative adversarial network (GAN) based model called DEGAIN to estimate the missing values in the dataset. We evaluate the performance of the presented method and compare the results with some of the existing methods on publicly available Letter Recognition and SPAM datasets. The Letter dataset consists of 20,000 samples and 16 input features and the SPAM dataset consists of 4601 samples and 57 input features. The results show that the proposed DEGAIN outperforms the existing ones in terms of root mean square error and Frechet inception distance metrics.

This article is focused on the automatic detection of the corrupted or inappropriate responses in questionnaire data using unsupervised outliers detection. The questionnaire surveys are often used in psychology research to collect self-report data and their preprocessing takes a lot of manual effort. Unlike with numerical data where the distance-based outliers prevail, the records in questionnaires have to be assessed from various perspectives that do not relate so much. We identify the most frequent types of errors in questionnaires. For each of them, we suggest different outliers detection methods ranking the records with the usage of normalized scores. Considering the similarity between pairs of outlier scores (some are highly uncorrelated), we propose an ensemble based on the union of outliers detected by different methods. Our outlier detection framework consists of some well-known algorithms but we also propose novel approaches addressing the typical issues of questionnaires. The selected methods are based on distance, entropy, and probability. The experimental section describes the process of assembling the methods and selecting their parameters for the final model detecting significant outliers in the real-world HBSC dataset. Web of Science 206 art. no. 117809

In the industrial domain, maintenance is essential to guarantee the correct operations, availability, and efficiency of machinery and systems. With the advent of Industry 4.0, solutions based on machine learning can be used for the prediction of future failures by exploiting historical failure data. Most of the time, these historical data have been collected by companies without a specific structure, schema, or even best practices, resulting in a potential loss of knowledge. In this paper, we analyze the historical data on maintenance alerts of the components of a revamping topping plant (referred to as RT2) belonging to the SARAS group. This analysis is done in collaboration with the ITALTELECO company, a partner of SARAS, that provided the necessary data. The pre-processing methodology to clean and fill these data and extract features useful for a prediction task will be shown. More in detail, we show the process to fill missing fields of these data to provide (i) a category for each fault by using simple natural language processing techniques and performing a clustering, and (ii) a data structure that can enable machine learning models and statistical approaches to perform reliable failure predictions. The data domain in which this methodology is applied is oil and gas, but it may be generalized and reformulated in various industrial and/or academic fields. The ultimate goal of our work is to obtain a procedure that is simple and can be applied to provide strategic support for the definition of an adequate maintenance plan.

Background Predicting protein-ligand binding sites is a fundamental step in understanding the functional characteristics of proteins, which plays a vital role in elucidating different biological functions and is a crucial step in drug discovery. A protein exhibits its true nature after binding to its interacting molecule known as a ligand that binds only in the favorable binding site of the protein structure. Different computational methods exploiting the features of proteins have been developed to identify the binding sites in the protein structure, but none seems to provide promising results, and therefore, further investigation is required. Results In this study, we present a deep learning model PUResNet and a novel data cleaning process based on structural similarity for predicting protein-ligand binding sites. From the whole scPDB (an annotated database of druggable binding sites extracted from the Protein DataBank) database, 5020 protein structures were selected to address this problem, which were used to train PUResNet. With this, we achieved better and justifiable performance than the existing methods while evaluating two independent sets using distance, volume and proportion metrics.

With an increasing application of Phase Measurement Units in the smart grid, it is becoming inevitable for PMUs to operate in severe conditions, which results in outliers and missing data. However, conventional techniques take excessive time to clean outliers and fill missing data due to lacking support from a big data platform. In this paper, a flexible ensemble algorithm is proposed to implement a precise and scalable data clean by the existing big data platform “Apache Spark.” In the proposed scheme, an ensemble model based on a soft voting approach utilizes principal component analysis in conjunction with the K-means, Gaussian mixture model, and isolation forest technique to detect outliers. The proposed scheme uses a gradient boosting decision tree for each extracted feature of PMUs for the data filling process after detecting outliers. The test results demonstrate that the proposed model achieves high accuracy and recall by comparing simulated and real-world Phase measurement unit data using the local outlier factor algorithm and Density-Based Spatial Clustering of Application with Noise (DBSCAN). The mean absolute error, root mean square error and R2-score criteria are used to validate the proposed method’s data filling results against contemporary techniques such as decision tree and linear regression algorithms.

This paper describes an active learning tool for the generation of Earth Observation (EO) benchmark datasets. This tool is able to generate training datasets, based on its active learning strategy with a classification accuracy of around 90%. Afterwards, a data cleaning tool is needed, in order to correct noisy data and provide a clean dataset to be stored in the benchmark database, and for subsequent benchmark verification. The data cleaning procedure is supported by unsupervised learning, using clustering algorithms to group similar patterns, and dimension reduction algorithms to embed them in lower dimension with annotated labels. Moreover, interactive visualizations are implemented in most modules to help better manipulate datasets and get better understandings.

With the increasing importance and complexity of data pipelines, data quality became one of the key challenges in modern software applications. The importance of data quality has been recognized beyond the field of data engineering and database management systems (DBMSs). Also, for machine learning (ML) applications, high data quality standards are crucial to ensure robust predictive performance and responsible usage of automated decision making. One of the most frequent data quality problems is missing values. Incomplete datasets can break data pipelines and can have a devastating impact on downstream ML applications when not detected. While statisticians and, more recently, ML researchers have introduced a variety of approaches to impute missing values, comprehensive benchmarks comparing classical and modern imputation approaches under fair and realistic conditions are underrepresented. Here, we aim to fill this gap. We conduct a comprehensive suite of experiments on a large number of datasets with heterogeneous data and realistic missingness conditions, comparing both novel deep learning approaches and classical ML imputation methods when either only test or train and test data are affected by missing data. Each imputation method is evaluated regarding the imputation quality and the impact imputation has on a downstream ML task. Our results provide valuable insights into the performance of a variety of imputation methods under realistic conditions. We hope that our results help researchers and engineers to guide their data preprocessing method selection for automated data quality improvement.

Data cleaning is an important first step in most statistical analyses, including efforts to map the genetic loci that contribute to variation in quantitative traits. Here we illustrate approaches to quality control and cleaning of array-based genotyping data for multiparent populations (experimental crosses derived from more than two founder strains), using MegaMUGA array data from a set of 291 Diversity Outbred (DO) mice. Our approach employs data visualizations that can reveal problems at the level of individual mice or with individual SNP markers. We find that the proportion of missing genotypes for each mouse is an effective indicator of sample quality. We use microarray probe intensities for SNPs on the X and Y chromosomes to confirm the sex of each mouse, and we use the proportion of matching SNP genotypes between pairs of mice to detect sample duplicates. We use a hidden Markov model (HMM) reconstruction of the founder haplotype mosaic across each mouse genome to estimate the number of crossovers and to identify potential genotyping errors. To evaluate marker quality, we find that missing data and genotyping error rates are the most effective diagnostics. We also examine the SNP genotype frequencies with markers grouped according to their minor allele frequency in the founder strains. For markers with high apparent error rates, a scatterplot of the allele-specific probe intensities can reveal the underlying cause of incorrect genotype calls. The decision to include or exclude low-quality samples can have a significant impact on the mapping results for a given study. We find that the impact of low-quality markers on a given study is often minimal, but reporting problematic markers can improve the utility of the genotyping array across many studies.

How can one best transform a paper-based publication into a living online resource? This is the theme of a project at The National Museum of Art, Architecture and Design in Norway, supported by the Arts Council Norway. The National Museum aims to create, publish and maintain an authority list of Norwegian artists, architects, designers and craftsmen. The objective is to ease the digitisation process for other museums, scholars and the public in general and contribute to better data quality in Norwegian online collections. The list will in part be based on the Norsk Kunstnerleksikon (Encyclopaedia of Norwegian Artists in English), published in 1982&ndash, 1986 and subsequently digitised in 2013. With the help of other public collections in Norway, the purpose is to make the new resource as complete as possible and available in both human- and machine-readable formats. Although the original paper publication contains biographical texts as well as lists of exhibitions, education, travels, publications and more, the data in the new authority list will be constrained to a set of core biographical data. It will however carry references to online biographical resources such as Norsk Kunstnerleksikon (NKL), Wikidata, Union List of Artist Names (ULAN) and Virtual International Authority File (VIAF). This article discusses the process of defining the scope of and setting constraints for the list, how to enrich and reconcile existing data, as well as strategies to ensure that other institutions contribute both as content publishers and end users. It will also shed light on issues concerning keeping such a resource updated and maintained.

Owing to urbanization, the output of construction waste is increasing yearly. Garbage treatment plays a vital role in urban development and construction. The accuracy and integrity of data are important for the implementation of construction waste treatment. Abnormal detection and incomplete filling occur when traditional cleaning algorithms are used. To improve the cleaning of construction waste data, a data cleaning algorithm based on multi-type construction waste was presented in this study. First, a multi-algorithm constraint model was designed to achieve accurate matching between the cleaning content and cleaning model. Thereafter, a natural language data cleaning model was proposed, and the spatial location data were separated from the general data through the content separation mechanism to effectively frame the area to be cleaned. Finally, a time series data cleaning model was constructed. By integrating “check” and “fill”, large-span and large-capacity time series data cleaning was realized. This algorithm was applied to the data collected by the pilot cities, which had precision and recall rates of 93.87% and 97.90% respectively, compared with the traditional algorithm, ultimately exhibiting a certain progressiveness. The algorithm proposed herein can be applied to urban environmental governance. Furthermore, this algorithm can markedly improve the control ability and work efficiency of construction waste treatment, and reduce the restriction of construction waste on the sustainable development of urban environments.

Battery is one of the most important and costly devices in electric vehicles (EVs). Developing an efficient battery management method is of great significance to enhancing vehicle safety and economy. Recently developed big-data and cloud platform computing technologies bring a bright perspective for efficient utilization and protection of vehicle batteries. However, a reliable data transmission network and a high-quality cloud battery dataset are indispensable to enable this benefit. This paper makes the first effort to systematically solve data quality problems in cloud-based vehicle battery monitoring and management by developing a novel integrated battery data cleaning framework. In the first stage, the outlier samples are detected by analyzing the temporal features in the battery data time series. The outlier data in the dataset can be accurately detected to avoid their impacts on battery monitoring and management. Then, the abnormal samples, including the noise polluted data and missing value, are restored by a novel future fusion data restoring model. The real electric bus operation data collected by a cloud-based battery monitoring and management platform are used to verify the performance of the developed data cleaning method. More than 93.3% of outlier samples can be detected, and the data restoring error can be limited to 2.11%, which validates the effectiveness of the developed methods. The proposed data cleaning method provides an effective data quality assessment tool in cloud-based vehicle battery management, which can further boost the practical application of the vehicle big data platform and Internet of vehicle.

Conference Name: EASE 2021: Evaluation and Assessment in Software Engineering Date of Conference: June 2021 Reviewer selection in modern code review is crucial for effective code reviews. Several techniques exist for recommending reviewers appropriate for a given pull request (PR). Most code reviewer recommendation techniques in the literature build and evaluate their models based on datasets collected from real projects using open-source or industrial practices. The techniques invariably presume that these datasets reliably represent the “ground truth.” In the context of a classification problem, ground truth refers to the objectively correct labels of a class used to build models from a dataset or evaluate a model’s performance. In a project dataset used to build a code reviewer recommendation system, the recommended code reviewer picked for a PR is usually assumed to be the best code reviewer for that PR. However, in practice, the recommended code reviewer may not be the best possible code reviewer, or even a qualified one. Recent code reviewer recommendation studies suggest that the datasets used tend to suffer from systematic labeling bias, making the ground truth unreliable. Therefore, models and recommendation systems built on such datasets may perform poorly in real practice. In this study, we introduce a novel approach to automatically detect and eliminate systematic labeling bias in code reviewer recommendation systems. The bias that we remove results from selecting reviewers that do not ensure a permanently successful fix for a bug-related PR. To demonstrate the effectiveness of our approach, we evaluated it on two open-source project datasets —HIVE and QT Creator— and with five code reviewer recommendation techniques —Profile-Based, RSTrace, Naive Bayes, k-NN, and Decision Tree. Our debiasing approach appears promising since it improved the Mean Reciprocal Rank (MRR) of the evaluated techniques up to 26% in the datasets used.

We demonstrate a system that enables a data-centric approach in understanding data quality. Instead of directly quantifying data quality as traditionally done, it disrupts the quality of the dataset and monitors the deviations in the output of an analytic task at hand. It computes the correlation factor between the disruption and the deviation and uses it as the quality metric. This allows users to understand not only the quality of their dataset but also the effect that present and future quality issues have to the intended analytic tasks. This is a novel data-centric approach aimed at complementing existing solutions. On top of the new information that it provides, and in contrast to existing techniques of data quality, it neither requires knowledge of the clean datasets, nor of the constraints on which the data should comply.

Data quality affects machine learning (ML) model performances, and data scientists spend considerable amount of time on data cleaning before model training. However, to date, there does not exist a rigorous study on how exactly cleaning affects ML -- ML community usually focuses on developing ML algorithms that are robust to some particular noise types of certain distributions, while database (DB) community has been mostly studying the problem of data cleaning alone without considering how data is consumed by downstream ML analytics. We propose a CleanML study that systematically investigates the impact of data cleaning on ML classification tasks. The open-source and extensible CleanML study currently includes 14 real-world datasets with real errors, five common error types, seven different ML models, and multiple cleaning algorithms for each error type (including both commonly used algorithms in practice as well as state-of-the-art solutions in academic literature). We control the randomness in ML experiments using statistical hypothesis testing, and we also control false discovery rate in our experiments using the Benjamini-Yekutieli (BY) procedure. We analyze the results in a systematic way to derive many interesting and nontrivial observations. We also put forward multiple research directions for researchers., published in ICDE 2021

BackgroundAutomated dietary assessment tools such as ASA24® are useful for collecting 24-hour recall data in large-scale studies. Modifications made during manual data cleaning may affect nutrient intakes.ObjectivesWe evaluated the effects of modifications made during manual data cleaning on nutrient intakes of interest: energy, carbohydrate, total fat, protein, and fiber.MethodsDifferences in mean intake before and after data cleaning modifications for all recalls and average intakes per subject were analyzed by paired t-tests. The Chi-squared test was used to determine whether unsupervised recalls had more open-ended text responses that required modification than supervised recalls. We characterized food types of text response modifications. Correlations between predictive energy requirements, measured total energy expenditure (TEE), and mean energy intake from raw and modified data were examined.ResultsAfter excluding 11 recalls with invalidating technical errors, 1499 valid recalls completed by 393 subjects were included in this analysis. We found significant differences before and after modifications for energy, carbohydrate, total fat, and protein intakes for all recalls (P < 0.05). Limiting to modified recalls, there were significant differences for all nutrients of interest, including fiber (P< 0.02). There was not a significantly greater proportion of text responses requiring modification for home compared with supervised recalls (P=0.271). Predicted energy requirements correlated highly with TEE. There was no significant difference in correlation of mean energy intake with TEE for modified compared with raw data. Mean intake for individual subjects was significantly different for energy, protein, and fat intakes following cleaning modifications (P< 0.001).ConclusionsManual modifications can change mean nutrient intakes for an entire cohort and individuals. However, modifications did not significantly affect the correlation of energy intake with predictive requirements and measured expenditure. Investigators can consider their research question and nutrients of interest when deciding to make cleaning modifications.

Checking data quality against domain knowledge is a common activity that pervades statistical analysis from raw data to output. The R package 'validate' facilitates this task by capturing and applying expert knowledge in the form of validation rules: logical restrictions on variables, records, or data sets that should be satisfied before they are considered valid input for further analysis. In the validate package, validation rules are objects of computation that can be manipulated, investigated, and confronted with data or versions of a data set. The results of a confrontation are then available for further investigation, summarization or visualization. Validation rules can also be endowed with metadata and documentation and they may be stored or retrieved from external sources such as text files or tabular formats. This data validation infrastructure thus allows for systematic, user-defined definition of data quality requirements that can be reused for various versions of a data set or by data correction algorithms that are parameterized by validation rules., Comment: To be published in the Journal of Statistical Software

Incorporating technology into virtual education encourages educational institutions to demand a migration from the current learning management system towards an intelligent virtual educational system, seeking greater benefit by exploiting the data generated by students in their day-to-day activities. Therefore, the design of these intelligent systems must be performed from a new perspective, which will take advantage of the new analytical functions provided by technologies such as artificial intelligence, big data, educational data mining techniques, and web analytics. This paper focuses on primary education in developing countries, showing the design of an intelligent virtual educational system to improve the efficiency of primary education through recommendations based on reliable data. The intelligent system is formed of four subsystems: data warehousing, analytical data processing, monitoring process and recommender system for educational agents. To illustrate this, the paper contains two dashboards that analyze, respectively, the digital resources usage time and an aggregate profile of teachers’ digital skills, in order to infer new activities that improve efficiency. These intelligent virtual educational systems focus the teaching–learning process on new forms of interaction on an educational future oriented to personalized teaching for the students, and new evaluation and teaching processes for each professor.

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Data cleaning is a time-consuming process that depends on the data analysis that users perform. Existing solutions treat data cleaning as a separate offline process that takes place before analysis begins. Applying data cleaning before analysis assumes a priori knowledge of the inconsistencies and the query workload, thereby requiring effort on understanding and cleaning the data that is unnecessary for the analysis. We propose an approach that performs probabilistic repair of denial constraint violations on-demand, driven by the exploratory analysis that users perform. We introduce Daisy, a system that seamlessly integrates data cleaning into the analysis by relaxing query results. Daisy executes analytical query-workloads over dirty data by weaving cleaning operators into the query plan. Our evaluation shows that Daisy adapts to the workload and outperforms traditional offline cleaning on both synthetic and real-world workloads., To appear in SIGMOD 2020 proceedings

The digitising efforts of herbaria aim to increase access to and impact of scientific collections, by making the data digitally accessible to the global community. Digitising the NHMUK’s botanical collection of around 5.1 million specimens is an ongoing process, but the majority of the type collections have already been imaged. The Chinese type collection has also been transcribed; however, during the recent georeferencing process, we realised that much of the data had been transcribed incorrectly, particularly the locality information in which 80% of the collection contained errors. We discovered 154 specimens that were mistakenly filed in China. We corrected the mistakes from the previous transcription and georeferenced the collection which consists of 3,736 records. In this paper, we discuss the problems and errors we encountered during the georeferencing process, detailing why there were mistakes, what made the transcription harder than expected and what could have led to errors. We also give a short description about the Chinese language and its difference from European languages, leading to complex problems for georeferencing. We provide a brief guide on how to georeference a Chinese collection, avoiding errors and making the georeferencing process easier and faster.

Data cleaning is a time-consuming process that depends on the data analysis that users perform. Existing solutions treat data cleaning as a separate offline process that takes place before analysis begins. Applying data cleaning before analysis assumes a priori knowledge of the inconsistencies and the query workload, thereby requiring effort on understanding and cleaning the data that is unnecessary for the analysis., We propose an approach that performs probabilistic repair of functional dependency violations on-demand, driven by the exploratory analysis that users perform. We introduce Daisy, a system that seamlessly integrates data cleaning into the analysis by relaxing query results. Daisy executes analytical query-workloads over dirty data by weaving cleaning operators into the query plan. Our evaluation shows that Daisy adapts to the workload and outperforms traditional offfine cleaning on both synthetic and real-world workloads.

PowerPoint slides to accompany the Galter Health Sciences Library & Learning Center class "10 Simple Rules for Data Cleaning with Excel."