Search

Predicting the severity of crashes has become a significant issue in research on road accidents. Traffic accident severity prediction is essential for protecting vulnerable road users and preventing traffic accidents. For practitioners to identify significant risk variables and set appropriate countermeasures in place, explainability of the forecast is also essential. Most previous research ignores the severity of property loss caused by traffic accidents and cannot differentiate between different levels of fatalities and property loss severity. Additionally, while an understandable structure of deep neural networks (DNN) is significantly lacking in existing works, understanding traditional systems is quite simple. An inability to use structural data when describing forecasting and the many attempts to incorporate neural networks afflict the absence of hidden layers. We propose a Deep Learning (DL) framework for forecasting traffic crash severity to overcome the accident severity prediction. It has three steps to process. Initially, collected input data are cleaned. Data cleaning is performed in a preprocessing step. We conduct experiments on two datasets, A Countrywide (US) Traffic Accident Dataset and UK Road Accident Dataset. The outcomes of the experiments demonstrate that the proposed technique outperformed other approaches and produced the best accuracy. [ABSTRACT FROM AUTHOR]

The combined complexity of wind turbine systems and harsh operating conditions pose significant challenges to the accuracy of operational data in Supervisory Control and Data Acquisition (SCADA) systems. Improving the precision of data cleaning for high proportions of stacked abnormalities remains an urgent problem. This paper deeply analyzes the distribution characteristics of abnormal data and proposes a novel method for abnormal data cleaning based on a classification processing framework. Firstly, the first type of abnormal data is cleaned based on operational criteria; secondly, the quartile method is used to eliminate sparse abnormal data to obtain a clearer boundary line; on this basis, the Random Sample Consensus (RANSAC) algorithm is employed to eliminate stacked abnormal data; finally, the effectiveness of the proposed algorithm in cleaning abnormal data with a high proportion of stacked abnormalities is verified through case studies, and evaluation indicators are introduced through comparative experiments to quantitatively assess the cleaning effect. The research results indicate that the algorithm excels in cleaning effectiveness, efficiency, accuracy, and rationality of data deletion. The cleaning accuracy improvement is particularly significant when dealing with a high proportion of stacked anomaly data, thereby bringing significant value to wind power applications such as wind power prediction, condition assessment, and fault detection. [ABSTRACT FROM AUTHOR]

Premise: The Flora e Funga do Brasil project is the most comprehensive effort to reliably document Brazilian plant and fungal diversity. It involves the collaborative work of hundreds of taxonomists, integrating detailed and standardized morphological descriptions, nomenclatural status, and geographic distribution information of plants, algae, and fungi collected throughout Brazil. Despite the extensive information available, managing the information from the Flora e Funga do Brasil website poses certain challenges. Methods and Results: florabr is an R package developed to facilitate the exploration and geographical analysis of species information derived from the Flora e Funga do Brasil. Unique to florabr is its ability to interact with the latest, or any other version of the dataset, which undergoes weekly updates. I illustrate the practical application of florabr in common tasks in biogeography and conservation studies. Conclusions: florabr is anticipated to be of significant interest to biogeographers, ecologists, curators of biological collections, and taxonomists actively contributing to the Flora e Funga do Brasil. [ABSTRACT FROM AUTHOR]

Context Poor anthropometric data quality affect the prevalence of malnutrition and could harm public policy planning. Objective This systematic review and meta-analysis was designed to identify different methods to evaluate and clean anthropometric data, and to calculate the frequency of implausible values for weight and height obtained from these methodologies. Data Sources Studies about anthropometric data quality and/or anthropometric data cleaning were searched for in the MEDLINE, LILACS, SciELO, Embase, Scopus, Web of Science, and Google Scholar databases in October 2020 and updated in January 2023. In addition, references of included studies were searched for the identification of potentially eligible studies. Data Extraction Paired researchers selected studies, extracted data, and critically appraised the selected publications. Data Analysis Meta-analysis of the frequency of implausible values and 95% confidence interval (CI) was estimated. Heterogeneity (I 2) and publication bias were examined by meta-regression and funnel plot, respectively. Results In the qualitative synthesis, 123 reports from 104 studies were included, and in the quantitative synthesis, 23 studies of weight and 14 studies of height were included. The study reports were published between 1980 and 2022. The frequency of implausible values for weight was 0.55% (95%CI, 0.29–0.91) and for height was 1.20% (95%CI, 0.44–2.33). Heterogeneity was not affected by the methodological quality score of the studies and publication bias was discarded. Conclusions Height had twice the frequency of implausible values compared with weight. Using a set of indicators of quality to evaluate anthropometric data is better than using indicators singly. Systematic Review Registration PROSPERO registration no. CRD42020208977. [ABSTRACT FROM AUTHOR]

Copyright of Computer Measurement & Control is the property of Magazine Agency of Computer Measurement & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

To address the background-bias problem in computer-aided cytology caused by microscopic slide deterioration, this article proposes a deep learning approach for cell segmentation and background removal without requiring cell annotation. A U-Net-based model was trained to separate cells from the background in an unsupervised manner by leveraging the redundancy of the background and the sparsity of cells in liquid-based cytology (LBC) images. The experimental results demonstrate that the U-Net-based model trained on a small set of cytology images can exclude background features and accurately segment cells. This capability is beneficial for debiasing in the detection and classification of the cells of interest in oral LBC. Slide deterioration can significantly affect deep learning-based cell classification. Our proposed method effectively removes background features at no cost of cell annotation, thereby enabling accurate cytological diagnosis through the deep learning of microscopic slide images. [ABSTRACT FROM AUTHOR]

While many instructors are aware of the Literary Digest 1936 poll as an example of biased sampling methods, this article details potential further explorations for the Digest's 1924–1936 quadrennial U.S. presidential election polls. Potential activities range from lessons in data acquisition, cleaning, and validation, to basic data literacy and visualization skills, to exploring one or more methods of adjustment to account for bias based on information collected at that time. Students can also compare how those methods would have performed. One option could be to give introductory students a first look at the idea of "sampling adjustment" and how this principle can be used to account for difficulties in modern polling, but the context is rich in other opportunities that can be discussed at various times in the course or in more advanced sampling courses. Supplementary materials for this article are available online. [ABSTRACT FROM AUTHOR]

In legacy industrial systems, discovering joinable information between database tables is important for applications such as data integration and data analysis. Locality-Sensitive Hashing-based methods have been proven to be capable of handling chaotic and diverse table relationships, but these methods often rely on an incorrect assumption—that the similarity of table columns in the database directly reflects their joinability, causing problems related to the accuracy of their results. To solve this problem, this study proposes a dynamic data-driven time-slicing Locality-Sensitive Hashing method for joinable table discovery. This method introduces database log information and within different time slices, uses the co-occurrence matrix of data tables to determine their joinability. Specifically, it first performs a MinHash dimensionality reduction on database columns and then uses Locality-Sensitive Hashing to calculate the static similarity. Next, it identifies business modular time slices through database logs, calculates the dynamic similarity of the slice time data, and builds a co-occurrence matrix between tables. Finally, the joinability between data tables is calculated using the static similarity, dynamic similarity, and co-occurrence matrix. The experimental results demonstrate that this method effectively excludes tables that only have similarity but no business relationship for data cleaning, and its accuracy exceeds that of methods that only depend on similarity. [ABSTRACT FROM AUTHOR]

While many instructors are aware of the Literary Digest 1936 poll as an example of biased sampling methods, this article details potential further explorations for the Digest’s 1924–1936 quadrennial U.S. presidential election polls. Potential activities range from lessons in data acquisition, cleaning, and validation, to basic data literacy and visualization skills, to exploring one or more methods of adjustment to account for bias based on information collected at that time. Students can also compare how those methods would have performed. One option could be to give introductory students a first look at the idea of “sampling adjustment” and how this principle can be used to account for difficulties in modern polling, but the context is rich in other opportunities that can be discussed at various times in the course or in more advanced sampling courses. Supplementary materials for this article are available online.

Urban water demand prediction is not only the foundation of water resource planning and management, but also an important component of water supply system optimization and scheduling. Therefore, predicting future water demand is of great significance. For univariate time series data, the issue of outliers can be solved through data preprocessing. Then, the data input dimension is increased through feature engineering, and finally, the LightGBM (Light Gradient Boosting Machine) model is used to predict future water demand. The results demonstrate that cubic polynomial interpolation outperforms the Prophet model and the linear method in the context of missing value interpolation tasks. In terms of predicting water demand, the LightGBM model demonstrates excellent forecasting performance and can effectively predict future water demand trends. The evaluation indicators MAPE (mean absolute percentage error) and NSE (Nash–Sutcliffe efficiency coefficient) on the test dataset are 4.28% and 0.94, respectively. These indicators can provide a scientific basis for short-term prediction of water supply enterprises. HIGHLIGHTS Interpolation of raw training data may not necessarily improve the performance of predictive models.; Accurate prediction of univariate data can be achieved through feature engineering and machine learning.;

The Industrial Internet of Things (IIoT) has become a transformative force in various healthcare applications, providing integrated services for daily life. The app healthcare based on the IIoT framework is broadly used to remotely monitor clients health using advanced biomedical sensors with wireless technologies, managing activities such as monitoring blood pressure, heart rate, and vital signs. Despite its widespread use, IIoT in healthcare faces challenges such as security concerns, inefficient work scheduling, and associated costs. To address these issues, this paper proposes and evaluates the Blockchain-Based Deep Reinforcement Learning System for Optimizing Healthcare (BDRL) framework. BDRL aims to enhance security protocols and maximize makespan efficiency in scheduling medical applications. It facilitates the sharing of legitimate and secure data among linked network nodes beyond the initial stages of data validation and assignment. This study presents the design, implementation, and statistical evaluation of BDRL using a new dataset and varying platform resources. The evaluation shows that BDRL is versatile and successfully addresses the security, privacy, and makespan needs of healthcare applications on distributed networks, while also delivering excellent performance. However, the framework utilizes high resources as the size of inserted data increases. [ABSTRACT FROM AUTHOR]

Image preprocessing is a method to transform raw image data into clean image data. The objective of preprocessing is to improve the image data by suppressing undesired distortions. Enhancement of some image features which are relevant for further processing of image and analysis task is also done in preprocessing. Screening and diagnosis of various eye diseases like diabetic retinopathy, Choroidal Neovascularization(CNV), DRUSEN, etc. are possible using digital retinal images. This paper aims to provide a better understanding and knowledge of the computer algorithms used for retinal image preprocessing. In this paper, various image preprocessing techniques are incorporated such as color correction, color space selection, noise reduction, and contrast enhancement on retinal images. Retinal blood vessels are better seen in Green color space instead of Red or Blue color space. Noise reduction through Block matching and 3D(BM3D) techniques show a significant result as compared to Total Variation Filter (TVF) and Bilateral Filter (BLF). Contrast enhancement through Contrast Limited Adaptive Histogram Equalization (CLAHE) outperforms Global Equalization (GE) or Adaptive Histogram Equalization (AHE). Evaluation parameters such as Mean square error, Peak Signal Noise ratio, Structured similarity index measures, and Normalized root mean square error values for BM3D noise filtering are 0.0029, 25.3370, 0.6839 and 0.0998 respectively which shows that BM3D outperforms the others. [ABSTRACT FROM AUTHOR]

Copyright of Revista Brasileira de Engenharia Agricola e Ambiental - Agriambi is the property of Revista Brasileira de Engenharia Agricola e Ambiental and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Mitigating the impact of class imbalance datasets on classifiers poses a challenge to the machine learning community. Conventional classifiers do not perform well as they are habitually biased toward the majority class. Among existing solutions, the synthetic minority oversampling technique (SMOTE) has shown great potential, aiming to improve the dataset rather than the classifier. However, SMOTE still needs improvement because of its equal oversampling to each minority instance. Based on the consensus that instances far from the borderline contribute less to classification, a refined method for oversampling borderline minority instances (OBMI) is proposed in this paper using a two-stage Tomek link-finding procedure. In the oversampling stage, the pairs of between-class instances nearest to each other are first found to form Tomek links. Then, these minority instances in Tomek links are extracted as base instances. Finally, new minority instances are generated, each of which is linearly interpolated between a base instance and one minority neighbor of the base instance. To address the overlap caused by oversampling, in the cleaning stage, Tomek links are employed again to remove the borderline instances from both classes. The OBMI is compared with ten baseline methods on 17 benchmark datasets. The results show that it performs better on most of the selected datasets in terms of the F1-score and G-mean. Statistical analysis also indicates its higher-level Friedman ranking. [ABSTRACT FROM AUTHOR]

Aim: Species occurrence data are valuable information that enables one to estimate geographical distributions, characterize niches and their evolution, and guide spatial conservation planning. Rapid increases in species occurrence data stem from increasing digitization and aggregation efforts, and citizen science initiatives. However, persistent quality issues in occurrence data can impact the accuracy of scientific findings, underscoring the importance of filtering erroneous occurrence records in biodiversity analyses. Innovation: We introduce an R package, occTest, that synthesizes a growing open‐source ecosystem of biodiversity cleaning workflows to prepare occurrence data for different modelling applications. It offers a structured set of algorithms to identify potential problems with species occurrence records by employing a hierarchical organization of multiple tests. The workflow has a hierarchical structure organized in testPhases (i.e. cleaning vs. testing) that encompass different testBlocks grouping different testTypes (e.g. environmental outlier detection), which may use different testMethods (e.g. Rosner test, jacknife,etc.). Four different testBlocks characterize potential problems in geographic, environmental, human influence and temporal dimensions. Filtering and plotting functions are incorporated to facilitate the interpretation of tests. We provide examples with different data sources, with default and user‐defined parameters. Compared to other available tools and workflows, occTest offers a comprehensive suite of integrated tests, and allows multiple methods associated with each test to explore consensus among data cleaning methods. It uniquely incorporates both coordinate accuracy analysis and environmental analysis of occurrence records. Furthermore, it provides a hierarchical structure to incorporate future tests yet to be developed. Main conclusions: occTest will help users understand the quality and quantity of data available before the start of data analysis, while also enabling users to filter data using either predefined rules or custom‐built rules. As a result, occTest can better assess each record's appropriateness for its intended application. [ABSTRACT FROM AUTHOR]

The precise prediction of NOx generation concentration in coal‐fired boilers serves as the foundational cornerstone for the judicious optimization and control of selective catalytic reduction denitrification (SCR) systems. Owing to the intricate nature of the denitrification process within SCR, there exists a temporal delay in regulating the ammonia injection rate based on the monitored data of NOx concentration at the SCR inlet. Such delays can give rise to ammonia leakage and subsequent obstruction of the air preheater. In light of this, a predictive model, CEEMDAN‐LSTM‐SA, is proposed as an amalgamation of data decomposition and the LSTM (long short‐term memory) fusion self‐attention mechanism within a deep learning network, which is introduced to forecast the NOx emission concentration at the SCR inlet of coal‐fired units. To mitigate the impact of data outliers on the training effectiveness of the model, a clustering method coupled with a statistical testing strategy is initially applied to refine the dataset first. CEEMDAN data decomposition technology is leveraged to facilitate the breakdown of data, alleviating its non‐stationary and intricate characteristics. Subsequently, through spectral analysis, the decomposed components are grouped and aggregated to form novel data elements, which are then subjected to prediction by the constructed LSTM‐SA deep learning network. The ultimate NOx emission concentration prediction value is derived through a process of fusion. Upon scrutinizing and comparing the predictions derived from various models using coal‐fired power plant data, it is evident that the performance metrics of CEEMDAN‐LSTM‐SA predictions exhibit a mean absolute error of 7.425, mean absolute percentage error of 2.415%, root mean square error of 9.715, R‐squared (R2) value of.789, mean absolute relative error of 2.109%, and a Theil's information criterion of.016. In contrast to other models, including traditional self‐attention networks, LSTM, and LSTM‐SA combination networks, CEEMDAN‐LSTM‐SA proposed in this study demonstrates superior prediction accuracy and enhanced generalization capabilities. Consequently, this predictive model stands poised to furnish an efficacious framework for the SCR ammonia injection strategy within thermal power units. [ABSTRACT FROM AUTHOR]

The evolution of information technology combined with artificial intelligence, IoT (Internet of Things) and robotics has made processes integrated and intelligent. The increased use of technology and the need for evidence-based decisions have contributed to the rapid expansion of a large volume of data in recent years. The quality of data generated mainly by humans must be given special attention, as errors can occur more frequently, making the pre-processing phase, such as data cleaning, a determining factor for better results in data analysis. The aim of this article is therefore to analyze data cleaning methods applied in Big Data environments by conducting a systematic review. The review method was based on the Kitchenham protocol, and the search databases were Scopus, Web of Science and CAPES. After searching and selecting the articles according to the protocol, 69 articles were analyzed, revealing the use of a wide variety of techniques, such as machine learning, data mining, natural language processing and others. The review also emphasized the various publication formats and the wide dissemination and discussion of research on data cleaning in Big Data in the academic community. Finally, this study provides the state of the art of data cleansing techniques that have been used in a Big Data context, offering insights and directions for future research. [ABSTRACT FROM AUTHOR]

Based on digital line graphic and true digital orthophoto map, a large amount of sample data that meets the requirements of deep learning can be automatically generated. However, there are often some erroneous information, which increases the difficulty of training neural network models and limits the improvement of ground feature extraction accuracy. A sample data automatic cleaning method based on selection weight iteration was proposed to address this issue. Firstly, a deep neural network model for data cleaning was constructed, and a network training method based on selection weight iteration was proposed. The method broke the assumption that all samples had the same weight for the calculation of loss function during network model training. The prediction accuracy of the samples during the data cleaning network model training process was used as the weight of the samples to be brought into the network training, and the sample weights were continuously updated through iterative training. Finally, samples with low weights were eliminated to achieve automatic data cleaning and sample database refinement. Training and accuracy comparison experiments were conducted on five classic semantic segmentation network models using a sample database before and after data cleaning. The results show that, the model trained using the sample database after data cleaning improves the average accuracy of building extraction by 2.36%, road extraction by 3.48%, and water extraction by 1.88%. This experiment proves that the data cleaning method proposed in this paper can effectively improve the accuracy of the network model in extracting ground features. [ABSTRACT FROM AUTHOR]

Using constrained factor mixture models (FMM) for careless response identification is still in its infancy. Existing models have overly restrictive statistical assumptions that do not identify all types of careless respondents. The current paper presents a novel constrained FMM model with more reasonable assumptions that capture both longstring and random careless respondents. We provide a comprehensive comparison of the statistical assumptions between the proposed model and two previous constrained models. The proposed model was evaluated using both real data (N = 1,455) and statistical simulation. The results showed that the model had a superior fit, stronger convergent validity with other indicators of careless responding, more accurate parameter recovery and more accurate identification of careless respondents when compared to its predecessors. The proposed model does not require additional data collection effort, and thus researchers can routinely use it to control careless responses. We provide user-friendly syntax with detailed explanations online to facilitate its use. [ABSTRACT FROM AUTHOR]

Nowadays, ubiquitous technology makes life easier, especially devices that use the internet (IoT). IoT devices have been used to generate data in various domains, including healthcare, industry, and education. However, there are often problems with this generated data such as missing values, duplication, and data errors, which can significantly affect data analysis results and lead to inaccurate decision making. Enhancing the quality of real-time data streams has become a challenging task as it is crucial for better decisions. In this paper, we propose a framework to improve the quality of a real-time data stream by considering different aspects, including context-awareness. The proposed framework tackles several issues in the data stream, including duplicated data, missing values, and outliers to improve data quality. The proposed framework also provides recommendations on appropriate data cleaning techniques to the user to help improve data quality in real time. Also, the data quality assessment is included in the proposed framework to provide insight to the user about the data stream quality for better decisions. We present a prototype to examine the concept of the proposed framework. We use a dataset that is collected in healthcare and process these data using a case study. The effectiveness of the proposed framework is verified by the ability to detect and repair stream data quality issues in selected context and to provide a recommended context and data cleaning techniques to the expert for better decision making in providing healthcare advice to the patient. We evaluate our proposed framework by comparing the proposed framework against previous works. [ABSTRACT FROM AUTHOR]

A common endocrine issue affecting millions globally is thyroid illness. For this ailment to be effectively treated and managed, an early and accurate diagnosis is essential. Machine learning algorithms have attracted a lot of attention recently in the healthcare industry and have the potential to improve thyroid disease diagnosis and categorization. The implementation of machine learning methods for the classification of thyroid disease is presented in this study. To create predictive models, the study makes use of a dataset that includes a variety of thyroid-related factors, including age, gender, and hormone levels. To evaluate the effectiveness of several machine learning techniques in classifying thyroid diseases, including random forest, support vector machines, XG-Boost, and ensemble classifier, they are implemented and compared. To ensure robust model performance, the methodology includes data preparation, feature selection, and model training, as well as strategies for hyperparameter adjustment and cross-validation. To assess the algorithms’ efficiency in differentiating between several thyroid illness classifications, such as hyperthyroidism, hypothyroidism, and the study measures the algorithms’ accuracy, precision, recall, F1-score, voting, and area under the ROC curve.Highlights: Machine Learning models has been proven important tool for disease diagnosis and classification. Here, in this research various machine learning models are implemented for thyroid disease classification. An analysis between various machine learning models is done to choose the best model for classification. [ABSTRACT FROM AUTHOR]

Abstract Premise The Flora e Funga do Brasil project is the most comprehensive effort to reliably document Brazilian plant and fungal diversity. It involves the collaborative work of hundreds of taxonomists, integrating detailed and standardized morphological descriptions, nomenclatural status, and geographic distribution information of plants, algae, and fungi collected throughout Brazil. Despite the extensive information available, managing the information from the Flora e Funga do Brasil website poses certain challenges. Methods and Results florabr is an R package developed to facilitate the exploration and geographical analysis of species information derived from the Flora e Funga do Brasil. Unique to florabr is its ability to interact with the latest, or any other version of the dataset, which undergoes weekly updates. I illustrate the practical application of florabr in common tasks in biogeography and conservation studies. Conclusions florabr is anticipated to be of significant interest to biogeographers, ecologists, curators of biological collections, and taxonomists actively contributing to the Flora e Funga do Brasil.

Abstract Background Understanding biodiversity patterns is a central topic in biogeography and ecology, and it is essential for conservation planning and policy development. Diversity estimates that consider the evolutionary relationships among species, such as phylogenetic diversity and phylogenetic endemicity indices, provide valuable insights into the functional diversity and evolutionary uniqueness of biological communities. These estimates are crucial for informed decision-making and effective global biodiversity management. However, the current methodologies used to generate these metrics encounter challenges in terms of efficiency, accuracy, and data integration. Results We introduce PhyloNext, a flexible and data-intensive computational pipeline designed for phylogenetic diversity and endemicity analysis. The pipeline integrates GBIF occurrence data and OpenTree phylogenies with the Biodiverse software. PhyloNext is free, open-source, and provided as Docker and Singularity containers for effortless setup. To enhance user accessibility, a user-friendly, web-based graphical user interface has been developed, facilitating easy and efficient navigation for exploring and executing the pipeline. PhyloNext streamlines the process of conducting phylogenetic diversity analyses, improving efficiency, accuracy, and reproducibility. The automated workflow allows for periodic reanalysis using updated input data, ensuring that conservation strategies remain relevant and informed by the latest available data. Conclusions PhyloNext provides researchers, conservationists, and policymakers with a powerful tool to facilitate a broader understanding of biodiversity patterns, supporting more effective conservation planning and policy development. This new pipeline simplifies the creation of reproducible and easily updatable phylogenetic diversity analyses. Additionally, it promotes increased interoperability and integration with other biodiversity databases and analytical tools.

Abstract Ecology increasingly relies on a massive volume of biodiversity occurrence records to draw insights into large‐scale biogeographical, ecological and evolutionary phenomena. This often involves defining a set of criteria that guides the collection, filtering and standardising of available records. These curation processes are often neither described in detail nor well documented. This hampers the comparability and reproducibility of studies and thus undermines the robustness of any ecological result. Yet, to date, there is no guide providing a friendly way to reason and document a complete data curation process. We reviewed the available literature on data curation, including tools such as R packages and workflows. From this assessment, we created a complete guide organised into five modules that allows users to consciously select and validate occurrence records based on these curation criteria. This is presented in the user‐friendly Shiny R OCCUR application, available at https://ecoinformatic.shinyapps.io/OCCUR/. OCCUR application provides a guide for researchers to deal with the trade‐off between data certainty and coverage generated in each data curation step. An interactive graph of these changes is provided within each module. OCCUR also produces a custom‐made final report including all steps used for data filtering. This report helps to streamline the writing of the methods section of manuscripts and technical reports, thus promoting the reproducibility of data curation processes. This Shiny application provides an interactive overview of the data curation methods and their use for handling occurrence records from public repositories. It brings together the taxonomic, temporal and spatial dimensions of data and also the identification of duplicates of the records. OCCUR can be applied for multiple purposes, such as teaching R to ecologists or enhancing reproducibility of macroecology and biogeography.

Abstract Mitigating the impact of class imbalance datasets on classifiers poses a challenge to the machine learning community. Conventional classifiers do not perform well as they are habitually biased toward the majority class. Among existing solutions, the synthetic minority oversampling technique (SMOTE) has shown great potential, aiming to improve the dataset rather than the classifier. However, SMOTE still needs improvement because of its equal oversampling to each minority instance. Based on the consensus that instances far from the borderline contribute less to classification, a refined method for oversampling borderline minority instances (OBMI) is proposed in this paper using a two-stage Tomek link-finding procedure. In the oversampling stage, the pairs of between-class instances nearest to each other are first found to form Tomek links. Then, these minority instances in Tomek links are extracted as base instances. Finally, new minority instances are generated, each of which is linearly interpolated between a base instance and one minority neighbor of the base instance. To address the overlap caused by oversampling, in the cleaning stage, Tomek links are employed again to remove the borderline instances from both classes. The OBMI is compared with ten baseline methods on 17 benchmark datasets. The results show that it performs better on most of the selected datasets in terms of the F1-score and G-mean. Statistical analysis also indicates its higher-level Friedman ranking.

Photovoltaic (PV) power generation has attracted widespread interest as a clean and sustainable energy source, with increasing global attention given to renewable energy. However, the operation and monitoring of PV power generation systems often result in large amounts of data containing missing values, outliers, and noise, posing challenges for data analysis and application. Therefore, PV data cleaning plays a crucial role in ensuring data quality, enhancing data availability and reliability. This study proposes a PV data cleaning method based on Rasterized Data Image Processing (RDIP) technology, which integrates rasterization and image processing techniques to select optimal contours and extract essential data. To validate the effectiveness of our method, we conducted comparative experiments using three data cleaning methods, including our RDIP algorithm, the Pearson correlation coefficient interpolation method, and cubic spline interpolation method. Subsequently, the cleaned datasets from these methods were utilized for power prediction using two linear regression models and two neural network models. The experimental results demonstrated that data cleaned using the RDIP algorithm improved the short-term forecast accuracy by approximately 1.0% and 3.7%, respectively, compared to the other two methods, indicating the feasibility and effectiveness of the RDIP approach. However, it is worth noting that the RDIP technique has limitations due to its reliance on integer parameters for grid division, potentially leading to coarse grid divisions. Future research efforts could focus on optimizing the selection of binarization thresholds to achieve better cleaning results and exploring other potential applications of RDIP in PV data analysis. [ABSTRACT FROM AUTHOR]

Persian poetry has profoundly affected all periods of Persian literature and the literature of other countries as well. It is a fundamental vehicle for expressing Persian culture and political opinion. This paper presents a corpus of Persian literary text mainly focusing on poetry, covering the ninth to twenty-first century annotated for century and style, with additional partial annotation of rhetorical figures. Our resource is the largest and the most diverse corpus available in Persian literary text, with a particularly broad temporal scope. This allows us to conduct several computational experiments to analyze poetic styles, authors and time periods, as well as context shifts over time, for which we rely both on supervised models and on Persian poetry-specific heuristics. The corpus, the tools, and experiments described in this paper can be used not only for digital humanities studies of Persian literature but also for processing Persian texts in general, as well as in other broader cross-linguistic applications. [ABSTRACT FROM AUTHOR]

The information contained in museum collections catalog databases is an invaluable resource for understanding and caring for collection items. The identification and grouping of items in a collection based on criteria such as location of manufacture, use, or materials composition allows for items at greater risk for long term preservation issues to be more appropriately monitored and cared for. However, identifying these items can be difficult, particularly when the collections data or terminology included in collections catalogs and databases vary significantly between items or across collections. This paper discusses the process and results of a data consolidation and cleaning campaign that was undertaken by the Smithsonian’s Glass Deterioration Working Group. This work was done in order to develop a consistent database of glass and glass-containing collection items that is able to be effectively queried for items of particular concern or interest from a preservation standpoint. [ABSTRACT FROM AUTHOR]

Plant trait data are used to quantify how plants respond to environmental factors and can act as indicators of ecosystem function. Measured trait values are influenced by genetics, trade‐offs, competition, environmental conditions, and phenology. These interacting effects on traits are poorly characterized across taxa, and for many traits, measurement protocols are not standardized. As a result, ancillary information about growth and measurement conditions can be highly variable, requiring a flexible data structure. In 2007, the TRY initiative was founded as an integrated database of plant trait data, including ancillary attributes relevant to understanding and interpreting the trait values. The TRY database now integrates around 700 original and collective datasets and has become a central resource of plant trait data. These data are provided in a generic long‐table format, where a unique identifier links different trait records and ancillary data measured on the same entity. Due to the high number of trait records, plant taxa, and types of traits and ancillary data released from the TRY database, data preprocessing is necessary but not straightforward. Here, we present the 'rtry' R package, specifically designed to support plant trait data exploration and filtering. By integrating a subset of existing R functions essential for preprocessing, 'rtry' avoids the need for users to navigate the extensive R ecosystem and provides the functions under a consistent syntax. 'rtry' is therefore easy to use even for beginners in R. Notably, 'rtry' does not support data retrieval or analysis; rather, it focuses on the preprocessing tasks to optimize data quality. While 'rtry' primarily targets TRY data, its utility extends to data from other sources, such as the National Ecological Observatory Network (NEON). The 'rtry' package is available on the Comprehensive R Archive Network (CRAN; https://cran.r‐project.org/package=rtry) and the GitHub Wiki (https://github.com/MPI‐BGC‐Functional‐Biogeography/rtry/wiki) along with comprehensive documentation and vignettes describing detailed data preprocessing workflows. [ABSTRACT FROM AUTHOR]

Ecology increasingly relies on a massive volume of biodiversity occurrence records to draw insights into large‐scale biogeographical, ecological and evolutionary phenomena. This often involves defining a set of criteria that guides the collection, filtering and standardising of available records. These curation processes are often neither described in detail nor well documented. This hampers the comparability and reproducibility of studies and thus undermines the robustness of any ecological result. Yet, to date, there is no guide providing a friendly way to reason and document a complete data curation process.We reviewed the available literature on data curation, including tools such as R packages and workflows. From this assessment, we created a complete guide organised into five modules that allows users to consciously select and validate occurrence records based on these curation criteria. This is presented in the user‐friendly Shiny R OCCUR application, available at https://ecoinformatic.shinyapps.io/OCCUR/.OCCUR application provides a guide for researchers to deal with the trade‐off between data certainty and coverage generated in each data curation step. An interactive graph of these changes is provided within each module. OCCUR also produces a custom‐made final report including all steps used for data filtering. This report helps to streamline the writing of the methods section of manuscripts and technical reports, thus promoting the reproducibility of data curation processes.This Shiny application provides an interactive overview of the data curation methods and their use for handling occurrence records from public repositories. It brings together the taxonomic, temporal and spatial dimensions of data and also the identification of duplicates of the records. OCCUR can be applied for multiple purposes, such as teaching R to ecologists or enhancing reproducibility of macroecology and biogeography. [ABSTRACT FROM AUTHOR]

It is common for some participants in self-report surveys to be careless, inattentive, or lacking in effort. Data quality can be severely compromised by responses that are not based on item content (non-content-based [nCB] responses), leading to strong biases in the results of data analysis and misinterpretation of individual scores. In this study, we propose a specification of factor mixture analysis (FMA) to detect nCB responses. We investigated the usefulness and effectiveness of the FMA model in detecting nCB responses using both simulated data (Study 1) and real data (Study 2). In the first study, FMA showed reasonably robust sensitivity (.60 to.86) and excellent specificity (.96 to.99) on mixed-worded scales, suggesting that FMA had superior properties as a screening tool under different sample conditions. However, FMA performance was poor on scales composed of only positive items because of the difficulty in distinguishing acquiescent patterns from valid responses representing high levels of the trait. In Study 2 (real data), FMA detected a minority of cases (6.5%) with highly anomalous response patterns. Removing these cases resulted in a large increase in the fit of the unidimensional model and a substantial reduction in spurious multidimensionality. [ABSTRACT FROM AUTHOR]

As high-throughput methods become more common, training undergraduates to analyze data must include having them generate informative summaries of large datasets. This flexible case study provides an opportunity for undergraduate students to become familiar with the capabilities of R programming in the context of high-throughput evolutionary data collected using macroarrays. The story line introduces a recent graduate hired at a biotech firm and tasked with analysis and visualization of changes in gene expression from 20,000 generations of the Lenski Lab's Long-Term Evolution Experiment (LTEE). Our main character is not familiar with R and is guided by a coworker to learn about this platform. Initially this involves a step-by-step analysis of the small Iris dataset built into R which includes sepal and petal length of three species of irises. Practice calculating summary statistics and correlations, and making histograms and scatter plots, prepares the protagonist to perform similar analyses with the LTEE dataset. In the LTEE module, students analyze gene expression data from the long-term evolutionary experiments, developing their skills in manipulating and interpreting large scientific datasets through visualizations and statistical analysis. Prerequisite knowledge is basic statistics, the Central Dogma, and basic evolutionary principles. The Iris module provides hands-on experience using R programming to explore and visualize a simple dataset; it can be used independently as an introduction to R for biological data or skipped if students already have some experience with R. Both modules emphasize understanding the utility of R, rather than creation of original code. Pilot testing showed the case study was well-received by students and faculty, who described it as a clear introduction to R and appreciated the value of R for visualizing and analyzing large datasets. [ABSTRACT FROM AUTHOR]