Search

Performance degradation assessment methods for rolling bearings under vibration signal monitoring typically involve extracting signal degradation features and inputting them directly into unoptimized assessment models. However, this method often fails to characterize degradation trends and detect early faults in bearings effectively. Moreover, it is susceptible to interference from outliers and false signal fluctuations, posing challenges for accurate performance degradation assessment. To solve the above problems, a novel performance degradation assessment method of rolling bearings based on optimized variational mode decomposition to construct the maximum information degradation feature set and improved hippopotamus optimization algorithm optimized fuzzy support vector data description is proposed. This method effectively suppresses the signal degradation feature pollution caused by the redundancy of irrelevant frequency components, realizes the screening of effective degradation feature sets, and further improves the accuracy of bearing performance degradation assessment. Through experimental verification, this research method uses early healthy rolling bearing samples to establish an assessment model, which can adaptively determine the starting point and degradation trend of bearing degradation. In comparison with other methods for performance degradation assessment under vibration monitoring, it offers distinct advantages. [ABSTRACT FROM AUTHOR]

Research has substantiated that the presence of outliers in data usually introduces additional errors and biases, which typically leads to a degradation in the precision of volatility forecasts. However, correcting outliers can mitigate these adverse effects. This study corrects the additive outliers through a weighting method and let these corrected values to replace the original outliers. Then, the model parameters are re-estimated based on this new return series. This approach reduces the extent to which outliers distort volatility estimates, allowing the model to better adapt to market conditions and improving the accuracy of volatility forecasts. This study introduces this approach for the first time to generalized autoregressive conditional heteroskedasticity mixed data sampling (GARCH-MIDAS) models, so as to establish an additional outliers corrected GARCH-MIDAS model (AO-GARCH-MIDAS). This pioneering approach marks a unique innovation. The research employs a diverse array of evaluation methods to validate the model's robustness and consistently demonstrates its dependable performance. Findings unequivocally reveal the substantial influence of outliers on the model's predictive capacity, with the AO-GARCH-MIDAS model exhibiting consistent superiority across all evaluation criteria. Additionally, while the GARCH model showcases stronger estimation capabilities compared to the GARCH-MIDAS model, the latter demonstrates heightened predictive prowess. Notably, regarding variable selection, the results underscore the greater predictive informational value inherent in realized volatility over other low-frequency factors. [ABSTRACT FROM AUTHOR]

This paper (i) explores the internal structure of two quantum mechanics datasets (QM7b, QM9), composed of several thousands of organic molecules and described in terms of electronic properties, and (ii) further explores an inverse design approach to molecular design consisting of using machine learning methods to approximate the atomic composition of molecules, using QM9 data. Understanding the structure and characteristics of this kind of data is important when predicting the atomic composition from physical‐chemical properties in inverse molecular designs. Intrinsic dimension analysis, clustering, and outlier detection methods were used in the study. They revealed that for both datasets the intrinsic dimensionality is several times smaller than the descriptive dimensions. The QM7b data is composed of well‐defined clusters related to atomic composition. The QM9 data consists of an outer region predominantly composed of outliers, and an inner, core region that concentrates clustered inliner objects. A significant relationship exists between the number of atoms in the molecule and its outlier/inliner nature. The spatial structure exhibits a relationship with molecular weight. Despite the structural differences between the two datasets, the predictability of variables of interest for inverse molecular design is high. This is exemplified by models estimating the number of atoms of the molecule from both the original properties and from lower dimensional embedding spaces. In the generative approach the input is given by a set of desired properties of the molecule and the output is an approximation of the atomic composition in terms of its constituent chemical elements. This could serve as the starting region for further search in the huge space determined by the set of possible chemical compounds. The quantum mechanic's dataset QM9 is used in the study, composed of 133,885 small organic molecules and 19 electronic properties. Different multi‐target regression approaches were considered for predicting the atomic composition from the properties, including feature engineering techniques in an auto‐machine learning framework. High‐quality models were found that predict the atomic composition of the molecules from their electronic properties, as well as from a subset of only 52.6% size. Feature selection worked better than feature generation. The results validate the generative approach to inverse molecular design. [ABSTRACT FROM AUTHOR]

Spike‐and‐slab and horseshoe regressions are arguably the most popular Bayesian variable selection approaches for linear regression models. However, their performance can deteriorate if outliers and heteroskedasticity are present in the data, which are common features in many real‐world statistics and machine learning applications. This work proposes a Bayesian nonparametric approach to linear regression that performs variable selection while accounting for outliers and heteroskedasticity. Our proposed model is an instance of a Dirichlet process scale mixture model with the advantage that we can derive the full conditional distributions of all parameters in closed‐form, hence producing an efficient Gibbs sampler for posterior inference. Moreover, we present how to extend the model to account for heavy‐tailed response variables. The model's performance is tested against competing algorithms on synthetic and real‐world datasets. [ABSTRACT FROM AUTHOR]

Exploratory data analysis (EDA) for functional data—data objects where observations are entire functions—is a difficult problem that has seen significant attention in recent literature. This surge in interest is motivated by the ubiquitous nature of functional data, which are prevalent in applications across fields such as meteorology, biology, medicine and engineering. Empirical probability density functions (PDFs) can be viewed as constrained functional data objects that must integrate to one and be nonnegative. They show up in contexts such as yearly income distributions, zooplankton size structure in oceanography and in connectivity patterns in the brain, among others. While PDF data are certainly common in modern research, little attention has been given to EDA specifically for PDFs. In this paper, we extend several methods for EDA on functional data for PDFs and compare them on simulated data that exhibit different types of variation, designed to mimic that seen in real‐world applications. We then use our new methods to perform EDA on the breakthrough curves observed in gas transport simulations for underground fracture networks. [ABSTRACT FROM AUTHOR]

This document discusses the emergence of online-to-offline logistics services, such as attended home delivery, bike sharing, and ride-hailing, which have become integral to modern on-demand lifestyles. The management of these services requires considering the behavior of customers and drivers in decision-making processes. The document highlights a special issue of OR Spectrum that focuses on data-driven operations research methods for online-to-offline logistics, covering topics such as parcel delivery management, attended home delivery, same-day delivery, vehicle sharing systems, and airline revenue management. The papers in the special issue explore various approaches, including predictive and prescriptive analytics, clustering and forecasting, and simulation, to address the challenges of managing these services effectively. [Extracted from the article]

This paper presents an automated approach for providing ranked lists of outliers in observed demand to support analysts in network revenue management. Such network revenue management, e.g. for railway itineraries, needs accurate demand forecasts. However, demand outliers across or in parts of a network complicate accurate demand forecasting, and the network structure makes such demand outliers hard to detect. We propose a two-step approach combining clustering with functional outlier detection to identify outlying demand from network bookings observed on the leg level. The first step clusters legs to appropriately partition and pools booking patterns. The second step identifies outliers within each cluster and uses a novel aggregation method across legs to create a ranked alert list of affected instances. Our method outperforms analyses that consider leg data without regard for network implications and offers a computationally efficient alternative to storing and analysing all data on the itinerary level, especially in highly-connected networks where most customers book multi-leg products. A simulation study demonstrates the robustness of the approach and quantifies the potential revenue benefits from adjusting demand forecasts for offer optimisation. Finally, we illustrate the applicability based on empirical data obtained from Deutsche Bahn. [ABSTRACT FROM AUTHOR]

Air quality prediction technology can provide effective technical means for environmental governance. In recent years, due to the strong nonlinearity of data, there has been extensive research on data analysis and preprocessing techniques. This paper aims to comprehensively summarize and analyze the methods used in air quality forecasting, specifically focusing on four categories: data decomposition, dimensionality reduction, data correction, and spatial interpolation. Each method's purpose, characteristics, improvements, and implementation details are described in detail. The evaluation of data preprocessing methods is based on popularity, accuracy improvements, time consumption, maturity, and implementation difficulty. Among the existing methods, data decomposition and feature selection are commonly used and well-developed. However, outlier detection and spatial interpolation have limited applications and require further research. Furthermore, this paper discusses current challenges in applying these methods and future development trends, providing a valuable reference for future research. [ABSTRACT FROM AUTHOR]

Monitoring snow depth variations aligned with the seasonal cycle is crucial for studies on climate change and its impacts due to global warming. Especially in terms of snow hydrology, the determination and continuous monitoring of snow depth to determine snow water equivalent is a priority in climate, water science, drought, flood, and inundation prediction studies, particularly in the water cycle. Recently, the so-called Global Navigation Satellite Systems – Interferometric Reflectometry (GNSS-IR) method enables to extract environmental radiometric and geometric characteristics of surface where the signals transmitted from the satellites reflect. The reflected signal follows an extra path than the direct one and causes a major error source for accurate point positioning but a useful tool for sensing the environment. However, outliers remain a challenge in long-term GNSS-IR estimations. In this study, we proposed a median-based robust outlier detection (ROD) approach for identifying outliers in long-term GNSS-IR snow depth estimations. To validate our approach, we analysed 5-year GNSS L1 SNR and L2 SNR data from 1 January 2015 to 31 December 2019 provided by AB33 and AB39 GNSS stations in Alaska, U.S.A. using the GNSS-IR method. We validated GNSS-IR estimations using snow depth measurements from the Coldfoot and Fort Yukon stations in the SNOTEL network. Applying ROD to long-term snow depth estimates increased the highest correlation from 91.58% to 94.93%, and reduced the lowest RMSE from 8.6 cm to 6.7 cm. In addition, the improvement rates calculated to assess the contribution of ROD to the results showed improvements of up to 6.9% in correlation and 26.1% in RMSE. Overall, the results demonstrate that ROD can be effectively used to detect outliers in long-term GNSS-IR snow depth time series. [ABSTRACT FROM AUTHOR]

Mass spectrometry (MS) is a powerful analytical technique in metabolomics, enabling the identification and quantification of metabolites. However, analyzing MS data poses challenges such as batch effects, outliers, and high-dimensional data. In this paper, we propose a comprehensive framework for MS data analysis. The framework integrates data calibration, outlier detection, and automatic classification modules. Data calibration is performed using a deep autoencoder to remove batch effects. Outlier detection combines multiple algorithms through ensemble learning to identify and remove outliers. Automatic classification utilizes a transformer model to handle high-dimensional data and capture global feature relationships. Experimental results on myocardial infarction (MI) and coronary heart disease (CHD) datasets demonstrate the effectiveness of the framework. It outperforms traditional classification models and achieves higher accuracy. The proposed framework provides a robust solution for MS data analysis, facilitating more accurate classification and enabling reliable biological insights in metabolomics research. [Display omitted] • Novel framework integrates three steps for MS data analysis. • Deep AE removes batch effects, while an ensemble approach removes outliers. • Transformer captures intrinsic relationships in MS data for accurate classification. • The framework provides a comprehensive solution to improve classification accuracy. • The framework also facilitates reliable biological insights. [ABSTRACT FROM AUTHOR]

Statistical depth, a commonly used analytic tool in nonparametric statistics, has been extensively studied for multivariate and functional observations over the past few decades. Although various forms of depth were introduced, they are mainly procedure based whose definitions are independent of the generative model for observations. To address this problem, we introduce a generative model-based approach to define statistical depth for both multivariate and functional data. The proposed model-based depth framework permits simple computation via a bootstrap sampling and improves the depth estimation accuracy. When applied to functional data, the proposed depth can capture important features such as continuity, smoothness or phase variability, depending on the defining criteria. We propose efficient algorithms to compute the proposed depths and establish estimation consistency. Through simulations and real data, we demonstrate that the proposed functional depths reveal important statistical information such as those captured by the median and quantiles, and detect outliers. [ABSTRACT FROM AUTHOR]

Clustering streaming data is challenging due to many temporal dynamics, such as concept drift, concept evolution, and feature evolution. Concept evolution is the most challenging of these. Due to concept evolution, new classes may emerge or existing classes may disappear, so it is crucial to process streaming data continuously. This paper proposes a novel online clustering method, specifically for streaming data with concept evolution. It consists of three phases: initialization, clustering and outlier handling. To identify recurrences of previous data in streaming data, it is critical to preserve the sequential properties of data chunks. In the proposed model, representatives from previous windows are added to the current window, making it distinct from existing models. The detection and handling of outliers are very challenging tasks in streaming data analysis. Outliers are often the first instances of a new cluster. The proposed model stores the outliers from each data window. When the number of outliers exceeds a certain threshold, the representatives of outliers are added to the next window to identify new classes. The lack of data sets made it necessary for us to create a synthetic data set with 22020 data instances and test the model on both synthetic and real datasets. Using Silhouette Coefficient, Calinski–Harabasz index, and Davies–Bouldin index analysis, this model yielded the most favourable results. [ABSTRACT FROM AUTHOR]

Outlier detection is an important aspect of statistical process monitoring (SPM) because outliers affect the performance of control charts. SPM researchers study the negative impact of outliers on control charts for monitoring location parameters. However, there is little research on outlier detection in multivariate charts for monitoring process dispersion. This study aims to investigate the impact of outliers in multivariate control charts for monitoring covariance matrix of a process, and then to recommend techniques for detecting potential outliers present from Phase I samples. We propose a new multivariate dispersion chart that employs the determinant of logarithm of estimated covariance matrix as the monitoring statistic. Through Monte Carlo simulations, the results show how outliers from the first phase affect the overall performance of multivariate charts. The results also demonstrate that the minimum volume ellipsoid (MVE) estimator is effective in reducing the effect of outliers on the proposed control scheme than the other compared estimators. [ABSTRACT FROM AUTHOR]

AbstractWe propose the Minimum Regularized Covariance Trace (MRCT) estimator, a novel method for robust covariance estimation and functional outlier detection designed primarily for dense functional data. The MRCT estimator employs a subset-based approach that prioritizes subsets exhibiting greater centrality based on the generalization of the Mahalanobis distance, resulting in a fast-MCD type algorithm. Notably, the MRCT estimator handles high-dimensional datasets without the need for preprocessing or dimension reduction techniques, due to the internal smoothening whose amount is determined by the regularization parameter α>0. The selection of α is automated. An extensive simulation study demonstrates the efficacy of the MRCT estimator in terms of robust covariance estimation and automated outlier detection, emphasizing the balance between noise exclusion and signal preservation achieved through appropriate selection of α. The method converges fast in practice and performs favorably when compared to other functional outlier detection methods. [ABSTRACT FROM AUTHOR]

In recent years, vision systems have become essential in the development of advanced driver assistance systems or autonomous vehicles. Although deep learning methods have been the center of focus in recent years to develop fast and reliable obstacle detection solutions, they face difficulties in complex and unknown environments where objects of varying types and shapes are present. In this study, a novel non-AI approach is presented for finding the ground-line and detecting the obstacles in roads using v-disparity data. The main motivation behind the study is that the ground-line estimation errors cause greater deviations at the output. Hence, a novel ground plane is defined as a region in the v-disparity map by using random variables to minimize these errors. In this new approach, weighted least squares regression, outlier detection, and camera height approximation were utilized for determining the ground region with higher accuracy. KITTY-2 dataset was chosen to conduct validation and evaluation experiments of the proposed approach. The experiment results were presented in GitHub, and the performance comparison shows that the proposed approach provides at least 20% improvement over Hough transform, which is a widely used non-AI algorithm. The results were also compared with a recently published article data and the best outcome was obtained among them for the recall metric. [ABSTRACT FROM AUTHOR]

Smog in China has been a major issue in recent years. As the main component of smog, PM 2.5 has received a lot of attention from the public, as the pollution caused by PM 2.5 is believed to have negative impacts in many aspects. Accordingly, modeling PM 2.5 concentrations has become research of interest for the sake of future prediction and control. Based on the data collected from monitoring stations nationwide, we find that PM 2.5 concentrations show both spatial and temporal dependency. In this paper, we adopt a grouped spatial-temporal model to depict the distribution of PM 2.5 , capturing the spatial heterogeneity in the patterns of PM 2.5 concentrations. An estimation method based on the EM algorithm is used since the group label assignment is unknown, and the results are further explained. Finally, an outlier detection procedure based on the grouped spatial-temporal model is presented. [ABSTRACT FROM AUTHOR]

Quality control in quantitative proteomics is a persistent challenge, particularly in identifying and managing outliers. Unsupervised learning models, which rely on data structure rather than predefined labels, offer potential solutions. However, without clear labels, their effectiveness might be compromised. Single models are susceptible to the randomness of parameters and initialization, which can result in a high rate of false positives. Ensemble models, on the other hand, have shown capabilities in effectively mitigating the impacts of such randomness and assisting in accurately detecting true outliers. Therefore, we introduced SEAOP, a Python toolbox that utilizes an ensemble mechanism by integrating multi-round data management and a statistics-based decision pipeline with multiple models. Specifically, SEAOP uses multi-round resampling to create diverse sub-data spaces and employs outlier detection methods to identify candidate outliers in each space. Candidates are then aggregated as confirmed outliers via a chi-square test, adhering to a 95% confidence level, to ensure the precision of the unsupervised approaches. Additionally, SEAOP introduces a visualization strategy, specifically designed to intuitively and effectively display the distribution of both outlier and non-outlier samples. Optimal hyperparameter models of SEAOP for outlier detection were identified by using a gradient-simulated standard dataset and Mann–Kendall trend test. The performance of the SEAOP toolbox was evaluated using three experimental datasets, confirming its reliability and accuracy in handling quantitative proteomics. [ABSTRACT FROM AUTHOR]

The problem of energy depletion has brought wind energy under consideration to replace oil- or chemical-based energy. However, the breakdown of wind turbines is a major concern. Accordingly, unsupervised learning was performed using the vibration signal of a wind power generator to achieve an outlier detection performance of 97%. We analyzed the vibration data through wavelet packet conversion and identified a specific frequency band that showed a large difference between the normal and abnormal data. To emphasize these specific frequency bands, high-pass filters were applied to maximize the difference. Subsequently, the dimensions of the data were reduced through principal component analysis, giving unique characteristics to the data preprocessing process. Normal data collected from a wind farm located in northern Sweden was first preprocessed and trained using a long short-term memory (LSTM) autoencoder to perform outlier detection. The LSTM Autoencoder is a model specialized for time-series data that learns the patterns of normal data and detects other data as outliers. Therefore, we propose a method for outlier detection through data preprocessing and unsupervised learning, utilizing the vibration signals from wind generators. This will facilitate the quick and accurate detection of wind power generator failures and provide alternatives to the problem of energy depletion. [ABSTRACT FROM AUTHOR]

State estimation is an essential issue in modern industry, which is used in a wide range of fields. Measurements corrupted by outliers have become an inevitable phenomenon leading to a degradation in the performance of estimators. To mitigate the negative impact of measurement outliers, the development of estimators with outlier suppression capabilities is crucial. In this paper, we propose a simple yet effective outlier suppression technique for discrete-time linear systems in the framework of moving horizon estimation (MHE) combined with the event-triggered mechanism. This is a novel attempt to integrate event-triggered outlier detection and correction mechanisms with the MHE approach. Specifically, we propose two event-based outlier detection methods that can effectively identify measurement outliers. Subsequently, two outlier correction techniques are designed according to the different detection schemes. Lastly, two MHE algorithms are proposed based on the corrected measurements. The proposed algorithms are applied to a target tracking simulation and compared with existing advanced outlier-robust estimators, demonstrating the effectiveness and superiority of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Topography complicates the illumination distribution over rugged terrains and hinders the applications of surface reflectance data over mountainous areas. Topographic correction is an essential process to remove the topographic effects in surface reflectance data. This study proposed a Physical model and image Simulation-based topographic Correction method (PSC) for atmospherically corrected surface reflectance images. In contrast with traditional methods, the proposed approach explicitly estimates the illumination distribution over rugged terrains based on image information and then corrects the topographic effects following physical laws. The proposed method was validated and compared with existing well-performed topographic correction methods, including path length correction (PLC), sun-canopy-sensor correction with c factor (SCSC), C correction (CC), and statistical empirical correction (SE). Simulated and satellite data obtained at different times and geolocations are used for evaluation and comparison. The results demonstrated that most existing methods face challenges in removing the biases induced by topography in surface reflectance images. The PLC method failed to obtain reasonable results for faint illumination conditions when the sun zenith angle is high. SE showed relatively poor performance in terms of physical consistency and outlier percentage. Besides, all the traditional methods failed in the correction of cast shadow regions. Comparably, our method consistently demonstrated superior performance in physical consistency, cast shadow correction, and outlier percentage across various regions, encompassing different sun zenith angles and illumination conditions. The proposed method offers an effective and physically consistent approach for the topographic correction of surface reflectance images, facilitating multi-temporal applications over mountainous areas. [ABSTRACT FROM AUTHOR]

Characterizing the processes underlying reproductive isolation between diverging lineages is central to understanding speciation. Here, we present RIDGE—Reproductive Isolation Detection using Genomic polymorphisms—a tool tailored for quantifying gene flow barrier proportion and identifying the relevant genomic regions. RIDGE relies on an Approximate Bayesian Computation with a model‐averaging approach to accommodate diverse scenarios of lineage divergence. It captures heterogeneity in effective migration rate along the genome while accounting for variation in linked selection and recombination. The barrier detection test relies on numerous summary statistics to compute a Bayes factor, offering a robust statistical framework that facilitates cross‐species comparisons. Simulations revealed RIDGE's efficiency in capturing signals of ongoing migration. Model averaging proved particularly valuable in scenarios of high model uncertainty where no migration or migration homogeneity can be wrongly assumed, typically for recent divergence times <0.1 2Ne generations. Applying RIDGE to four published crow data sets, we first validated our tool by identifying a well‐known large genomic region associated with mate choice patterns. Second, while we identified a significant overlap of outlier loci using RIDGE and traditional genomic scans, our results suggest that a substantial portion of previously identified outliers are likely false positives. Outlier detection relies on allele differentiation, relative measures of divergence and the count of shared polymorphisms and fixed differences. Our analyses also highlight the value of incorporating multiple summary statistics including our newly developed outlier ones that can be useful in challenging detection conditions. [ABSTRACT FROM AUTHOR]

Meta‐analysis is a widely used tool for synthesizing results from multiple studies. The collected studies are deemed heterogeneous when they do not share a common underlying effect size; thus, the factors attributable to the heterogeneity need to be carefully considered. A critical problem in meta‐analyses and systematic reviews is that outlying studies are frequently included, which can lead to invalid conclusions and affect the robustness of decision‐making. Outliers may be caused by several factors such as study selection criteria, low study quality, small‐study effects, and so on. Although outlier detection is well‐studied in the statistical community, limited attention has been paid to meta‐analysis. The conventional outlier detection method in meta‐analysis is based on a leave‐one‐study‐out procedure. However, when calculating a potentially outlying study's deviation, other outliers could substantially impact its result. This article proposes an iterative method to detect potential outliers, which reduces such an impact that could confound the detection. Furthermore, we adopt bagging to provide valid inference for sensitivity analyses of excluding outliers. Based on simulation studies, the proposed iterative method yields smaller bias and heterogeneity after performing a sensitivity analysis to remove the identified outliers. It also provides higher accuracy on outlier detection. Two case studies are used to illustrate the proposed method's real‐world performance. [ABSTRACT FROM AUTHOR]

The aim of this paper is to discuss the effect of the sensor on the acoustic emission (AE) signature and to develop a methodology to reduce the sensor effect. Pencil leads are broken on PMMA plates at different source–sensor distances, and the resulting waves are detected with different sensors. Several transducers, commonly used for acoustic emission measurements, are compared with regard to their ability to reproduce the characteristic shapes of plate waves. Their consequences for AE descriptors are discussed. Their different responses show why similar test specimens and test conditions can yield disparate results. This sensor effect will furthermore make the classification of different AE sources more difficult. In this context, a specific procedure is proposed to reduce the sensor effect and to propose an efficient selection of descriptors for data merging. Principal Component Analysis has demonstrated that using the Z-score normalized descriptor data in conjunction with the Krustal–Wallis test and identifying the outliers can help reduce the sensor effect. This procedure leads to the selection of a common descriptor set with the same distribution for all sensors. These descriptors can be merged to create a library. This result opens up new outlooks for the generalization of acoustic emission signature libraries. This aspect is a key point for the development of a database for machine learning. [ABSTRACT FROM AUTHOR]

Accurate and complete digital elevation models (DEMs) play an important fundamental role in geospatial analysis, supporting various engineering applications, human activities, and scientific research. Interferometric synthetic aperture radar (InSAR) plays an increasingly important role in DEM generation. Nonetheless, owing to its inherent characteristics, gaps often appear in regions marked by significant topographical fluctuations, necessitating an extra void-filling process. Traditional void-filling methods have operated directly on preexisting data, succeeding in relatively flat terrain. When facing mountainous regions, there will always be gross errors in elevation values. Regrettably, conventional methods have often disregarded this vital consideration. To this end, this research proposes a DEM void-filling method based on incorporating elevation outlier detection. It accounts for the detection and removal of elevation outliers, thereby mitigating the shortcomings of existing methods and ensuring robust DEM restoration in mountainous terrains. Experiments were conducted to validate the method applicability using TanDEM-X data from Sichuan, China, Hebei, China, and Oregon, America. The results underscore the superiority of the proposed method. Three traditional methods are selected for comparison. The proposed method has different degrees of improvement in filling accuracy, depending on the void status of the local terrain. Compared with the delta surface fill (DSF) method, the root mean squared error (RMSE) of the filling results has improved by 7.87% to 51.87%. The qualitative and quantitative experiments demonstrate that the proposed method is promising for large-scale DEM void-filling tasks. [ABSTRACT FROM AUTHOR]

Background: Outliers, data points that significantly deviate from the norm, can have a substantial impact on statistical inference and provide valuable insights in data analysis. Multiple methods have been developed for outlier detection, however, almost all available approaches fail to consider the spatial dependence and heterogeneity in spatial data. Spatial data has diverse formats and semantics, requiring specialized outlier detection methodology to handle these unique properties. For now, there is limited research exists on robust spatial outlier detection methods designed specifically under the spatial error model (SEM) structure. Method: We propose the Spatial-Θ-Iterative Procedure for Outlier Detection (Spatial-Θ-IPOD), which utilizes a mean-shift vector to identify outliers within the SEM. Our method enables an effective detection of spatial outliers while also providing robust coefficient estimates. To assess the performance of our approach, we conducted extensive simulations and applied it to a real-world empirical study using life expectancy data from multiple countries. Results: Simulation results showed that the masking and JD (Joint Detection) indicators of our Spatial-Θ-IPOD method outperformed several commonly used methods, even in high-dimensional scenarios, demonstrating stable performance. Conversely, the Θ-IPOD method proved to be ineffective in detecting outliers when spatial correlation was present. Moreover, our model successfully provided reliable coefficient estimation alongside outlier detection. The proposed method consistently outperformed other models (both robust and non-robust) in most cases. In the empirical study, our proposed model successfully detected outliers and provided valuable insights in the modeling process. Conclusions: Our proposed Spatial-Θ-IPOD offers an effective solution for detecting spatial outliers for SEM while providing robust coefficient estimates. Notably, our approach showcases its relative superiority even in the presence of high leverage points. By successfully identifying outliers, our method enhances the overall understanding of the data and provides valuable insights for further analysis. [ABSTRACT FROM AUTHOR]

Diabetes is a chronic disease that affects millions of people worldwide. Accurate and timely diagnosis of diabetes is crucial for its effective treatment and management. While machine learning has shown promise in predicting the disease, missing data, outliers, class imbalance and limitations of classifiers can hinder accuracy. To address these challenges, we propose a novel machine learning approach that combines adaptive iterative imputation (AII) for missing value imputation, dynamic ensemble isolation forest (DE-IF) for outlier detection and removal, Iterated KMeans SMOTEENN (IKMSENN) for class imbalance, and an adaptive extra tree classifier (AETC) for classification. Our approach is evaluated using the Pima Indian Diabetes Dataset (PIDD), a widely used benchmark dataset in diabetes disease prediction. Experimental results show that our approach outperforms several state-of-the-art machine learning models in terms of accuracy, precision, recall, f -measure, and the area under the receiver operating characteristic (ROC) curve (AUC-ROC). Our approach achieved an accuracy of 98.58%, with a precision of 0.986, recall of 0.987, f -measure of 0.985, and ROC of 0.965 on the PIDD dataset. Our research presents a significant contribution to the field of diabetes disease prediction by introducing novel machine learning approaches that address common challenges such as missing data, outliers and class imbalance, as well as limitations of classifiers. Our approach has the potential to greatly improve the accuracy and effectiveness of diabetes disease prediction and has important implications for the diagnosis and management of the disease. This research work focuses on the development of a machine learning approach to improve the accuracy of diabetes diagnosis. Diabetes is a chronic disease that affects millions of people globally, and timely diagnosis is essential for its effective treatment and management. However, traditional diagnosis methods can be limited by missing data, outliers, class imbalance, and classifier limitations. To overcome these challenges, the proposed approach combines Adaptive Iterative Imputation (AII) for missing value imputation, Dynamic Ensemble Isolation Forest (DE-IF) for outlier detection and removal, Iterated KMeans SMOTEENN (IKMSENN) for class imbalance, and an Adaptive Extra Tree classifier (AETC) for classification. The approach is evaluated using the Pima Indian Diabetes Dataset (PIDD), a widely used benchmark dataset in diabetes disease prediction. Overall, this research work provides a novel and effective machine learning approach for accurate diabetes diagnosis. [ABSTRACT FROM AUTHOR]

The Capra operation is a graph theoretical operation that transforms the given graph into a structurally different and larger graph. It is commonly used in statistical analysis, data processing, and scientific research to compare and analyze data. The Capra operation involves subtracting the reference value from each numerical value, taking the absolute value of the resultand raising it to a specified power. This operation is used to create a new set of values that are more sensitive to differences between numerical values and can help to identify outliers and other anomalous data points. The Capra operation has numerous applications in fields such as finance, engineering, and biology, where it is used to analyze data and make predictions about future trends. Its ability to transform data and highlight differences between values makes it a valuable tool for researchers and analysts seeking to gain deeper insights into the patterns and structures within their data. In this paper, we studied the distance-based topological indices of some graphs under the Capra operation. [ABSTRACT FROM AUTHOR]

Determination of the precision of the designed observations in a geodetic network referred as the Second Order Design is an essential element of the network design process. Although the precision requirements are usually of key importance, ensuring an adequate level of reliability, understood as the possibility of outliers detection can be also vital. The subject of this study is the optimization of the observations' precision distribution to get the balanced observation reliability indices. The objective of the work is to test usability of two optimization methods based on optimization algorithms, (simulated annealing and Hooke–Jeeves optimization), to solve the mentioned problem. An analytical method proposed by Amiri-Simkooei was applied as a reference. The performance of the above-mentioned methods was tested on two simulated angular-linear networks. Due to acceptable working time and the possibility of defining the boundary conditions on the final solution, the Hooke–Jeeves method appeared to be the most suitable to solve the analysed problem. [ABSTRACT FROM AUTHOR]

In this article, we implement the classification of nonstationary streaming data. Due to the inability to obtain full data in the context of streaming data, we adopt a strategy based on clustering structure for data classification. Specifically, this strategy involves dynamically maintaining clustering structures to update the model, thereby updating the objective function for classification. Simultaneously, incoming samples are monitored in real‐time to identify the emergence of new classes or the presence of outliers. Moreover, this strategy can also deal with the concept drift problem, where the distribution of data changes with the inflow of data. Regarding the handling of novel instances, we introduce a buffer analysis mechanism to delay their processing, which in turn improves the prediction performance of the model. In the process of model updating, we also introduce a novel renewable strategy for the covariance matrix. Numerical simulations and experiments on datasets show that our method has significant advantages. [ABSTRACT FROM AUTHOR]

The minimum covariance determinant (MCD) estimate is an important subsample method in robust statistics. It is commonly used to estimate parameters of multivariate normal distributions when outliers exist because of its good robustness properties. However, MCD has some disadvantages including low efficiency when there is no outliers and poor performance when there are clustered outliers. This paper first introduces an adaptive subsample method, and shows that the adaptive MCD estimator can possess both full asymptotic efficiency and maximum breakdown value. We then propose a minimum distance subsample estimate to handle the situations where there are clustered outliers. Simulation results indicate that the adaptive version of the minimum distance subsample estimate has satisfactory performance for various types of outliers. [ABSTRACT FROM AUTHOR]

Density based methods are significant approaches in outlier detection for high dimensional datasets and Local correlation integral (LOCI) is one of the best of them. To extend LOCI for fuzzy datasets, we should employ suitable metrics to measure the distance between two fuzzy numbers. Euclidean distance measure is a classic one in metric learning, but to overcome curse of dimensionality, we apply fractional distance metric too. Then, after introducing the FLOCI outlier detection algorithm for identifying the fuzzy outliers, we study the efficiency of the proposed method by doing some numerical experiments, in which the obtained results were completely successfull. We also compared the results with Fuzzy versions of Distance based ABOD and SOD methods to prove robustness of this approache. More than the above, one of the main advantages of the new approach is the determination of outlierness factor for each data which is not presented in classical LOCI method. [ABSTRACT FROM AUTHOR]

Roller bearings are critical components in various mechanical systems, and the timely detection of potential failures is essential for preventing costly downtimes and avoiding substantial machinery breakdown. This research focuses on finding and verifying a robust method that can detect failures early, without creating false positive failure states. Therefore, this paper introduces a novel algorithm for the early detection of roller bearing failures, particularly tailored to high-precision bearings and automotive test bed systems. The featured method (AFI—Advanced Failure Indicator) utilizes the Fast Fourier Transform (FFT) of wideband accelerometers to calculate the spectral content of vibration signals emitted by roller bearings. By calculating the frequency bands and tracking the movement of these bands within the spectra, the method provides an indicator of the machinery's health, mainly focusing on the early stages of bearing failure. The calculated channel can be used as a trend indicator, enabling the method to identify subtle deviations associated with impending failures. The AFI algorithm incorporates a non-static limit through moving average calculations and volatility analysis methods to determine critical changes in the signal. This thresholding mechanism ensures the algorithm's responsiveness to variations in operating conditions and environmental factors, contributing to its robustness in diverse industrial settings. Further refinement was achieved through an outlier detection filter, which reduces false positives and enhances the algorithm's accuracy in identifying genuine deviations from the normal operational state. To benchmark the developed algorithm, it was compared with three industry-standard algorithms: VRMS calculations per ISO 10813-3, Mean Absolute Value of Extremums (MAVE), and Envelope Frequency Band (EFB). This comparative analysis aimed to evaluate the efficacy of the novel algorithm against the established methods in the field, providing valuable insights into its potential advantages and limitations. In summary, this paper presents an innovative algorithm for the early detection of roller bearing failures, leveraging FFT-based spectral analysis, trend monitoring, adaptive thresholding, and outlier detection. Its ability to confirm the first failure state underscores the algorithm's effectiveness. [ABSTRACT FROM AUTHOR]

Cellwise outliers are widespread in real world data analysis. Traditional robust methods may fail when applied to datasets under such contamination. We introduce a variable selection procedure, that uses the Gaussian rank estimator to obtain an initial empirical covariance matrix among the response and potential predictors. We re-parameterize the classical linear regression model design matrix and the response vector such that we are able to take advantage of these robustly estimated components before applying the adaptive Lasso to obtain consistent variable selection results. The procedure is robust to cellwise outliers in low and high-dimensional settings. Empirical results show good performance compared with recently proposed robust techniques, particularly in the challenging environment when contamination rates are high but the magnitude of outliers is moderate. [ABSTRACT FROM AUTHOR]

Beyond the challenge of keeping up to date with current best practices regarding the diagnosis and treatment of outliers, an additional difficulty arises concerning the mathematical implementation of the recommended methods. Here, we provide an overview of current recommendations and best practices and demonstrate how they can easily and conveniently be implemented in the R statistical computing software, using the {performance} package of the easystats ecosystem. We cover univariate, multivariate, and model-based statistical outlier detection methods, their recommended threshold, standard output, and plotting methods. We conclude by reviewing the different theoretical types of outliers, whether to exclude or winsorize them, and the importance of transparency. A preprint of this paper is available at: 10.31234/osf.io/bu6nt. [ABSTRACT FROM AUTHOR]

This study introduces parallel coordinates plot as an innovative visualization method, offering an alternative to traditional score plots in Principal Component Analysis (PCA) for exploratory data analysis (EDA). EDA is pivotal for unraveling intricate data structures, and PCA serves as a powerful tool for simplifying complex datasets. The paper explores the application of parallel coordinates plot in visualizing and interpreting PCA results, emphasizing their effectiveness in revealing multivariate patterns and detecting outliers. Through the examination of simulated datasets, where two distinct classes are generated in three-dimensional and five-dimensional spaces and real Raman spectroscopic data, the research demonstrates the utility of parallel coordinates plot in enhancing data exploration and providing valuable insights in pattern recognition methodologies. The results highlight the complementary nature of traditional score plots and parallel coordinates plots, offering a comprehensive approach to understanding high-dimensional datasets. This innovative visualization technique proves valuable in discerning subtle patterns and relationships within complex data structures, contributing to a more profound exploration of datasets in various domains. [ABSTRACT FROM AUTHOR]

With the development of technology and the widespread collection of data, high-dimensional data analysis has become a research hotspot in many fields. Traditional parameter methods often face problems such as dimensional disasters in high-dimensional data analysis. Non parametric methods have broad application prospects in high-dimensional data because they do not rely on specific parameter distribution assumptions. The Bayesian rule is more suitable for dealing with noise and outliers in high-dimensional data because it takes uncertainty into account. Therefore, it is of great significance to combine non parametric methods with Bayesian methods for application research in high-dimensional data analysis. In this paper, the nonparametric Bayesian method was applied to the analysis of high-dimensional data, and the Dirichlet process Mixture model was used to cluster high-dimensional data. The regression analysis of high-dimensional data was carried out through the prediction model of nonparametric Bayesian regression. In this paper, the nonparametric Bayesian method based on Bayesian sparse linear model was used for feature selection of high-dimensional data. In order to determine the superiority of nonparametric Bayesian methods in high-dimensional data analysis, this paper conducted experiments on nonparametric Bayesian methods and traditional parametric methods in high-dimensional data analysis from five aspects of cluster analysis, classification analysis, regression analysis, feature selection and anomaly detection, and evaluated them through multiple indicators. This article explored the application of non parametric Bayesian methods in high-dimensional data analysis from these aspects through simulation experiments. The experimental results show that the clustering accuracy of the non parametric Bayesian clustering algorithm was 0.93, and the accuracy of the non parametric Bayesian classification algorithm was between 0.93 and 0.99; the coefficient of determination of nonparametric Bayesian regression algorithm was 0.98; the F1 values of non parametric Bayesian methods in anomaly detection ranged from 0.86 to 0.91, which was superior to traditional methods. Non parametric Bayesian methods have broad application prospects in high-dimensional data analysis, and can be applied in multiple fields such as clustering, classification, regression, etc. [ABSTRACT FROM AUTHOR]

In this research, it was proposed to create three new robust multivariate charts corresponding to the |S|‐ chart, which are robust to outliers, using three methods, an algorithm, namely the Rousseuw and Leroy algorithm, Maronna and Zamar, and the family of 'concentration algorithms' by Olive and Hawkins. Then the comparison between the proposed and classical method of the researcher Shewhart depends on the total variance (trace variance matrix), the general variance (determinant of the variance matrix), and the difference between the upper and lower control limits to obtain the most efficient charts against outliers through simulation and real data and using a program in the MATLAB language designed for this purpose. The study concluded that the proposed charts dealt with the problem of the influence of outliers and were more efficient than the classical method, in addition, the proposed robust chart (Orthogonalized Gnanadesikan‐Kettenring) was more efficient than the rest of the proposed charts. [ABSTRACT FROM AUTHOR]

Motivation Cell-type clustering is a crucial first step for single-cell RNA-seq data analysis. However, existing clustering methods often provide different results on cluster assignments with respect to their own data pre-processing, choice of distance metrics, and strategies of feature extraction, thereby limiting their practical applications. Results We propose Cross-Tabulation Ensemble Clustering (CTEC) method that formulates two re-clustering strategies (distribution- and outlier-based) via cross-tabulation. Benchmarking experiments on five scRNA-Seq datasets illustrate that the proposed CTEC method offers significant improvements over the individual clustering methods. Moreover, CTEC-DB outperforms the state-of-the-art ensemble methods for single-cell data clustering, with 45.4% and 17.1% improvement over the single-cell aggregated from ensemble clustering method (SAFE) and the single-cell aggregated clustering via Mixture model ensemble method (SAME), respectively, on the two-method ensemble test. Availability and implementation The source code of the benchmark in this work is available at the GitHub repository https://github.com/LWCHN/CTEC.git. [ABSTRACT FROM AUTHOR]

This study introduces an enhanced recursive least-squares algorithm that applies the adaptive accelerated proximal gradient method to identify Auto-Regressive Exogenous models with output outliers. First, the outlier problem was converted into a robust principal component analysis problem. The adaptive accelerated proximal gradient method is then introduced to recover the information matrix, and the recursive least squares algorithm is applied to estimate the model parameters. Furthermore, we demonstrated that the parameter estimation is unbiased and that the parameter estimation error converges to zero under persistent excitation. A series of bench tests consistently highlighted the practicality and effectiveness of the proposed algorithm. • The accelerated proximal gradient algorithm is introduced for effective outlier recovery. • The proposed adaptive accelerated proximal gradient algorithm can adaptively adjust the step-size. • A comprehensive framework is established by integrating accelerated proximal gradient and recursive least squares algorithms. • The proposed algorithm is validated through a state of charge estimation problem, demonstrating its effectiveness. [ABSTRACT FROM AUTHOR]

Classical estimators fail to be efficient in practical scenarios when data is riddled with extreme values known as outliers. Robust estimation strategies are insensitive to outliers and may be used in such cases. The current work is focused on developing a novel robust estimation strategy using Huber M-estimation. A new chain-product type estimator for population mean has been suggested utilizing data on two auxiliary variables. A numerical comparison has been carried out between the proposed robust estimator and the corresponding classical estimator using real and simulated data containing outliers. Recommendations have been made for its practical use based on the encouraging results. [ABSTRACT FROM AUTHOR]

A straightforward algorithm is proposed for L1-norm minimisation. The proposed algorithm is based on grey wolf optimisation which is a meta-heuristic method. Although L1-norm is an efficient tool for robust estimation and outlier detection, the complexity of its implementation made it less useful than L2-norm since after formulation of the L1-norm minimisation for a certain problem one must solve a linear programming problem by a search method while here we only need to set the corresponding L1-norm target function. Two geodetic examples approve the efficiency of the proposed approach. [ABSTRACT FROM AUTHOR]

Aim: Species occurrence data are valuable information that enables one to estimate geographical distributions, characterize niches and their evolution, and guide spatial conservation planning. Rapid increases in species occurrence data stem from increasing digitization and aggregation efforts, and citizen science initiatives. However, persistent quality issues in occurrence data can impact the accuracy of scientific findings, underscoring the importance of filtering erroneous occurrence records in biodiversity analyses. Innovation: We introduce an R package, occTest, that synthesizes a growing open‐source ecosystem of biodiversity cleaning workflows to prepare occurrence data for different modelling applications. It offers a structured set of algorithms to identify potential problems with species occurrence records by employing a hierarchical organization of multiple tests. The workflow has a hierarchical structure organized in testPhases (i.e. cleaning vs. testing) that encompass different testBlocks grouping different testTypes (e.g. environmental outlier detection), which may use different testMethods (e.g. Rosner test, jacknife,etc.). Four different testBlocks characterize potential problems in geographic, environmental, human influence and temporal dimensions. Filtering and plotting functions are incorporated to facilitate the interpretation of tests. We provide examples with different data sources, with default and user‐defined parameters. Compared to other available tools and workflows, occTest offers a comprehensive suite of integrated tests, and allows multiple methods associated with each test to explore consensus among data cleaning methods. It uniquely incorporates both coordinate accuracy analysis and environmental analysis of occurrence records. Furthermore, it provides a hierarchical structure to incorporate future tests yet to be developed. Main conclusions: occTest will help users understand the quality and quantity of data available before the start of data analysis, while also enabling users to filter data using either predefined rules or custom‐built rules. As a result, occTest can better assess each record's appropriateness for its intended application. [ABSTRACT FROM AUTHOR]

The OLS model is built on the assumption of normality in the distribution of error terms. However, this assumption can be easily violated, especially when there are outliers in the data. A single outlier can disrupt the normality assumption of error terms, making the OLS model less effective. In such situations, M-estimators (MEs) come into play to obtain reliable estimates. We introduce a redescending M-estimators (RME) for robust regression to handle datasets with outliers. The proposed RME produces more robust estimates by effectively managing the influence of outliers, even at lower values of the tuning constant. We compared the performance of this estimator with existing RMEs using real-life data examples and an extensive simulation study. The results show that our suggested RME is more efficient than the compared ME in various situations. [ABSTRACT FROM AUTHOR]

The detection of water quality indicators such as Temperature, pH, Turbidity, Conductivity, and TDS involves five national standard methods. Chemically based measurement techniques may generate liquid residue, causing secondary pollution. The water quality monitoring and data analysis system can effectively address the issues that conventional methods require multiple pieces of equipment and repeated measurements. This paper analyzes the distribution characteristics of the historical data from five sensors at a specific time, displays them graphically in real time, and provides an early warning of exceeding the standard; It selects four water samples from different sections of the Li River, based on the national standard method, the average measurement errors of Temperature, PH, TDS, Conductivity and Turbidity are 0.98%, 2.23%, 2.92%, 3.05% and 3.98%.;It further uses the quartile method to analyze the outlier data over 100,000 records and five historical periods are selected. Experiment results show the system is relatively stable in measuring Temperature, PH and TDS, and the proportion of outlier is 0.42%, 0.84% and 1.24%. When Turbidity and Conductivity are measured, the proportion is 3.11% and 2.92%. In the experiment of using 7 methods to fill outlier, K nearest neighbor algorithm is better than others. The analysis of data trends, outliers, means, and extreme values assists in making decisions, such as updating and maintaining equipment, addressing extreme water quality situations, and enhancing regional water quality oversight. [ABSTRACT FROM AUTHOR]