3,120 results on '"data smoothing"'
Search Results
2. Application of data smoothing and principal component analysis to develop a parameter ranking system for the anaerobic digestion process
- Author
-
Kim, Moonil, Chul, Park, Kim, Wan, and Cui, Fenghao
- Published
- 2022
- Full Text
- View/download PDF
3. Predicting Heart Diseases by Selective Machine Learning Algorithms.
- Author
-
UMAR, N., HASSAN, S. K., UMAR, A., and AHMED, S. S.
- Abstract
Heart disease is among the leading causes of mortality worldwide. As a result, it’s critical to diagnose patients appropriately and promptly. Consequently, the objective of this paper was to predict heart diseases using selective machine learning algorithms. The leverage technique was evaluated using the Cleveland heart disease dataset. In this study five classifiers were trained and tested with the unsmooth Cleveland dataset and the smooth Cleveland dataset. The results obtained showed all the classifiers performed better when tested with the smooth dataset with an accuracy of 98.11% than when tested with the unsmooth dataset with an accuracy of 89.71% The leverage technique performed better than works found in literature reviewed. These results show that feature engineering using data smoothing is effective for improved heart disease prediction. [ABSTRACT FROM AUTHOR]
- Published
- 2025
- Full Text
- View/download PDF
4. Identifying Cyclical Patterns of Behavior Using a Moving-Average, Data-Smoothing Manipulation.
- Author
-
Retzlaff, Billie J., Craig, Andrew R., Owen, Todd M., Greer, Brian D., O'Donnell, Alex, and Fisher, Wayne W.
- Subjects
- *
SLEEP , *STATISTICAL smoothing , *MOVING average process , *DATA analysis , *MEDICAL personnel - Abstract
For some individuals, rates of destructive behavior change in a predictable manner, irrespective of the contingencies programmed. Identifying such cyclical patterns can lead to better prediction of destructive behavior and may allow for the identification of relevant biological processes. However, identifying cyclical patterns of behavior can be difficult when using traditional methods of visual analysis. We describe a data-manipulation method, called data smoothing, in which one averages the data across time points within a specified window (e.g., 3, 5, or 7 days). This approach minimizes variability in the data and can increase the saliency of cyclical behavior patterns. We describe two cases for which we identified cyclical patterns in daily occurrences of destructive behavior, and we demonstrate the importance of analyzing smoothed data across various windows when using this approach. We encourage clinicians to analyze behavioral data in this way when rates vary independently of programmed contingencies and other potentially controlling variables have been ruled out (e.g., behavior variability related to sleep behavior). [ABSTRACT FROM AUTHOR]
- Published
- 2024
- Full Text
- View/download PDF
5. Improving Radio Signal from Baghdad University Radio Telescope Using the Savitzky-Golay Filter.
- Author
-
Hussein, Zahraa A. and Mahdi, Hareth S.
- Subjects
COLLEGE radio stations ,RADIO astronomy ,ASTRONOMICAL observations ,GEODETIC astronomy ,STATISTICAL smoothing - Abstract
Copyright of Iraqi Journal of Physics is the property of Republic of Iraq Ministry of Higher Education & Scientific Research (MOHESR) and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
- Published
- 2024
- Full Text
- View/download PDF
6. A filter calibration method for laser-scanned weld toe geometries
- Author
-
Finn Renken, Matthias Jung, Sören Ehlers, and Moritz Braun
- Subjects
Weld toe measurement ,Laser scanning ,Filter calibration ,Data smoothing ,Universal filter ,Engineering (General). Civil engineering (General) ,TA1-2040 - Abstract
The scanning of weld seams can be used to evaluate the local weld toe geometry for fatigue assessments. Laser scanned weld seam profiles often contain noise which complicates the accurate measurement of the weld toe geometry. For that reason, filtering of the scanned data is necessary. The issue at hand is that a filtering method can significantly affect the measurement results. Therefore, a calibration of the filter input parameters is needed. In this study, a calibration method for filtered laser-scanned weld profiles is presented by using artificial weld toe geometries. The adjustment of different filter functions is achieved by using an optimization method on predefined weld toes with an artificial noise. The resulting input data for the filter functions is tested on a real specimen to verify the method. Through the calibration method it is possible to achieve satisfactory measurement results with precisely set input parameters for the filter functions. The most suitable filter functions for the measurement of the weld toe are the Gaussian and the Lowpass filter. Both functions are adequate as a universally applicable filter. For the evaluation of the measurement results of the radii and angles, a tolerance range is introduced, which is defined by the theoretically minimum measurable radii and angles. Using an adjusted Lowpass filter and a point distance of 0.07 mm set by the laser scanner, a measurement within the tolerance range of 0.2 mm is achievable for the weld toe radius. For the weld toe angle, the tolerance range of 1.5° is achieved for the majority of measurements.
- Published
- 2024
- Full Text
- View/download PDF
7. Improving Radio Signal from Baghdad University Radio Telescope Using the Savitzky-Golay Filter
- Author
-
Zahraa A. Hussein and Hareth S. Mahdi
- Subjects
Baghdad University Radio Telescope BURT ,Data Smoothing ,Radio Astronomy ,Savitzky Golay Filter ,Signal Processing ,Physics ,QC1-999 - Abstract
Astronomical radio observations from a small radio telescope suffer from various types of noise. Hence, astronomers continuously search for new techniques to eliminate or reduce such noise and obtain more accurate results. This research investigates the impact of implementing the Savitzky-Golay filter on enhancing radio observation signals retrieved from the Baghdad University Radio Telescope (BURT). Observations from BURT were carried out for different Galactic coordinates, and then a MATLAB code was written and used to implement the Savitzky-Golay filter for the collected data. This process provides an assessment of the ability of the filter to reduce noise and improve the quality of the signal. The results of this research clearly showed that applying the Savitzky-Golay filter reduces the noise and enhances the signal of astronomical radio observations. However, the filter should be used appropriately to preserve the original features of the signal. In conclusion, the filter is considered an efficient tool for enhancing the radio signal by reducing the noise and smoothing the signal. Therefore, the filter provides a substantial contribution and improvement to the field of radio astronomy.
- Published
- 2024
- Full Text
- View/download PDF
8. Modeling Infectious Disease Trend using Sobolev Polynomials
- Author
-
Rolly Czar Joseph Castillo, Victoria May Mendoza, Jose Ernie Lope, and Renier Mendoza
- Subjects
data smoothing ,sobolev polynomials ,covid-19 ,mpox ,schistosomiasis ,whittaker-henderson method ,Biology (General) ,QH301-705.5 ,Mathematics ,QA1-939 - Abstract
Trend analysis plays an important role in infectious disease control. An analysis of the underlying trend in the number of cases or the mortality of a particular disease allows one to characterize its growth. Trend analysis may also be used to evaluate the effectiveness of an intervention to control the spread of an infectious disease. However, trends are often not readily observable because of noise in data that is commonly caused by random factors, short-term repeated patterns, or measurement error. In this paper, a smoothing technique that generalizes the Whittaker-Henderson method to infinite dimension and whose solution is represented by a polynomial is applied to extract the underlying trend in infectious disease data. The solution is obtained by projecting the problem to a finite-dimensional space using an orthonormal Sobolev polynomial basis obtained from Gram-Schmidt orthogonalization procedure and a smoothing parameter computed using the Philippine Eagle Optimization Algorithm, which is more efficient and consistent than a hybrid model used in earlier work. Because the trend is represented by the polynomial solution, extreme points, concavity, and periods when infectious disease cases are increasing or decreasing can be easily determined. Moreover, one can easily generate forecast of cases using the polynomial solution. This approach is applied in the analysis of trends, and in forecasting cases of different infectious diseases.
- Published
- 2023
- Full Text
- View/download PDF
9. Extended Smoothing Methods for Sparse Test Data Based on Zero-Padding.
- Author
-
Zhou, Pan, Shi, Tuo, Xin, Jianghui, Li, Yaowei, Lv, Tian, and Zang, Liguo
- Subjects
DISCRETE Fourier transforms ,WAVENUMBER ,STATISTICAL smoothing ,TEST methods - Abstract
Aiming at the problem of sparse measurement points due to test conditions in engineering, a smoothing method based on zero-padding in the wavenumber domain is proposed to increase data density. Firstly, the principle of data extension and smoothing is introduced. The core idea of this principle is to extend the discrete data series by zero-padding in the wavenumber domain. The conversion between the spatial and wavenumber domains is achieved using the Discrete Fourier Transform (DFT) and the Inverse Discrete Fourier Transform (IDFT). Then, two sets of two-dimensional discrete random data are extended and smoothed, respectively, and the results verify the effectiveness and feasibility of the algorithm. The method can effectively increase the density of test data in engineering tests, achieve smoothing and extend the application to areas related to data processing. [ABSTRACT FROM AUTHOR]
- Published
- 2023
- Full Text
- View/download PDF
10. On smoothing of data using Sobolev polynomials
- Author
-
Rolly Czar Joseph Castillo and Renier Mendoza
- Subjects
data smoothing ,whittaker-henderson method ,sobolev polynomials ,high-frequency data ,approximation ,generalized cross validation score ,Mathematics ,QA1-939 - Abstract
Data smoothing is a method that involves finding a sequence of values that exhibits the trend of a given set of data. This technique has useful applications in dealing with time series data with underlying fluctuations or seasonality and is commonly carried out by solving a minimization problem with a discrete solution that takes into account data fidelity and smoothness. In this paper, we propose a method to obtain the smooth approximation of data by solving a minimization problem in a function space. The existence of the unique minimizer is shown. Using polynomial basis functions, the problem is projected to a finite dimension. Unlike the standard discrete approach, the complexity of our method does not depend on the number of data points. Since the calculated smooth data is represented by a polynomial, additional information about the behavior of the data, such as rate of change, extreme values, concavity, etc., can be drawn. Furthermore, interpolation and extrapolation are straightforward. We demonstrate our proposed method in obtaining smooth mortality rates for the Philippines, analyzing the underlying trend in COVID-19 datasets, and handling incomplete and high-frequency data.
- Published
- 2022
- Full Text
- View/download PDF
11. Data smoothing with applications to edge detection
- Author
-
Al-Jamal Mohammad F., Baniabedalruhman Ahmad, and Alomari Abedel-Karrem
- Subjects
data smoothing ,numerical differentiation ,noisy data ,diffusion equation ,regularization ,edge detection ,65f22 ,47a52 ,35k20 ,65d19 ,Mathematics ,QA1-939 - Abstract
The aim of this paper is to present a new stable method for smoothing and differentiating noisy data defined on a bounded domain Ω⊂RN\Omega \subset {{\mathbb{R}}}^{N} with N≥1N\ge 1. The proposed method stems from the smoothing properties of the classical diffusion equation; the smoothed data are obtained by solving a diffusion equation with the noisy data imposed as the initial condition. We analyze the stability and convergence of the proposed method and we give optimal convergence rates. One of the main advantages of this method lies in multivariable problems, where some of the other approaches are not easily generalized. Moreover, this approach does not require strong smoothness assumptions on the underlying data, which makes it appealing for detecting data corners or edges. Numerical examples demonstrate the feasibility and robustness of the method even with the presence of a large amounts of noise.
- Published
- 2022
- Full Text
- View/download PDF
12. On smoothing of data using Sobolev polynomials.
- Author
-
Castillo, Rolly Czar Joseph and Mendoza, Renier
- Subjects
SOBOLEV spaces ,APPROXIMATION theory ,GENERALIZATION ,INTERPOLATION ,SMOOTHNESS of functions - Abstract
Data smoothing is a method that involves finding a sequence of values that exhibits the trend of a given set of data. This technique has useful applications in dealing with time series data with underlying fluctuations or seasonality and is commonly carried out by solving a minimization problem with a discrete solution that takes into account data fidelity and smoothness. In this paper, we propose a method to obtain the smooth approximation of data by solving a minimization problem in a function space. The existence of the unique minimizer is shown. Using polynomial basis functions, the problem is projected to a finite dimension. Unlike the standard discrete approach, the complexity of our method does not depend on the number of data points. Since the calculated smooth data is represented by a polynomial, additional information about the behavior of the data, such as rate of change, extreme values, concavity, etc., can be drawn. Furthermore, interpolation and extrapolation are straightforward. We demonstrate our proposed method in obtaining smooth mortality rates for the Philippines, analyzing the underlying trend in COVID-19 datasets, and handling incomplete and high-frequency data. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF
13. Data filtering methods for SARS-CoV-2 wastewater surveillance
- Author
-
Rezgar Arabzadeh, Daniel Martin Grünbacher, Heribert Insam, Norbert Kreuzinger, Rudolf Markt, and Wolfgang Rauch
- Subjects
data smoothing ,pandemic management ,sars-cov-2 ,signal filtering ,virus monitoring ,wastewater-based epidemiology ,Environmental technology. Sanitary engineering ,TD1-1066 - Abstract
In the case of SARS-CoV-2 pandemic management, wastewater-based epidemiology aims to derive information on the infection dynamics by monitoring virus concentrations in the wastewater. However, due to the intrinsic random fluctuations of the viral signal in wastewater caused by several influencing factors that cannot be determined in detail (e.g. dilutions; number of people discharging; variations in virus excretion; water consumption per day; transport and fate processes in sewer system), the subsequent prevalence analysis may result in misleading conclusions. It is thus helpful to apply data filtering techniques to reduce the noise in the signal. In this paper we investigate 13 smoothing algorithms applied to the virus signals monitored in four wastewater treatment plants in Austria. The parameters of the algorithms have been defined by an optimization procedure aiming for performance metrics. The results are further investigated by means of a cluster analysis. While all algorithms are in principle applicable, SPLINE, Generalized Additive Model and Friedman's Super Smoother are recognized as superior methods in this context (with the latter two having a tendency to over-smoothing). A first analysis of the resulting datasets indicates the positive effect of filtering to the correlation of the viral signal to monitored incidence values. HIGHLIGHTS The random component in the timeline of SARS-CoV-2 virus concentration makes data filtering necessary.; Thirteen common filtering techniques are investigated for their potential to smooth the virus signals.; SPLINE, GAM and Friedman's Super Smoother are seen as superior algorithms for smoothing SARS-CoV-2 signals.;
- Published
- 2021
- Full Text
- View/download PDF
14. Modelling Norm Scores with the cNORM Package in R
- Author
-
Sebastian Gary, Wolfgang Lenhard, and Alexandra Lenhard
- Subjects
regression-based norming ,continuous norming ,inferential norming ,data smoothing ,curve fitting ,percentile estimation ,Psychology ,BF1-990 - Abstract
In this article, we explain and demonstrate how to model norm scores with the cNORM package in R. This package is designed specifically to determine norm scores when the latent ability to be measured covaries with age or other explanatory variables such as grade level. The mathematical method used in this package draws on polynomial regression to model a three-dimensional hyperplane that smoothly and continuously captures the relation between raw scores, norm scores and the explanatory variable. By doing so, it overcomes the typical problems of classical norming methods, such as overly large age intervals, missing norm scores, large amounts of sampling error in the subsamples or huge requirements with regard to the sample size. After a brief introduction to the mathematics of the model, we describe the individual methods of the package. We close the article with a practical example using data from a real reading comprehension test.
- Published
- 2021
- Full Text
- View/download PDF
15. Score filtering for contextualized noise suppression of Poisson distributed geophysical signals
- Author
-
Altdorff, Daniel, Schrön, Martin, Altdorff, Daniel, and Schrön, Martin
- Abstract
Geophysical and remote sensing products that rely on Poisson-distributed measurement signals, such as cosmic-ray neutron sensing (CRNS) and gamma spectrometry, often face challenges due to inherent Poisson noise. Common techniques to enhance signal stability include data aggregation or smoothing (e.g., moving averages and interpolation). However, these methods typically reduce the ability to resolve detailed temporal (stationary data) and spatial (mobile data) features. In this study, we introduced a method for contextual noise suppression tailored to Poisson-distributed data, utilizing a discrete score attribution system. This score filter evaluates each observation against eight different criteria to assess its consistency with surrounding values, assigning a score between 0 (very unlikely) and 8 (very likely) to indicate whether the observation is likely to act as noise. These scores can then be used to flag or remove data points based on user-defined thresholds. We tested the score filter's effectiveness on both stationary and mobile CRNS data, as well as on gamma-ray spectrometry and electromagnetic induction (EMI) recordings. In our examples, the score filter consistently outperformed established filters, for example Savitzky–Golay and Kalman, in direct competition when applied to CRNS time series data. Additionally, the score filter substantially reduced Poisson noise in mobile CRNS, gamma-ray spectrometry and EMI data. The scoring system also provides a context-sensitive evaluation of individual observations or aggregates, assessing their conformity within the dataset. Given its general applicability, customizable criteria and very low computational demands, the proposed filter is easy to implement and holds promise as a valuable tool for denoising geophysical data and applications in other fields.
- Published
- 2024
16. A filter calibration method for laser-scanned weld toe geometries
- Author
-
Renken, Finn, Jung, Matthias, Ehlers, Sören, Braun, Moritz, Renken, Finn, Jung, Matthias, Ehlers, Sören, and Braun, Moritz
- Abstract
The scanning of weld seams can be used to evaluate the local weld toe geometry for fatigue assessments. Laser scanned weld seam profiles often contain noise which complicates the accurate measurement of the weld toe geometry. For that reason, filtering of the scanned data is necessary. The issue at hand is that a filtering method can significantly affect the measurement results. Therefore, a calibration of the filter input parameters is needed. In this study, a calibration method for filtered laser-scanned weld profiles is presented by using artificial weld toe geometries. The adjustment of different filter functions is achieved by using an optimization method on predefined weld toes with an artificial noise. The resulting input data for the filter functions is tested on a real specimen to verify the method. Through the calibration method it is possible to achieve satisfactory measurement results with precisely set input parameters for the filter functions. The most suitable filter functions for the measurement of the weld toe are the Gaussian and the Lowpass filter. Both functions are adequate as a universally applicable filter. For the evaluation of the measurement results of the radii and angles, a tolerance range is introduced, which is defined by the theoretically minimum measurable radii and angles. Using an adjusted Lowpass filter and a point distance of 0.07 mm set by the laser scanner, a measurement within the tolerance range of 0.2 mm is achievable for the weld toe radius. For the weld toe angle, the tolerance range of 1.5° is achieved for the majority of measurements.
- Published
- 2024
17. A binary search algorithm for univariate data approximation and estimation of extrema by piecewise monotonic constraints.
- Author
-
Demetriou, Ioannis C.
- Subjects
SEARCH algorithms ,APPROXIMATION algorithms ,LEAST squares ,CONVEX functions ,GEOPHYSICS ,FORTRAN - Abstract
The piecewise monotonic approximation problem makes the least changes to n univariate noisy data so that the piecewise linear interpolant to the new values is composed of at most k monotonic sections. The term "least changes" is defined in the sense of a global sum of strictly convex functions of changes. The main difficulty in this calculation is that the extrema of the interpolant have to be found automatically, but the number of all possible combinations of extrema can be O (n k - 1) , which makes not practicable to test each one separately. It is known that the case k = 1 is straightforward, and that the case k > 1 reduces to partitioning the data into at most k disjoint sets of adjacent data and solving a k = 1 problem for each set. Some ordering relations of the extrema are studied that establish three quite efficient algorithms by using a binary search method for partitioning the data. In the least squares case the total work is only O (n σ + k σ log 2 σ) computer operations when k ≥ 3 and is only O (n) when k = 1 or 2, where σ - 2 is the number of sign changes in the sequence of the first differences of the data. Fortran software has been written for this case and the numerical results indicate superior performance to existing algorithms. Some examples with real data illustrate the method. Many applications of the method arise from bioinformatics, energy, geophysics, medical imaging, and peak finding in spectroscopy, for instance. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF
18. Principal component analysis-assisted selection of optimal denoising method for oil well transient data
- Author
-
Bing Zhang, Khafiz Muradov, and Akindolu Dada
- Subjects
Intelligent well ,Downhole gauge ,Pressure and temperature transient analysis (PTTA) ,Data smoothing ,wavelet threshold denoising ,Principal component analysis (PCA) ,Petroleum refining. Petroleum products ,TP690-692.5 ,Petrology ,QE420-499 - Abstract
Abstract Oil and gas production wells are often equipped with modern, permanent or temporary in-well monitoring systems, either electronic or fiber-optic, typically for measurement of downhole pressure and temperature. Consequently, novel methods of pressure and temperature transient analysis (PTTA) have emerged in the past two decades, able to interpret subtle thermodynamic effects. Such analysis demands high-quality data. High-level reduction in data noise is often needed in order to ensure sufficient reliability of the PTTA. This paper considers the case of a state-of-the-art intelligent well equipped with fiber-optic, high-precision, permanent downhole gauges. This is followed by screening, development, verification and application of data denoising methods that can overcome the limitation of the existing noise reduction methods. Firstly, the specific types of noise contained in the original data are analyzed by wavelet transform, and the corresponding denoising methods are selected on the basis of the wavelet analysis. Then, the wavelet threshold denoising method is used for the data with white noise and white Gaussian noise, while a data smoothing method is used for the data with impulse noise. The paper further proposes a comprehensive evaluation index as a useful denoising success metrics for optimal selection of the optimal combination of the noise reduction methods. This metrics comprises a weighted combination of the signal-to-noise ratio and smoothness value where the principal component analysis was used to determine the weights. Thus the workflow proposed here can be comprehensively defined solely by the data via its processing and analysis. Finally, the effectiveness of the optimal selection methods is confirmed by the robustness of the PTTA results derived from the de-noised measurements from the above-mentioned oil wells.
- Published
- 2020
- Full Text
- View/download PDF
19. Extended Smoothing Methods for Sparse Test Data Based on Zero-Padding
- Author
-
Pan Zhou, Tuo Shi, Jianghui Xin, Yaowei Li, Tian Lv, and Liguo Zang
- Subjects
two-dimensional discrete data ,data extensions ,data smoothing ,zero-padding ,Technology ,Engineering (General). Civil engineering (General) ,TA1-2040 ,Biology (General) ,QH301-705.5 ,Physics ,QC1-999 ,Chemistry ,QD1-999 - Abstract
Aiming at the problem of sparse measurement points due to test conditions in engineering, a smoothing method based on zero-padding in the wavenumber domain is proposed to increase data density. Firstly, the principle of data extension and smoothing is introduced. The core idea of this principle is to extend the discrete data series by zero-padding in the wavenumber domain. The conversion between the spatial and wavenumber domains is achieved using the Discrete Fourier Transform (DFT) and the Inverse Discrete Fourier Transform (IDFT). Then, two sets of two-dimensional discrete random data are extended and smoothed, respectively, and the results verify the effectiveness and feasibility of the algorithm. The method can effectively increase the density of test data in engineering tests, achieve smoothing and extend the application to areas related to data processing.
- Published
- 2023
- Full Text
- View/download PDF
20. Improvement of Norm Score Quality via Regression-Based Continuous Norming.
- Author
-
Lenhard, Wolfgang and Lenhard, Alexandra
- Subjects
- *
PSYCHOMETRICS , *REGRESSION analysis , *STATISTICAL sampling , *STATISTICAL models - Abstract
The interpretation of psychometric test results is usually based on norm scores. We compared semiparametric continuous norming (SPCN) with conventional norming methods by simulating results for test scales with different item numbers and difficulties via an item response theory approach. Subsequently, we modeled the norm scores based on random samples with varying sizes either with a conventional ranking procedure or SPCN. The norms were then cross-validated by using an entirely representative sample of N = 840,000 for which different measures of norming error were computed. This process was repeated 90,000 times. Both approaches benefitted from an increase in sample size, with SPCN reaching optimal results with much smaller samples. Conventional norming performed worse on data fit, age-related errors, and number of missings in the norm tables. The data fit in conventional norming of fixed subsample sizes varied with the granularity of the age brackets, calling into question general recommendations for sample sizes in test norming. We recommend that test norms should be based on statistical models of the raw score distributions instead of simply compiling norm tables via conventional ranking procedures. [ABSTRACT FROM AUTHOR]
- Published
- 2021
- Full Text
- View/download PDF
21. Comparison of Methods for Smoothing Environmental Data with an Application to Particulate Matter PM10
- Author
-
Martina Čampulová
- Subjects
data smoothing ,trend filtering ,environmental data ,particulate matter PM10 ,Agriculture ,Biology (General) ,QH301-705.5 - Abstract
Data smoothing is often required within the environmental data analysis. A number of methods and algorithms that can be applied for data smoothing have been proposed. This paper gives an overview and compares the performance of different smoothing procedures that estimate the trend in the data, based on the surrounding noisy observations that can be applied on environmental data. The considered methods include kernel regression with both global and local bandwidth, moving average, exponential smoothing, robust repeated median regression, trend filtering and approach based on discrete Fourier and discrete wavelet transform. The methods are applied to real data obtained by measurement of PM10 concentrations and compared in a simulation study.
- Published
- 2018
- Full Text
- View/download PDF
22. Functional approach to analysis of daily tax revenues
- Author
-
Jovita Gudan and Alfredas Račkauskas
- Subjects
functional data analysis ,data smoothing ,registration ,prediction ,Mathematics ,QA1-939 - Abstract
We present a functional data analysis approach to modeling and analyzing daily tax revenues. The main features of daily tax revenue we need to extract are some patterns within calendar months which can be used for prediction. As standard seasonal time series techniques cannot be used due to varying number of banking days per calendar month and presence of seasonality between and within months we interpret monthly tax revenues as curves obtained from daily data. Standard smoothing techniques and registration taking into account time variability are used for data preparation.
- Published
- 2019
- Full Text
- View/download PDF
23. Functional QTL mapping and genomic prediction of canopy height in wheat measured using a robotic field phenotyping platform.
- Author
-
Lyra, Danilo H, Virlet, Nicolas, Sadeghi-Tehran, Pouria, Hassall, Kirsty L, Wingen, Luzie U, Orford, Simon, Griffiths, Simon, Hawkesford, Malcolm J, and Slavov, Gancho T
- Subjects
- *
FORECASTING , *GROWING season , *STATISTICAL power analysis , *ALTITUDES , *PREDICTION models , *WHEAT - Abstract
Genetic studies increasingly rely on high-throughput phenotyping, but the resulting longitudinal data pose analytical challenges. We used canopy height data from an automated field phenotyping platform to compare several approaches to scanning for quantitative trait loci (QTLs) and performing genomic prediction in a wheat recombinant inbred line mapping population based on up to 26 sampled time points (TPs). We detected four persistent QTLs (i.e. expressed for most of the growing season), with both empirical and simulation analyses demonstrating superior statistical power of detecting such QTLs through functional mapping approaches compared with conventional individual TP analyses. In contrast, even very simple individual TP approaches (e.g. interval mapping) had superior detection power for transient QTLs (i.e. expressed during very short periods). Using spline-smoothed phenotypic data resulted in improved genomic predictive abilities (5–8% higher than individual TP prediction), while the effect of including significant QTLs in prediction models was relatively minor (<1–4% improvement). Finally, although QTL detection power and predictive ability generally increased with the number of TPs analysed, gains beyond five or 10 TPs chosen based on phenological information had little practical significance. These results will inform the development of an integrated, semi-automated analytical pipeline, which will be more broadly applicable to similar data sets in wheat and other crops. [ABSTRACT FROM AUTHOR]
- Published
- 2020
- Full Text
- View/download PDF
24. Improving the simultaneous application of the DSN-PC and NOAA GFS datasets.
- Author
-
Vas, Ádám, Owino, Oluoch Josphat, and Tóth, László
- Subjects
- *
DISTRIBUTED sensors , *SENSOR networks , *WEATHER forecasting , *MEASUREMENT errors , *STATISTICAL smoothing - Abstract
Our surface-based sensor network, called Distributed Sensor Network for Prediction Calculations (DSN-PC) obviously has limitations in terms of vertical atmospheric data. While efforts are being made to approximate these upper-air parameters from surface-level, as a first step it was necessary to test the network's capability of making distributed computations by applying a hybrid approach. We accessed public databases like NOAA Global Forecast System (GFS) and the initial values for the 2-dimensional computational grid were produced by using both DSN-PC measurements and NOAA GFS data for each grid point. However, though the latter consists of assimilated and initialized (smoothed) data the stations of the DSN-PC network provide raw measurements which can cause numerical instability due to measurement errors or local weather phenomena. Previously we simultaneously interpolated both DSN-PC and GFS data. As a step forward, we wanted for our network to have a more significant role in the production of the initial values. Therefore it was necessary to apply 2D smoothing algorithms on the initial conditions. We found significant difference regarding numerical stability between calculating with raw and smoothed initial data. Applying the smoothing algorithms greatly improved the prediction reliability compared to the cases when raw data were used. The size of the grid portion used for smoothing has a significant impact on the goodness of the forecasts and it's worth further investigation. We could verify the viability of direct integration of DSN-PC data since it provided forecast errors similar to the previous approach. In this paper we present one simple method for smoothing our initial data and the results of the weather prediction calculations. [ABSTRACT FROM AUTHOR]
- Published
- 2020
- Full Text
- View/download PDF
25. Wavelet-Based Kalman Smoothing Method for Uncertain Parameters Processing: Applications in Oil Well-Testing Data Denoising and Prediction
- Author
-
Xin Feng, Qiang Feng, Shaohui Li, Xingwei Hou, Mengqiu Zhang, and Shugui Liu
- Subjects
low-distortion processing ,oil well-testing data ,wavelet analysis ,Kalman prediction ,data smoothing ,data compression ,Chemical technology ,TP1-1185 - Abstract
The low-distortion processing of well-testing geological parameters is a key way to provide decision-making support for oil and gas field development. However, the classical processing methods face many problems, such as the stochastic nature of the data, the randomness of initial parameters, poor denoising ability, and the lack of data compression and prediction mechanisms. These problems result in poor real-time predictability of oil operation status and difficulty in offline interpreting the played back data. Given these, we propose a wavelet-based Kalman smoothing method for processing uncertain oil well-testing data. First, we use correlation and reconstruction errors as analysis indicators and determine the optimal combination of decomposition scale and vanishing moments suitable for wavelet analysis of oil data. Second, we build a ground pressure measuring platform and use the pressure gauge equipped with the optimal combination parameters to complete the downhole online wavelet decomposition, filtering, Kalman prediction, and data storage. After the storage data are played back, the optimal Kalman parameters obtained by particle swarm optimization are used to complete the data smoothing for each sample. The experiments compare the signal-to-noise ratio and the root mean square error before and after using different classical processing models. In addition, robustness analysis is added. The proposed method, on the one hand, has the features of decorrelation and compressing data, which provide technical support for real-time uploading of downhole data; on the other hand, it can perform minimal variance unbiased estimates of the data, filter out the interference and noise, reduce the reconstruction error, and make the data have a high resolution and strong robustness.
- Published
- 2020
- Full Text
- View/download PDF
26. Deriving a Process Viscosity for Complex Particulate Nanofibrillar Cellulose Gel-containing Suspensions
- Author
-
Dimic-Misic Katarina, Nieminen Kaarlo, Gane Patrick, Maloney Thaddeus, Sixta Herbert, and Paltakari Jouni
- Subjects
data smoothing ,rheology of gel suspension ,dewatering ,immobilisation ,nanocellulose ,phase separable process ,Materials of engineering and construction. Mechanics of materials ,TA401-492 - Abstract
Phase-separable particulate-containing gel structures constitute complex fluids. In many cases they may incorporate component concentration inhomogeneities within the ensemble matrix. When formulated into high consistency suspensions, these can lead to unpredictable time-dependent variations in rheological response, particularly under shear in simple parallel plate and cylindrical rotational geometries. Smoothing function algorithms are primarily designed to cope with random noise. In the case studied here, namely nanocellulose-based high consistency aqueous suspensions, the system is not randomised but based on a series of parallel and serial spatial and time related mechanisms. These include: phase separation, wall slip, stress relaxation, breakdown of elastic structure and inhomogeneous time-dependent and induced structure rebuild. When vacuum dewatering is applied to such a suspension while under shear, all these effects are accompanied by the development of an uneven solid content gradient within the sample, which further adds to transitional phenomena in the recorded rheological data due to spatial and temporal differences in yield stress distribution. Although these phenomena are strictly speaking not noise, it is nevertheless necessary to apply relevant data smoothing in order to extract apparent/process viscosity parameters in respect to averaging across the structural ensemble. The control parameters in the measurement of the rheological properties, to which smoothing is applied, are focused on parallel plate gap, surface geometry, shear rate, oscillation frequency and strain variation, and relaxation time between successive applications of strain. The smoothing algorithm follows the Tikhonov regularisation procedure.
- Published
- 2014
- Full Text
- View/download PDF
27. Data filtering methods for SARS-CoV-2 wastewater surveillance
- Author
-
Wolfgang Rauch, Rezgar Arabzadeh, Rudolf Markt, Norbert Kreuzinger, Daniel Martin Grünbacher, and Heribert Insam
- Subjects
FOS: Computer and information sciences ,Environmental Engineering ,Computer science ,Context (language use) ,Wastewater ,Disease cluster ,Signal ,Statistics - Applications ,Environmental technology. Sanitary engineering ,Statistics ,pandemic management ,Humans ,Applications (stat.AP) ,data smoothing ,Quantitative Biology - Populations and Evolution ,TD1-1066 ,Water Science and Technology ,Noise (signal processing) ,Generalized additive model ,Populations and Evolution (q-bio.PE) ,COVID-19 ,virus monitoring ,signal filtering ,wastewater-based epidemiology ,Spline (mathematics) ,sars-cov-2 ,Austria ,FOS: Biological sciences ,Smoothing - Abstract
In the case of SARS-CoV-2 pandemic management, wastewater-based epidemiology aims to derive information on the infection dynamics by monitoring virus concentrations in the wastewater. However, due to the intrinsic random fluctuations of the viral signal in the wastewater (due to e.g., dilution; transport and fate processes in sewer system; variation in the number of persons discharging; variations in virus excretion and water consumption per day) the subsequent prevalence analysis may result in misleading conclusions. It is thus helpful to apply data filtering techniques to reduce the noise in the signal. In this paper we investigate 13 smoothing algorithms applied to the virus signals monitored in four wastewater treatment plants in Austria. The parameters of the algorithms have been defined by an optimization procedure aiming for performance metrics. The results are further investigated by means of a cluster analysis. While all algorithms are in principle applicable, SPLINE, Generalized Additive Model and Friedman Super Smoother are recognized as superior methods in this context (with the latter two having a tendency to over-smoothing). A first analysis of the resulting datasets indicates the influence of catchment size for wastewater-based epidemiology as smaller communities both reveal a signal threshold before any relation with infection dynamics is visible and also a higher sensitivity towards infection clusters., Comment: 22 pages 6 Figures
- Published
- 2021
28. The Optimal Threshold and Vegetation Index Time Series for Retrieving Crop Phenology Based on a Modified Dynamic Threshold Method
- Author
-
Xin Huang, Jianhong Liu, Wenquan Zhu, Clement Atzberger, and Qiufeng Liu
- Subjects
crop phenology ,time series ,crop development ,data smoothing ,land surface phenology ,vegetation index ,ndvi ,evi ,Science - Abstract
Crop phenology is an important parameter for crop growth monitoring, yield prediction, and growth simulation. The dynamic threshold method is widely used to retrieve vegetation phenology from remotely sensed vegetation index time series. However, crop growth is not only driven by natural conditions, but also modified through field management activities. Complicated planting patterns, such as multiple cropping, makes the vegetation index dynamics less symmetrical. These impacts are not considered in current approaches for crop phenology retrieval based on the dynamic threshold method. Thus, this paper aimed to (1) investigate the optimal thresholds for retrieving the start of the season (SOS) and the end of the season (EOS) of different crops, and (2) compare the performances of the Normalized Difference Vegetation Index (NDVI) and Enhanced Vegetation Index (EVI) in retrieving crop phenology with a modified version of the dynamic threshold method. The reference data included SOS and EOS ground observations of three major crop types in 2015 and 2016, which includes rice, wheat, and maize. Results show that (1) the modification of the original method ensures a 100% retrieval rate, which was not guaranteed using the original method. The modified dynamic threshold method is more suitable to retrieve crop SOS/EOS because it considers the asymmetry of crop vegetation index time series. (2) It is inappropriate to retrieve SOS and EOS with the same threshold for all crops, and the commonly used 20% or 50% thresholds are not the optimal thresholds for all crops. (3) For single and late rice, the accuracies of the SOS estimations based on EVI are generally higher compared to those based on NDVI. However, for spring maize and summer maize, results based on NDVI give higher accuracies. In terms of EOS, for early rice and summer maize, estimates based on EVI result in higher accuracies, but, for late rice and winter wheat, results based on NDVI are closer to the ground records.
- Published
- 2019
- Full Text
- View/download PDF
29. LiDAR DEM Smoothing and the Preservation of Drainage Features
- Author
-
John B. Lindsay, Anthony Francioni, and Jaclyn M. H. Cockburn
- Subjects
DEM ,LiDAR ,data smoothing ,denoise ,roughness ,micro-topography ,hydrology ,geomorphometry ,streams ,Science - Abstract
Fine-resolution Light Detection and Ranging (LiDAR) data often exhibit excessive surface roughness that can hinder the characterization of topographic shape and the modeling of near-surface flow processes. Digital elevation model (DEM) smoothing methods, commonly low-pass filters, are sometimes applied to LiDAR data to subdue the roughness. These techniques can negatively impact the representation of topographic features, most notably drainage features, such as headwater streams. This paper presents the feature-preserving DEM smoothing (FPDEMS) method, which modifies surface normals to smooth the topographic surface in a similar manner to approaches that were originally designed for de-noising three-dimensional (3D) meshes. The FPDEMS method has been optimized for application with raster DEM data. The method was compared with several low-pass filters while using a 0.5-m resolution LiDAR DEM of an agricultural area in southwestern Ontario, Canada. The findings demonstrated that the technique was better at removing roughness, when compared with mean, median, and Gaussian filters, while also preserving sharp breaks-in-slope and retaining the topographic complexity at broader scales. Optimal smoothing occurred with kernel sizes of 11−21 grid cells, threshold angles of 10°−20°, and 3−15 elevation-update iterations. These parameter settings allowed for the effective reduction in roughness and DEM noise and the retention of terrace scarps, channel banks, gullies, and headwater streams.
- Published
- 2019
- Full Text
- View/download PDF
30. Smooth, hierarchical competitor clustering using agglomerative hierarchical clustering
- Author
-
Botha, Christiaan (author) and Botha, Christiaan (author)
- Abstract
Clustering forms a major part of showing different relations between data points. Real-time clustering algorithms can visualise relationships between elements in a 3D environment, provide an analysis of data that is separate from the underlying structure and show how the data changes over time. This paper analyses whether conventional clustering algorithms can be adapted to real-time dynamic data while remaining stable over time. By implementing an agglomerative hierarchical clustering algorithm combined with an exponential decay smoothing function, this paper tested several different distance functions and compared their resulting clusterings. It then derives a stable distance function for clustering sailboat competitors during a regatta and compared different smoothing values to see the impact on the final result. The paper shows that an adaptively chosen smoothing value provides the best balance between cluster stability and an intuitive visualisation. This paper concludes this solution can be used and adapted to fit a multitude of applications by changing the distance function and the clustering depth., CSE3000 Research Project, Computer Science and Engineering
- Published
- 2022
31. COMPARISON OF METHODS FOR SMOOTHING ENVIRONMENTAL DATA WITH AN APPLICATION TO PARTICULATE MATTER PM10.
- Author
-
Čampulová, Martina
- Subjects
PARTICULATE matter ,STATISTICAL smoothing ,MEAN square algorithms ,PARAMETER estimation ,REGRESSION analysis - Abstract
Data smoothing is often required within the environmental data analysis. A number of methods and algorithms that can be applied for data smoothing have been proposed. This paper gives an overview and compares the performance of different smoothing procedures that estimate the trend in the data, based on the surrounding noisy observations that can be applied on environmental data. The considered methods include kernel regression with both global and local bandwidth, moving average, exponential smoothing, robust repeated median regression, trend filtering and approach based on discrete Fourier and discrete wavelet transform. The methods are applied to real data obtained by measurement of PM
10 concentrations and compared in a simulation study. [ABSTRACT FROM AUTHOR]- Published
- 2018
- Full Text
- View/download PDF
32. Y-Spect: A Multi-Method Gamma Spectrometry Analysis Program
- Author
-
P.I. Yazid
- Subjects
Gamma Spectrometry ,Peak Searching ,Peak Fitting ,Data Smoothing ,Region of Interest ,Escape-/Sum Peak Identification. ,Nuclear engineering. Atomic power ,TK9001-9401 - Abstract
To accomplish a more accurate, precise and correct interpretation and analysis of spectrum data collecting from a gamma spectrometry counting system, a fully interactive computer code, named Y-Spect, has been developed by using the Delphi 7.0 programming language. The code combines several popular methods for peak search, i.e.: Mariscotti, Phillips-Marlow, Robertson et al., Routti-Prussin, Black, Sterlinski, Savitzky-Golay and Block et al. Any combinations of those methods can be chosen during a peak searching process, which can be performed in automatic or manual mode. Moving Window Average- and Savitzky-Golay-methods are available for spectrum data smoothing. Peak fitting is done by using a non-linear least square method of Levenberg-Marquardt for either a pure Gaussian peak shape or one with an additional Right/Left Tail function. Other than standard features, such as: peak identification and determination of: continuum, region of interest (ROI), and peak area, etc., Y-Spect has also a special feature which can predict the existence of escape- and/or sum peaks that belong to a certain radioisotope. Aside from displaying the complete spectrum graph, including: singlet or multiplet ROIs and peak identifications, Y-Spect can also display the first- or second-derivative of the spectrum data. Data evaluation is given as: isotope names, peak energy, Net-Count(-Rate), etc. Y-Spect is provided with a complete ENDF/B-VII.0 gamma-ray library file that contains of 16089 gamma energy lines from 1420 different radioisotopes. Other general specifications are: maximum number of: spectrum's channels = 16*1024; ROIs = 2*1024; ROI’s width = 2*1024 channels; Overlapping peaks (multiplet) = 20; Identified isotopes = 3*1024, and Isotope library's energy lines = 16*1024
- Published
- 2013
33. Principal component analysis-assisted selection of optimal denoising method for oil well transient data
- Author
-
Khafiz Muradov, Akindolu Oluwakanyinsola Dada, and Bing Zhang
- Subjects
Noise reduction ,02 engineering and technology ,Downhole gauge ,010502 geochemistry & geophysics ,Impulse noise ,01 natural sciences ,symbols.namesake ,Wavelet ,020401 chemical engineering ,Data smoothing ,wavelet threshold denoising ,0204 chemical engineering ,Petroleum refining. Petroleum products ,0105 earth and related environmental sciences ,Petrology ,QE420-499 ,Wavelet transform ,White noise ,Geotechnical Engineering and Engineering Geology ,Intelligent well ,Noise ,General Energy ,Additive white Gaussian noise ,Principal component analysis (PCA) ,symbols ,Algorithm ,Smoothing ,TP690-692.5 ,Pressure and temperature transient analysis (PTTA) - Abstract
Oil and gas production wells are often equipped with modern, permanent or temporary in-well monitoring systems, either electronic or fiber-optic, typically for measurement of downhole pressure and temperature. Consequently, novel methods of pressure and temperature transient analysis (PTTA) have emerged in the past two decades, able to interpret subtle thermodynamic effects. Such analysis demands high-quality data. High-level reduction in data noise is often needed in order to ensure sufficient reliability of the PTTA. This paper considers the case of a state-of-the-art intelligent well equipped with fiber-optic, high-precision, permanent downhole gauges. This is followed by screening, development, verification and application of data denoising methods that can overcome the limitation of the existing noise reduction methods. Firstly, the specific types of noise contained in the original data are analyzed by wavelet transform, and the corresponding denoising methods are selected on the basis of the wavelet analysis. Then, the wavelet threshold denoising method is used for the data with white noise and white Gaussian noise, while a data smoothing method is used for the data with impulse noise. The paper further proposes a comprehensive evaluation index as a useful denoising success metrics for optimal selection of the optimal combination of the noise reduction methods. This metrics comprises a weighted combination of the signal-to-noise ratio and smoothness value where the principal component analysis was used to determine the weights. Thus the workflow proposed here can be comprehensively defined solely by the data via its processing and analysis. Finally, the effectiveness of the optimal selection methods is confirmed by the robustness of the PTTA results derived from the de-noised measurements from the above-mentioned oil wells.
- Published
- 2020
34. A hidden Markov space–time model for mapping the dynamics of global access to food
- Author
-
Alessio Farcomeni and Francesco Bartolucci
- Subjects
Statistics and Probability ,Economics and Econometrics ,MCMC ,data augmentation ,data smoothing ,prediction ,Watanabe-Akaike information criterion ,Watanabe–Akaike information criterion ,Statistics, Probability and Uncertainty ,Settore SECS-S/01 ,Social Sciences (miscellaneous) - Abstract
In order to analyse worldwide data about access to food, coming from a series of Gallup’s world polls, we propose a hidden Markov model with both a spatial and a temporal component. This model is estimated by an augmented data MCMC algorithm in a Bayesian framework. Data are referred to a sample of more than 750 thousand individuals in 166 countries, widespread in more than two thousand areas, and cover the period 2007–2014. The model is based on a discrete latent space, with the latent state corresponding to a certain area and time occasion that depends on the states of neighbouring areas at the same time occasion, and on the previous state for the same area. The latent model also accounts for area-time-specific covariates. Moreover, the binary response variable (access to food, in our case) observed at individual level is modelled on the basis of individual-specific covariates through a logistic model with a vector of parameters depending on the latent state. Model selection, in particular for the number of latent states, is based on the Watanabe–Akaike information criterion. The application shows the potential of the approach in terms of clustering the areas, data smoothing and prediction of prevalence for areas without sample units and over time.
- Published
- 2022
35. Estimates of gradient Richardson numbers from vertically smoothed data in the Gulf Stream region
- Author
-
Paul van Gastel and Josep L. Pelegrí
- Subjects
shear mixing ,geostrophic shear ,ageostrophic motions ,richardson number ,data smoothing ,Aquaculture. Fisheries. Angling ,SH1-691 - Abstract
We use several hydrographic and velocity sections crossing the Gulf Stream to examine how the gradient Richardson number, Ri, is modified due to both vertical smoothing of the hydrographic and/or velocity fields and the assumption of parallel or geostrophic flow. Vertical smoothing of the original (25 m interval) velocity field leads to a substantial increase in the Ri mean value, of the same order as the smoothing factor, while its standard deviation remains approximately constant. This contrasts with very minor changes in the distribution of the Ri values due to vertical smoothing of the density field over similar lengths. Mean geostrophic Ri values remain always above the actual unsmoothed Ri values, commonly one to two orders of magnitude larger, but the standard deviation is typically a factor of five larger in geostrophic than in actual Ri values. At high vertical wavenumbers (length scales below 3 m) the geostrophic shear only leads to near critical conditions in already rather mixed regions. At these scales, hence, the major contributor to shear mixing is likely to come from the interaction of the background flow with internal waves. At low vertical wavenumbers (scales above 25 m) the ageostrophic motions provide the main source for shear, with cross-stream movements having a minor but non-negligible contribution. These large-scale motions may be associated with local accelerations taking place during frontogenetic phases of meanders.
- Published
- 2004
- Full Text
- View/download PDF
36. Adaptive penalized splines for data smoothing.
- Author
-
Yang, Lianqiang and Hong, Yongmiao
- Subjects
- *
SPLINES , *STATISTICAL smoothing , *REGRESSION analysis , *HETEROGENEITY , *DATA analysis - Abstract
Data driven adaptive penalized splines are considered via the principle of constrained regression. A locally penalized vector based on the local ranges of the data is generated and added into the penalty matrix of the classical penalized splines, which remarkably improves the local adaptivity of the model for data heterogeneity. The algorithm complexity and simulations are studied. The results show that the adaptive penalized splines outperform the smoothing splines, l 1 trend filtering and classical penalized splines in estimating functions with inhomogeneous smoothness. [ABSTRACT FROM AUTHOR]
- Published
- 2017
- Full Text
- View/download PDF
37. Data Smoothing and Numerical Differentiation by a Regularization Method
- Author
-
Stickel, J.
- Published
- 2010
- Full Text
- View/download PDF
38. Navigation and Obstacle Avoidance System in Unknown Environment
- Author
-
Marwan Dhuheir, Jaspreet Singh, Mohsen Guizani, Aiman Erbad, Amr Mohamed, and Ahmed Refaey
- Subjects
Search and rescue operations ,Data smoothing ,Computer science ,Real-time computing ,Unknown environments ,System modeling ,Aircraft detection ,Obstacle detection ,Ultrasonic applications ,Environment mapping ,Obstacle detectors ,Obstacle-avoidance system ,Obstacle ,Obstacle avoidance ,Savitzky-Golay filter ,Search and rescue ,Reflection mapping ,Drones - Abstract
Recently, drones have been used in many different applications such as search and rescue operations, extinguishing fires, and environment mapping. As the number of moving drones increases in the sky, the collisions risk increases. In this paper, we present a system model, prototype, and preliminary evaluation for UAV obstacle avoidance. The obstacle avoidance system prototype uses ultrasonic sensors for obstacle detection, S-BUS communication protocol for drone control, and Savitzky-Golay filter for data smoothing. 2020 IEEE. Scopus
- Published
- 2020
- Full Text
- View/download PDF
39. Adaptive evolutionary clustering.
- Author
-
Xu, Kevin, Kliger, Mark, and Hero III, Alfred
- Subjects
ADAPTIVE computing systems ,EVOLUTIONARY algorithms ,ROBUST control ,ALGORITHMS ,SMOOTHNESS of functions ,COST functions - Abstract
In many practical applications of clustering, the objects to be clustered evolve over time, and a clustering result is desired at each time step. In such applications, evolutionary clustering typically outperforms traditional static clustering by producing clustering results that reflect long-term trends while being robust to short-term variations. Several evolutionary clustering algorithms have recently been proposed, often by adding a temporal smoothness penalty to the cost function of a static clustering method. In this paper, we introduce a different approach to evolutionary clustering by accurately tracking the time-varying proximities between objects followed by static clustering. We present an evolutionary clustering framework that adaptively estimates the optimal smoothing parameter using shrinkage estimation, a statistical approach that improves a naïve estimate using additional information. The proposed framework can be used to extend a variety of static clustering algorithms, including hierarchical, k-means, and spectral clustering, into evolutionary clustering algorithms. Experiments on synthetic and real data sets indicate that the proposed framework outperforms static clustering and existing evolutionary clustering algorithms in many scenarios. [ABSTRACT FROM AUTHOR]
- Published
- 2014
- Full Text
- View/download PDF
40. An Application of Best L1 Piecewise Monotonic Data Approximation to Signal Restoration.
- Author
-
Demetriou, I. C.
- Subjects
- *
MONOTONIC functions , *DATA analysis , *APPROXIMATION theory , *SIGNAL restoration , *COMPUTER software - Abstract
We consider an application of the best L1 piecewise monotonic data approximation method to univariate signal restoration. We extend numerical examples concerned with the L2 analogous method to the L1 case and we show the efficacy of a relevant software package that implements the method in data fitting and in denoising data from a medical image. The piecewise monotonic approximation method makes the smallest change to the data such that the first differences of the smoothed data change sign a prescribed number of times. Our results exhibit some strengths and certain advantages of the method. Therefore, they may be helpful to the development of new algorithms that are suitable to signal restoration calculations. [ABSTRACT FROM AUTHOR]
- Published
- 2013
41. Reconciling continuum and non-continuum data with industrial application
- Author
-
Gonzalez, Ruben, Huang, Biao, Xu, Fangwei, Espejo, Aris, Amalraj, Joseph, and Lam, William
- Subjects
- *
MATHEMATICAL continuum , *INDUSTRIAL applications , *PRINCIPAL components analysis , *PERFORMANCE evaluation , *DISTRIBUTION (Probability theory) , *DIGITAL filters (Mathematics) , *CASE studies - Abstract
Abstract: In order to perform data reconciliation, it is important that noises in the data have well-defined distributions. The motivation behind this study was to enable the comparison between a discrete and continuous data set so that means can be compared for gross error over the short term; this required that local variables exhibit similar distributions. A case study was done on a system where non-continuum loads from a dump truck were to be reconciled with two downstream continuum weightometers. An algorithm was developed using the binomial distribution and time delay in order to simulate the effect of the dump pocket. Regression analysis based on principal components was used to evaluate the performance of the smoothing algorithm and to determine the most likely maximum hopper capacity that locates between the two weightometers. [Copyright &y& Elsevier]
- Published
- 2011
- Full Text
- View/download PDF
42. Data Smoothing and Interpolation Using Eighth-order Algebraic Splines.
- Author
-
Simon, Dan
- Subjects
- *
IMAGE processing , *SPLINES , *DIGITAL filters (Mathematics) , *EQUATIONS , *MATHEMATICAL optimization , *BANDWIDTHS - Abstract
A new type of algebraic spline is used to derive a filter for smoothing or interpolating discrete data points. The spline is dependent on control parameters that specify the relative importance of data fitting and the derivatives of the spline. A general spline of arbitrary order is first formulated using matrix equations. We then focus on eighth-order splines because of the continuity of their first three derivatives (desirable for motor and robotics applications). The spline's matrix equations are rewritten to give a recursive filter that can be implemented in real time for lengthy data sequences. The filter is lowpass with a bandwidth that is dependent on the spline's control parameters. Numerical results, including a simple image processing application, show the tradeoffs that can be achieved using the algebraic splines. [ABSTRACT FROM AUTHOR]
- Published
- 2004
- Full Text
- View/download PDF
43. A theorem for piecewise convex–concave data approximation
- Author
-
Demetriou, I.C.
- Subjects
- *
CONVEX functions , *SET theory , *APPROXIMATION theory - Abstract
We are given univariate data that include random errors. We consider the problem of calculating a best approximation to the data by minimizing a strictly convex function of the errors subject to the condition that there are at most
q sign changes in the sequence of the second divided differences of the approximation, whereq is a prescribed integer. There are aboutO(nq) combinations of positions of sign changes, which make an exhaustive approach prohibitively expensive. However, Demetriou and Powell (Approximation Theory and Optimization, Cambridge University Press, Cambridge, 1997, pp. 109–132), have proved the remarkable property that there exists a partitioning of the data into(q+1) disjoint subsets such that the approximation may be calculated by a separate convex programming calculation on each subset. Based on this result, we provide a characterization theorem that reduces the problem to an equivalent one, where the unknowns are the positions of the sign changes subject to feasibility restrictions at the sign changes. Furthermore, we present counterexamples on two conjectures that investigate whether the search for optimal sign changes may be restricted to certain subranges of the data. [Copyright &y& Elsevier]- Published
- 2004
- Full Text
- View/download PDF
44. Functional QTL mapping and genomic prediction of canopy height in wheat measured using a robotic field phenotyping platform
- Author
-
Luzie U. Wingen, Pouria Sadeghi-Tehran, Gancho T. Slavov, Nicolas Virlet, Simon Griffiths, Kirsty L. Hassall, Simon Orford, Danilo Hottis Lyra, and Malcolm J. Hawkesford
- Subjects
0106 biological sciences ,0301 basic medicine ,Canopy ,Data smoothing ,Physiology ,function-valued traits ,factor-analytic model ,Plant Science ,Computational biology ,Dynamic QTLs ,Biology ,Quantitative trait locus ,01 natural sciences ,Statistical power ,Field (computer science) ,genomic selection ,Factor-analytic model ,03 medical and health sciences ,Phenomics ,Robotic Surgical Procedures ,Humans ,Triticum ,dimensionality reduction ,Genomic selection ,AcademicSubjects/SCI01210 ,dynamic QTLs ,Chromosome Mapping ,Contrast (statistics) ,food and beverages ,phenomics ,Genomics ,Function-valued traits ,Research Papers ,Dimensionality reduction ,Phenotype ,030104 developmental biology ,Growth and Development ,Predictive modelling ,Smoothing ,010606 plant biology & botany - Abstract
Functional analysis of longitudinal phenotypic data for five and 10 time points saturates QTL detection power and genomic predictive ability for canopy height in wheat., Genetic studies increasingly rely on high-throughput phenotyping, but the resulting longitudinal data pose analytical challenges. We used canopy height data from an automated field phenotyping platform to compare several approaches to scanning for quantitative trait loci (QTLs) and performing genomic prediction in a wheat recombinant inbred line mapping population based on up to 26 sampled time points (TPs). We detected four persistent QTLs (i.e. expressed for most of the growing season), with both empirical and simulation analyses demonstrating superior statistical power of detecting such QTLs through functional mapping approaches compared with conventional individual TP analyses. In contrast, even very simple individual TP approaches (e.g. interval mapping) had superior detection power for transient QTLs (i.e. expressed during very short periods). Using spline-smoothed phenotypic data resulted in improved genomic predictive abilities (5–8% higher than individual TP prediction), while the effect of including significant QTLs in prediction models was relatively minor (
- Published
- 2020
45. Univariate and Bivariate Loglinear Models for Discrete Test Score Distributions.
- Author
-
Holland, Paul W. and Thayer, Dorothy T.
- Abstract
The well-developed theory of exponential families of distributions is applied to the problem of fitting the univariate histograms and discrete bivariate frequency distributions that often arise in the analysis of test scores. These models are powerful tools for many forms of parametric data smoothing and are particularly well-suited to problems in which there is little or no theory to guide a choice of probability models, e.g., smoothing a distribution to eliminate roughness and zero frequencies in order to equate scores from different tests. Attention is given to efficient computation of the maximum likelihood estimates of the parameters using Newton's Method and to computationally efficient methods for obtaining the asymptotic standard errors of the fitted frequencies and proportions. We discuss tools that can be used to diagnose the quality of the fitted frequencies for both the univariate and the bivariate cases. Five examples, using real data, are used to illustrate the methods of this paper. [ABSTRACT FROM PUBLISHER]
- Published
- 2000
- Full Text
- View/download PDF
46. The Optimal Threshold and Vegetation Index Time Series for Retrieving Crop Phenology Based on a Modified Dynamic Threshold Method
- Author
-
Clement Atzberger, Wenquan Zhu, Xin Huang, Jianhong Liu, and Qiufeng Liu
- Subjects
010504 meteorology & atmospheric sciences ,Science ,Reference data (financial markets) ,0211 other engineering and technologies ,02 engineering and technology ,evi ,Multiple cropping ,crop development ,01 natural sciences ,Normalized Difference Vegetation Index ,crop phenology ,time series ,data smoothing ,land surface phenology ,vegetation index ,NDVI ,EVI ,Crop ,Yield (wine) ,Statistics ,021101 geological & geomatics engineering ,0105 earth and related environmental sciences ,Mathematics ,Sowing ,Enhanced vegetation index ,ndvi ,General Earth and Planetary Sciences ,Smoothing - Abstract
Crop phenology is an important parameter for crop growth monitoring, yield prediction, and growth simulation. The dynamic threshold method is widely used to retrieve vegetation phenology from remotely sensed vegetation index time series. However, crop growth is not only driven by natural conditions, but also modified through field management activities. Complicated planting patterns, such as multiple cropping, makes the vegetation index dynamics less symmetrical. These impacts are not considered in current approaches for crop phenology retrieval based on the dynamic threshold method. Thus, this paper aimed to (1) investigate the optimal thresholds for retrieving the start of the season (SOS) and the end of the season (EOS) of different crops, and (2) compare the performances of the Normalized Difference Vegetation Index (NDVI) and Enhanced Vegetation Index (EVI) in retrieving crop phenology with a modified version of the dynamic threshold method. The reference data included SOS and EOS ground observations of three major crop types in 2015 and 2016, which includes rice, wheat, and maize. Results show that (1) the modification of the original method ensures a 100% retrieval rate, which was not guaranteed using the original method. The modified dynamic threshold method is more suitable to retrieve crop SOS/EOS because it considers the asymmetry of crop vegetation index time series. (2) It is inappropriate to retrieve SOS and EOS with the same threshold for all crops, and the commonly used 20% or 50% thresholds are not the optimal thresholds for all crops. (3) For single and late rice, the accuracies of the SOS estimations based on EVI are generally higher compared to those based on NDVI. However, for spring maize and summer maize, results based on NDVI give higher accuracies. In terms of EOS, for early rice and summer maize, estimates based on EVI result in higher accuracies, but, for late rice and winter wheat, results based on NDVI are closer to the ground records.
- Published
- 2019
47. Outlier Detection and Smoothing Process for Water Level Data Measured by Ultrasonic Sensor in Stream Flows
- Author
-
Inhyeok Bae and Un Ji
- Subjects
lcsh:Hydraulic engineering ,modified Z-scores ,exponentially weighted moving average ,Computer science ,0208 environmental biotechnology ,Geography, Planning and Development ,Robust statistics ,02 engineering and technology ,Aquatic Science ,outlier detection ,01 natural sciences ,Biochemistry ,lcsh:Water supply for domestic and industrial purposes ,ultrasonic sensor ,lcsh:TC1-978 ,Range (statistics) ,data smoothing ,Median absolute deviation ,EWMA chart ,Water Science and Technology ,lcsh:TD201-500 ,business.industry ,010401 analytical chemistry ,Pattern recognition ,020801 environmental engineering ,0104 chemical sciences ,Outlier ,median absolute deviation ,Anomaly detection ,Ultrasonic sensor ,Artificial intelligence ,business ,Smoothing ,water level monitoring - Abstract
Water level data sets acquired by ultrasonic sensors in stream-scale channels exhibit relatively large numbers of outliers that are off the measurement range between the ultrasonic sensor and water surface, as well as data dispersion of approximately 2 cm due to random errors such as water waves. Therefore, this study develops a data processing algorithm for outlier removal and smoothing for water level data measured by ultrasonic sensors to consider these characteristics. The outlier removal process includes an initial cutoff process to remove outliers out of the measurement range and an outlier detection process using modified Z-scores based on the median absolute deviation (MAD) of a robust estimator. In addition, an exponentially weighted moving average (EWMA) method is applied to smooth the processed data. Sensitivity analyses are performed for factors that are subjectively set by the user, including the window size for the MAD outlier detection stage, the rejection criterion for the modified Z-score outlier removal stage, and the smoothing constant for the EWMA smoothing stage, based on four different water level data sets acquired by ultrasonic sensors in stream-scale experiments.
- Published
- 2019
- Full Text
- View/download PDF
48. The influence of alternative data smoothing prediction techniques on the performance of a two-stage short-term urban travel time prediction framework
- Author
-
Fangce Guo, John W. Polak, Rajesh Krishnan, and Engineering & Physical Science Research Council (EPSRC)
- Subjects
Technology ,Engineering ,TRAFFIC VOLUME ,Joint influence ,1507 Transportation And Freight Services ,Aerospace Engineering ,Transportation ,machine learning method ,02 engineering and technology ,computer.software_genre ,Traffic prediction ,RANDOM FORESTS ,0102 Applied Mathematics ,Prediction methods ,0502 economics and business ,0202 electrical engineering, electronic engineering, information engineering ,data smoothing ,NONPARAMETRIC REGRESSION ,intelligent transportation systems (ITS) ,Intelligent transportation system ,050210 logistics & transportation ,Science & Technology ,business.industry ,Applied Mathematics ,Transportation Science & Technology ,Logistics & Transportation ,05 social sciences ,short-term traffic prediction ,TRANSFORM ,WAVELET ,Computer Science Applications ,Term (time) ,MODEL ,Travel time ,Control and Systems Engineering ,Automotive Engineering ,020201 artificial intelligence & image processing ,Stage (hydrology) ,Data mining ,business ,computer ,Software ,Smoothing ,Information Systems - Abstract
This article investigates the impact of alternative data smoothing and traffic prediction methods on the accuracy of the performance of a two-stage short-term urban travel time prediction framework. Using this framework, we test the influence of the combination of two different data smoothing and four different prediction methods using travel time data from two substantially different urban traffic environments and under both normal and abnormal conditions. This constitutes the most comprehensive empirical evaluation of the joint influence of smoothing and predictor choice to date. The results indicate that the use of data smoothing improves prediction accuracy regardless of the prediction method used and that this is true in different traffic environments and during both normal and abnormal (incident) conditions. Moreover, the use of data smoothing in general has a much greater influence on prediction performance than the choice of specific prediction method, and this is independent of the specific smoothing method used. In normal traffic conditions, the different prediction methods produce broadly similar results but under abnormal conditions, lazy learning methods emerge as superior.
- Published
- 2017
- Full Text
- View/download PDF
49. Data Smoothing: Prediction of Human Behavior, Detection of Behavioral Patterns, and Monitoring Treatment Effectiveness in Single-Subject Behavioral Studies.
- Author
-
Sideridis, Georgios
- Abstract
Data-smoothing can be particularly useful in predicting human behavior, detecting behavioral patterns, and monitoring treatment effectiveness in highly variable single-subject behavioral experiments that cannot be determined by only visual inspection of their graphs. Using an example from the applied behavior analytic literature, the use of moving-average and exponential data-smoothing aided the detection of the unique behavioral patterns of a child with autism across different treatments. Furthermore, the utility of the data-smoothing procedures to monitor and control the effectiveness of an intervention is discussed. The ease of their calculations suggest use of data-smoothing by behavior analysts whenever the effects of particular interventions are questionable. [ABSTRACT FROM AUTHOR]
- Published
- 1997
- Full Text
- View/download PDF
50. Contour Extraction Based on Adaptive Thresholding in Sonar Images.
- Author
-
Andreatos, Antonios and Leros, Apostolos
- Subjects
- *
SONAR imaging , *THRESHOLDING algorithms , *AUTOMATIC target recognition , *STATISTICAL smoothing , *ACOUSTIC imaging , *CURVE fitting - Abstract
A common problem in underwater side-scan sonar images is the acoustic shadow generated by the beam. Apart from that, there are a number of reasons impairing image quality. In this paper, an innovative algorithm with two alternative histogram approximation methods is presented. Histogram approximation is based on automatically estimating the optimal threshold for converting the original gray scale images into binary images. The proposed algorithm clears the shadows and masks most of the impairments in side-scan sonar images. The idea is to select a proper threshold towards the rightmost local minimum of the histogram, i.e., closest to the white values. For this purpose, the histogram envelope is approximated by two alternative contour extraction methods: polynomial curve fitting and data smoothing. Experimental results indicate that the proposed algorithm produces superior results than popular thresholding methods and common edge detection filters, even after corrosion expansion. The algorithm is simple, robust and adaptive and can be used in automatic target recognition, classification and storage in large-scale multimedia databases. [ABSTRACT FROM AUTHOR]
- Published
- 2021
- Full Text
- View/download PDF
Catalog
Discovery Service for Jio Institute Digital Library
For full access to our library's resources, please sign in.