Search

In this paper, we provide a geometric analysis of a new hysteresis model that is based upon singular perturbations. Here hysteresis refers to a type of regularization of piecewise smooth differential equations where the past of a trajectory, in a small neighborhood of the discontinuity set, determines the vector-field at present. In fact, in the limit where the neighborhood of the discontinuity vanishes, hysteresis converges in an appropriate sense to Filippov’s sliding vector-field. Recently (2022), however, Bonet and Seara showed that hysteresis, in contrast to regularization through smoothing, leads to chaos in the regularization of grazing bifurcations, even in two dimensions. The hysteresis model we analyze in the present paper—which was developed by Bonet et al in a paper from 2017 as an attempt to unify different regularizations of piecewise smooth systems—involves two singular perturbation parameters and includes a combination of slow–fast and nonsmooth effects. The description of this model is therefore—from the perspective of singular perturbation theory—challenging, even in two dimensions. Using blowup as our main technical tool, we prove existence of an invariant cylinder carrying fast dynamics in the azimuthal direction and a slow drift in the axial direction. We find that the slow drift is given by Filippov’s sliding vector-field to leading order. Moreover, in the case of grazing, we identify two important parameter regimes that relate the model to smoothing (through a saddle-node bifurcation of limit cycles) and hysteresis (through chaotic dynamics, due to a folded saddle and a novel return mechanism).

Inferring relations from correlational data allows researchers across the sciences to uncover complex connections between variables for insights into the underlying mechanisms. The researchers often represent inferred relations using Gaussian graphical models, requiring regularization to sparsify the models. Acknowledging that the modular structure of these inferred networks is often studied, we suggest module-based regularization to balance under- and overfitting. Compared with the graphical lasso, a standard approach using the Gaussian log-likelihood for estimating the regularization strength, this approach better recovers and infers modular structure in noisy synthetic and real data. The module-based regularization technique improves the usefulness of Gaussian graphical models in the many applications where they are employed.

As the amount of high dimensional data becomes increasingly accessible and common, the need for reliable methods to combat problems such as overfitting and multicollinearity increases. Models need to be able to manage large data sets where predictor variables often outnumber the amount of observations. In this study the frequentist and Bayesian framework is tested against each other based on three different simulated situations. One where the amount of predictor variables greatly outnumber the observations, one where the simulated data has a high correlation between variables and one where a situation is created where the coefficients to be estimated are known beforehand. This enables comparisons between true values and estimated values. Three different approaches are used from both of the statistical frameworks. The frequentist models consist of Ridge regression, least absolute shrinkage and selection operator (LASSO) regression as well as the combined model Elastic net regression. The Bayesian models consist of three regressions with different prior beliefs regarding the coefficients’ probability distributions. The Normal distribution, the Cauchy distribution and the Horseshoe distribution were chosen in this thesis. To compare the different frameworks, different loss functions have been used such as predictability on new data, amount of explained variance and the amount of unnecessary predictor variables the model successfully regularizes. The results of the study show that the Bayesian Horseshoe model has the greatest overall performance regarding predictability, variable selection and parameter estimation. The LASSO regres- sion performs better variable selection on highly correlated data than all of the other models. The frequentist models are also more easily computed if compu- tational power or time is a limited resource, in the other cases the Horseshoe model is to prefer.

In this work, an advanced 2D nonparametric correlogram method is presented to cope with output-only measurements of linear (slow) time-varying systems. The proposed method is a novel generalization of the kernel function-based regularization techniques that have been developed for estimating linear time-invariant impulse response functions. In the proposed system identification technique, an estimation method is provided that can estimate the time-varying auto- and cross-correlation function and indirectly, the time-varying auto- and cross-correlation power spectrum estimates based on real-life measurements without measuring the perturbation signals. The (slow) time-varying behavior means that the dynamic of the system changes as a function of time. In this work, a tailored regularization cost function is considered to impose assumptions such as smoothness and stability on the 2D auto- and cross-correlation function resulting in robust and uniquely determined estimates. The proposed method is validated on two examples: a simulation to check the numerical correctness of the method, and a flutter test measurement of a scaled airplane model to illustrate the power of the method on a real-life challenging problem., SCOPUS: ar.j, info:eu-repo/semantics/published

Sampling in control applications is increasingly done non-equidistantly in time. This includes applications in motion control, networked control, resource-aware control, and event-based control. Some of these applications, like the ones where displacement is tracked using incremental encoders, are driven by signals that are only measured when their values cross fixed thresholds in the amplitude domain. This paper introduces a non-parametric estimator of the impulse response and transfer function of continuous-time systems based on such amplitude-equidistant sampling strategy, known as Lebesgue sampling. To this end, kernel methods are developed to formulate an algorithm that adequately takes into account the bounded output uncertainty between the event timestamps, which ultimately leads to more accurate models and more efficient output sampling compared to the equidistantly-sampled kernel-based approach. The efficacy of our proposed method is demonstrated through a mass–spring damper example with encoder measurements and extensive Monte Carlo simulation studies on system benchmarks., Team Jan-Willem van Wingerden

Sampling in control applications is increasingly done non-equidistantly in time. This includes applications in motion control, networked control, resource-aware control, and event-based control. Some of these applications, like the ones where displacement is tracked using incremental encoders, are driven by signals that are only measured when their values cross fixed thresholds in the amplitude domain. This paper introduces a non-parametric estimator of the impulse response and transfer function of continuous-time systems based on such amplitude-equidistant sampling strategy, known as Lebesgue sampling. To this end, kernel methods are developed to formulate an algorithm that adequately takes into account the bounded output uncertainty between the event timestamps, which ultimately leads to more accurate models and more efficient output sampling compared to the equidistantly-sampled kernel-based approach. The efficacy of our proposed method is demonstrated through a mass–spring damper example with encoder measurements and extensive Monte Carlo simulation studies on system benchmarks.

Masked Autoencoders (MAEs) represent a significant shift in self-supervised learning (SSL) due to their independence from augmentation techniques for generating positive (and/or negative) pairs as in contrastive frameworks. Their masking and reconstruction strategy also aligns well with SSL approaches in natural language processing. Most MAEs are built upon Transformer-based architectures where visual features are not regularized as opposed to their convolutional neural network (CNN) based counterparts, which can potentially limit their effectiveness. To address this, we introduce a novel batch-wide layer-wise regularization loss applied to representations of different Transformer layers. We demonstrate that by plugging in the proposed regularization loss, one can significantly improve the performance of MAE-based baselines., Computer Science

Inferring relations from correlational data allows researchers across the sciences to uncover complex connections between variables for insights into the underlying mechanisms. The researchers often represent inferred relations using Gaussian graphical models, requiring regularization to sparsify the models. Acknowledging that the modular structure of these inferred networks is often studied, we suggest module-based regularization to balance under- and overfitting. Compared with the graphical lasso, a standard approach using the Gaussian log-likelihood for estimating the regularization strength, this approach better recovers and infers modular structure in noisy synthetic and real data. The module-based regularization technique improves the usefulness of Gaussian graphical models in the many applications where they are employed.

The tissue electrical properties of conductivity and permittivity affect the interactions of electromagnetic fields in the body. These properties vary throughout the different tissues as the tissue structure and composition varies. In this thesis, medical imaging and diagnosis is used as primary example to motivate exploration of a novel regularization approach to an MRI-based electrical properties tomography (EPT) method. Total variation (TV) regularization has been shown to perform noise reduction in the iterative Contrast Source Inversion EPT (CSI-EPT) method. The Jacobi matrix inversion regularization, an alternative to the known conjugate gradient formulation, is elaborated and applied to an E-polarized MRI fields scenario such that this thesis presents the Jacobi step regularized CSI-EPT. The alternative regularization method outperforms the known regularization method in the reconstruction qualities of noise-suppression and edge-preservation in the simulated MRI experiments using a virtual body model. Further advancements are also described, such as multiple inner-iterations Jacobi regularization and an anatomical prior initialization of the contrast function. Important future research topics are the incorporation and evaluation of the Jacobi step regularization into more advanced CSI-EPT versions, which are the three-dimensional and transceive phase based algorithms to correct realistic MRI data., Electrical Engineering | Signals and Systems

Fast-ion loss detectors (FILDs) are crucial for analyzing fast-ion dynamics in magnetically confined fusion plasmas. A core challenge is to derive an accurate ion velocity distribution, requiring treatment of thousands of remapped camera frames for a full discharge. The ill-posed nature of this task necessitates regularization with a well-chosen regularization parameter and computationally efficient methods. In this work, we introduce the ‘resolution principle,’ a novel criterion for selecting the optimal regularization parameter, providing a distinction between genuine features and artefacts smaller than the diagnostic resolution in the reconstruction, thereby preventing misinterpretations. This principle, coupled with three iterative reconstruction techniques—Kaczmarz’s method, coordinate descent, and Cimmino’s method—demonstrates enhanced reconstruction capabilities compared to conventional methods like Tikhonov regularization. Utilizing these techniques allows rapid processing of measurements from full discharges, removing the computational bottleneck and facilitating between-discharge reconstructions. By reconstructing 6000 camera frames from an ELMy H-mode discharge at ASDEX Upgrade, we capture the temporal evolution of gyroradii and pitch angles, unveiling a direct correlation between pitch-angle behavior and changes in the toroidal magnetic field for a specific subset of lost ions accelerated by edge-localized modes (ELMs) to energies approximately twice that of the injection energy.

We introduce Sauron, a filter pruning method that eliminates redundant feature maps of convolutional neural networks (CNNs). Sauron optimizes, jointly with the loss function, a regularization term that promotes feature maps clustering at each convolutional layer by reducing the distance between feature maps. Sauron then eliminates the filters corresponding to the redundant feature maps by using automatically adjusted layer-specific thresholds. Unlike most filter pruning methods, Sauron requires minimal changes to typical neural network optimization because it prunes and optimizes CNNs jointly, which, in turn, accelerates the optimization over time. Moreover, unlike with other cluster-based approaches, the user does not need to specify the number of clusters in advance, a hyperparameter that is difficult to tune. We evaluated Sauron and five state-of-the-art filter pruning methods on four medical image segmentation tasks. This is an area where little attention has been paid to filter pruning, but where smaller CNN models are desirable for local deployment, mitigating privacy concerns associated with cloud-based solutions. Sauron was the only method that achieved a reduction in model size of over 90% without deteriorating substantially the performance. Sauron also achieved, overall, the fastest models at inference time in machines with and without GPUs. Finally, we show through experiments that the feature maps of models pruned with Sauron are highly interpretable, which is essential for medical image segmentation.

Important advances have recently been achieved in developing procedures yielding uniformly valid inference for a low dimensional causal parameter when high-dimensional nuisance models must be estimated. In this paper, we review the literature on uniformly valid causal inference and discuss the costs and benefits of using uniformly valid inference procedures. Naive estimation strategies based on regularisation, machine learning, or a preliminary model selection stage for the nuisance models have finite sample distributions which are badly approximated by their asymptotic distributions. To solve this serious problem, estimators which converge uniformly in distribution over a class of data generating mechanisms have been proposed in the literature. In order to obtain uniformly valid results in high-dimensional situations, sparsity conditions for the nuisance models need typically to be made, although a double robustness property holds, whereby if one of the nuisance model is more sparse, the other nuisance model is allowed to be less sparse. While uniformly valid inference is a highly desirable property, uniformly valid procedures pay a high price in terms of inflated variability. Our discussion of this dilemma is illustrated by the study of a double-selection outcome regression estimator, which we show is uniformly asymptotically unbiased, but is less variable than uniformly valid estimators in the numerical experiments conducted., Originally included in thesis in manuscript form.

Complex diseases like neurodegenerative or cancer disorders are characterized by deregulations in multiple genes and proteins. Previous research has shown that neighboring genes in a molecular network tend to undergo coordinated expression changes. We describe an approach that allows identifying such jointly differentially expressed genes from input expression data and a graph encoding pairwise functional associations between genes (such as protein interactions). We cast this as a feature selection problem in penalized two-class (cases vs. controls) classification, and we propose a novel pairwise elastic net (PEN) penalty that favors the selection of discriminative genes according to their connectedness in the interaction graph. Experiments on large-scale gene expression data for Parkinson’s disease demonstrate marked improvements in feature grouping over competitive methods.

Inverse problems arise in a wide variety of applications including biomedicine, environmental sciences, astronomy, and more. Computing reliable solutions to these problems requires the inclusion of prior knowledge in a process that is often referred to as regularization. Most regularization techniques require suitable choices of regularization parameters. In this work, we will describe new approaches that use deep neural networks (DNN) to estimate these regularization parameters. We will train multiple networks to approximate mappings from observation data to individual regularization parameters in a supervised learning approach. Once the networks are trained, we can efficiently compute regularization parameters for newly-obtained data by forward propagation through the DNNs. The network-obtained regularization parameters can be computed more efficiently and may even lead to more accurate solutions compared to existing regularization parameter selection methods. Numerical results for tomography demonstrate the potential benefits of using DNNs to learn regularization parameters.

Machine learning models are typically configured by minimizing the training error over a given training dataset. On the other hand, the main objective is to obtain models that can generalize, i.e., perform well on data unseen during training. A fundamental challenge is that a low training error does not guarantee a low generalization error. While the classical wisdom from statistical learning theory states that models which perfectly fit the training data are unlikely to generalize, recent empirical results show that massively overparameterized models can defy this notion, leading to double-descent curves. This thesis investigates this phenomenon and characterizes the generalization performance of linear models across various scenarios. The first part of this thesis focuses on characterizing the generalization performance as a function of the level of model mismatch between the patterns in the data and the assumed model. Model mismatch is undesirable but often present in practice. We reveal the trade-offs between the number of samples, the number of features, and the possibly incorrect statistical assumptions on the assumed model. We show that fake features, i.e., features present in the model but not in the data, can significantly improve the generalization performance even though they are not correlated with the features in the underlying system. The second part of this thesis focuses on generalization under distributed learning. We focus on the scenario where the model parameters are distributed over a network of learners. We consider two settings: the single-task setting, where the network learns a single task, and the continual learning setting, where the network learns multiple tasks sequentially. The obtained results show that for a wide family of feature probability distributions, the generalization performance heavily depends on how the model parameters are partitioned over the network. In particular, if the number of parameters per learner and the number o

Copyright © 1992 Published by Elsevier B.V. The authors are grateful to Professors G. Leibbrandt and J.C. Taylor for valuable conversations and encouragement. GG was supported by the Italian Consiglio Nazionale delle Ricerche, CPM by the National Science and Engineering Research Council of Canada under grant No. A-8063, and FRR by The Commission of the European Communities through contract No. SC1000488 with The Niels Bohr Institute. Their work also benefited from partial support from CICYT, Spain., We study quantum Chern-Simons theory as the large-mass limit of the limit D --> 3 of dimensionally regularized topologically massive Yang-Mills theory. This approach can also be interpreted as a BRS-invariant hybrid regularization of Chern-Simons theory, consisting of a higher-covariant derivative Yang-Mills term plus dimensional regularization. Working in the Landau gauge, we compute radiative corrections up to second order in perturbation theory and show that there is no two-loop correction to the one-loop shift k --> k + c(v), k being the bare Chern-Simons parameter. In passing, we also prove by explicit computation that topologically massive Yang Mills theory is UV finite., Italian Consiglio Nazionale delle Ricerche, National Science and Engineering Research Council of Canada, Commission of the European Communities, CICYT, Spain, Niels Bohr Institute, Depto. de Física Teórica, Fac. de Ciencias Físicas, TRUE, pub

In the last decade, significant progress in sensing and data storage technologies has ushered in a new era of spatiotemporal data analysis. This progress has dramatically increased the availability and scale of spatiotemporal data, presenting both exciting opportunities and complex challenges in this field. Spatiotemporal data from various sources exhibit unique patterns, but they share two common characteristics: (i) anomalies in spatiotemporal data are typically sparse, meaning that the number of anomalies is significantly less than the number of normal data, and (ii) anomalies cause deviations from the normal patterns of the data. To illustrate the practical application of these principles, consider a water distribution system (WDS). A sudden burst in the system can lead to a substantial drop in water pressure compared to normal conditions. To efficiently detect such anomalies, a penalized regression model based on basis expansion is introduced. This model effectively captures the features of hydraulic measurements through basis coefficients. It encourages sparsity in the anomalies (such as bursts) by applying an $L_1$ penalty. Furthermore, it encourages normal measurements to align closely with the sample mean through an $L_2$ regularization term. The model is solved using an optimization algorithm, and its performance is evaluated through a simulated case study. In the context of additive manufacturing, where images capture the manufacturing process, the profile of objects being produced must be extracted accurately. This is particularly challenging due to variations in pixel intensities between the cured profile and the background, which are caused by differences in optical properties. To address this, a tensor decomposition-based method is introduced. Difference matrices are employed to penalize variations in pixel intensities both vertically and horizontally, promoting smoothness in the background. Simultaneously, an $L_1$ regularization term enforces sparsi

Regularization is a common tool in variational inverse problems to impose assumptions on the parameters of the problem. One such assumption is sparsity,which is commonly promoted using lasso and total variation-like regularization.Although the solutions to many such regularized inverse problems can be considered as points of maximum probability of well-chosen posterior distributions, samples from these distributions are generally not sparse. Int his paper, we present a framework for implicitly defining a probability distribution that combines the effects of sparsity imposing regularization with Gaussian distributions. Unlike continuous distributions, these implicit distributions can assign positive probability to sparse vectors. We study these regularized distributions for various regularization functions including total variation regularization and piecewise linear convex functions. We apply the developed theory to uncertainty quantification for Bayesian linear inverse problems and derive a Gibbs sampler for a Bayesian hierarchical model. To illustrate the difference between our sparsity-inducing framework and continuous distributions, we apply our framework to small-scale deblurring and computed tomography examples.

When meta-analyzing heterogeneous bodies of literature, meta-regression can be used to account for potentially relevant between-studies differences. A key challenge is that the number of candidate moderators is often high relative to the number of studies. This introduces risks of overfitting, spurious results, and model non-convergence. To overcome these challenges, we introduce Bayesian Regularized Meta-Analysis (BRMA), which selects relevant moderators from a larger set of candidates by shrinking small regression coefficients towards zero with regularizing (LASSO or horseshoe) priors. This method is suitable when there are many potential moderators, but it is not known beforehand which of them are relevant. A simulation study compared BRMA against state-of-the-art random effects meta-regression using restricted maximum likelihood (RMA). Results indicated that BRMA outperformed RMA on three metrics: BRMA had superior predictive performance, which means that the results generalized better; BRMA was better at rejecting irrelevant moderators, and worse at detecting true effects of relevant moderators, while the overall proportion of Type I and Type II errors was equivalent to RMA. BRMA regression coefficients were slightly biased towards zero (by design), but its residual heterogeneity estimates were less biased than those of RMA. BRMA performed well with as few as 20 studies, suggesting its suitability as a small sample solution. We present free open source software implementations in the R-package pema (for penalized meta-analysis) and in the stand-alone statistical program JASP. An applied example demonstrates the use of the R-package.

Machine learning models are typically configured by minimizing the training error over a given training dataset. On the other hand, the main objective is to obtain models that can generalize, i.e., perform well on data unseen during training. A fundamental challenge is that a low training error does not guarantee a low generalization error. While the classical wisdom from statistical learning theory states that models which perfectly fit the training data are unlikely to generalize, recent empirical results show that massively overparameterized models can defy this notion, leading to double-descent curves. This thesis investigates this phenomenon and characterizes the generalization performance of linear models across various scenarios. The first part of this thesis focuses on characterizing the generalization performance as a function of the level of model mismatch between the patterns in the data and the assumed model. Model mismatch is undesirable but often present in practice. We reveal the trade-offs between the number of samples, the number of features, and the possibly incorrect statistical assumptions on the assumed model. We show that fake features, i.e., features present in the model but not in the data, can significantly improve the generalization performance even though they are not correlated with the features in the underlying system. The second part of this thesis focuses on generalization under distributed learning. We focus on the scenario where the model parameters are distributed over a network of learners. We consider two settings: the single-task setting, where the network learns a single task, and the continual learning setting, where the network learns multiple tasks sequentially. The obtained results show that for a wide family of feature probability distributions, the generalization performance heavily depends on how the model parameters are partitioned over the network. In particular, if the number of parameters per learner and the number o

This paper presents a novel method for the reconstruction of high-resolution temporal images in dynamic tomographic imaging, particularly for discrete objects with smooth boundaries that vary over time. Addressing the challenge of limited measurements per time point, we propose a technique that synergistically incorporates spatial and temporal information of the dynamic objects. This is achieved through the application of the level-set method for image segmentation and the representation of motion via a sinusoidal basis. The result is a computationally efficient and easily optimizable variational framework that enables the reconstruction of high-quality 2D or 3D image sequences with a single projection per frame. Compared to current methods, our proposed approach demonstrates superior performance on both synthetic and pseudo-dynamic real X-ray tomography datasets. The implications of this research extend to improved visualization and analysis of dynamic processes in tomographic imaging, finding potential applications in diverse scientific and industrial domains.

Machine learning models are typically configured by minimizing the training error over a given training dataset. On the other hand, the main objective is to obtain models that can generalize, i.e., perform well on data unseen during training. A fundamental challenge is that a low training error does not guarantee a low generalization error. While the classical wisdom from statistical learning theory states that models which perfectly fit the training data are unlikely to generalize, recent empirical results show that massively overparameterized models can defy this notion, leading to double-descent curves. This thesis investigates this phenomenon and characterizes the generalization performance of linear models across various scenarios. The first part of this thesis focuses on characterizing the generalization performance as a function of the level of model mismatch between the patterns in the data and the assumed model. Model mismatch is undesirable but often present in practice. We reveal the trade-offs between the number of samples, the number of features, and the possibly incorrect statistical assumptions on the assumed model. We show that fake features, i.e., features present in the model but not in the data, can significantly improve the generalization performance even though they are not correlated with the features in the underlying system. The second part of this thesis focuses on generalization under distributed learning. We focus on the scenario where the model parameters are distributed over a network of learners. We consider two settings: the single-task setting, where the network learns a single task, and the continual learning setting, where the network learns multiple tasks sequentially. The obtained results show that for a wide family of feature probability distributions, the generalization performance heavily depends on how the model parameters are partitioned over the network. In particular, if the number of parameters per learner and the number o

Este proyecto se centra en el diseño, la implementación y la evaluación de Redes Neuronales de Función de Base Radial (RBFNN), comparando el modelo gaussiano con una nueva versión que utiliza la función de activación Ricker. La forma de esta función ha sido observada en las señales de neuronas de distintas partes del cerebro humano, a menudo produciendo una señal negativa (inhibitoria) conocida como inhibición lateral. Se han desarrollado dos modelos de RBFNN, incorporando técnicas de Machine Learning (ML) y estadística como la regularización L2 y el algoritmo sigest para mejorar su rendimiento. También se implementan técnicas adicionales, como la estimación de un parámetro k sobredimensionado y la AIC backward selection, para mejorar la eficiencia. En este estudio, los modelos desarrollados se prueban con conjuntos de datos de diferente naturaleza, evaluando su rendimiento con datos sintéticos y realistas, y midiendo sus resultados con problemas de varios niveles de ruido y dificultad. Además, también se realiza una comparación de los modelos para observar qué RBFNN funciona mejor en determinadas condiciones, así como para analizar la diferencia en el número de neuronas y el parámetro de suavizado estimado. La evaluación experimental confirma la eficacia de los modelos RBFNN, proporcionando estimaciones precisas y demostrando su adaptabilidad con problemas de dificultad variable. El análisis comparativo revela que el modelo Ricker tiende a exhibir un rendimiento superior en presencia de altos niveles de ruido, mientras que ambos modelos tienen un rendimiento similar en condiciones de bajo ruido. Estos resultados sugieren la potencial influencia de la inhibición lateral, que podría ser explorada en más profundidad en futuros estudios., This project focuses on the design, implementation and evaluation of Radial Basis Function Neural Networks (RBFNN), comparing the gaussian model with a new version using the Ricker Wavelet activation function. The shape of this wavelet has been observed in the signals of neurons from different parts of the human brain, often producing a negative (inhibitory) signal known as lateral inhibition. Two RBFNN models have been developed, incorporating Machine Learning (ML) and statistical techniques such as L2 regularization and the sigest algorithm for improved performance. Additional techniques, such as estimating an oversized k parameter and using AIC backward selection, are implemented to enhance efficiency. In this study, the developed models are tested with datasets of different nature, evaluating their performance with synthetic and realistic data and measuring their results with problems of various levels of noise and difficulty. Furthermore, a comparison of the models is also made in order to observe which RBFNN performs better on certain conditions, as well as to analyze the difference in the number of neurons and the estimated smoothing parameter. The experimental evaluation confirms the effectiveness of the RBFNN models, yielding accurate estimations and demonstrating their adaptability to problems of varying difficulty. Comparative analysis reveals that the Ricker model tends to exhibit superior performance in the presence of high levels of noise, while both models perform similarly under low noise conditions. These results suggest the potential influence of lateral inhibition, which could be explored further in future studies.

Deep Learning architectures in which neural layers alternate with mappings to infinitedimensional feature spaces have been proposed in recent years, showing improvements on the results obtained when using either technique separately. However, these new algorithms have been presented without delving into the rich mathematical structure that sustains kernel methods. The main focus of this thesis is not only to review these advances in the field of Deep Learning, but to extend and generalize them by defining a broader family of models that operate under the mathematical framework defined by the composition of a neural layerwith a kernel mapping, all of which operate in reproducing kernel Hilbert spaces thatare then concatenated. Each of these spaces has a specific reproducing kernel that we can characterize. Together all of this defines a regularization-based learning optimization problem, for which we prove that minimizers exist. This strong mathematical background is complemented by the presentation of a new a model, the Kernel Network, which manages to produce successful results on many classification problems.

In a regular PCP the verifier queries each proof symbol in the same number of tests. This number is called the degree of the proof, and it is at least 1/(sq) where s is the soundness error and q is the number of queries. It is incredibly useful to have regularity and reduced degree in PCP. There is an expander-based transformation by Papadimitriou and Yannakakis that transforms any PCP with a constant number of queries and constant soundness error to a regular PCP with constant degree. There are also transformations for low error projection and unique PCPs. Other PCPs are constructed especially to be regular. In this work we show how to regularize and reduce degree of PCPs with a possibly large number of queries and low soundness error. As an application, we prove NP-hardness of an unweighted variant of the collective minimum monotone satisfying assignment problem, which was introduced by Hirahara (FOCS'22) to prove NP-hardness of MCSP^* (the partial function variant of the Minimum Circuit Size Problem) under randomized reductions. We present a simplified proof and sufficient conditions under which MCSP^* is NP-hard under the standard notion of reduction: MCSP^* is NP-hard under deterministic polynomial-time many-one reductions if there exists a function in E that satisfies certain direct sum properties.

This note provides unregularized and regularized closed-form analytical expressions for the depletion-induced or injection-induced pre-slip Coulomb stresses in two-dimensional displaced dip-slip faults. The regularization serves to remove logarithmic singularities and jump-discontinuities in the unregularized formulation. The expressions are identical to those in Appendices A and B of Jansen & Meulenbroek (2022): Netherlands Journal of Geosciences 101 e13, except for the correction of a small error in the regularized formulation. In numerical examples the difference of the correction is hardly noticeable, but it ensures that the corrected formulation is internally consistent in the sense that integrals of stresses and pressure along a fault are identical for the unregularized and regularized expressions., Note for file to document the correction of a minor error in an earlier publication., Civil Engineering & Geosciences

Subspace distance is an invaluable tool exploited in a wide range of feature selection methods. The power of subspace distance is that it can identify a representative subspace, including a group of features that can efficiently approximate the space of original features. On the other hand, employing intrinsic statistical information of data can play a significant role in a feature selection process. Nevertheless, most of the existing feature selection methods founded on the subspace distance are limited in properly fulfilling this objective. To pursue this void, we propose a framework that takes a subspace distance into account which is called “Variance–Covariance subspace distance”. The approach gains advantages from the correlation of information included in the features of data, thus determines all the feature subsets whose corresponding Variance–Covariance matrix has the minimum norm property. Consequently, a novel, yet efficient unsupervised feature selection framework is introduced based on the Variance–Covariance distance to handle both the dimensionality reduction and subspace learning tasks. The proposed framework has the ability to exclude those features that have the least variance from the original feature set. Moreover, an efficient update algorithm is provided along with its associated convergence analysis to solve the optimization side of the proposed approach. An extensive number of experiments on nine benchmark datasets are also conducted to assess the performance of our method from which the results demonstrate its superiority over a variety of state-of-the-art unsupervised feature selection methods. The source code is available at https://github.com/SaeedKarami/VCSDFS. © 2023 The Author(s), Funding: We acknowledge the support of following EU projects: CALLISTO (101004152), E-Vita (101016453), and the ScaDS.AI (Center for Scalable Data Analytics and Artificial Intelligence) Dresden/Leipzig (IS18026A-F). This work was also supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI, and grants 336033, 352986, 358246) and EU (H2020 grant 101016775 and NextGenerationEU)., CALLISTO, E-Vita, ScaDS.AI

In this paper, we propose a novel ROM stabilization strategy for under-resolved convection-dominated flows, the approximate deconvolution Leray ROM (ADL-ROM). The new ADL-ROM introduces AD as a new means to increase the accuracy of the classical Leray ROM (L-ROM) without degrading its numerical stability. We also introduce two new AD ROM strategies: the Tikhonov and van Cittert methods. Our numerical investigation for convection-dominated systems shows that, when the filter radius is relatively large, the new ADL-ROM is more accurate than the standard L-ROM. Furthermore, the new ADL-ROM is less sensitive with respect to model parameters than L-ROM.

Pruning is a model compression method that removes redundant parameters and accelerates the inference speed of deep neural networks (DNNs) while maintaining accuracy. Most available pruning methods impose various conditions on parameters or features directly. In this paper, we propose a simple and effective regularization strategy to improve the structured sparsity and structured pruning in DNNs from a new perspective of evolution of features. In particular, we consider the trajectories connecting features of adjacent hidden layers, namely feature flow. We propose feature flow regularization (FFR) to penalize the length and the total absolute curvature of the trajectories, which implicitly increases the structured sparsity of the parameters. The principle behind FFR is that short and straight trajectories will lead to an efficient network that avoids redundant parameters. Experiments on CIFAR-10 and ImageNet datasets show that FFR improves structured sparsity and achieves pruning results comparable to or even better than those state-of-the-art methods.

While self-supervised learning techniques are often used to mine hidden knowledge from unlabeled data via modeling multiple views, it is unclear how to perform effective representation learning in a complex and inconsistent context. To this end, we propose a new multi-view self-supervised learning method, namely consistency and complementarity network (CoCoNet), to comprehensively learn global inter-view consistent and local cross-view complementarity-preserving representations from multiple views. To capture crucial common knowledge which is implicitly shared among views, CoCoNet employs a global consistency module that aligns the probabilistic distribution of views by utilizing an efficient discrepancy metric based on the generalized sliced Wasserstein distance. To incorporate cross-view complementary information, CoCoNet proposes a heuristic complementarity-aware contrastive learning approach, which extracts a complementarity-factor jointing cross-view discriminative knowledge and uses it as the contrast to guide the learning of view-specific encoders. Theoretically, the superiority of CoCoNet is verified by our information-theoretical-based analyses. Empirically, our thorough experimental results show that CoCoNet outperforms the state-of-the-art self-supervised methods by a significant margin, for instance, CoCoNet beats the best benchmark method by an average margin of 1.1% on ImageNet.

Data-Driven Predictive Control (DDPC) has been recently proposed as an effective alternative to traditional Model Predictive Control (MPC), in that the same constrained optimization problem can be addressed without the need to explicitly identify a full model of the plant. However, DDPC is built upon input/output trajectories. Therefore, the finite sample effect of stochastic data, due to, e.g., measurement noise, may have a detrimental impact on closed-loop performance. Exploiting a formal statistical analysis of the prediction error, in this paper we propose the first systematic approach to deal with uncertainty due to finite sample effects. To this end, we introduce two regularization strategies for which, differently from existing regularization-based DDPC techniques, we propose a tuning rationale allowing us to select the regularization hyper-parameters before closing the loop and without additional experiments. Simulation results confirm the potential of the proposed strategy when closing the loop.

The setting where an unknown number m of the largest data is missing from an underlying Pareto-type distribution is considered. So-lutions are provided for estimating the extreme value index, the number of missing data and extreme quantiles. Asymptotic results of the parameter estimators and an adaptive selection method for the number of top data used in the estimation are proposed for the case where all missing data are beyond the observed data. An estimator of the number of missing extremes spread over the largest observed data is also proposed. To this purpose, a key component is a likelihood solution based on exponential representations of spacings between the largest observations. An effective and fast optimization procedure is established using regularization, and simulation experiments are provided. The methodology is illustrated with a dataset from the diamond mining industry, where large-carat diamonds are expected to be missing.

The setting where an unknown number m of the largest data is missing from an underlying Pareto-type distribution is considered. So-lutions are provided for estimating the extreme value index, the number of missing data and extreme quantiles. Asymptotic results of the parameter estimators and an adaptive selection method for the number of top data used in the estimation are proposed for the case where all missing data are beyond the observed data. An estimator of the number of missing extremes spread over the largest observed data is also proposed. To this purpose, a key component is a likelihood solution based on exponential representations of spacings between the largest observations. An effective and fast optimization procedure is established using regularization, and simulation experiments are provided. The methodology is illustrated with a dataset from the diamond mining industry, where large-carat diamonds are expected to be missing.

Data modeling and online monitoring are two critical stages for data-driven anomaly detection. Regarding data modeling, deep neural networks (DNNs) can learn good decision boundaries to separate the anomaly and normal regions, due to their flexible model structures and excellent fitting ability. However, DNNs, using nonlinear activations with specific boundaries, may indirectly cause a limited anomaly detection margin, especially when there are samples far from centroids. Moreover, an anomaly detection model with a narrow detection margin is deemed insensitive to general faults. An anomaly detection model with a tight detection margin will suffer a severe performance degradation. To mitigate the intrinsic drawbacks of DNNs, we develop a new regularizer based on the maximum likelihood of complete data (i.e., observations and latent variables). The regularizer is neuronwise and mathematically acts as compressing neurons, dragging the marginal points into the centroids. Combining the regularizer with the encoding-decoding structure networks, we perform an industrial case study to verify the superiority of the proposed method. © 2005-2012 IEEE.

Capacity constrained optimal transport is a variant of optimal transport, which adds extra constraints on the set of feasible couplings in the original optimal transport problem to limit the mass transported between each pair of source and sink. Based on this setting, constrained optimal transport has numerous applications, e.g., finance, network flow. However, due to the large number of constraints in this problem, existing algorithms are both time-consuming and space-consuming. In this paper, inspired by entropic regularization for the classical optimal transport problem, we introduce a novel regularization term for capacity constrained optimal transport. The regularized problem naturally satisfies the capacity constraints and consequently makes it possible to analyze the duality. Unlike the matrix-vector multiplication in the alternate iteration scheme for solving classical optimal transport, in our algorithm, each alternate iteration step is to solve several single-variable equations. Fortunately, we find that each of these equations corresponds to a single-variable monotonic function, and we convert solving these equations into finding the unique zero point of each single-variable monotonic function with Newton’s method. Theoretical analysis further provides a convergence guarantee to our algorithm. Extensive numerical experiments demonstrate that our proposed method has a significant advantage in terms of accuracy, efficiency, and memory consumption compared with existing methods. ©2023 Global-Science Press.

Dimensionality reduction techniques (DR) enhance data interpretability and reduce space complexity, though at the cost of information loss. Such methods have been prevalent in the Structural Health Monitoring (SHM) anomaly detection literature. While DR is favorable in supervised anomaly detection, where possible novelties are known a priori, the efficacy is less clear in unsupervised detection. In this work, we perform a detailed assessment of the DR performance trade-offs to determine whether the information loss imposed by DR can impact SHM performance for previously unseen novelties. As a basis for our analysis, we rely on an SHM anomaly detection method operating on input signals' fast Fourier transform (FFT). FFT is regarded as a raw, frequency-domain feature that allows studying various DR techniques. We design extensive experiments comparing various DR techniques, including neural autoencoder models, to capture the impact on two SHM benchmark datasets exclusively. Results imply the loss of information to be more detrimental, reducing the novelty detection accuracy by up to 60\% with autoencoder-based DR. Regularization can alleviate some of the challenges though unpredictable. Dimensions of substantial vibrational information mostly survive DR; thus, the regularization impact suggests that these dimensions are not reliable damage-sensitive features regarding unseen faults. Consequently, we argue that designing new SHM anomaly detection methods that can work with high-dimensional raw features is a necessary research direction and present open challenges and future directions.

This paper provides a rationale for the commonly observed numerical efficiency of staggered C-grid discretizations for solving the inviscid shallow water equations. In particular, using the key concepts of nonstandard calculus, we aim to show that the grid staggering of the primitive variables (surface elevation and normal velocity components) is capable of dealing with flow discontinuities. After a brief introduction of hyperreals through the notion of infinitesimal increments, a nonstandard rendition of the governing equations is derived that essentially turns into a finite procedure and permits a convenient way of modeling the hydraulic jumps in open channel flow. A central result of this paper is that the discrete formulations thus obtained are distinguished by the topological structures of the solution fields and subsequently provide a natural framework for the staggered discretization of the governing equations. Another key of the present study is to demonstrate that the discretization naturally regularizes the solution of the inviscid flow passing through the hydraulic jump without the need of non-physical dissipation. The underlying justification is provided by analytically studying the distributions of the flow variables across an infinitesimal thin hydraulic jump along with the use of hyperreal Heaviside step functions. This main finding is shown to be useful to comprehend the importance of the application of staggered finite difference schemes to accurately predict rapidly varying free-surface flows. A numerical experiment is provided to confirm this result., Environmental Fluid Mechanics

OBJECTIVE: To investigate cardiac activation maps estimated using electrocardiographic imaging and to find methods reducing line-of-block (LoB) artifacts, while preserving real LoBs.METHODS: Body surface potentials were computed for 137 simulated ventricular excitations. Subsequently, the inverse problem was solved to obtain extracellular potentials (EP) and transmembrane voltages (TMV). From these, activation times (AT) were estimated using four methods and compared to the ground truth. This process was evaluated with two cardiac mesh resolutions. Factors contributing to LoB artifacts were identified by analyzing the impact of spatial and temporal smoothing on the morphology of source signals.RESULTS: AT estimation using a spatiotemporal derivative performed better than using a temporal derivative. Compared to deflection-based AT estimation, correlation-based methods were less prone to LoB artifacts but performed worse in identifying real LoBs. Temporal smoothing could eliminate artifacts for TMVs but not for EPs, which could be linked to their temporal morphology. TMVs led to more accurate ATs on the septum than EPs. Mesh resolution had a negligible effect on inverse reconstructions, but small distances were important for cross-correlation-based estimation of AT delays.CONCLUSION: LoB artifacts are mainly caused by the inherent spatial smoothing effect of the inverse reconstruction. Among the configurations evaluated, only deflection-based AT estimation in combination with TMVs and strong temporal smoothing can prevent LoB artifacts, while preserving real LoBs.SIGNIFICANCE: Regions of slow conduction are of considerable clinical interest and LoB artifacts observed in non-invasive ATs can lead to misinterpretations. We addressed this problem by identifying factors causing such artifacts and methods to reduce them.

Filtering for GRACE temporal gravity fields is a necessary step before calculating surface mass anomalies. In this study, we propose a new denoising and decorrelation kernel (DDK) filtering scheme called adaptive DDK filter. The involved error covariance matrix (ECM) adopts nothing but the monthly time‐variable released by several data centers. The signal covariance matrix (SCM) involved is monthly time‐variable also. Specifically, it is parameterized into two parameters, namely the regularization coefficient and the power index of signal covariances, which are adaptively determined from the data themselves according to the generalized cross validation (GCV) criterion. The regularization coefficient controls the global constraint on the signal variances of all degrees, while the power index adjusts the attenuation of the signal variances from low to high degrees, namely local constraint. By tuning these two parameters for the monthly SCM, the adaptability to the data and the optimality of filtering strength can be expected. In addition, we also devise a half-weight polygon area (HWPA) of the filter kernel to measure the filtering strength of the anisotropic filter more reasonably. The proposed adaptive DDK filter and filtering strength metric are tested based on CSR GRACE temporal gravity solutions with their ECMs from January 2004 to December 2010. Results show that the selected optimal power indices range from 3.5 to 6.9, with the corresponding regularization parameters range from 1 × 1014 to 5 × 1019. The adaptive DDK filter can retain comparable/more signal amplitude and suppress more high‐degree noise than the conven-tional DDK filters. Compared with the equivalent smoothing radius (ESR) of filtering strength, the HWPA has stronger a distinguishing ability, especially when the filtering strength is similar., Physical and Space Geodesy

Immigrant legalization policies pose an ethical dilemma between justice and the rule of law. On the one hand, liberal democracies aspire to the principles of individual liberty and equality. Building on liberal ideals of justice, compelling arguments have been made for granting legal status and a path to citizenship to unauthorized migrants by virtue of the social ties they have developed, their contributions to the host society, and their vulnerability to exploitation. On the other hand, legalization poses a challenge to another important value, the rule of law, which requires government to operate within a framework of law in accordance with well-established public norms. Immigrant legalization programmes are said to undermine the rule of law because they reward lawbreaking, allow queue-jumping, and incentivize further unauthorized migration. This article clarifies each horn of the dilemma, focusing on rule of law arguments. We offer a critical reappraisal of immigrant legalization policies by reflecting on the normative meaning of the rule of law and by examining empirical evidence assessing the effects of legalization. Our central contention is that legalization policies can enhance the rule of law. We offer five rule of law arguments in support of legalization, which help to mitigate the dilemma between justice and the rule of law. We conclude by discussing some policy implications of our analysis.

Graphical models reveal the conditional dependence structure between random variables. By estimating the joint density or conditional density, we can detect edges and recover the structure of a graphical model. We propose new nonparametric methods to learn edges for graphical models under a consolidated framework of smoothing spline ANOVA (SS ANOVA) decomposition. We first develop an automatic nonparametric edge detection method by estimating the joint density function with an L1 penalty to interactions in the SS ANOVA decomposition. In the second project, we work directly on the conditional dependence structure and develop a fully nonparametric neighborhood selection method. We detect edges by applying an L1 regularization to interactions in the SS ANOVA decomposition of conditional density functions. These two methods are flexible and contain many existing models as special cases. They also provide a unified framework without any restrictions on the type of each random variable. The joint density approach requires a large computer memory and is thus computationally feasible only when the dimension is small. The neighborhood selection approach overcomes this disadvantage and is more computationally efficient.We propose iterative procedures to compute the estimates and establish the convergence rates for both the joint and conditional density as well as interactions. Simulations indicate that both joint and neighborhood selection methods perform well under Gaussian and non-Gaussian settings. We illustrate the proposed methods using real data examples.

In agricultural studies it is often important to predict the performance of genetically different plants. To make sure predictions are done well, it is necessary to make sure they are not influenced by effects of the field on which they are planted. These field effects or spatial effects are in practice often quite complicated and can be due to a wide variety of reasons. To get a better view of these field effects a good mathematical model is desired. In this paper a model is presented which helps to find these field effects. This model tries to estimate the field effect by comparing data of the same plant on different positions of the field. Data is obtained in a finite amount of positions, which means that the model finds the field effect in a finite amount of positions as well. This field effect is found using a cross-validation technique obtained from Tikhonov regularization. The field effect in a finite amount of positions is extended to a field effect in every position of the field. To do this in a good way a kernel method is used, the advantage of which is that it does not depend on a mesh. This kernel method is here applied with a kernel function that is based on Gaussian distribution. This model is applied to several fields of crops to get a view of the performance of the model on real data., Applied Mathematics

Atmospheric inverse modeling describes the process of estimating greenhouse gas fluxes or air pollution emissions at the Earth's surface using observations of these gases collected in the atmosphere. The launch of new satellites, the expansion of surface observation networks, and a desire for more detailed maps of surface fluxes have yielded numerous computational and statistical challenges for standard inverse modeling frameworks that were often originally designed with much smaller data sets in mind. In this article, we discuss computationally efficient methods for large-scale atmospheric inverse modeling and focus on addressing some of the main computational and practical challenges. We develop generalized hybrid projection methods, which are iterative methods for solving large-scale inverse problems, and specifically we focus on the case of estimating surface fluxes. These algorithms confer several advantages. They are efficient, in part because they converge quickly, they exploit efficient matrix-vector multiplications, and they do not require inversion of any matrices. These methods are also robust because they can accurately reconstruct surface fluxes, they are automatic since regularization or covariance matrix parameters and stopping criteria can be determined as part of the iterative algorithm, and they are flexible because they can be paired with many different types of atmospheric models. We demonstrate the benefits of generalized hybrid methods with a case study from NASA's Orbiting Carbon Observatory 2 (OCO-2) satellite. We then address the more challenging problem of solving the inverse model when the mean of the surface fluxes is not known a priori; we do so by reformulating the problem, thereby extending the applicability of hybrid projection methods to include hierarchical priors. We further show that by exploiting mathematical relations provided by the generalized hybrid method, we can efficiently calculate an approximate posterior variance, there

Gene expression data have become increasingly important in machine learning and computational biology over the past few years. In the field of gene expression analysis, several matrix factorization-based dimensionality reduction methods have been developed. However, such methods can still be improved in terms of efficiency and reliability. In this paper, an innovative approach to feature selection, called Dual Regularized Unsupervised Feature Selection Based on Matrix Factorization and Minimum Redundancy (DR-FS-MFMR), is introduced. The major focus of DR-FS-MFMR is to discard redundant features from the set of original features. In order to reach this target, the primary feature selection problem is defined in terms of two aspects: (1) the matrix factorization of data matrix in terms of the feature weight matrix and the representation matrix, and (2) the correlation information related to the selected features set. Then, the objective function is enriched by employing two data representation characteristics along with an inner product regularization criterion to perform both the redundancy minimization process and the sparsity task more precisely. To demonstrate the proficiency of the DR-FS-MFMR method, a large number of experimental studies are conducted on nine gene expression datasets. The obtained computational results indicate the efficiency and productivity of DR-FS-MFMR for the gene selection task. © 2022 The Author(s)

Gene expression data have become increasingly important in machine learning and computational biology over the past few years. In the field of gene expression analysis, several matrix factorization-based dimensionality reduction methods have been developed. However, such methods can still be improved in terms of efficiency and reliability. In this paper, an innovative approach to feature selection, called Dual Regularized Unsupervised Feature Selection Based on Matrix Factorization and Minimum Redundancy (DR-FS-MFMR), is introduced. The major focus of DR-FS-MFMR is to discard redundant features from the set of original features. In order to reach this target, the primary feature selection problem is defined in terms of two aspects: (1) the matrix factorization of data matrix in terms of the feature weight matrix and the representation matrix, and (2) the correlation information related to the selected features set. Then, the objective function is enriched by employing two data representation characteristics along with an inner product regularization criterion to perform both the redundancy minimization process and the sparsity task more precisely. To demonstrate the proficiency of the DR-FS-MFMR method, a large number of experimental studies are conducted on nine gene expression datasets. The obtained computational results indicate the efficiency and productivity of DR-FS-MFMR for the gene selection task. © 2022 The Author(s)

Vessels constructed in electrically conductive materials give rise to frequency-dependent, induced magnetic fields when waves of water cause them to roll in the Earth's magnetic field. These fields, typically referred to as roll-induced magnetic vortex fields, are a component of the ship's overall signature, where signature refers to measurable quantities which can reveal or identify objects. It is crucial for military platforms to keep the signature low and thereby increase the possibilities of operation. For magnetic signatures, this is done through strategic design and construction of the platform or by using magnetic silencing systems. The signature is then decreased to minimize the risk of detection from naval mines and marine detection systems. This report covers the initial research on the subject of an inverse source model for roll induced magnetic fields. By limiting the analysis to two basic objects and applying a time variant magnetic field to them, we induce a magnetic field which we then model. The inverse modeling is done using magnetic dipoles as sources which are placed around the area of the object. The parameters of the model are then found by applying a least squares algorithm coupled with Tikhonov regularization. The focus of this report is the configuration of this setup in terms of measurements and sources, as well as finding a proper regularization parameter. Since the applied magnetic field is dependent on the roll frequency, also the inverse model depends on a frequency parameter in addition to the geometry and material of the object. The objects here studied are of two simple geometries, a rectangular block and a hollow cylinder. Both objects are constructed in an aluminum alloy with well known material parameters. Measurement data is gathered using a numerical solver utilizing the finite element method for solving the partial differential equations. The numerical measurement data is compared to physical measurements as well. The physical

We use mollification to regularize the problem of deconvolution of random variables. This regularization method offers a unifying and generalizing framework in order to compare the benefits of various filter-type techniques like deconvolution kernels, Tikhonov or spectral cut- off methods. In particular, the mollifier approach allows to relax some restrictive assumptions required for the deconvolution kernels, and has better stabilizing properties compared to spectral cutoff or Tikhonov. We show that this approach achieves optimal rates of convergence both for finitely and infinitely smoothing convolution operators under Besov and Sobolev smoothness assumptions on the unknown probability density. The qualification can be arbitrarily high depending on the choice of the mollifier function. We propose an adaptive choice of the regularization parameter using the Lepskii method and we provide simulations to compare the finite sample properties of our estimator with respect to the well-known regularization methods.

Graphical models reveal the conditional dependence structure between random variables. By estimating the joint density or conditional density, we can detect edges and recover the structure of a graphical model. We propose new nonparametric methods to learn edges for graphical models under a consolidated framework of smoothing spline ANOVA (SS ANOVA) decomposition. We first develop an automatic nonparametric edge detection method by estimating the joint density function with an L1 penalty to interactions in the SS ANOVA decomposition. In the second project, we work directly on the conditional dependence structure and develop a fully nonparametric neighborhood selection method. We detect edges by applying an L1 regularization to interactions in the SS ANOVA decomposition of conditional density functions. These two methods are flexible and contain many existing models as special cases. They also provide a unified framework without any restrictions on the type of each random variable. The joint density approach requires a large computer memory and is thus computationally feasible only when the dimension is small. The neighborhood selection approach overcomes this disadvantage and is more computationally efficient.We propose iterative procedures to compute the estimates and establish the convergence rates for both the joint and conditional density as well as interactions. Simulations indicate that both joint and neighborhood selection methods perform well under Gaussian and non-Gaussian settings. We illustrate the proposed methods using real data examples.

The recent surge in highly successful, but opaque, machine-learning models has given rise to a dire need for interpretability. This work addresses the problem of interpretability with novel definitions, methodology, and scientific investigations, ensuring that interpretations are useful by grounding them in the context of real-world problems and audiences. We begin by defining what we mean by interpretability and some desiderata surrounding it, emphasizing the underappreciated role of context. We then dive into novel methods for interpreting/improving neural network models, focusing on how to best score, use, and distill interactions. Next, we turn from neural networks to relatively simple rule-based models, where we investigate how to improve predictive performance while maintaining an extremely concise model. Finally, we conclude with work on open-source software and data for facilitating interpretable data science. In each case, we dive into a specific context which motivates the proposed methodology, ranging from cosmology to cell biology to medicine. Code for everything is available at github.com/csinva.

Immigrant legalization policies pose an ethical dilemma between justice and the rule of law. On the one hand, liberal democracies aspire to the principles of individual liberty and equality. Building on liberal ideals of justice, compelling arguments have been made for granting legal status and a path to citizenship to unauthorized migrants by virtue of the social ties they have developed, their contributions to the host society, and their vulnerability to exploitation. On the other hand, legalization poses a challenge to another important value, the rule of law, which requires government to operate within a framework of law in accordance with well-established public norms. Immigrant legalization programmes are said to undermine the rule of law because they reward lawbreaking, allow queue-jumping, and incentivize further unauthorized migration. This article clarifies each horn of the dilemma, focusing on rule of law arguments. We offer a critical reappraisal of immigrant legalization policies by reflecting on the normative meaning of the rule of law and by examining empirical evidence assessing the effects of legalization. Our central contention is that legalization policies can enhance the rule of law. We offer five rule of law arguments in support of legalization, which help to mitigate the dilemma between justice and the rule of law. We conclude by discussing some policy implications of our analysis.