Search

The technical world of today fundamentally relies on structural analysis in the form of design and structural mechanic simulations. A traditional and robust simulation method is the physics-based finite element method (FEM) simulation. FEM simulations in structural mechanics are known to be very accurate; however, the higher the desired resolution, the more computational effort is required. Surrogate modeling provides a robust approach to address this drawback. Nonetheless, finding the right surrogate model and its hyperparameters for a specific use case is not a straightforward process. In this paper, we discuss and compare several classes of mesh-free surrogate models based on traditional and thriving machine learning (ML) and deep learning (DL) methods. We show that relatively simple algorithms (such as k-nearest neighbor regression) can be competitive in applications with low geometrical complexity and extrapolation requirements. With respect to tasks exhibiting higher geometric complexity, our results show that recent DL methods at the forefront of literature (such as physics-informed neural networks) are complicated to train and to parameterize and thus, require further research before they can be put to practical use. In contrast, we show that already well-researched DL methods, such as the multi-layer perceptron, are superior with respect to interpolation use cases and can be easily trained with available tools. With our work, we thus present a basis for the selection and practical implementation of surrogate models. [ABSTRACT FROM AUTHOR]

In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that the resulting optimization problem suffers from two severe issues: First, for deterministic DNNs, either the IB functional is infinite for almost all values of network parameters, making the optimization problem ill-posed, or it is piecewise constant, hence not admitting gradient-based optimization methods. Second, the invariance of the IB functional under bijections prevents it from capturing properties of the learned representation that are desirable for classification, such as robustness and simplicity. We argue that these issues are partly resolved for stochastic DNNs, DNNs that include a (hard or soft) decision rule, or by replacing the IB functional with related, but more well-behaved cost functions. We conclude that recent successes reported about training DNNs using the IB framework must be attributed to such solutions. As a side effect, our results indicate limitations of the IB framework for the analysis of DNNs. We also note that rather than trying to repair the inherent problems in the IB functional, a better approach may be to design regularizers on latent representation enforcing the desired properties directly. [ABSTRACT FROM AUTHOR]

Acknowledging the issues in [[12]], these NNs are trained to minimize an upper bound on HT ht , combining variational and non-parametric approaches for bounding. In this case, it is known that sweeping the paramter HT ht in (1) is not sufficient to sweep the Pareto frontier of optimal HT ht pairs [[12]]. The authors show that this shortcoming can be removed by optimizing HT ht instead, where I u i is a strictly convex function. [Extracted from the article]

In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals. [ABSTRACT FROM AUTHOR]