Search

The evidence lower bound (ELBO) is one of the most central objectives for probabilistic unsupervised learning. For the ELBOs of several generative models and model classes, we here prove convergence to entropy sums. As one result, we provide a list of generative models for which entropy convergence has been shown, so far, along with the corresponding expressions for entropy sums. Our considerations include very prominent generative models such as probabilistic PCA, sigmoid belief nets or Gaussian mixture models. However, we treat more models and entire model classes such as general mixtures of exponential family distributions. Our main contributions are the proofs for the individual models. For each given model we show that the conditions stated in Theorem 1 or Theorem 2 of [arXiv:2209.03077] are fulfilled such that by virtue of the theorems the given model's ELBO is equal to an entropy sum at all stationary points. The equality of the ELBO at stationary points applies under realistic conditions: for finite numbers of data points, for model/data mismatches, at any stationary point including saddle points etc, and it applies for any well behaved family of variational distributions., Comment: 16 Pages

In this paper, we present a nonlinear model predictive control (NMPC) algorithm for systems modeled by semi-explicit stochastic differential-algebraic equations (DAEs) of index 1. The NMPC combines a continuous-discrete extended Kalman filter (CD-EKF) with an optimal control problem (OCP) for setpoint tracking. We discretize the OCP using direct multiple shooting. We apply an explicit singly diagonal implicit Runge-Kutta (ESDIRK) integration scheme to solve systems of DAEs, both for the one-step prediction in the CD-EKF and in each shooting interval of the discretized OCP. The ESDIRK methods use an iterated internal numerical differentiation approach for precise sensitivity computations. These sensitivities are used to provide accurate gradient information in the OCP and to efficiently integrate the covariance information in the CD-EKF. Subsequently, we present a simulation case study where we apply the NMPC to a simple alkaline electrolyzer stack model. We use the NMPC to track a time-varying setpoint for the stack temperature subject to input bound constraints.

This study presents the design, discretization and implementation of the continuous-time linear-quadratic model predictive control (CT-LMPC). The control model of the CT-LMPC is parameterized as transfer functions with time delays, and they are separated into deterministic and stochastic parts for relevant control and filtering algorithms. We formulate time-delay, finite-horizon CT linear-quadratic optimal control problems (LQ-OCPs) for the CT-LMPC. By assuming piece-wise constant inputs and constraints, we present the numerical discretization of the proposed LQ-OCPs and show how to convert the discrete-time (DT) equivalent into a standard quadratic program. The performance of the CT-LMPC is compared with the conventional DT-LMPC algorithm. Our numerical experiments show that, under fixed tunning parameters, the CT-LMPC shows better closed-loop performance as the sampling time increases than the conventional DT-LMPC.

Recent advances in multimodal foundation models have set new standards in few-shot anomaly detection. This paper explores whether high-quality visual features alone are sufficient to rival existing state-of-the-art vision-language models. We affirm this by adapting DINOv2 for one-shot and few-shot anomaly detection, with a focus on industrial applications. We show that this approach does not only rival existing techniques but can even outmatch them in many settings. Our proposed vision-only approach, AnomalyDINO, is based on patch similarities and enables both image-level anomaly prediction and pixel-level anomaly segmentation. The approach is methodologically simple and training-free and, thus, does not require any additional data for fine-tuning or meta-learning. Despite its simplicity, AnomalyDINO achieves state-of-the-art results in one- and few-shot anomaly detection (e.g., pushing the one-shot performance on MVTec-AD from an AUROC of 93.1% to 96.6%). The reduced overhead, coupled with its outstanding few-shot performance, makes AnomalyDINO a strong candidate for fast deployment, e.g., in industrial contexts.

This paper details the efforts of the WisPerMed team in the BioLaySumm2024 Shared Task on automatic lay summarization in the biomedical domain, aimed at making scientific publications accessible to non-specialists. Large language models (LLMs), specifically the BioMistral and Llama3 models, were fine-tuned and employed to create lay summaries from complex scientific texts. The summarization performance was enhanced through various approaches, including instruction tuning, few-shot learning, and prompt variations tailored to incorporate specific context information. The experiments demonstrated that fine-tuning generally led to the best performance across most evaluated metrics. Few-shot learning notably improved the models' ability to generate relevant and factually accurate texts, particularly when using a well-crafted prompt. Additionally, a Dynamic Expert Selection (DES) mechanism to optimize the selection of text outputs based on readability and factuality metrics was developed. Out of 54 participants, the WisPerMed team reached the 4th place, measured by readability, factuality, and relevance. Determined by the overall score, our approach improved upon the baseline by approx. 5.5 percentage points and was only approx 1.5 percentage points behind the first place., Comment: 4 pages, 6 figure, 3 tables, acctepted at: BIONLP 2024 and Shared Tasks @ ACL 2024

This study aims to leverage state of the art language models to automate generating the "Brief Hospital Course" and "Discharge Instructions" sections of Discharge Summaries from the MIMIC-IV dataset, reducing clinicians' administrative workload. We investigate how automation can improve documentation accuracy, alleviate clinician burnout, and enhance operational efficacy in healthcare facilities. This research was conducted within our participation in the Shared Task Discharge Me! at BioNLP @ ACL 2024. Various strategies were employed, including few-shot learning, instruction tuning, and Dynamic Expert Selection (DES), to develop models capable of generating the required text sections. Notably, utilizing an additional clinical domain-specific dataset demonstrated substantial potential to enhance clinical language processing. The DES method, which optimizes the selection of text outputs from multiple predictions, proved to be especially effective. It achieved the highest overall score of 0.332 in the competition, surpassing single-model outputs. This finding suggests that advanced deep learning methods in combination with DES can effectively automate parts of electronic health record documentation. These advancements could enhance patient care by freeing clinician time for patient interactions. The integration of text selection strategies represents a promising avenue for further research., Comment: 8 pages, 6 tables, 8 figures, submitted to: BioNLP 2024 and Shared Tasks @ ACL 2024

The Danish system operator, Energinet, has recently introduced an innovative grid code called the P90 requirement, which allows stochastic flexible resources to bid their flexibility in Nordic ancillary service markets, contingent upon a minimum 90% probability of successfully realizing the reserve capacity bid. For limited-energy resources, Energinet imposes additional requirements for participation in these markets. Given these requirements, this paper presents a chance-constrained optimization model designed for aggregators of electric vehicles, aiming to optimally place reserve capacity bids in the Nordic Frequency Containment Reserve for Disturbances (FCR-D) market while accounting for uncertainty in future consumption baselines. We analyze both FCR-D up and down markets, reformulating and solving the proposed joint chance-constrained model using two sample-based methods. Using real data from 1,400 charging stations in Denmark from March 2022 to March 2023, we demonstrate the out-of-sample profit potential. Our findings indicate that vehicle owners could save between 6% and 10% on their annual electricity bills by providing FCR-D services. Additionally, we observed a synergy effect, where having more vehicles in a single portfolio enables larger bids per vehicle compared to a collective bid from multiple portfolios with the same total number of vehicles., Comment: Submitted to IEEE Transactions on Power Systems

In this study, we introduce numerical methods for discretizing continuous-time linear-quadratic optimal control problems (LQ-OCPs). The discretization of continuous-time LQ-OCPs is formulated into differential equation systems, and we can obtain the discrete equivalent by solving these systems. We present the ordinary differential equation (ODE), matrix exponential, and a novel step-doubling method for the discretization of LQ-OCPs. Utilizing Euler-Maruyama discretization with a fine step, we reformulate the costs of continuous-time stochastic LQ-OCPs into a quadratic form, and show that the stochastic cost follows the $\chi^2$ distribution. In the numerical experiment, we test and compare the proposed numerical methods. The results ensure that the discrete-time LQ-OCP derived using the proposed numerical methods is equivalent to the original problem., Comment: This paper (7 pages) has been accepted by the 22nd European Control Conference (ECC) in Stockholm, Sweden

In this paper, we compare the impact of iterated and direct approaches to sensitivity computation in fixed-step explicit singly diagonally-implicit Runge-Kutta (ESDIRK) methods when applied to optimal control problems (OCPs). We use the principle of internal numerical differentiation (IND) strictly for the iterated approach, i.e., reusing the iteration matrix factorizations, the number of Newton-type iterations, and Newton iterates, to compute the sensitivities. The direct method computes the sensitivities without using the Newton schemes. We compare the impact of the iterated and direct sensitivity computations in OCPs for the quadruple tank system. We benchmark the iterated and direct approaches with a base case. This base case is an OCP that applies an ESDIRK method that refactorizes the iteration matrix in every Newton iteration and uses a direct approach for sensitivity computations. In these OCPs, we vary the number of integration steps between control intervals and we evaluate the performance based on the number of SQP and QPs iterations, KKT violations, and the total number of function evaluations, Jacobian updates, and iteration matrix factorizations. The results indicate that the iterated approach outperforms the direct approach but yields similar performance to the base case., Comment: 6 pages, 5 figures, 2 tables. Submitted for European Control Conference 2024 (ECC2024). Stockholm, Sweden

With the ever increasing number of active satellites in space, the rising demand for larger formations of small satellites and the commercialization of the space industry (so-called New Space), the realization of manufacturing processes in orbit comes closer to reality. Reducing launch costs and risks, allowing for faster on-demand deployment of individually configured satellites as well as the prospect for possible on-orbit servicing for satellites makes the idea of realizing an in-orbit factory promising. In this paper, we present a novel approach to an in-orbit factory of small satellites covering a digital process twin, AI-based fault detection, and teleoperated robot-control, which are being researched as part of the "AI-enabled Cyber-Physical In-Orbit Factory" project. In addition to the integration of modern automation and Industry 4.0 production approaches, the question of how artificial intelligence (AI) and learning approaches can be used to make the production process more robust, fault-tolerant and autonomous is addressed. This lays the foundation for a later realisation of satellite production in space in the form of an in-orbit factory. Central aspect is the development of a robotic AIT (Assembly, Integration and Testing) system where a small satellite could be assembled by a manipulator robot from modular subsystems. Approaches developed to improving this production process with AI include employing neural networks for optical and electrical fault detection of components. Force sensitive measuring and motion training helps to deal with uncertainties and tolerances during assembly. An AI-guided teleoperated control of the robot arm allows for human intervention while a Digital Process Twin represents process data and provides supervision during the whole production process. Approaches and results towards automated satellite production are presented in detail.

Standard probabilistic sparse coding assumes a Laplace prior, a linear mapping from latents to observables, and Gaussian observable distributions. We here derive a solely entropy-based learning objective for the parameters of standard sparse coding. The novel variational objective has the following features: (A) unlike MAP approximations, it uses non-trivial posterior approximations for probabilistic inference; (B) unlike for previous non-trivial approximations, the novel objective is fully analytical; and (C) the objective allows for a novel principled form of annealing. The objective is derived by first showing that the standard ELBO objective converges to a sum of entropies, which matches similar recent results for generative models with Gaussian priors. The conditions under which the ELBO becomes equal to entropies are then shown to have analytical solutions, which leads to the fully analytical objective. Numerical experiments are used to demonstrate the feasibility of learning with such entropy-based ELBOs. We investigate different posterior approximations including Gaussians with correlated latents and deep amortized approximations. Furthermore, we numerically investigate entropy-based annealing which results in improved learning. Our main contributions are theoretical, however, and they are twofold: (1) for non-trivial posterior approximations, we provide the (to the knowledge of the authors) first analytical ELBO objective for standard probabilistic sparse coding; and (2) we provide the first demonstration on how a recently shown convergence of the ELBO to entropy sums can be used for learning.

In this paper, we consider a model reduction technique for stabilizable and detectable stochastic systems. It is based on a pair of Gramians that we analyze in terms of well-posedness. Subsequently, dominant subspaces of the stochastic systems are identified exploiting these Gramians. An associated balancing related scheme is proposed that removes unimportant information from the stochastic dynamics in order to obtain a reduced system. We show that this reduced model preserves important features like stabilizability and detectability. Additionally, a comprehensive error analysis based on eigenvalues of the Gramian pair product is conducted. This provides an a-priori criterion for the reduction quality which we illustrate in numerical experiments.

In recent work, Darmon, Pozzi and Vonk explicitly construct a modular form whose spectral coefficients are $p$-adic logarithms of Gross-Stark units and Stark-Heegner points. Here we describe how this construction gives rise to a practical algorithm for explicitly computing these logarithms to specified precision, and how to recover the exact values of the Gross-Stark units and Stark-Heegner points from them. Key tools are overconvergent modular forms, reduction theory of quadratic forms and Newton polygons. As an application, we tabulate Brumer-Stark units in narrow Hilbert class fields of real quadratic fields with discriminants up to $10000$, for primes less than $20$, as well as Stark-Heegner points on elliptic curves., Comment: 23 pages, 4 tables, 2 figures

In the course of the past few years, diffusion models (DMs) have reached an unprecedented level of visual quality. However, relatively little attention has been paid to the detection of DM-generated images, which is critical to prevent adverse impacts on our society. In contrast, generative adversarial networks (GANs), have been extensively studied from a forensic perspective. In this work, we therefore take the natural next step to evaluate whether previous methods can be used to detect images generated by DMs. Our experiments yield two key findings: (1) state-of-the-art GAN detectors are unable to reliably distinguish real from DM-generated images, but (2) re-training them on DM-generated images allows for almost perfect detection, which remarkably even generalizes to GANs. Together with a feature space analysis, our results lead to the hypothesis that DMs produce fewer detectable artifacts and are thus more difficult to detect compared to GANs. One possible reason for this is the absence of grid-like frequency artifacts in DM-generated images, which are a known weakness of GANs. However, we make the interesting observation that diffusion models tend to underestimate high frequencies, which we attribute to the learning objective., Comment: Accepted at VISAPP 2024. This is the extended version with additional experiments and supplemental material. Code and data: https://github.com/jonasricker/diffusion-model-deepfake-detection