Search

Reliable evaluation of AI models is critical for scientific progress and practical application. While existing VLM benchmarks provide general insights into model capabilities, their heterogeneous designs and limited focus on a few imaging domains pose significant challenges for both cross-domain performance comparison and targeted domain-specific evaluation. To address this, we propose three key contributions: (1) a framework for the resource-efficient creation of domain-specific VLM benchmarks enabled by task augmentation for creating multiple diverse tasks from a single existing task, (2) the release of new VLM benchmarks for seven domains, created according to the same homogeneous protocol and including 162,946 thoroughly human-validated answers, and (3) an extensive benchmarking of 22 state-of-the-art VLMs on a total of 37,171 tasks, revealing performance variances across domains and tasks, thereby supporting the need for tailored VLM benchmarks. Adoption of our methodology will pave the way for the resource-efficient domain-specific selection of models and guide future research efforts toward addressing core open questions.

Exploring the equation of state of dense matter is an essential part of interpreting the observable properties of neutron stars. We present here the first results for dense matter in the zero-temperature limit generated by the MUSES Calculation Engine, a composable workflow management system that orchestrates calculation and data processing stages comprising a collection of software modules designed within the MUSES framework. The modules presented in this work calculate equations of state using algorithms spanning three different theories/models: (1) Crust Density Functional Theory, valid starting at low densities, (2) Chiral Effective Field Theory, valid around saturation density, and (3) the Chiral Mean Field model, valid beyond saturation density. Lepton contributions are added through the Lepton module to each equation of state, ensuring charge neutrality and the possibility of $\beta$-equilibrium. Using the Synthesis module, we match the three equations of state using different thermodynamic variables and different methods. We then couple the complete equation of state to a novel full-general-relativity solver (QLIMR) module that calculates neutron star properties. We find that the matching performed using different thermodynamic variables affects differently the range obtained for neutron star masses and radii (although never beyond a few percent difference). We also investigate the universality of equation of state-independent relations for our matched stars. Finally, for the first time, we use the Flavor Equilibration module to estimate bulk viscosity and flavor relaxation charge fraction and rates (at low temperature) for Chiral Effective Field Theory and the Chiral Mean Field model., Comment: 38 pages, 17 figures, 6 tables

The Euler characteristic of a very affine variety encodes the number of critical points of the likelihood equation on this variety. In this paper, we study the Euler characteristic of the complement of a hypersurface arrangement with generic hypersurfaces added. For hyperplane arrangements, it depends on the characteristic polynomial coefficients and generic hypersurface degrees. As a corollary, we show that adding a degree-two hypersurface to a real hyperplane arrangement enables efficient sampling of a single interior point from each region in the complement. We compare the method to existing alternatives and demonstrate its efficiency. For hypersurface arrangements, the Euler characteristic is expressed in terms of Milnor numbers and generic hypersurface degrees. This formulation further yields a novel upper bound on the number of regions in the complement of a hypersurface arrangement., Comment: 19 pages, 3 figures, 2 tables

The intersection numbers of moduli spaces of stable curves $\overline{\mathcal{M}}_{g,m}$ are well-studied and are known to have rich combinatorial structure. We introduce a natural class of these intersection numbers $\omega_{G,g,m}$ indexed by finite simple graphs $G=(V,E)$. In genus zero, these numbers are closely related to several previously-studied quantities, including maximum likelihood degrees in algebraic statistics, counts of regions of certain hyperplane arrangements, and Kapranov degrees. We give two proofs of a simple closed formula $\omega_{G,g,m}=(-1)^{\left\lvert V \right\rvert}\chi_G(-(2g-2+m)),$ where $\chi_G$ is the chromatic polynomial of $G$ -- one proof via intersection theory on moduli spaces of stable curves, and the other using the theory of hyperplane arrangements. We discuss several related questions and speculations, including new candidates for the chromatic polynomial of a directed graph., Comment: 28 pages, comments welcome

Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50% of papers do not assess performance variability at all. Moreover, only one (0.5%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice., Comment: Paper accepted at MICCAI 2024 conference

In this paper we explore independently for the first time three chemical potentials (baryon $\mu_B$, charged $\mu_Q$, and strange $\mu_S$) in the Chiral mean-field (CMF) model. We designed and implemented \texttt{CMF++}, a new version of the CMF model rewritten in \texttt{C++} that is optimized, modular, and well-documented. \texttt{CMF++} has been integrated into the MUSES Calculation Engine as a free and open source software module. The runtime improved in more than 4 orders of magnitude across all 3 chemical potentials, when compared to the legacy code. Here we focus on the zero temperature case and study stable, as well as metastable and unstable, vacuum, hadronic, and quark phases, showing how phase boundaries vary with the different chemical potentials. Due to the significant numerical improvements in \texttt{CMF++}, we can calculate for the first time high-order susceptibilities within the CMF framework to study the properties of the quark deconfinement phase transition. We found phases of matter that include a light hadronic phase, strangeness-dominated hadronic phase, and quark deconfinement within our $\mu_B$, $\mu_S$, $\mu_Q$ phase space. The phase transitions are of first, second (quantum critical point), and third order between these phases and we even identified a tricritical point., Comment: 63 pages, 22 figures

Plasma detachment involves interactions of the plasma with impurities and neutral particles, leading to significant losses of plasma power, momentum, and particles. Accurate mapping of plasma emissivity in the divertor and X-point region is essential for assessing the relationship between particle flux and radiative detachment. The recently validated InfraRed Video Bolometer (IRVB) diagnostic, in MAST-U enables this mapping with higher spatial resolution than more established methods like resistive bolometers. In previous preliminary work, the evolution of radiative detachment was characterised in L-mode (power entering the scrape-off layer, PSOL ~0.4MW). With a conventional divertor the inner leg consistently detached ahead of the outer leg, and radiative detachment preceded particle flux detachment. This work presents results also from the third MAST-U experimental campaign, fuelled from the low field side instead of the high field side, including Ohmic and beam heated L-mode shots (with a power exiting the core up to PSOL ~1-1.5MW). The radiation peak moves upstream from the target at lower upstream densities than the ion target flux roll-over (typically considered the detachment onset), while the inner leg detaches before the outer one. The movement of the radiation is in partial agreement with the expectations from the DLS model, predicting a sudden shift from the target to the X-point. The energy confinement is found to be related to detachment, but there seems to be some margin between the radiation on the inner leg reaching the X-point and confinement being affected, a beneficial characteristic if it could be extrapolated to future reactors. For increasing PSOL the particle flux roll over is almost unaffected, while radiative detachment occurs at higher density in both legs, but much higher on the outer, suggesting an uneven distribution of the power exiting the core., Comment: Submitted to Nuclear Materials and Energy

This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies' internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA., Comment: Accepted at ECCV 2024, preprint, Computer Vision, Data Annotation

Fusion power plants require ELM-free, detached operation to prevent divertor damage and erosion. The separatrix operational space (SepOS) is proposed as a tool for identifying access to the type-I ELM-free quasi-continuous exhaust regime. In this work, we recast the SepOS framework using simple parameters and present dedicated ASDEX Upgrade discharges to demonstrate how to interpret its results. Analyzing an extended ASDEX Upgrade database consisting of 6688 individual measurements, we show that SepOS accurately describes how the H-mode boundary varies with plasma current and magnetic field strength. We then introduce a normalized SepOS framework and LH minimum scaling and show that normalized boundaries across multiple machines are nearly identical, suggesting that the normalized SepOS can be used to translate results between different machines. The LH minimum density predicted by SepOS is found to closely match an experimentally determined multi-machine scaling, which provides a further indirect validation of SepOS across multiple devices. Finally, we demonstrate how SepOS can be used predictively, identifying a viable QCE operational point for SPARC, at a separatrix density of 4e20/m3, a separatrix temperature of 156eV and an alpha-t of 0.7 - a value solidly within the QCE operational space on ASDEX Upgrade. This demonstrates how SepOS provides a concise, intuitive method for scoping ELM-free operation on next-step devices., Comment: Conference paper for PSI-26 (Plasma Surface Interactions)

Fusion power plants will require detachment to mitigate sputtering and keep divertor heat fluxes at tolerable levels. Controlling detachment on these devices may require the use of real-time scrape-off-layer modeling to complement the limited set of available diagnostics. In this work, we use the configurable Hermes-3 edge modeling framework to perform time-dependent, fixed-fraction-impurity 1D detachment simulations. Although currently far from real-time, these simulations are used to investigate time-dependent effects and the minimum physics set required for control-relevant modeling. We show that these simulations reproduce the expected rollover of the target ion flux - a typical characteristic of detachment onset. We also perform scans of the input heat flux and impurity concentration and show that the steady-state results closely match the scalings predicted by the 0D time-independent Lengyel-Goedheer model. This allows us to indirectly compare to SOLPS simulations, which find a similar scaling but a lower value for the impurity concentration required for detachment for given upstream conditions. We use this result to suggest a series of improvements for the Hermes simulations, and finally show simulations demonstrating the impact of time-dependence., Comment: Conference paper for PSI-26 (26th Conference on Plasma Surface Interactions, Marseille)

Conditions similar to those at the end of the divertor leg in a tokamak were replicated in the linear plasma machine Magnum-PSI. The neutral pressure in the target chamber is then increased to cause the target to transition from an attached to a detached state. Superimposed to this steady state regime, ELM-like pulses are reproduced, resulting in a sudden increase in plasma temperature and density, such that the heat flux increases transiently by half an order of magnitude. Visible light emission, target thermography, and Thomson scattering are used to demonstrate that the higher the neutral pressure the more energy is removed from the ELM-like pulse in the volume. If the neutral pressure is sufficiently high, the ELM-like pulse can be prevented from affecting the target and the plasma energy is fully dissipated in the volume instead (ID 4 in Table 1). The visible light images allow the division of the ELM-plasma interaction process of ELM energy dissipation into 3 "stages" ranging from no dissipation to full dissipation (the target plasma is detached). In the second publication related to this study, spectroscopic data is analysed with a Bayesian approach, to acquire insights into the significance of molecular processes in dissipating the plasma energy and particles., Comment: 16 pages, 17 figures, Submitted to Nuclear Fusion

Despite the many recent achievements in developing and deploying social robotics, there are still many underexplored environments and applications for which systematic evaluation of such systems by end-users is necessary. While several robotic platforms have been used in gerontological healthcare, the question of whether or not a social interactive robot with multi-modal conversational capabilities will be useful and accepted in real-life facilities is yet to be answered. This paper is an attempt to partially answer this question, via two waves of experiments with patients and companions in a day-care gerontological facility in Paris with a full-sized humanoid robot endowed with social and conversational interaction capabilities. The software architecture, developed during the H2020 SPRING project, together with the experimental protocol, allowed us to evaluate the acceptability (AES) and usability (SUS) with more than 60 end-users. Overall, the users are receptive to this technology, especially when the robot perception and action skills are robust to environmental clutter and flexible to handle a plethora of different interactions.

Instance segmentation of neurons in volumetric light microscopy images of nervous systems enables groundbreaking research in neuroscience by facilitating joint functional and morphological analyses of neural circuits at cellular resolution. Yet said multi-neuron light microscopy data exhibits extremely challenging properties for the task of instance segmentation: Individual neurons have long-ranging, thin filamentous and widely branching morphologies, multiple neurons are tightly inter-weaved, and partial volume effects, uneven illumination and noise inherent to light microscopy severely impede local disentangling as well as long-range tracing of individual neurons. These properties reflect a current key challenge in machine learning research, namely to effectively capture long-range dependencies in the data. While respective methodological research is buzzing, to date methods are typically benchmarked on synthetic datasets. To address this gap, we release the FlyLight Instance Segmentation Benchmark (FISBe) dataset, the first publicly available multi-neuron light microscopy dataset with pixel-wise annotations. In addition, we define a set of instance segmentation metrics for benchmarking that we designed to be meaningful with regard to downstream analyses. Lastly, we provide three baselines to kick off a competition that we envision to both advance the field of machine learning regarding methodology for capturing long-range data dependencies, and facilitate scientific discovery in basic neuroscience., Comment: CVPR2024, Project page: https://kainmueller-lab.github.io/fisbe

Given two rational univariate polynomials, the Wasserstein distance of their associated measures is an algebraic number. We determine the algebraic degree of the squared Wasserstein distance, serving as a measure of algebraic complexity of the corresponding optimization problem. The computation relies on the structure of a subpolytope of the Birkhoff polytope, invariant under a transformation induced by complex conjugation., Comment: 17 pages, 3 figures

The real Grassmannian is both a projective variety (via Pl\"ucker coordinates) and an affine variety (via orthogonal projections). We connect these two representations, and we develop the commutative algebra of the latter variety. We introduce the squared Grassmannian, and we study applications to determinantal point processes in statistics., Comment: 19 pages. Theorem 3.8 and Remark 5.11 are new

Although several studies have focused on why school climate is important, the timing of the collection of climate measures should be considered. This is of particular interest to schools that gauge school improvement efforts within a school year and are interested in how climate changes from the beginning to the end of the academic year. We show that there is a tendency for school-level climate measures to fluctuate in a predictable, nontrivial manner (ds = 0.25-0.47). Findings are based on data from 26 secondary schools (using over 20,000 student responses) that had school climate measures taken in fall and spring over 18 months. We show that in the fall, on average, students consistently had a more favorable outlook of the school based on five climate measures. [This is the online version of an article published in "School Psychology Review."]

The present study investigated the relationship between teacher stress and job satisfaction, and examined the role of coping as a moderator. Based on the transactional and Coping-Competence-Context models of stress we expected that coping would serve as a protective factor in the relationship between teacher stress and their job satisfaction. Stress, coping, and satisfaction were measured using single-item scales which are cost-effective and practical instruments for measuring and monitoring teacher stress. A total of 2347 teachers from 93 schools formed the sample for this study. Regression analysis with job satisfaction as the outcome, and stress, coping, and their interaction entered separately were used to test the primary hypothesis. Results showed that there was a negative correlation between stress and job satisfaction, and coping had a significant moderating effect. Increasing levels of stress had less of a negative impact on job satisfaction for teachers with high coping ratings compared to those with average or low coping ratings. Overall, our results suggest that the development of coping skills could be beneficial for mitigating the effects of stress on job satisfaction. [This is the online first version of an article published in "Psychology in the Schools."]

The linear plasma machine Magnum-PSI can replicate similar conditions to those found in a tokamak at the end of the divertor leg. A dedicated capacitor bank, in parallel to the plasma source, can release a sudden burst of energy, leading to a rapid increase in plasma temperature and density, resulting in a transient heat flux increase of half of an order of magnitude, a so called ELM-like pulse. Throughout both the steady state and the pulse, the neutral pressure in the target chamber is then increased, causing the target to transition from an attached to a detached state. In the first paper related to this study\cite{Federici} direct measurements of the plasma properties are used to qualitatively determine the effect of detachment on the ELM-like pulse. This is used to show the importance of molecular assisted reactions. Molecular processes, and especially molecular activated dissociation, are found to be important in the exchange of potential energy with the plasma, while less so in radiating the energy from the ELM-like pulse. At low target chamber pressure, the plasma generated via ionisation during the part of the ELM-like pulse with the higher temperature is more than that produced by the plasma source, a unique case in linear machines. At high target chamber pressure molecular activated recombination contributes up to a third of the total recombination rate, contributing to the reduction of the target particle flux. Some metrics that estimate the energy lost by the plasma per interactions with neutrals, potentially relevant for the portion of the tokamak divertor leg below $\sim10eV$, are then tentatively obtained., Comment: 25 pages, 22 figures. To be published in Nuclear Fusion. This paper is the second related to a single experimental session, the first is: "Effect of detachment on Magnum-PSI ELM-like pulses: I. Direct observations and qualitative results", to be published in Nuclear Fusion

This paper introduces panoptica, a versatile and performance-optimized package designed for computing instance-wise segmentation quality metrics from 2D and 3D segmentation maps. panoptica addresses the limitations of existing metrics and provides a modular framework that complements the original intersection over union-based panoptic quality with other metrics, such as the distance metric Average Symmetric Surface Distance. The package is open-source, implemented in Python, and accompanied by comprehensive documentation and tutorials. panoptica employs a three-step metrics computation process to cover diverse use cases. The efficacy of panoptica is demonstrated on various real-world biomedical datasets, where an instance-wise evaluation is instrumental for an accurate representation of the underlying clinical task. Overall, we envision panoptica as a valuable tool facilitating in-depth evaluation of segmentation methods., Comment: 15 pages, 6 figures, 3 tables

Deep Neural Networks (DNNs) became the standard tool for function approximation with most of the introduced architectures being developed for high-dimensional input data. However, many real-world problems have low-dimensional inputs for which standard Multi-Layer Perceptrons (MLPs) are the default choice. An investigation into specialized architectures is missing. We propose a novel DNN layer called Univariate Radial Basis Function (U-RBF) layer as an alternative. Similar to sensory neurons in the brain, the U-RBF layer processes each individual input dimension with a population of neurons whose activations depend on different preferred input values. We verify its effectiveness compared to MLPs in low-dimensional function regressions and reinforcement learning tasks. The results show that the U-RBF is especially advantageous when the target function becomes complex and difficult to approximate.

Current deep learning-based solutions for image analysis tasks are commonly incapable of handling problems to which multiple different plausible solutions exist. In response, posterior-based methods such as conditional Diffusion Models and Invertible Neural Networks have emerged; however, their translation is hampered by a lack of research on adequate validation. In other words, the way progress is measured often does not reflect the needs of the driving practical application. Closing this gap in the literature, we present the first systematic framework for the application-driven validation of posterior-based methods in inverse problems. As a methodological novelty, it adopts key principles from the field of object detection validation, which has a long history of addressing the question of how to locate and match multiple object instances in an image. Treating modes as instances enables us to perform mode-centric validation, using well-interpretable metrics from the application perspective. We demonstrate the value of our framework through instantiations for a synthetic toy example and two medical vision use cases: pose estimation in surgery and imaging-based quantification of functional tissue parameters for diagnostics. Our framework offers key advantages over common approaches to posterior validation in all three examples and could thus revolutionize performance assessment in inverse problems., Comment: Accepted at Medical Image Analysis. Shared first authors: Tim J. Adler and Jan-Hinrich N\"olke. 24 pages, 9 figures, 1 table

While clinical trials are the state-of-the-art methods to assess the effect of new medication in a comparative manner, benchmarking in the field of medical image analysis is performed by so-called challenges. Recently, comprehensive analysis of multiple biomedical image analysis challenges revealed large discrepancies between the impact of challenges and quality control of the design and reporting standard. This work aims to follow up on these results and attempts to address the specific question of the reproducibility of the participants methods. In an effort to determine whether alternative interpretations of the method description may change the challenge ranking, we reproduced the algorithms submitted to the 2019 Robust Medical Image Segmentation Challenge (ROBUST-MIS). The leaderboard differed substantially between the original challenge and reimplementation, indicating that challenge rankings may not be sufficiently reproducible., Comment: Accepted at BVM 2023