Search

Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare ten public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting and supervised fine-tuning regimes for medical question-answering (QA). For instance, across all tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 22.7% of cases, reach a (statistical) tie in 36.8% of cases, and are significantly worse than their base models in the remaining 40.5% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately in zero-/few-shot prompting; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Meanwhile, we find that after fine-tuning on specific QA tasks, medical LLMs can show performance improvements, but the benefits do not carry over to tasks based on clinical notes. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies., Comment: Extended version of EMNLP 2024 paper arXiv:2411.04118. Includes additional results on clinical note QA tasks and supervised fine-tuning evaluations

Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare seven public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting regime for medical question-answering (QA) tasks. For instance, across the tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 12.1% of cases, reach a (statistical) tie in 49.8% of cases, and are significantly worse than their base models in the remaining 38.2% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies., Comment: This version was published at EMNLP 2024 Main Conference as a Long Paper (Oral). See the extended version (arXiv:2411.08870) for additional results on QA tasks based on clinical notes and evaluations in the supervised fine-tuning regime

With hash tables being one of the most used data structures, Lehmann, Sanders and Walzer propose a novel, light-weight hash table, referred to as Slick Hash. Their idea is to hit a sweet spot between space consumption and speed. Building on the theoretical ideas by the authors, an implementation and experiments are required to evaluate the practical performance of Slick Hash. This work contributes to fulfilling this requirement by providing a basic implementation of Slick Hash, an analysis of its performance, and an evaluation of the entry deletion, focusing on the impact of backyard cleaning. The findings are discussed, and a conclusion is drawn.

The possibilities of robot control have multiplied across various domains through the application of deep reinforcement learning. To overcome safety and sampling efficiency issues, deep reinforcement learning models can be trained in a simulation environment, allowing for faster iteration cycles. This can be enhanced further by parallelizing the training process using GPUs. NVIDIA's open-source robot learning framework Orbit leverages this potential by wrapping tensor-based reinforcement learning libraries for high parallelism and building upon Isaac Sim for its simulations. We contribute a detailed description of the implementation of a benchmark reinforcement learning task, namely box pushing, using Orbit. Additionally, we benchmark the performance of our implementation in comparison to a CPU-based implementation and report the performance metrics. Finally, we tune the hyper parameters of our implementation and show that we can generate significantly more samples in the same amount of time by using Orbit.

In June 1976, a pristine meteorite stone weighing approximately 1 kg, fully covered with a fresh black fusion crust, was collected on a mountain road in the high-altitude Alpine environment. The recovery took place while clearing the remnants of a snow avalanche, 2 km northwest of Ischgl in Austria. Subsequent to its retrieval, the specimen remained in the finder's private residence without undergoing any scientific examination or identification until 2008, when it was brought to the University of Innsbruck. The sample was classified as a well-preserved LL6 chondrite, with a W0 weathering grade, implying a relatively short time between the meteorite fall and its retrieval. To investigate the potential connection between the Ischgl meteorite and a recorded fireball event, we have reviewed all documented fireballs ever photographed by German fireball camera stations. This examination led us to identify the fireball EN241170 observed in Germany by ten different European Network stations on the night of November 23/24, 1970, as the most likely candidate. We employed state-of-the-art techniques to reconstruct the fireball's trajectory, and to reproduce both its luminous and dark flight phases in detail. We find that the determined strewn field and the generated heat map closely align with the recovery location of the Ischgl meteorite. Furthermore, the measured radionuclide data reported here indicate that the pre-atmospheric size of the Ischgl meteoroid is consistent with the mass estimate inferred from our deceleration analysis along the trajectory. Our findings strongly support the conclusion that the Ischgl meteorite originated from the EN241170 fireball, effectively establishing it as a confirmed meteorite fall. This discovery enables to determine, along with the physical properties, also the heliocentric orbit and cosmic history of the Ischgl meteorite., Comment: Accepted for publication in Meteoritics & Planetary Science

The presence of inequity is a fundamental problem in the outcomes of decision-making systems, especially when human lives are at stake. Yet, estimating notions of unfairness or inequity is difficult, particularly if they rely on hard-to-measure concepts such as risk. Such measurements of risk can be accurately obtained when no unobserved confounders have jointly influenced past decisions and outcomes. However, in the real world, this assumption rarely holds. In this paper, we show a surprising result that one can still give meaningful bounds on treatment rates to high-risk individuals, even when entirely eliminating or relaxing the assumption that all relevant risk factors are observed. We use the fact that in many real-world settings (e.g., the release of a new treatment) we have data from prior to any allocation to derive unbiased estimates of risk. This result is of immediate practical interest: we can audit unfair outcomes of existing decision-making systems in a principled manner. For instance, in a real-world study of Paxlovid allocation, our framework provably identifies that observed racial inequity cannot be explained by unobserved confounders of the same strength as important observed covariates., Comment: AISTATS 2024

The third ML4H symposium was held in person on December 10, 2023, in New Orleans, Louisiana, USA. The symposium included research roundtable sessions to foster discussions between participants and senior researchers on timely and relevant topics for the \ac{ML4H} community. Encouraged by the successful virtual roundtables in the previous year, we organized eleven in-person roundtables and four virtual roundtables at ML4H 2022. The organization of the research roundtables at the conference involved 17 Senior Chairs and 19 Junior Chairs across 11 tables. Each roundtable session included invited senior chairs (with substantial experience in the field), junior chairs (responsible for facilitating the discussion), and attendees from diverse backgrounds with interest in the session's topic. Herein we detail the organization process and compile takeaways from these roundtable discussions, including recent advances, applications, and open challenges for each topic. We conclude with a summary and lessons learned across all roundtables. This document serves as a comprehensive review paper, summarizing the recent advancements in machine learning for healthcare as contributed by foremost researchers in the field., Comment: ML4H 2023, Research Roundtables

Drawing causal inferences from observational studies (OS) requires unverifiable validity assumptions; however, one can falsify those assumptions by benchmarking the OS with experimental data from a randomized controlled trial (RCT). A major limitation of existing procedures is not accounting for censoring, despite the abundance of RCTs and OSes that report right-censored time-to-event outcomes. We consider two cases where censoring time (1) is independent of time-to-event and (2) depends on time-to-event the same way in OS and RCT. For the former, we adopt a censoring-doubly-robust signal for the conditional average treatment effect (CATE) to facilitate an equivalence test of CATEs in OS and RCT, which serves as a proxy for testing if the validity assumptions hold. For the latter, we show that the same test can still be used even though unbiased CATE estimation may not be possible. We verify the effectiveness of our censoring-aware tests via semi-synthetic experiments and analyze RCT and OS data from the Women's Health Initiative study., Comment: Artificial Intelligence and Statistics (AISTATS) 2024

Randomized Controlled Trials (RCT)s are relied upon to assess new treatments, but suffer from limited power to guide personalized treatment decisions. On the other hand, observational (i.e., non-experimental) studies have large and diverse populations, but are prone to various biases (e.g. residual confounding). To safely leverage the strengths of observational studies, we focus on the problem of falsification, whereby RCTs are used to validate causal effect estimates learned from observational data. In particular, we show that, given data from both an RCT and an observational study, assumptions on internal and external validity have an observable, testable implication in the form of a set of Conditional Moment Restrictions (CMRs). Further, we show that expressing these CMRs with respect to the causal effect, or "causal contrast", as opposed to individual counterfactual means, provides a more reliable falsification test. In addition to giving guarantees on the asymptotic properties of our test, we demonstrate superior power and type I error of our approach on semi-synthetic and real world datasets. Our approach is interpretable, allowing a practitioner to visualize which subgroups in the population lead to falsification of an observational study., Comment: Artificial Intelligence and Statistics 2023

Randomized Controlled Trials (RCTs) represent a gold standard when developing policy guidelines. However, RCTs are often narrow, and lack data on broader populations of interest. Causal effects in these populations are often estimated using observational datasets, which may suffer from unobserved confounding and selection bias. Given a set of observational estimates (e.g. from multiple studies), we propose a meta-algorithm that attempts to reject observational estimates that are biased. We do so using validation effects, causal effects that can be inferred from both RCT and observational data. After rejecting estimators that do not pass this test, we generate conservative confidence intervals on the extrapolated causal effects for subgroups not observed in the RCT. Under the assumption that at least one observational estimator is asymptotically normal and consistent for both the validation and extrapolated effects, we provide guarantees on the coverage probability of the intervals output by our algorithm. To facilitate hypothesis testing in settings where causal effect transportation across datasets is necessary, we give conditions under which a doubly-robust estimator of group average treatment effects is asymptotically normal, even when flexible machine learning methods are used for estimation of nuisance parameters. We illustrate the properties of our approach on semi-synthetic and real world datasets, and show that it compares favorably to standard meta-analysis techniques., Comment: Conference on Neural Information Processing Systems, 2022

We give a method for proactively identifying small, plausible shifts in distribution which lead to large differences in model performance. These shifts are defined via parametric changes in the causal mechanisms of observed variables, where constraints on parameters yield a "robustness set" of plausible distributions and a corresponding worst-case loss over the set. While the loss under an individual parametric shift can be estimated via reweighting techniques such as importance sampling, the resulting worst-case optimization problem is non-convex, and the estimate may suffer from large variance. For small shifts, however, we can construct a local second-order approximation to the loss under shift and cast the problem of finding a worst-case shift as a particular non-convex quadratic optimization problem, for which efficient algorithms are available. We demonstrate that this second-order approximation can be estimated directly for shifts in conditional exponential family models, and we bound the approximation error. We apply our approach to a computer vision task (classifying gender from images), revealing sensitivity to shifts in non-causal attributes., Comment: NeurIPS 2022; Equal Contribution by Nikolaj/Michael, order determined by coin flip

Several problems in statistics involve the combination of high-variance unbiased estimators with low-variance estimators that are only unbiased under strong assumptions. A notable example is the estimation of causal effects while combining small experimental datasets with larger observational datasets. There exist a series of recent proposals on how to perform such a combination, even when the bias of the low-variance estimator is unknown. To build intuition for the differing trade-offs of competing approaches, we argue for examining the finite-sample estimation error of each approach as a function of the unknown bias. This includes understanding the bias threshold -- the largest bias for which a given approach improves over using the unbiased estimator alone. Though this lens, we review several recent proposals, and observe in simulation that different approaches exhibits qualitatively different behavior. We also introduce a simple alternative approach, which compares favorably in simulation to recent alternatives, having a higher bias threshold and generally making a more conservative trade-off between best-case performance (when the bias is zero) and worst-case performance (when the bias is adversarially chosen). More broadly, we prove that for any amount of (unknown) bias, the MSE of this estimator can be bounded in a transparent way that depends on the variance / covariance of the underlying estimators that are being combined.

Mars exploration motivates the search for extraterrestrial life, the development of space technologies, and the design of human missions and habitations. Here we seek new insights and pose unresolved questions relating to the natural history of Mars, habitability, robotic and human exploration, planetary protection, and the impacts on human society. Key observations and findings include:(1)high escape rates of early Mars' atmosphere, including loss of water, impact present-day habitability;(2)putative fossils on Mars will likely be ambiguous biomarkers for life;(3)microbial contamination resulting from human habitation is unavoidable;(4)based on Mars' current planetary protection category, robotic payload(s) should characterize the local martian environment for any life-forms prior to human habitation. Some of the outstanding questions are:(1)which interpretation of the hemispheric dichotomy of the planet is correct;(2)to what degree did deep-penetrating faults transport subsurface liquids to Mars' surface;(3)in what abundance are carbonates formed by atmospheric processes;(4)what properties of martian meteorites could be used to constrain their source locations;(5)the origin(s) of organic macromolecules;(6)was/is Mars inhabited;(7)how can missions designed to uncover microbial activity in the subsurface eliminate potential false positives caused by microbial contaminants from Earth;(8)how can we ensure that humans and microbes form a stable and benign biosphere;(9)should humans relate to putative extraterrestrial life from a biocentric viewpoint (preservation of all biology), or anthropocentric viewpoint of expanding habitation of space? Studies of Mars' evolution can shed light on the habitability of extrasolar planets. In addition, Mars exploration can drive future policy developments and confirm (or put into question) the feasibility and/or extent of human habitability of space.

Individuals often make different decisions when faced with the same context, due to personal preferences and background. For instance, judges may vary in their leniency towards certain drug-related offenses, and doctors may vary in their preference for how to start treatment for certain types of patients. With these examples in mind, we present an algorithm for identifying types of contexts (e.g., types of cases or patients) with high inter-decision-maker disagreement. We formalize this as a causal inference problem, seeking a region where the assignment of decision-maker has a large causal effect on the decision. Our algorithm finds such a region by maximizing an empirical objective, and we give a generalization bound for its performance. In a semi-synthetic experiment, we show that our algorithm recovers the correct region of heterogeneity accurately compared to baselines. Finally, we apply our algorithm to real-world healthcare datasets, recovering variation that aligns with existing clinical knowledge., Comment: To appear in NeurIPS 2021

Acoustic meta-atoms serve as the building blocks of metamaterials, with linear properties designed to achieve functions such as beam steering, cloaking and focusing. They have also been used to shape the characteristics of incident acoustic fields, which led to the manipulation of acoustic radiation force and torque for development of acoustic tweezers with improved spatial resolution. However, acoustic radiation force and torque also depend on the shape of the object, which strongly affects its scattering properties. We show that by designing linear properties of an object using metamaterial concepts, the nonlinear acoustic effects of radiation force and torque can be controlled. Trapped objects are typically small compared to the wavelength, and are described as particles, inducing monopole and dipole scattering. We extend such models to a polarizability tensor including Willis coupling terms, as a measure of asymmetry, capturing the significance of geometrical features. We apply our model to a three-dimensional, sub-wavelength meta-atom with maximal Willis coupling, demonstrating that the force and the torque can be reversed relative to an equivalent symmetrical particle. By considering shape asymmetry in the acoustic radiation force and torque, Gorkov's fundamental theory of acoustophoresis is thereby extended. Asymmetrical shapes influence the acoustic fields by shifting the stable trapping location, highlighting a potential for tunable, shape-dependent particle sorting.

We investigate the feasibility and demonstrate the merits of using Mars Orbiter Laser Altimeter (MOLA) profiles to retrieve seasonal height variations of CO2 snow/ice cap in Mars' polar areas by applying a co-registration strategy. We present a prototype analysis on the research region of [85.75{\deg}S, 86.25{\deg}S, 300{\deg}E, 330{\deg}E] that is located on the residual south polar cap. Our method comprises the recomputation of MOLA footprint coordinates with an updated Mars Global Surveyor (MGS) ephemeris and a revised Mars rotation model. The reprocessed MOLA dataset at the South Pole of Mars (poleward of 78{\deg}S) is then self-registered to form a coherent reference digital terrain model (DTM). We co-register segments of reprocessed MOLA profiles to the self-registered MOLA reference DTM to obtain the temporal height differences at either footprints or cross-overs. Subsequently, a two-step Regional Pseudo Cross-over Adjustment (RPCA) procedure is proposed and applied to post-correct the aforementioned temporal height differences for a temporal systematic bias and other residual errors. These pseudo cross-overs are formed by profile pairs that do not necessarily intersect, but are connected through the underlaying DTM. Finally, CO2 snow/ice temporal height variation is obtained by median-filtering those post-corrected temporal height differences. The precision of the derived height change time series is ~4.9 cm. The peak-to-peak height variation is estimated to be ~2 m. In addition, a pronounced "pit" (transient height accumulation) of ~0.5 m in magnitude centered at Ls=210{\deg} in southern spring is observed. The proposed method opens the possibility to map the seasonal CO2 snow/ice height variations at the entire North and South polar regions of Mars.

Acoustophoresis deals with the manipulation of sub-wavelength scatterers in an incident acoustic field. The geometric details of manipulated particles are often neglected by replacing them with equivalent symmetric geometries such as spheres, spheroids, cylinders or disks. It has been demonstrated that geometric asymmetry, represented by Willis coupling terms, can strongly affect the scattering of a small object, hence neglecting these terms may miss important force contributions. In this work, we present a generalized formalism of acoustic radiation force and radiation torque based on the polarizability tensor, where Willis coupling terms are included to account for geometric asymmetry. Following Gorkov's approach, the effects of geometric asymmetry are explicitly formulated as additional terms in the radiation force and torque expressions. By breaking the symmetry of a sphere along one axis using intrusion and protrusion, we characterize the changes in the force and torque in terms of partial components, associated with the direct and Willis Coupling coefficients of the polarizability tensor. We investigate in detail the cases of standing and travelling plane waves, showing how the equilibrium positions and angles are shifted by these additional terms. We show that while the contributions of asymmetry to the force are often negligible for small particles, these terms greatly affect the radiation torque. Our presented theory, providing a way of calculating radiation force and torque directly from polarizability coefficients, shows that in general it is essential to account for shape of objects undergoing acoustophoretic manipulation, and this may have important implications for applications such as the manipulation of biological cells., Comment: 11 pages, 6 figures

Acoustic focusing plays a pivotal role in a wide variety of applications ranging from medical science to nondestructive testing. Previous works have shown that acoustic metagratings can overcome the inherent efficiency limitations of gradient metasurfaces in beam steering. In this work, we propose a new design principle for acoustic metalenses, based on metagratings, to achieve efficient ultrasonic focusing. We achieve beam focusing by locally controlling the excitation of a single diffraction order with the use of adiabatically varying metagratings over the lens aperture. A set of metagratings is optimized by a semi-analytical approach using a genetic algorithm, enabling efficient anomalous reflection for a wide range of reflection angles. Numerical results reveal that our metalens can effectively focus impinging ultrasonic waves to a focal point of FWHM = 0.364{\lambda}. The focusing performance of the metalens is demonstrated experimentally, validating our proposed approach.

We propose a method for learning linear models whose predictive performance is robust to causal interventions on unobserved variables, when noisy proxies of those variables are available. Our approach takes the form of a regularization term that trades off between in-distribution performance and robustness to interventions. Under the assumption of a linear structural causal model, we show that a single proxy can be used to create estimators that are prediction optimal under interventions of bounded strength. This strength depends on the magnitude of the measurement noise in the proxy, which is, in general, not identifiable. In the case of two proxy variables, we propose a modified estimator that is prediction optimal under interventions up to a known strength. We further show how to extend these estimators to scenarios where additional information about the "test time" intervention is available during training. We evaluate our theoretical findings in synthetic experiments and using real data of hourly pollution levels across several cities in China., Comment: ICML 2021 (to appear)