Search

Many training methods, such as Adam(W) and Shampoo, learn a positive-definite curvature matrix and apply an inverse root before preconditioning. Recently, non-diagonal training methods, such as Shampoo, have gained significant attention; however, they remain computationally inefficient and are limited to specific types of curvature information due to the costly matrix root computation via matrix decomposition. To address this, we propose a Riemannian optimization approach that dynamically adapts spectral-factorized positive-definite curvature estimates, enabling the efficient application of arbitrary matrix roots and generic curvature learning. We demonstrate the efficacy and versatility of our approach in positive-definite matrix optimization and covariance adaptation for gradient-free optimization, as well as its efficiency in curvature learning for neural net training., Comment: technical report

Sparse variational Gaussian process (GP) approximations based on inducing points have become the de facto standard for scaling GPs to large datasets, owing to their theoretical elegance, computational efficiency, and ease of implementation. This paper introduces a provably tighter variational approximation by relaxing the standard assumption that the conditional approximate posterior given the inducing points must match that in the prior. The key innovation is to modify the conditional posterior to have smaller variances than that of the prior at the training points. We derive the collapsed bound for the regression case, describe how to use the proposed approximation in large data settings, and discuss its application to handle orthogonally structured inducing points and GP latent variable models. Extensive experiments on regression benchmarks, classification, and latent variable models demonstrate that the proposed approximation consistently matches or outperforms standard sparse variational GPs while maintaining the same computational cost. An implementation will be made available in all popular GP packages.

Vision-language models (VLMs) excel in tasks such as visual question answering and image captioning. However, VLMs are often limited by their use of pretrained image encoders, like CLIP, leading to image understanding errors that hinder overall performance. On top of that, real-world applications often require the model to be continuously adapted as new and often limited data continuously arrive. To address this, we propose LoRSU (Low-Rank Adaptation with Structured Updates), a robust and computationally efficient method for selectively updating image encoders within VLMs. LoRSU introduces structured and localized parameter updates, effectively correcting performance on previously error-prone data while preserving the model's general robustness. Our approach leverages theoretical insights to identify and update only the most critical parameters, achieving significant resource efficiency. Specifically, we demonstrate that LoRSU reduces computational overhead by over 25x compared to full VLM updates, without sacrificing performance. Experimental results on VQA tasks in the few-shot continual learning setting, validate LoRSU's scalability, efficiency, and effectiveness, making it a compelling solution for image encoder adaptation in resource-constrained environments.

The mean apparent magnitude of Starlink Mini Direct-To-Cell (DTC) satellites observed in brightness mitigation mode is 5.16, while the mean of magnitudes adjusted to a uniform distance of 1,000 km is 6.47. The DTCs have faded since early in 2024 because SpaceX subsequently adjusted the spacecraft attitudes to dim them. A physical model for satellite brightness that fits the observations is described.

High Mountain Asia holds the largest concentration of frozen water outside the polar regions, serving as a crucial water source for more than 1.9 billion people. In the face of climate change, precipitation represents the largest source of uncertainty for hydrological modelling in this area. Future precipitation predictions remain challenging due to complex orography, lack of in situ hydrological observations, and limitations in climate model resolution and parametrisation for this region. To address the uncertainty posed by these challenges, climate models are often aggregated into multi-model ensembles. While multi-model ensembles are known to improve the predictive accuracy and analysis of future climate projections, consensus regarding how models are aggregated is lacking. In this study, we propose a probabilistic machine learning framework to systematically combine 13 regional climate models from the Coordinated Regional Downscaling Experiment (CORDEX) over High Mountain Asia. Our approach accounts for seasonal and spatial biases within the models, enabling the prediction of more faithful precipitation distributions. The framework is validated against gridded historical precipitation data and is used to generate projections for the near-future (2036-2065) and far-future (2066-2095) under RCP4.5 and RCP8.5 scenarios., Comment: 16 pages 8 figures (main text), 32 pages 14 figures (total)

Galaxy groups with $M_{tot} \lesssim 10^{14}$ $M_\odot$ and up to a few tens of members are the most common galaxy environment, marking the transition between field and massive clusters. Identifying groups plays a crucial role in understanding structure formation and galaxy evolution. Modern deep surveys allow us to build well-characterized samples of groups up to the regime where structures were taking shape. We aimed to build the largest deep catalog of galaxy groups to date over the COSMOS-Web field effective area of 0.45 deg$^2$, leveraging the deep high quality data of the new COSMOS-Web photometric catalog resulted from the James Webb Space Telescope observations of the COSMOS-Web field. We performed the group search with the AMICO algorithm, a linear matched filter based on an analytical model for the group signal. AMICO has already been tested in wide and deep field surveys, including COSMOS data up to $z=2$. In this work, we tested the algorithm performances at even higher redshift and searched for protocluster cores at $z>2$. We compiled a list of known protoclusters in COSMOS at $2 \leq z \leq 3.7$, matched them with our detections and studied the clustering of the detected cores. We estimated purity and completeness of our sample by creating data-driven mocks with the SinFoniA code and linked signal-to-noise to purity. We detected 1678 groups in the COSMOS-Web field up to $z=3.7$, including lists of members extending nearly two magnitudes deeper than the previous AMICO-COSMOS catalog. 756 groups were detected with purity of 80\%. More than 500 groups have their redshift confirmed by assigning spectroscopic counterparts. This group catalog offers a unique opportunity to explore galaxy evolution in different environments spanning $\sim$12 Gyr and to study groups, from the least rich population to the formation of the most massive clusters., Comment: 17 pages, 14 figures, 2 tables (including appendices), submitted to Astronomy & Astrophysics (A&A)

Biased quorum-sensing active particles (bQSAPs) are shown to extend notions of dynamic critical phenomena beyond active phase separation into the prototypical nonequilibrium setting of driven transport, where characteristic emergent behaviour is not stationary. To do so, we construct an effective field theory in a single order-parameter -- a non-stationary analogue of active Model B -- which accounts for the fact that different aspects of bQSAPs can only be cast in terms of passive thermodynamics under an appropriate choice of inertial frame. This codifies the movement of phase boundaries due to nonequilibrium fluxes between coexisting bulk phases in terms of a difference in effective chemical potentials and therefore an unequal tangent construction on a bulk free energy density. The result is an anomalous phase structure; binodals are permitted to cross spinodal lines so that criticality is no longer constrained to a single point. Instead, criticality -- with exponents that are seemingly unchanged from symmetric QSAPs -- is shown to exist along a line that marks the entry to an otherwise forbidden region of phase space. The interior of this region is not critical in the conventional sense but retains certain features of criticality, which we term pseudo-critical. Since a Ginzburg criterion cannot be satisfied, fluctuations cannot be ignored, no matter how small, and manifest at the scale of macroscopic features. However, finite-wavenumber fluctuations grow at non-vanishing rates and are characterized by non-trivial dispersion relations. The resulting interplay is used to explain how different areas of phase space correspond to different types of micro- and meso-phase separation.

Prostate cancer is a leading cause of cancer-related deaths among men. The recent development of high frequency, micro-ultrasound imaging offers improved resolution compared to conventional ultrasound and potentially a better ability to differentiate clinically significant cancer from normal tissue. However, the features of prostate cancer remain subtle, with ambiguous borders with normal tissue and large variations in appearance, making it challenging for both machine learning and humans to localize it on micro-ultrasound images. We propose a novel Mask Enhanced Deeply-supervised Micro-US network, termed MedMusNet, to automatically and more accurately segment prostate cancer to be used as potential targets for biopsy procedures. MedMusNet leverages predicted masks of prostate cancer to enforce the learned features layer-wisely within the network, reducing the influence of noise and improving overall consistency across frames. MedMusNet successfully detected 76% of clinically significant cancer with a Dice Similarity Coefficient of 0.365, significantly outperforming the baseline Swin-M2F in specificity and accuracy (Wilcoxon test, Bonferroni correction, p-value<0.05). While the lesion-level and patient-level analyses showed improved performance compared to human experts and different baseline, the improvements did not reach statistical significance, likely on account of the small cohort. We have presented a novel approach to automatically detect and segment clinically significant prostate cancer on B-mode micro-ultrasound images. Our MedMusNet model outperformed other models, surpassing even human experts. These preliminary results suggest the potential for aiding urologists in prostate cancer diagnosis via biopsy and treatment decision-making.

The eROSITA instrument on board Spectrum-Roentgen-Gamma has completed four scans of the X-ray sky, leading to the detection of almost one million X-ray sources in eRASS1 only, including multiple new X-ray binary candidates. We report on analysis of the X-ray binary 1eRASS J085039.9-421151, using a ~55\,ks long NuSTAR observation, following its detection in each eROSITA scan. Analysis of the eROSITA and NuSTAR X-ray spectra in combination with X-shooter data of the optical counterpart provide evidence of an X-ray binary with a red supergiant (RSG) companion, confirming previous results, although we determine a cooler spectral type of M2-3, owing to the presence of TiO bands in the optical and near infrared spectra. The X-ray spectrum is well-described by an absorbed power law with a high energy cutoff typically applied for accreting high mass X-ray binaries. In addition, we detect a strong fluorescent neutral iron line with an equivalent width of ~700\,eV and an absorption edge, the latter indicating strong absorption by a partial covering component. It is unclear if the partial absorber is ionised. There is no significant evidence of a cyclotron resonant scattering feature. We do not detect any pulsations in the NuSTAR lightcurves, possibly on account of a large spin period that goes undetected due to insufficient statistics at low frequencies or potentially large absorption that causes pulsations to be smeared out. Even so, the low persistent luminosity, the spectral parameters observed (photon index, $\Gamma<1.0$), and the minuscule likelihood of detection of RSG-black hole systems, suggest that the compact object is a neutron star., Comment: 17 pages, 11 figures, accepted by A&A

Modelling partial differential equations (PDEs) is of crucial importance in science and engineering, and it includes tasks ranging from forecasting to inverse problems, such as data assimilation. However, most previous numerical and machine learning approaches that target forecasting cannot be applied out-of-the-box for data assimilation. Recently, diffusion models have emerged as a powerful tool for conditional generation, being able to flexibly incorporate observations without retraining. In this work, we perform a comparative study of score-based diffusion models for forecasting and assimilation of sparse observations. In particular, we focus on diffusion models that are either trained in a conditional manner, or conditioned after unconditional training. We address the shortcomings of existing models by proposing 1) an autoregressive sampling approach that significantly improves performance in forecasting, 2) a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths, and 3) a hybrid model which employs flexible pre-training conditioning on initial conditions and flexible post-training conditioning to handle data assimilation. We empirically show that these modifications are crucial for successfully tackling the combination of forecasting and data assimilation, a task commonly encountered in real-world scenarios., Comment: Accepted at NeurIPS 2024

Astronomical solutions provide calculated orbital and rotational parameters of solar system bodies based on the dynamics and physics of the solar system. Application of astronomical solutions in the Earth sciences has revolutionized our understanding in at least two areas of active research. (i) The Astronomical (or Milankovic) forcing of climate on time scales > ~10 kyr and (ii) the dating of geologic archives. The latter has permitted the development of the astronomical time scale, widely used today to reconstruct highly accurate geological dates and chronologies. The tasks of computing vs. applying astronomical solutions are usually performed by investigators from different backgrounds, which has led to confusion and recent inaccurate results on the side of the applications. Here we review astronomical solutions and Milankovic forcing in the Earth sciences, primarily aiming at clarifying the astronomical basis, applicability, and limitations of the solutions. We provide a summary of current up-to-date and outdated astronomical solutions and their valid time span. We discuss the fundamental limits imposed by dynamical solar system chaos on astronomical calculations and geological/astrochronological applications. We illustrate basic features of chaotic behavior using a simple mechanical system, i.e., the driven pendulum. Regarding so-called astronomical "metronomes", we point out that the current evidence does not support the notion of generally stable and prominent metronomes for universal use in astrochronology and cyclostratigraphy. We also describe amplitude and frequency modulation of astronomical forcing signals and the relation to their expression in cyclostratigraphic sequences. Furthermore, the various quantities and terminology associated with Earth's axial precession are discussed in detail. Finally, we provide some suggestions regarding practical considerations., Comment: Earth-Science Reviews. Final revised version in press

Background: Machine learning methods for clinical named entity recognition and entity normalization systems can utilize both labeled corpora and Knowledge Graphs (KGs) for learning. However, infrequently occurring concepts may have few mentions in training corpora and lack detailed descriptions or synonyms, even in large KGs. For Disease Entity Recognition (DER) and Disease Entity Normalization (DEN), this can result in fewer high quality training examples relative to the number of known diseases. Large Language Model (LLM) generation of synthetic training examples could improve performance in these information extraction tasks. Methods: We fine-tuned a LLaMa-2 13B Chat LLM to generate a synthetic corpus containing normalized mentions of concepts from the Unified Medical Language System (UMLS) Disease Semantic Group. We measured overall and Out of Distribution (OOD) performance for DER and DEN, with and without synthetic data augmentation. We evaluated performance on 3 different disease corpora using 4 different data augmentation strategies, assessed using BioBERT for DER and SapBERT and KrissBERT for DEN. Results: Our synthetic data yielded a substantial improvement for DEN, in all 3 training corpora the top 1 accuracy of both SapBERT and KrissBERT improved by 3-9 points in overall performance and by 20-55 points in OOD data. A small improvement (1-2 points) was also seen for DER in overall performance, but only one dataset showed OOD improvement. Conclusion: LLM generation of normalized disease mentions can improve DEN relative to normalization approaches that do not utilize LLMs to augment data with synthetic mentions. Ablation studies indicate that performance gains for DEN were only partially attributable to improvements in OOD performance. The same approach has only a limited ability to improve DER. We make our software and dataset publicly available., Comment: 21 pages, 3 figures, 7 tables

Many important problems require modelling large-scale spatio-temporal datasets, with one prevalent example being weather forecasting. Recently, transformer-based approaches have shown great promise in a range of weather forecasting problems. However, these have mostly focused on gridded data sources, neglecting the wealth of unstructured, off-the-grid data from observational measurements such as those at weather stations. A promising family of models suitable for such tasks are neural processes (NPs), notably the family of transformer neural processes (TNPs). Although TNPs have shown promise on small spatio-temporal datasets, they are unable to scale to the quantities of data used by state-of-the-art weather and climate models. This limitation stems from their lack of efficient attention mechanisms. We address this shortcoming through the introduction of gridded pseudo-token TNPs which employ specialised encoders and decoders to handle unstructured observations and utilise a processor containing gridded pseudo-tokens that leverage efficient attention mechanisms. Our method consistently outperforms a range of strong baselines on various synthetic and real-world regression tasks involving large-scale data, while maintaining competitive computational efficiency. The real-life experiments are performed on weather data, demonstrating the potential of our approach to bring performance and computational benefits when applied at scale in a weather modelling pipeline.

XTE J1829-098 is a transient X-ray pulsar with a period of ~7.8 s. It is a candidate Be star system, although the evidence for this is not yet definitive. We investigated the twenty-year long X-ray light curve using the Rossi X-ray Timing Explorer Proportional Counter Array (PCA), Neil Gehrels Swift Observatory Burst Alert Telescope (BAT), and the Monitor of All-sky X-ray Image (MAXI). We find that all three light curves are clearly modulated on the ~244 day orbital period previously reported from PCA monitoring observations, with outbursts confined to a narrow phase range. The light curves also show that XTE J1829-098 was in an inactive state between approximately December 2008 and April 2018 and no strong outbursts occurred. Such behavior is typical of Be X-ray binary systems, with the absence of outbursts likely related to the dissipation of the Be star's decretion disk. The mean outburst shapes can be approximated with a triangular profile and, from a joint fit of this to all three light curves, we refine the orbital period to 243.95 +/- 0.04 days. The mean outburst profile does not show any asymmetry and has a total phase duration of 0.140 +/- 0.007. However, the PCA light curve shows that there is considerable cycle-to-cycle variability of the individual outbursts. We compare the properties of XTE J1829-098 with other sources that show short phase-duration outbursts, in particular GS 1843-02 (2S 1845-024) which has a very similar orbital period, but longer pulse period, and whose orbit is known to be highly eccentric., Comment: Accepted for publication in the Astrophysical Journal. 21 pages

When training transformers on graph-structured data, incorporating information about the underlying topology is crucial for good performance. Topological masking, a type of relative position encoding, achieves this by upweighting or downweighting attention depending on the relationship between the query and keys in a graph. In this paper, we propose to parameterise topological masks as a learnable function of a weighted adjacency matrix -- a novel, flexible approach which incorporates a strong structural inductive bias. By approximating this mask with graph random features (for which we prove the first known concentration bounds), we show how this can be made fully compatible with linear attention, preserving $\mathcal{O}(N)$ time and space complexity with respect to the number of input tokens. The fastest previous alternative was $\mathcal{O}(N \log N)$ and only suitable for specific graphs. Our efficient masking algorithms provide strong performance gains for tasks on image and point cloud data, including with $>30$k nodes.

Observed magnitudes of Qianfan spacecraft range from 4 when they are near zenith to 8 when low in the sky. Nearly all of the observations can be modeled with a nadir-facing flat antenna panel and the underside of a zenith-facing solar array, both with Lambertian reflectance properties. These satellites will impact astronomical research unless their brightness is reduced.

Rb-82 is a radioactive isotope widely used for cardiac PET imaging. Despite numerous benefits of 82-Rb, there are several factors that limits its image quality and quantitative accuracy. First, the short half-life of 82-Rb results in noisy dynamic frames. Low signal-to-noise ratio would result in inaccurate and biased image quantification. Noisy dynamic frames also lead to highly noisy parametric images. The noise levels also vary substantially in different dynamic frames due to radiotracer decay and short half-life. Existing denoising methods are not applicable for this task due to the lack of paired training inputs/labels and inability to generalize across varying noise levels. Second, 82-Rb emits high-energy positrons. Compared with other tracers such as 18-F, 82-Rb travels a longer distance before annihilation, which negatively affect image spatial resolution. Here, the goal of this study is to propose a self-supervised method for simultaneous (1) noise-aware dynamic image denoising and (2) positron range correction for 82-Rb cardiac PET imaging. Tested on a series of PET scans from a cohort of normal volunteers, the proposed method produced images with superior visual quality. To demonstrate the improvement in image quantification, we compared image-derived input functions (IDIFs) with arterial input functions (AIFs) from continuous arterial blood samples. The IDIF derived from the proposed method led to lower AUC differences, decreasing from 11.09% to 7.58% on average, compared to the original dynamic frames. The proposed method also improved the quantification of myocardium blood flow (MBF), as validated against 15-O-water scans, with mean MBF differences decreased from 0.43 to 0.09, compared to the original dynamic frames. We also conducted a generalizability experiment on 37 patient scans obtained from a different country using a different scanner., Comment: 15 Pages, 10 Figures, 5 tables. Paper Under review. Oral Presentation at IEEE MIC 2023

The magnetic cloud (MC) of the Coronal Mass Ejection on April 24, 2023, contains sub-Alfv\'{e}nic solar wind, transforming Earth's magnetosphere from conventional bow-shock magnetotail configuration to Alfv\'{e}n wings. Utilizing measurements from the Magnetosphere Multiscale (MMS) mission, we present for the first time electron distribution signatures as the spacecraft traverses through various magnetic topologies during this transformation. Specifically, we characterize electrons inside the sub-Alfv\'{e}nic MC, on the dawn-dusk wing field lines and on the closed field lines. The signatures include strahl electrons in MC regions and energetic keV electrons streaming along the dawn and dusk wing field lines. We demonstrate the distribution signatures of dual wing reconnection, defined as reconnection between dawn-dusk Alfv\'{e}n wing field lines and the IMF. These signatures include four electron populations comprised of partially-depleted MC electrons and bi-directional energetic electrons with variations in energy and pitch-angle. The distributions reveal evidence of bursty magnetic reconnection under northward IMF., Comment: 11pages 4 figures

Mitigating methane emissions is the fastest way to stop global warming in the short-term and buy humanity time to decarbonise. Despite the demonstrated ability of remote sensing instruments to detect methane plumes, no system has been available to routinely monitor and act on these events. We present MARS-S2L, an automated AI-driven methane emitter monitoring system for Sentinel-2 and Landsat satellite imagery deployed operationally at the United Nations Environment Programme's International Methane Emissions Observatory. We compile a global dataset of thousands of super-emission events for training and evaluation, demonstrating that MARS-S2L can skillfully monitor emissions in a diverse range of regions globally, providing a 216% improvement in mean average precision over a current state-of-the-art detection method. Running this system operationally for six months has yielded 457 near-real-time detections in 22 different countries of which 62 have already been used to provide formal notifications to governments and stakeholders.

Finding proper collective variables for complex systems and processes is one of the most challenging tasks in simulations, which limits the interpretation of experimental and simulated data and the application of enhanced sampling techniques. Here, we propose a machine learning approach able to distill few, physically relevant variables by associating instantaneous configurations of the system to their corresponding inherent structures as defined in liquids theory. We apply this approach to the challenging case of structural transitions in nanoclusters, managing to characterize and explore the structural complexity of an experimentally relevant system constituted by 147 gold atoms. Our inherent-structure variables are shown to be effective at computing complex free-energy landscapes, transition rates, and at describing non-equilibrium melting and freezing processes. The effectiveness of this machine learning strategy guided by the generally-applicable concept of inherent structures shows promise to devise collective variables for a vast range of systems, including liquids, glasses, and proteins.

Vision language models (VLMs) demonstrate impressive capabilities in visual question answering and image captioning, acting as a crucial link between visual and language models. However, existing open-source VLMs heavily rely on pretrained and frozen vision encoders (such as CLIP). Despite CLIP's robustness across diverse domains, it still exhibits non-negligible image understanding errors. These errors propagate to the VLM responses, resulting in sub-optimal performance. In our work, we propose an efficient and robust method for updating vision encoders within VLMs. Our approach selectively and locally updates encoders, leading to substantial performance improvements on data where previous mistakes occurred, while maintaining overall robustness. Furthermore, we demonstrate the effectiveness of our method during continual few-shot updates. Theoretical grounding, generality, and computational efficiency characterize our approach.

The BlueWalker 3 satellite is now fainter than during the first months after deployment. The greatest improvement is that the average maximum luminosity near zenith has been reduced from magnitude 1.0 to 2.2. However, the spacecraft is still usually bright enough to interfere with astronomical research.

The mean apparent magnitude of Starlink Mini Direct-To-Cell (DTC) satellites is 4.62 while the mean of magnitudes adjusted to a uniform distance of 1000 km is 5.50. DTCs average 4.9 times brighter than other Starlink Mini spacecraft at a common distance. We cannot currently separate the effects of the DTC antenna itself, the different attitude modes that may be required for DTC operations and to what extent brightness mitigation procedures were in place at the times of our observations. In a best case scenario, where DTC brightness mitigation is as successful as that for other Minis and the DTC antenna does not add significantly to brightness, we estimate that DTCs will be about 2.6 times as bright as the others based upon their lower altitudes. The DTCs spend a greater fraction of their time in the Earth's shadow than satellites at higher altitudes. That will offset some of their impact on astronomical observing.

Neural processes (NPs) are a powerful family of meta-learning models that seek to approximate the posterior predictive map of the ground-truth stochastic process from which each dataset in a meta-dataset is sampled. There are many cases in which practitioners, besides having access to the dataset of interest, may also have access to other datasets that share similarities with it. In this case, integrating these datasets into the NP can improve predictions. We equip NPs with this functionality and describe this paradigm as in-context in-context learning. Standard NP architectures, such as the convolutional conditional NP (ConvCNP) or the family of transformer neural processes (TNPs), are not capable of in-context in-context learning, as they are only able to condition on a single dataset. We address this shortcoming by developing the in-context in-context learning pseudo-token TNP (ICICL-TNP). The ICICL-TNP builds on the family of PT-TNPs, which utilise pseudo-token-based transformer architectures to sidestep the quadratic computational complexity associated with regular transformer architectures. Importantly, the ICICL-TNP is capable of conditioning on both sets of datapoints and sets of datasets, enabling it to perform in-context in-context learning. We demonstrate the importance of in-context in-context learning and the effectiveness of the ICICL-TNP in a number of experiments.

Equivariant deep learning architectures exploit symmetries in learning problems to improve the sample efficiency of neural-network-based models and their ability to generalise. However, when modelling real-world data, learning problems are often not exactly equivariant, but only approximately. For example, when estimating the global temperature field from weather station observations, local topographical features like mountains break translation equivariance. In these scenarios, it is desirable to construct architectures that can flexibly depart from exact equivariance in a data-driven way. Current approaches to achieving this cannot usually be applied out-of-the-box to any architecture and symmetry group. In this paper, we develop a general approach to achieving this using existing equivariant architectures. Our approach is agnostic to both the choice of symmetry group and model architecture, making it widely applicable. We consider the use of approximately equivariant architectures in neural processes (NPs), a popular family of meta-learning models. We demonstrate the effectiveness of our approach on a number of synthetic and real-world regression experiments, showing that approximately equivariant NP models can outperform both their non-equivariant and strictly equivariant counterparts.

The effectiveness of neural processes (NPs) in modelling posterior prediction maps -- the mapping from data to posterior predictive distributions -- has significantly improved since their inception. This improvement can be attributed to two principal factors: (1) advancements in the architecture of permutation invariant set functions, which are intrinsic to all NPs; and (2) leveraging symmetries present in the true posterior predictive map, which are problem dependent. Transformers are a notable development in permutation invariant set functions, and their utility within NPs has been demonstrated through the family of models we refer to as TNPs. Despite significant interest in TNPs, little attention has been given to incorporating symmetries. Notably, the posterior prediction maps for data that are stationary -- a common assumption in spatio-temporal modelling -- exhibit translation equivariance. In this paper, we introduce of a new family of translation equivariant TNPs that incorporate translation equivariance. Through an extensive range of experiments on synthetic and real-world spatio-temporal data, we demonstrate the effectiveness of TE-TNPs relative to their non-translation-equivariant counterparts and other NP baselines.

The need for regression models to predict circular values arises in many scientific fields. In this work we explore a family of expressive and interpretable distributions over circle-valued random functions related to Gaussian processes targeting two Euclidean dimensions conditioned on the unit circle. The resulting probability model has connections with continuous spin models in statistical physics. Moreover, its density is very simple and has maximum-entropy, unlike previous Gaussian process-based approaches, which use wrapping or radial marginalization. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling. We argue that transductive learning in these models favors a Bayesian approach to the parameters. We present experiments applying this model to the prediction of (i) wind directions and (ii) the percentage of the running gait cycle as a function of joint angles., Comment: Contribution to the Structured Probabilistic Inference & Generative Modeling workshop of ICML 2024

Many high-stakes applications require machine learning models that protect user privacy and provide well-calibrated, accurate predictions. While Differential Privacy (DP) is the gold standard for protecting user privacy, standard DP mechanisms typically significantly impair performance. One approach to mitigating this issue is pre-training models on simulated data before DP learning on the private data. In this work we go a step further, using simulated data to train a meta-learning model that combines the Convolutional Conditional Neural Process (ConvCNP) with an improved functional DP mechanism of Hall et al. [2013] yielding the DPConvCNP. DPConvCNP learns from simulated data how to map private data to a DP predictive model in one forward pass, and then provides accurate, well-calibrated predictions. We compare DPConvCNP with a DP Gaussian Process (GP) baseline with carefully tuned hyperparameters. The DPConvCNP outperforms the GP baseline, especially on non-Gaussian data, yet is much faster at test time and requires less tuning.

Expectation propagation (EP) is a family of algorithms for performing approximate inference in probabilistic models. The updates of EP involve the evaluation of moments -- expectations of certain functions -- which can be estimated from Monte Carlo (MC) samples. However, the updates are not robust to MC noise when performed naively, and various prior works have attempted to address this issue in different ways. In this work, we provide a novel perspective on the moment-matching updates of EP; namely, that they perform natural-gradient-based optimisation of a variational objective. We use this insight to motivate two new EP variants, with updates that are particularly well-suited to MC estimation. They remain stable and are most sample-efficient when estimated with just a single sample. These new variants combine the benefits of their predecessors and address key weaknesses. In particular, they are easier to tune, offer an improved speed-accuracy trade-off, and do not rely on the use of debiasing estimators. We demonstrate their efficacy on a variety of probabilistic inference tasks.

Random features (RFs) are a popular technique to scale up kernel methods in machine learning, replacing exact kernel evaluations with stochastic Monte Carlo estimates. They underpin models as diverse as efficient transformers (by approximating attention) to sparse spectrum Gaussian processes (by approximating the covariance function). Efficiency can be further improved by speeding up the convergence of these estimates: a variance reduction problem. We tackle this through the unifying lens of optimal transport, finding couplings to improve RFs defined on both Euclidean and discrete input spaces. They enjoy theoretical guarantees and sometimes provide strong downstream gains, including for scalable approximate inference on graphs. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm, showing that other properties of the coupling should be optimised for attention estimation in efficient transformers.

Starlink satellites can become extremely bright when sunlight reflects specularly to an observer on the ground. The observed brightness of such flares is consistent with a bidirectional reflectance function of the Starlink satellite chassis. These findings are applied to the case of an extreme flare that was reported as an Unidentified Aerial Phenomena by the pilots of two commercial aircraft.