Search

Understanding how biological visual systems process information is challenging because of the nonlinear relationship between visual input and neuronal responses. Artificial neural networks allow computational neuroscientists to create predictive models that connect biological and machine vision. Machine learning has benefited tremendously from benchmarks that compare different model on the same task under standardized conditions. However, there was no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we established the Sensorium 2023 Benchmark Competition with dynamic input, featuring a new large-scale dataset from the primary visual cortex of ten mice. This dataset includes responses from 78,853 neurons to 2 hours of dynamic stimuli per neuron, together with the behavioral measurements such as running speed, pupil dilation, and eye movements. The competition ranked models in two tracks based on predictive performance for neuronal responses on a held-out test set: one focusing on predicting in-domain natural stimuli and another on out-of-distribution (OOD) stimuli to assess model generalization. As part of the NeurIPS 2023 competition track, we received more than 160 model submissions from 22 teams. Several new architectures for predictive models were proposed, and the winning teams improved the previous state-of-the-art model by 50%. Access to the dataset as well as the benchmarking infrastructure will remain online at www.sensorium-competition.net.

Diffusion models (DMs) have gained attention in Missing Data Imputation (MDI), but there remain two long-neglected issues to be addressed: (1). Inaccurate Imputation, which arises from inherently sample-diversification-pursuing generative process of DMs. (2). Difficult Training, which stems from intricate design required for the mask matrix in model training stage. To address these concerns within the realm of numerical tabular datasets, we introduce a novel principled approach termed Kernelized Negative Entropy-regularized Wasserstein gradient flow Imputation (KnewImp). Specifically, based on Wasserstein gradient flow (WGF) framework, we first prove that issue (1) stems from the cost functionals implicitly maximized in DM-based MDI are equivalent to the MDI's objective plus diversification-promoting non-negative terms. Based on this, we then design a novel cost functional with diversification-discouraging negative entropy and derive our KnewImp approach within WGF framework and reproducing kernel Hilbert space. After that, we prove that the imputation procedure of KnewImp can be derived from another cost functional related to the joint distribution, eliminating the need for the mask matrix and hence naturally addressing issue (2). Extensive experiments demonstrate that our proposed KnewImp approach significantly outperforms existing state-of-the-art methods.

Developing therapeutics is a lengthy and expensive process that requires the satisfaction of many different criteria, and AI models capable of expediting the process would be invaluable. However, the majority of current AI approaches address only a narrowly defined set of tasks, often circumscribed within a particular domain. To bridge this gap, we introduce Tx-LLM, a generalist large language model (LLM) fine-tuned from PaLM-2 which encodes knowledge about diverse therapeutic modalities. Tx-LLM is trained using a collection of 709 datasets that target 66 tasks spanning various stages of the drug discovery pipeline. Using a single set of weights, Tx-LLM simultaneously processes a wide variety of chemical or biological entities(small molecules, proteins, nucleic acids, cell lines, diseases) interleaved with free-text, allowing it to predict a broad range of associated properties, achieving competitive with state-of-the-art (SOTA) performance on 43 out of 66 tasks and exceeding SOTA on 22. Among these, Tx-LLM is particularly powerful and exceeds best-in-class performance on average for tasks combining molecular SMILES representations with text such as cell line names or disease names, likely due to context learned during pretraining. We observe evidence of positive transfer between tasks with diverse drug types (e.g.,tasks involving small molecules and tasks involving proteins), and we study the impact of model size, domain finetuning, and prompting strategies on performance. We believe Tx-LLM represents an important step towards LLMs encoding biochemical knowledge and could have a future role as an end-to-end tool across the drug discovery development pipeline.

Structured output prediction problems are ubiquitous in machine learning. The prominent approach leverages neural networks as powerful feature extractors, otherwise assuming the independence of the outputs. These outputs, however, jointly encode an object, e.g. a path in a graph, and are therefore related through the structure underlying the output space. We discuss the semantic loss, which injects knowledge about such structure, defined symbolically, into training by minimizing the network's violation of such dependencies, steering the network towards predicting distributions satisfying the underlying structure. At the same time, it is agnostic to the arrangement of the symbols, and depends only on the semantics expressed thereby, while also enabling efficient end-to-end training and inference. We also discuss key improvements and applications of the semantic loss. One limitations of the semantic loss is that it does not exploit the association of every data point with certain features certifying its membership in a target class. We should therefore prefer minimum-entropy distributions over valid structures, which we obtain by additionally minimizing the neuro-symbolic entropy. We empirically demonstrate the benefits of this more refined formulation. Moreover, the semantic loss is designed to be modular and can be combined with both discriminative and generative neural models. This is illustrated by integrating it into generative adversarial networks, yielding constrained adversarial networks, a novel class of deep generative models able to efficiently synthesize complex objects obeying the structure of the underlying domain., Comment: Preprint of Ch. 22 "Semantic Loss Functions for Neuro-Symbolic Structured Prediction" in "Compendium of Neurosymbolic Artificial Intelligence", https://ebooks.iospress.nl/ISBN/978-1-64368-406-2. arXiv admin note: substantial text overlap with arXiv:2201.11250, arXiv:2007.13197

Many clinical tasks require an understanding of specialized data, such as medical images and genomics, which is not typically found in general-purpose large multimodal models. Building upon Gemini's multimodal models, we develop several models within the new Med-Gemini family that inherit core capabilities of Gemini and are optimized for medical use via fine-tuning with 2D and 3D radiology, histopathology, ophthalmology, dermatology and genomic data. Med-Gemini-2D sets a new standard for AI-based chest X-ray (CXR) report generation based on expert evaluation, exceeding previous best results across two separate datasets by an absolute margin of 1% and 12%, where 57% and 96% of AI reports on normal cases, and 43% and 65% on abnormal cases, are evaluated as "equivalent or better" than the original radiologists' reports. We demonstrate the first ever large multimodal model-based report generation for 3D computed tomography (CT) volumes using Med-Gemini-3D, with 53% of AI reports considered clinically acceptable, although additional research is needed to meet expert radiologist reporting quality. Beyond report generation, Med-Gemini-2D surpasses the previous best performance in CXR visual question answering (VQA) and performs well in CXR classification and radiology VQA, exceeding SoTA or baselines on 17 of 20 tasks. In histopathology, ophthalmology, and dermatology image classification, Med-Gemini-2D surpasses baselines across 18 out of 20 tasks and approaches task-specific model performance. Beyond imaging, Med-Gemini-Polygenic outperforms the standard linear polygenic risk score-based approach for disease risk prediction and generalizes to genetically correlated diseases for which it has never been trained. Although further development and evaluation are necessary in the safety-critical medical domain, our results highlight the potential of Med-Gemini across a wide range of medical tasks.

In this paper, we explore the challenges of ensuring security and privacy for users from diverse demographic backgrounds. We propose a threat modeling approach to identify potential risks and countermeasures for product inclusion in security and privacy. We discuss various factors that can affect a user's ability to achieve a high level of security and privacy, including low-income demographics, poor connectivity, shared device usage, ML fairness, etc. We present results from a global security and privacy user experience survey and discuss the implications for product developers. Our work highlights the need for a more inclusive approach to security and privacy and provides a framework for researchers and practitioners to consider when designing products and services for a diverse range of users.

Scientific machine learning (SciML) has emerged as a versatile approach to address complex computational science and engineering problems. Within this field, physics-informed neural networks (PINNs) and deep operator networks (DeepONets) stand out as the leading techniques for solving partial differential equations by incorporating both physical equations and experimental data. However, training PINNs and DeepONets requires significant computational resources, including long computational times and large amounts of memory. In search of computational efficiency, training neural networks using half precision (float16) rather than the conventional single (float32) or double (float64) precision has gained substantial interest, given the inherent benefits of reduced computational time and memory consumed. However, we find that float16 cannot be applied to SciML methods, because of gradient divergence at the start of training, weight updates going to zero, and the inability to converge to a local minima. To overcome these limitations, we explore mixed precision, which is an approach that combines the float16 and float32 numerical formats to reduce memory usage and increase computational speed. Our experiments showcase that mixed precision training not only substantially decreases training times and memory demands but also maintains model accuracy. We also reinforce our empirical observations with a theoretical analysis. The research has broad implications for SciML in various computational applications., Comment: 25 pages, 7 figures

Gamma-Ray Bursts (GRBs), due to their high luminosities are detected up to redshift 10, and thus have the potential to be vital cosmological probes of early processes in the universe. Fulfilling this potential requires a large sample of GRBs with known redshifts, but due to observational limitations, only 11\% have known redshifts ($z$). There have been numerous attempts to estimate redshifts via correlation studies, most of which have led to inaccurate predictions. To overcome this, we estimated GRB redshift via an ensemble supervised machine learning model that uses X-ray afterglows of long-duration GRBs observed by the Neil Gehrels Swift Observatory. The estimated redshifts are strongly correlated (a Pearson coefficient of 0.93) and have a root mean square error, namely the square root of the average squared error $\langle\Delta z^2\rangle$, of 0.46 with the observed redshifts showing the reliability of this method. The addition of GRB afterglow parameters improves the predictions considerably by 63\% compared to previous results in peer-reviewed literature. Finally, we use our machine learning model to infer the redshifts of 154 GRBs, which increase the known redshifts of long GRBs with plateaus by 94\%, a significant milestone for enhancing GRB population studies that require large samples with redshift., Comment: 12 Figures, 24 pages. Accepted for publication at The Astrophysical Journal Supplement Series

HISPEC is a new, high-resolution near-infrared spectrograph being designed for the W.M. Keck II telescope. By offering single-shot, R=100,000 between 0.98 - 2.5 um, HISPEC will enable spectroscopy of transiting and non-transiting exoplanets in close orbits, direct high-contrast detection and spectroscopy of spatially separated substellar companions, and exoplanet dynamical mass and orbit measurements using precision radial velocity monitoring calibrated with a suite of state-of-the-art absolute and relative wavelength references. MODHIS is the counterpart to HISPEC for the Thirty Meter Telescope and is being developed in parallel with similar scientific goals. In this proceeding, we provide a brief overview of the current design of both instruments, and the requirements for the two spectrographs as guided by the scientific goals for each. We then outline the current science case for HISPEC and MODHIS, with focuses on the science enabled for exoplanet discovery and characterization. We also provide updated sensitivity curves for both instruments, in terms of both signal-to-noise ratio and predicted radial velocity precision., Comment: 25 pages, 9 figures. To appear in the Proceedings of SPIE: Techniques and Instrumentation for Detection of Exoplanets XI, vol. 12680 (2023)

Understanding how biological visual systems process information is challenging due to the complex nonlinear relationship between neuronal responses and high-dimensional visual input. Artificial neural networks have already improved our understanding of this system by allowing computational neuroscientists to create predictive models and bridge biological and machine vision. During the Sensorium 2022, we introduced benchmarks for vision models with static input. However, animals operate and excel in dynamic environments, making it crucial to study and understand how the brain functions under these conditions. Moreover, many biological theories, such as predictive coding, suggest that previous input is crucial for current input processing. Currently, there is no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we propose the Sensorium 2023 Benchmark Competition with dynamic input. It includes the collection of a new large-scale dataset from the primary visual cortex of ten mice, containing responses from over 78,000 neurons to over 2 hours of dynamic stimuli per neuron. Participants in the main benchmark track will compete to identify the best predictive models of neuronal responses for dynamic input. We will also host a bonus track in which submission performance will be evaluated on out-of-domain input, using withheld neuronal responses to dynamic input stimuli whose statistics differ from the training set. Both tracks will offer behavioral data along with video stimuli. As before, we will provide code, tutorials, and strong pre-trained baseline models to encourage participation. We hope this competition will continue to strengthen the accompanying Sensorium benchmarks collection as a standard tool to measure progress in large-scale neural system identification models of the entire mouse visual hierarchy and beyond., Comment: arXiv admin note: substantial text overlap with arXiv:2206.08666

Variational autoencoders (VAEs) are powerful tools for learning latent representations of data used in a wide range of applications. In practice, VAEs usually require multiple training rounds to choose the amount of information the latent variable should retain. This trade-off between the reconstruction error (distortion) and the KL divergence (rate) is typically parameterized by a hyperparameter $\beta$. In this paper, we introduce Multi-Rate VAE (MR-VAE), a computationally efficient framework for learning optimal parameters corresponding to various $\beta$ in a single training run. The key idea is to explicitly formulate a response function that maps $\beta$ to the optimal parameters using hypernetworks. MR-VAEs construct a compact response hypernetwork where the pre-activations are conditionally gated based on $\beta$. We justify the proposed architecture by analyzing linear VAEs and showing that it can represent response functions exactly for linear VAEs. With the learned hypernetwork, MR-VAEs can construct the rate-distortion curve without additional training and can be deployed with significantly less hyperparameter tuning. Empirically, our approach is competitive and often exceeds the performance of multiple $\beta$-VAEs training with minimal computation and memory overheads., Comment: 22 pages, 9 figures

We present the design of SCALES (Slicer Combined with Array of Lenslets for Exoplanet Spectroscopy) a new 2-5 micron coronagraphic integral field spectrograph under construction for Keck Observatory. SCALES enables low-resolution (R~50) spectroscopy, as well as medium-resolution (R~4,000) spectroscopy with the goal of discovering and characterizing cold exoplanets that are brightest in the thermal infrared. Additionally, SCALES has a 12x12" field-of-view imager that will be used for general adaptive optics science at Keck. We present SCALES's specifications, its science case, its overall design, and simulations of its expected performance. Additionally, we present progress on procuring, fabricating and testing long lead-time components., Comment: 19 pages, 12 Figures, SPIE Astronomical Telescopes + Instrumentation, (#12184-18), 2022, Montreal, Canada

Liger is a next-generation near-infrared imager and integral field spectrograph (IFS) planned for the W.M. Keck Observatory. Liger is designed to take advantage of improved adaptive optics (AO) from the Keck All-Sky Precision Adaptive Optics (KAPA) upgrade currently underway. Liger operates at 0.84-2.45 $\mu$m with spectral resolving powers of R$\sim$4,000-10,000. Liger makes use of a sequential imager and spectrograph design allowing for simultaneous observations. There are two spectrograph modes: a lenslet with high spatial sampling of 14 and 31 mas, and a slicer with 75 and 150 mas sampling with an expanded field of view. Two pick-off mirrors near the imager detector direct light to these two IFS channels. We present the design and structural analysis for the imager detector and IFS pick-off mirror mounting assembly that will be used to align and maintain stability throughout its operation. A piezoelectric actuator will be used to step through $\rm3\,mm$ of travel during alignment of the instrument to determine the optimal focus for both the detector and pick-off mirrors which will be locked in place during normal operation. We will demonstrate that the design can withstand the required gravitational and shipping loads and can be aligned within the positioning tolerances for the optics., Comment: 10 pages, 7 figures, 3 tables

Liger is an adaptive optics (AO) fed imager and integral field spectrograph (IFS) designed to take advantage of the Keck All-sky Precision Adaptive-optics (KAPA) upgrade for the W.M. Keck Observatory. We present the design and analysis of the imager optical assembly including the spectrograph Re-Imaging Optics (RIO) which transfers the beam path from the imager focal plane to the IFS slicer module and lenslet array. Each imager component and the first two RIO mechanisms are assembled and individually aligned on the same optical plate. Baffling suppresses background radiation and scattered light, and a pupil viewing camera allows the imager detector to focus on an image of the telescope pupil. The optical plate mounts on an adapter frame for alignment of the overall system. The imager and RIO will be characterized in a cryogenic test chamber before installation in the final science cryostat., Comment: 8 pages, 5 figures

A next-generation instrument named, Slicer Combined with Array of Lenslets for Exoplanet Spectroscopy (SCALES), is being planned for the W. M. Keck Observatory. SCALES will have an integral field spectrograph (IFS) and a diffraction-limited imaging channel to discover and spectrally characterize the directly imaged exoplanets. Operating at thermal infrared wavelengths (1-5 micron, and a goal of 0.6-5 micron), the imaging channel of the SCALES is designed to cover a 12"x 12" field of view with low distortions and high throughput. Apart from expanding the mid-infrared science cases and providing a potential upgrade/alternative for the NIRC2, the H2RG detector of the imaging channel can take high-resolution images of the pupil to aid the alignment process.Further, the imaging camera would also assist in small field acquisition for the IFS arm. In this work, we present the optomechanical design of the imager and evaluate its capabilities and performances., Comment: 11 pages, 9 Figures, SPIE Astronomical Telescopes + Instrumentation, (#12188-65), 2022, Montreal, Canada

In structured prediction, the goal is to jointly predict many output variables that together encode a structured object -- a path in a graph, an entity-relation triple, or an ordering of objects. Such a large output space makes learning hard and requires vast amounts of labeled data. Different approaches leverage alternate sources of supervision. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. It extracts supervision from unlabeled examples, but remains agnostic to the structure of the output space. Conversely, neuro-symbolic approaches exploit the knowledge that not every prediction corresponds to a valid structure in the output space. Yet, they does not further restrict the learned output distribution. This paper introduces a framework that unifies both approaches. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object. It is obtained by restricting entropy regularization to the distribution over only valid structures. This loss is efficiently computed when the output constraint is expressed as a tractable logic circuit. Moreover, it seamlessly integrates with other neuro-symbolic losses that eliminate invalid predictions. We demonstrate the efficacy of our approach on a series of semi-supervised and fully-supervised structured-prediction experiments, where we find that it leads to models whose predictions are more accurate and more likely to be valid.

The Keck Planet Imager and Characterizer (KPIC) is a purpose-built instrument to demonstrate new technological and instrumental concepts initially developed for the exoplanet direct imaging field. Located downstream of the current Keck II adaptive optic system, KPIC contains a fiber injection unit (FIU) capable of combining the high-contrast imaging capability of the adaptive optics system with the high dispersion spectroscopy capability of the current Keck high resolution infrared spectrograph (NIRSPEC). Deployed at Keck in September 2018, this instrument has already been used to acquire high resolution spectra ($R > 30,000$) of multiple targets of interest. In the near term, it will be used to spectrally characterize known directly imaged exoplanets and low-mass brown dwarf companions visible in the northern hemisphere with a spectral resolution high enough to enable spin and planetary radial velocity measurements as well as Doppler imaging of atmospheric weather phenomena. Here we present the design of the FIU, the unique calibration procedures needed to operate a single-mode fiber instrument and the system performance., Comment: 31 pages, 17 figures, submitted to JATIS

Understanding the behavior of learned classifiers is an important task, and various black-box explanations, logical reasoning approaches, and model-specific methods have been proposed. In this paper, we introduce probabilistic sufficient explanations, which formulate explaining an instance of classification as choosing the "simplest" subset of features such that only observing those features is "sufficient" to explain the classification. That is, sufficient to give us strong probabilistic guarantees that the model will behave similarly when all features are observed under the data distribution. In addition, we leverage tractable probabilistic reasoning tools such as probabilistic circuits and expected predictions to design a scalable algorithm for finding the desired explanations while keeping the guarantees intact. Our experiments demonstrate the effectiveness of our algorithm in finding sufficient explanations, and showcase its advantages compared to Anchors and logical explanations.

Given a convex $n$-gon $P$ and a positive integer $m$ such that $3\le m\le n-1$, let $Q$ denote the largest area convex $m$-gon contained in $P$. We are interested in the minimum value of $\Delta(Q)/\Delta(P)$, the ratio of the areas of these two polygons. More precisely, given positive integers $n$ and $m$, with $3 \le m \le n-1$, define \begin{equation*} f_n(m)=\min_{P\in \mathcal {P}_n} \max_{Q \subset P,|Q|=m} \frac{\Delta(Q)}{\Delta(P)} \end{equation*} where the maximum is taken over all $m$-gons contained in $P$, and the minimum is taken over $\mathcal{P}_n$, the entire class of convex $n$-gons. The values of $f_4(3)$, $f_5(4)$ and $f_6(3)$ are known. In this paper we compute the values of $f_5(3)$, $f_6(5)$ and $f_6(4)$. In addition, we prove that for all $n\ge 6$ we have \begin{equation*} \frac{4}{n}\cdot\sin^2\left(\frac{\pi}{n}\right)\le 1-f_n(n-1)\le \min\left(\frac{1}{n}, \frac{4}{n}\cdot\sin^2\left(\frac{2\pi}{n}\right)\right). \end{equation*} These bounds can be used to improve the known estimates for $f_n(m)$., Comment: 15 pages, 3 figures

Liger is a next-generation near-infrared imager and integral field spectrograph (IFS) for the W.M. Keck Observatory designed to take advantage of the Keck All-Sky Precision Adaptive Optics (KAPA) upgrade. Liger will operate at spectral resolving powers between R$\sim$4,000 - 10,000 over a wavelength range of 0.8-2.4$\mu$m. Liger takes advantage of a sequential imager and spectrograph design that allows for simultaneous observations between the two channels using the same filter wheel and cold pupil stop. We present the design for the filter wheels and pupil mask and their location and tolerances in the optical design. The filter mechanism is a multi-wheel design drawing from the heritage of the current Keck/OSIRIS imager single wheel design. The Liger multi-wheel configuration is designed to allow future upgrades to the number and range of filters throughout the life of the instrument. The pupil mechanism is designed to be similarly upgradeable with the option to add multiple pupil mask options. A smaller wheel mechanism allows the user to select the desired pupil mask with open slots being designed in for future upgrade capabilities. An ideal pupil would match the shape of the image formed of the primary and would track its rotation. For different pupil shapes without tracking we model the additional exposure time needed to achieve the same signal to noise of an ideal pupil and determine that a set of fixed masks of different shapes provides a mechanically simpler system with little compromise in performance., Comment: 9 pages, 7 figures, 1 table

Liger is a next generation adaptive optics (AO) fed integral field spectrograph (IFS) and imager for the W. M. Keck Observatory. This new instrument is being designed to take advantage of the upgraded AO system provided by Keck All-Sky Precision Adaptive-optics (KAPA). Liger will provide higher spectral resolving power (R$\sim$4,000-10,000), wider wavelength coverage ($\sim$0.8-2.4 $\mu$m), and larger fields of view than any current IFS. We present the design and analysis for a custom-made dewar chamber for characterizing the Liger opto-mechanical system. This dewar chamber is designed to test and assemble the Liger imaging camera and slicer IFS components while being adaptable for future experiments. The vacuum chamber will operate below $10^{-5}$ Torr with a cold shield that will be kept below 90 K. The dewar test chamber will be mounted to an optical vibration isolation platform and further isolated from the cryogenic and vacuum systems with bellows. The cold head and vacuums will be mounted to a custom cart that will also house the electronics and computer that interface with the experiment. This test chamber will provide an efficient means of calibrating and characterizing the Liger instrument and performing future experiments., Comment: 8 pages, 6 figures

We study the problem of entity-relation extraction in the presence of symbolic domain knowledge. Such knowledge takes the form of an ontology defining relations and their permissible arguments. Previous approaches set out to integrate such knowledge in their learning approaches either through self-training, or through approximations that lose the precise meaning of the logical expressions. By contrast, our approach employs semantic loss which captures the precise meaning of a logical sentence through maintaining a probability distribution over all possible states, and guiding the model to solutions which minimize any constraint violations. With a focus on low-data regimes, we show that semantic loss outperforms the baselines by a wide margin., Comment: 8 pages, 2 figures