Search

In large language models (LLMs), code and reasoning reinforce each other: code offers an abstract, modular, and logic-driven structure that supports reasoning, while reasoning translates high-level goals into smaller, executable steps that drive more advanced code intelligence. In this study, we examine how code serves as a structured medium for enhancing reasoning: it provides verifiable execution paths, enforces logical decomposition, and enables runtime validation. We also explore how improvements in reasoning have transformed code intelligence from basic completion to advanced capabilities, enabling models to address complex software engineering tasks through planning and debugging. Finally, we identify key challenges and propose future research directions to strengthen this synergy, ultimately improving LLM's performance in both areas., Comment: Project Repo: https://github.com/dayuyang1999/Awesome-Code-Reasoning

Fine-tuning provides an effective means to specialize pre-trained models for various downstream tasks. However, fine-tuning often incurs high memory overhead, especially for large transformer-based models, such as LLMs. While existing methods may reduce certain parts of the memory required for fine-tuning, they still require caching all intermediate activations computed in the forward pass to update weights during the backward pass. In this work, we develop TokenTune, a method to reduce memory usage, specifically the memory to store intermediate activations, in the fine-tuning of transformer-based models. During the backward pass, TokenTune approximates the gradient computation by backpropagating through just a subset of input tokens. Thus, with TokenTune, only a subset of intermediate activations are cached during the forward pass. Also, TokenTune can be easily combined with existing methods like LoRA, further reducing the memory cost. We evaluate our approach on pre-trained transformer models with up to billions of parameters, considering the performance on multiple downstream tasks such as text classification and question answering in a few-shot learning setup. Overall, TokenTune achieves performance on par with full fine-tuning or representative memory-efficient fine-tuning methods, while greatly reducing the memory footprint, especially when combined with other methods with complementary memory reduction mechanisms. We hope that our approach will facilitate the fine-tuning of large transformers, in specializing them for specific domains or co-training them with other neural components from a larger system. Our code is available at https://github.com/facebookresearch/tokentune., Comment: EMNLP 2024

Liver transplant can be a life-saving procedure for patients with end-stage liver disease. With the introduction of modern immunosuppressive therapies, short-term survival has significantly improved. However, long-term survival has not substantially improved in decades. Consequently, causes of death are now more likely to be due to the toxicities and side-effects of long-term immunosuppression rather than rejection. In order to study the balance of immunosuppression and rejection, we developed the first mechanistic mathematical model of liver transplant and immune system dynamics. We determined key cells and interactions in the model using literature information; we then used sensitivity analysis to determine key pathways driving the health status of the transplanted liver. We found that dynamics related to cytotoxic T cells and IL-2, in addition to the liver itself, are key determinants of liver graft injury. This has significant implications for the use of tests to monitor patients, and therapeutic strategies to prevent or treat liver transplantation rejection. Future work to collect appropriate data and parametrize the model would be valuable in improving our understanding of the dynamics of this system. We also note that our model could be tailored to model transplant of other organs., Comment: 34 pages, 5 figures

Quasar feedback may play a key role in the evolution of massive galaxies. The dust-reddened quasar, F2M110648.35$+$480712 at $z = 0.4352$ is one of the few cases at its redshift that exhibits powerful quasar feedback through bipolar outflows. Our new observation with the integral field unit mode of Near-infrared Spectrograph onboard JWST opens a new window to examine this spectacular outflow through Pa$\alpha$ emission line with $\sim$3$\times$ better spatial resolution than previous work. The morphology and kinematics of the Pa$\alpha$ nebula confirm the existence of a bipolar outflow extending on a scale of $\sim$17$\times$14 kpc and with a velocity reaching $\sim$1100 km s$^{-1}$. The higher spatial resolution of our new observation leads to more reliable measurements of outflow kinematics. Considering only the spatially resolved outflow and assuming an electron density of 100 cm$^{-2}$, the mass, momentum and kinetic energy outflow rates are $\sim$50-210 M$_{\odot}$ yr$^{-1}$, $\sim$0.3-1.7$\times$10$^{36}$ dynes ($\sim$14-78\% of the quasar photon momentum flux) and $\sim$0.16-1.27$\times$10$^{44}$ erg s$^{-1}$ ($\sim$0.02-0.20\% of the quasar bolometric luminosity), respectively. The local instantaneous outflow rates generally decrease radially. We infer that the quasar is powerful enough to drive the outflow, while stellar processes cannot be overlooked as a contributing energy source. The mass outflow rate is $\sim$0.4-1.5 times the star formation rate, and the ratio of kinetic energy outflow rate to the quasar bolometric luminosity is comparable to the minimum value required for negative quasar feedback in simulations. This outflow may help regulate the star formation activity within the system to some extent., Comment: 14 pages, 6 figures, 1 table, ApJ in review

Materials scientists utilize image segmentation of micrographs to create large curve ensembles representing grain boundaries of material microstructures. Observations of these collections of shapes can facilitate inferences about material properties and manufacturing processes. We seek to bolster this application, and related engineering/scientific tasks, using novel pattern recognition formalisms and inference over large ensembles of segmented curves -- i.e., facilitate principled assessments for quantifying differences in distributions of shapes. To this end, we apply a composite integral operator to motivate accurate and efficient numerical representations of discrete planar curves over matrix manifolds. The main result is a rigid-invariant orthonormal decomposition of curve component functions into separable forms of scale variations and complementary features of undulation. We demonstrate how these separable shape tensors -- given thousands of curves in an ensemble -- can inform explainable binary classification of segmented images by utilizing a product maximum mean discrepancy to distinguish the shape distributions; absent labelled data, building interpretable feature spaces in seconds without high performance computation, and detecting discrepancies below cursory visual inspections., Comment: 20 pages, 7 figures

How much does AI assistance impact developer productivity? To date, the software engineering literature has provided a range of answers, targeting a diversity of outcomes: from perceived productivity to speed on task and developer throughput. Our randomized controlled trial with 96 full-time Google software engineers contributes to this literature by sharing an estimate of the impact of three AI features on the time developers spent on a complex, enterprise-grade task. We found that AI significantly shortened the time developers spent on task. Our best estimate of the size of this effect, controlling for factors known to influence developer time on task, stands at about 21\%, although our confidence interval is large. We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI. Product and future research considerations are discussed. In particular, we invite further research that explores the impact of AI at the ecosystem level and across multiple suites of AI-enhanced tools, since we cannot assume that the effect size obtained in our lab study will necessarily apply more broadly, or that the effect of AI found using internal Google tooling in the summer of 2024 will translate across tools and over time., Comment: 12 pages, 7 figures, 3 tables

In this article, we focus on the communication costs of three symmetric matrix computations: i) multiplying a matrix with its transpose, known as a symmetric rank-k update (SYRK) ii) adding the result of the multiplication of a matrix with the transpose of another matrix and the transpose of that result, known as a symmetric rank-2k update (SYR2K) iii) performing matrix multiplication with a symmetric input matrix (SYMM). All three computations appear in the Level 3 Basic Linear Algebra Subroutines (BLAS) and have wide use in applications involving symmetric matrices. We establish communication lower bounds for these kernels using sequential and distributed-memory parallel computational models, and we show that our bounds are tight by presenting communication-optimal algorithms for each setting. Our lower bound proofs rely on applying a geometric inequality for symmetric computations and analytically solving constrained nonlinear optimization problems. The symmetric matrix and its corresponding computations are accessed and performed according to a triangular block partitioning scheme in the optimal algorithms., Comment: 43 pages, 6 figures. To be published in ACM Transactions on Parallel Computing

With the emergence of Foundational Machine Learning Interatomic Potential (FMLIP) models trained on extensive datasets, transferring data between different ML architectures has become increasingly important. In this work, we examine the extent to which training data optimised for one machine-learning forcefield algorithm may be re-used to train different models, aiming to accelerate FMLIP fine-tuning and to reduce the need for costly iterative training. As a test case, we train models of an organic liquid mixture that is commonly used as a solvent in rechargeable battery electrolytes, making it an important target for reactive MLIP development. We assess model performance by analysing the properties of molecular dynamics trajectories, showing that this is a more stringent test than comparing prediction errors for fixed datasets. We consider several types of training data, and several popular MLIPs - notably the recent MACE architecture, a message-passing neural network designed for high efficiency and smoothness. We demonstrate that simple training sets constructed without any ab initio dynamics are sufficient to produce stable models of molecular liquids. For simple neural-network architectures, further iterative training is required to capture thermodynamic and kinetic properties correctly, but MACE performs well with extremely limited datsets. We find that configurations designed by human intuition to correct systematic model deficiencies transfer effectively between algorithms, but active-learned data that are generated by one MLIP do not typically benefit a different algorithm. Finally, we show that any training data which improve model performance also improve its ability to generalise to similar unseen molecules. This suggests that trajectory failure modes are connected with chemical structure rather than being entirely system-specific., Comment: 23 pages, 7 figures

Resource sharing is a crucial part of a multi-robot system. We propose a Boolean satisfiability based approach to resource sharing. Our key contributions are an algorithm for converting any constrained assignment to a weighted-SAT based optimization. We propose a theorem that allows optimal resource assignment problems to be solved via repeated application of a SAT solver. Additionally we show a way to encode continuous time ordering constraints using Conjunctive Normal Form (CNF). We benchmark our new algorithms and show that they can be used in an ad-hoc setting. We test our algorithms on a fleet of simulated and real world robots and show that the algorithms are able to handle real world situations. Our algorithms and test harnesses are opensource and build on Open-RMFs fleet management system., Comment: IROS 2024

Below its Jahn--Teller transition temperature, $T_\mathrm{JT}$, NaNiO$_2$ has a monoclinic layered structure consisting of alternating layers of edge-sharing NaO$_6$ and Jahn-Teller-distorted NiO$_6$ octahedra. Above $T_\mathrm{JT}$ where NaNiO$_2$ is rhombohedral, diffraction measurements show the absence of a cooperative Jahn-Teller distortion, accompanied by an increase in the unit cell volume. Using neutron total scattering, solid-state Nuclear Magnetic Resonance (NMR), and extended X-ray absorption fine structure (EXAFS) experiments as local probes of the structure we find direct evidence for a displacive, as opposed to order-disorder Jahn-Teller transition at $T_\mathrm{JT}$. This is supported by \textit{ab initio} molecular dynamics (AIMD) simulations. To our knowledge this study is the first to show a displacive Jahn-Teller transition in any material using direct observations with local probe techniques., Comment: Under review. For SI, email the authors

Large language models (LLMs) often struggle with temporal reasoning, crucial for tasks like historical event analysis and time-sensitive information retrieval. Despite advancements, state-of-the-art models falter in handling temporal information, especially when faced with irrelevant or noisy contexts. This paper addresses this gap by empirically examining the robustness of temporal question-answering (TQA) systems trained on various context types, including relevant, irrelevant, slightly altered, and no context. Our findings indicate that training with a mix of these contexts enhances model robustness and accuracy. Additionally, we show that the position of context relative to the question significantly impacts performance, with question-first positioning yielding better results. We introduce two new context-rich TQA datasets, ContextAQA and ContextTQE, and provide comprehensive evaluations and guidelines for training robust TQA models. Our work lays the foundation for developing reliable and context-aware temporal QA systems, with broader implications for enhancing LLM robustness against diverse and potentially adversarial information.

We demonstrate monolithically grown germanium-tin (GeSn) on silicon avalanche photodiodes (APDs) for infrared light detection. A relatively thinner Ge buffer design was adopted to allow effective photo carriers to transport from the GeSn absorber to the Si multiplication layer such that clear punch-through behavior and a saturated primary responsivity of 0.3 A/W at 1550 nm were observed before avalanche breakdown in GeSn/Si APDs for the first time. The spectral response covers 1500 to 1700 nm. The measured punch-through and breakdown voltages are 15 and 17 V, respectively. Undisputed multiplication gain was obtained with the maximum value of 4.5 at 77 K, and 1.4 at 250 K, directly in reference to the saturated primary responsivity from the same device rather than a different GeSn p-i-n photodiode in previous reports. A peak responsivity was measured as 1.12 A/W at 1550 nm and 77 K., Comment: 8 pages, 5 figures, invited paper

Let $\Omega$ be a bounded, smooth domain of $\mathbb{R}^{N},$ $N\geq2.$ For $p>N$ and $1\leq q(p)<\infty$ set \[ \lambda_{p,q(p)}:=\inf\left\{ \int_{\Omega}\left\vert \nabla u\right\vert ^{p}\mathrm{d}x:u\in W_{0}^{1,p}(\Omega)\text{ \ and \ }\int_{\Omega }\left\vert u\right\vert ^{q(p)}\mathrm{d}x=1\right\} \] and let $u_{p,q(p)}$ denote a corresponding positive extremal function. We show that if $\lim\limits_{p\rightarrow\infty}q(p)=\infty$, then $\lim\limits_{p\rightarrow\infty}\lambda_{p,q(p)}^{1/p}=\left\Vert d_{\Omega }\right\Vert _{\infty}^{-1}$, where $d_{\Omega}$ denotes the distance function to the boundary of $\Omega.$ Moreover, in the hyperdiffusive case: $\lim\limits_{p\rightarrow\infty}\frac{q(p)}{p}=\infty,$ we prove that each sequence $u_{p_{n},q(p_{n})},$ with $p_{n}\rightarrow\infty,$ admits a subsequence converging uniformly in $\overline{\Omega}$ to a viscosity solution to the problem \[ \left\{ \begin{array} [c]{lll} -\Delta_{\infty}u=0 & \text{in} & \Omega\setminus M\\ u=0 & \text{on} & \partial\Omega\\ u=1 & \text{in} & M, \end{array} \right. \] where $M$ is a closed subset of the set of all maximum points of $d_{\Omega}.$, Comment: 16 pages

We use 62 electron density profiles collected by the Radio Occultation Science Experiment (ROSE), on MAVEN, when Mars was hit by CIRs and ICMEs close to aphelion (April 2021) and during two dust storms (June-July 2022) to examine the response of the Martian ionosphere to solar events and to solar events hitting during dust storms. We do so through three proxies - variation in total electron content between 80 and 300 km altitude, peak density, and peak altitude - of the aforementioned 62 ROSE electron density profiles, relative to a characterisation of the ionosphere through solar minimum leading to solar maximum, specific to local time sector and season, presented in Segale et al., (COMPANION). We observe an increased Total Electron Content (TEC) between 80 and 300 km altitude up to 2.5 x 10(15) m(-2) in April 2021 and up to 5 x 10(15) m(-2) in June-July 2022 compared to the baseline photochemically produced ionosphere. This increase in TEC corresponds mainly to increases in the solar energetic particles flux (detected by MAVEN SEP) and electron fluxes (detected by MAVEN SWEA). In addition to solar events, in June-July 2022, an A storm and a B storm were occurring and merging on the surface of Mars. We observe a raise in peak altitude in general lower than expected during dust storms, possibly due to high values of solar wind dynamic pressure (derived from MAVEN SWIA). From 31 ROSE profiles collected in this time period that showed both the M2 and M1 layer, we observe that, on average, M1 and M2 peak altitudes raise the same amount, suggesting that the thermosphere might loft as a unit during dust storms. During this time period, several proton aurora events of variable brightness were detected with MAVEN IUVS underlining the complex and multifaceted impact of dust activity and extreme solar activity on the Martian ionosphere.

Graph neural networks (GNNs) have achieved remarkable success across a wide range of applications, such as recommendation, drug discovery, and question answering. Behind the success of GNNs lies the backpropagation (BP) algorithm, which is the de facto standard for training deep neural networks (NNs). However, despite its effectiveness, BP imposes several constraints, which are not only biologically implausible, but also limit the scalability, parallelism, and flexibility in learning NNs. Examples of such constraints include storage of neural activities computed in the forward pass for use in the subsequent backward pass, and the dependence of parameter updates on non-local signals. To address these limitations, the forward-forward algorithm (FF) was recently proposed as an alternative to BP in the image classification domain, which trains NNs by performing two forward passes over positive and negative data. Inspired by this advance, we propose ForwardGNN in this work, a new forward learning procedure for GNNs, which avoids the constraints imposed by BP via an effective layer-wise local forward training. ForwardGNN extends the original FF to deal with graph data and GNNs, and makes it possible to operate without generating negative inputs (hence no longer forward-forward). Further, ForwardGNN enables each layer to learn from both the bottom-up and top-down signals without relying on the backpropagation of errors. Extensive experiments on real-world datasets show the effectiveness and generality of the proposed forward graph learning framework. We release our code at https://github.com/facebookresearch/forwardgnn., Comment: ICLR 2024

Symmetric Nonnegative Matrix Factorization (SymNMF) is a technique in data analysis and machine learning that approximates a symmetric matrix with a product of a nonnegative, low-rank matrix and its transpose. To design faster and more scalable algorithms for SymNMF we develop two randomized algorithms for its computation. The first algorithm uses randomized matrix sketching to compute an initial low-rank approximation to the input matrix and proceeds to rapidly compute a SymNMF of the approximation. The second algorithm uses randomized leverage score sampling to approximately solve constrained least squares problems. Many successful methods for SymNMF rely on (approximately) solving sequences of constrained least squares problems. We prove theoretically that leverage score sampling can approximately solve nonnegative least squares problems to a chosen accuracy with high probability. Additionally, we prove sampling complexity results for previously proposed hybrid sampling techniques which deterministically include high leverage score rows. This hybrid scheme is crucial for obtaining speeds ups in practice. Finally we demonstrate that both methods work well in practice by applying them to graph clustering tasks on large real world data sets. These experiments show that our methods approximately maintain solution quality and achieve significant speed ups for both large dense and large sparse problems.

Machine-learned force fields have transformed the atomistic modelling of materials by enabling simulations of ab initio quality on unprecedented time and length scales. However, they are currently limited by: (i) the significant computational and human effort that must go into development and validation of potentials for each particular system of interest; and (ii) a general lack of transferability from one chemical system to the next. Here, using the state-of-the-art MACE architecture we introduce a single general-purpose ML model, trained on a public database of 150k inorganic crystals, that is capable of running stable molecular dynamics on molecules and materials. We demonstrate the power of the MACE-MP-0 model - and its qualitative and at times quantitative accuracy - on a diverse set problems in the physical sciences, including the properties of solids, liquids, gases, chemical reactions, interfaces and even the dynamics of a small protein. The model can be applied out of the box and as a starting or "foundation model" for any atomistic system of interest and is thus a step towards democratising the revolution of ML force fields by lowering the barriers to entry., Comment: 119 pages, 63 figures, 37MB PDF

We report that device architecture engineering has a substantial impact on the reverse bias instability that has been reported as a critical issue in commercializing perovskite solar cells. We demonstrate breakdown voltages exceeding -15 V in typical pin structured perovskite solar cells via two steps: i) using polymer hole transporting materials; ii) using a more electrochemically stable gold electrode. While device degradation can be exacerbated by higher reverse bias and prolonged exposure, our as-fabricated perovskite solar cells completely recover their performance even after stressing at -7 V for 9 hours both in the dark and under partial illumination. Following these observations, we systematically discuss and compare the reverse bias driven degradation pathways in perovskite solar cells with different device architectures. Our model highlights the role of electrochemical reaction rates and species in dictating the reverse bias stability of perovskite solar cells.

In this work, we design, analyze, and optimize sequential and shared-memory parallel algorithms for partitioned local depths (PaLD). Given a set of data points and pairwise distances, PaLD is a method for identifying strength of pairwise relationships based on relative distances, enabling the identification of strong ties within dense and sparse communities even if their sizes and within-community absolute distances vary greatly. We design two algorithmic variants that perform community structure analysis through triplet comparisons of pairwise distances. We present theoretical analyses of computation and communication costs and prove that the sequential algorithms are communication optimal, up to constant factors. We introduce performance optimization strategies that yield sequential speedups of up to $29\times$ over a baseline sequential implementation and parallel speedups of up to $19.4\times$ over optimized sequential implementations using up to $32$ threads on an Intel multicore CPU.

Floods are one of the most common natural disasters, with a disproportionate impact in developing countries that often lack dense streamflow gauge networks. Accurate and timely warnings are critical for mitigating flood risks, but hydrological simulation models typically must be calibrated to long data records in each watershed. Using AI, we achieve reliability in predicting extreme riverine events in ungauged watersheds at up to a 5-day lead time that is similar to or better than the reliability of nowcasts (0-day lead time) from a current state of the art global modeling system (the Copernicus Emergency Management Service Global Flood Awareness System). Additionally, we achieve accuracies over 5-year return period events that are similar to or better than current accuracies over 1-year return period events. This means that AI can provide flood warnings earlier and over larger and more impactful events in ungauged basins. The model developed in this paper was incorporated into an operational early warning system that produces publicly available (free and open) forecasts in real time in over 80 countries. This work highlights a need for increasing the availability of hydrological data to continue to improve global access to reliable flood warnings.

Quasar-driven galactic outflows are a major driver of the evolution of massive galaxies. We report observations of a powerful galactic-scale outflow in a $z=3$ extremely red, intrinsically luminous ($L_{\rm bol}\simeq 5\times 10^{47}$erg s$^{-1}$) quasar SDSSJ1652+1728 with the Near Infrared Spectrograph (NIRSpec) on board JWST. We analyze the kinematics of rest-frame optical emission lines and identify the quasar-driven outflow extending out to $\sim 10$ kpc from the quasar with a velocity offset of ($v_{r}=\pm 500$ km s$^{-1}$) and high velocity dispersion (FWHM$=700-2400$ km s$^{-1}$). Due to JWST's unprecedented surface brightness sensitivity in the near-infrared -- we unambiguously show that the powerful high velocity outflow in an extremely red quasar (ERQ) encompasses a large swath of the host galaxy's interstellar medium (ISM). Using the kinematics and dynamics of optical emission lines, we estimate the mass outflow rate -- in the warm ionized phase alone -- to be at least $2300\pm1400$ $M_{\odot}$ yr$^{-1}$. We measure a momentum flux ratio between the outflow and the quasar accretion disk of $\sim$1 on kpc scale, indicating that the outflow was likely driven in a relatively high ($>10^{23}$cm$^{-2}$) column density environment through radiation pressure on dust grains. We find a coupling efficiency between the bolometric luminosity of the quasar and the outflow of 0.1$\%$, matching the theoretical prediction of the minimum coupling efficiency necessary for negative quasar feedback. The outflow has sufficient energetics to drive the observed turbulence seen in shocked regions of the quasar host galaxy, likely directly responsible for prolonging the time it takes for gas to cool efficiently., Comment: 14 pages, 7 figures, 1 table, submitted for publication in ApJ