Search

Energy-based learning algorithms, such as predictive coding (PC), have garnered significant attention in the machine learning community due to their theoretical properties, such as local operations and biologically plausible mechanisms for error correction. In this work, we rigorously analyze the stability, robustness, and convergence of PC through the lens of dynamical systems theory. We show that, first, PC is Lyapunov stable under mild assumptions on its loss and residual energy functions, which implies intrinsic robustness to small random perturbations due to its well-defined energy-minimizing dynamics. Second, we formally establish that the PC updates approximate quasi-Newton methods by incorporating higher-order curvature information, which makes them more stable and able to converge with fewer iterations compared to models trained via backpropagation (BP). Furthermore, using this dynamical framework, we provide new theoretical bounds on the similarity between PC and other algorithms, i.e., BP and target propagation (TP), by precisely characterizing the role of higher-order derivatives. These bounds, derived through detailed analysis of the Hessian structures, show that PC is significantly closer to quasi-Newton updates than TP, providing a deeper understanding of the stability and efficiency of PC compared to conventional learning methods., Comment: 29 pages, 9 theorems

We propose a novel factorization algorithm that leverages the theory underlying the SQUFOF method, including reduced quadratic forms, infrastructural distance, and Gauss composition. We also present an analysis of our method, which has a computational complexity of $O \left( \exp \left( \frac{3}{\sqrt{8}} \sqrt{\ln N \ln \ln N} \right) \right)$, making it more efficient than the classical SQUFOF and CFRAC algorithms. Additionally, our algorithm is polynomial-time, provided knowledge of a (not too large) multiple of the regulator of $\mathbb{Q}(N)$., Comment: 19 pages

Unexpected stimuli induce "error" or "surprise" signals in the brain. The theory of predictive coding promises to explain these observations in terms of Bayesian inference by suggesting that the cortex implements variational inference in a probabilistic graphical model. However, when applied to machine learning tasks, this family of algorithms has yet to perform on par with other variational approaches in high-dimensional, structured inference problems. To address this, we introduce a novel predictive coding algorithm for structured generative models, that we call divide-and-conquer predictive coding (DCPC). DCPC differs from other formulations of predictive coding, as it respects the correlation structure of the generative model and provably performs maximum-likelihood updates of model parameters, all without sacrificing biological plausibility. Empirically, DCPC achieves better numerical performance than competing algorithms and provides accurate inference in a number of problems not previously addressed with predictive coding. We provide an open implementation of DCPC in Pyro on Github., Comment: 22 pages, 5 figures, accepted to Neural Information Processing Systems (NeurIPS) 2024

This paper describes a discrete state-space model -- and accompanying methods -- for generative modelling. This model generalises partially observed Markov decision processes to include paths as latent variables, rendering it suitable for active inference and learning in a dynamic setting. Specifically, we consider deep or hierarchical forms using the renormalisation group. The ensuing renormalising generative models (RGM) can be regarded as discrete homologues of deep convolutional neural networks or continuous state-space models in generalised coordinates of motion. By construction, these scale-invariant models can be used to learn compositionality over space and time, furnishing models of paths or orbits; i.e., events of increasing temporal depth and itinerancy. This technical note illustrates the automatic discovery, learning and deployment of RGMs using a series of applications. We start with image classification and then consider the compression and generation of movies and music. Finally, we apply the same variational principles to the learning of Atari-like games., Comment: 64 pages, 28 figures

Data imputation and data generation have important applications for many domains, like healthcare and finance, where incomplete or missing data can hinder accurate analysis and decision-making. Diffusion models have emerged as powerful generative models capable of capturing complex data distributions across various data modalities such as image, audio, and time series data. Recently, they have been also adapted to generate tabular data. In this paper, we propose a diffusion model for tabular data that introduces three key enhancements: (1) a conditioning attention mechanism, (2) an encoder-decoder transformer as the denoising network, and (3) dynamic masking. The conditioning attention mechanism is designed to improve the model's ability to capture the relationship between the condition and synthetic data. The transformer layers help model interactions within the condition (encoder) or synthetic data (decoder), while dynamic masking enables our model to efficiently handle both missing data imputation and synthetic data generation tasks within a unified framework. We conduct a comprehensive evaluation by comparing the performance of diffusion models with transformer conditioning against state-of-the-art techniques, such as Variational Autoencoders, Generative Adversarial Networks and Diffusion Models, on benchmark datasets. Our evaluation focuses on the assessment of the generated samples with respect to three important criteria, namely: (1) Machine Learning efficiency, (2) statistical similarity, and (3) privacy risk mitigation. For the task of data imputation, we consider the efficiency of the generated samples across different levels of missing features., Comment: 25 pages, 7 figures, 6 tables

In this work, we tackle the problems of efficiency and scalability for predictive coding networks in machine learning. To do so, we first propose a library called PCX, whose focus lies on performance and simplicity, and provides a user-friendly, deep-learning oriented interface. Second, we use PCX to implement a large set of benchmarks for the community to use for their experiments. As most works propose their own tasks and architectures, do not compare one against each other, and focus on small-scale tasks, a simple and fast open-source library adopted by the whole community would address all of these concerns. Third, we perform extensive benchmarks using multiple algorithms, setting new state-of-the-art results in multiple tasks and datasets, as well as highlighting limitations inherent to PC that should be addressed. Thanks to the efficiency of PCX, we are able to analyze larger architectures than commonly used, providing baselines to galvanize community efforts towards one of the main open problems in the field: scalability. The code for PCX is available at \textit{https://github.com/liukidar/pcax}., Comment: 33 pages, 25 figures

We study the construction of local subtraction schemes through the lenses of tropical geometry. We focus on individual Feynman integrals in parametric presentation, and think of them as particular instances of Euler integrals. We provide a necessary and sufficient condition for a combination of Euler integrands to be locally finite, i.e. to be expandable as a Taylor series in the exponent variables directly under sign of integration. We use this to construct a local subtraction scheme that is applicable to a class of Euler integrals that satisfy a certain geometric property. We apply this to compute the Laurent expansion in the dimensional regulator $\epsilon$ of various Feynman integrals involving both UV and IR singularities, as well as to generalizations of Feynman integrals that arise in effective field theories and in phase-space integrations.

We present a characterization of the algebraic integers with continued fraction expansions of the form $[a_0, \overline{a_1, \ldots, a_n, s}]$, where $(a_1, \ldots, a_n)$ is a palindrome and $s \in \mathbb{N}_{\geq 1}$. In particular, we focus on the special case where $(a_1, \ldots, a_n) = (m, \ldots, m)$, providing a detailed characterizations of the corresponding algebraic integers and $s$ in terms of Fibonacci polynomials. Then, we derive new expansions of square roots of integers with these periods, given $m$ and $n$. Moreover, we explicitly determine the fundamental solutions of both positive and negative Pell's equations corresponding to this family of integers.

Our understanding of organs at risk is progressing to include physical small tissues such as coronary arteries and the radiosensitivities of many small organs and tissues are high. Therefore, the accurate segmentation of small volumes in external radiotherapy is crucial to protect them from over-irradiation. Moreover, with the development of the particle therapy and on-board imaging, the treatment becomes more accurate and precise. The purpose of this work is to optimize organ segmentation algorithms for small organs. We used 50 three-dimensional (3-D) computed tomography (CT) head and neck images from StructSeg2019 challenge to develop a general-purpose V-Net model to segment 20 organs in the head and neck region. We applied specific strategies to improve the segmentation accuracy of the small volumes in this anatomical region, i.e., the lens of the eye. Then, we used 17 additional head images from OSF healthcare to validate the robustness of the V Net model optimized for small-volume segmentation. With the study of the StructSeg2019 images, we found that the optimization of the image normalization range and classification threshold yielded a segmentation improvement of the lens of the eye of approximately 50%, compared to the use of the V-Net not optimized for small volumes. We used the optimized model to segment 17 images acquired using heterogeneous protocols. We obtained comparable Dice coefficient values for the clinical and StructSeg2019 images (0.61 plus/minus 0.07 and 0.58 plus/minus 0.10 for the left and right lens of the eye, respectively)

One major criticism of deep learning centers around the biological implausibility of the credit assignment schema used for learning -- backpropagation of errors. This implausibility translates into practical limitations, spanning scientific fields, including incompatibility with hardware and non-differentiable implementations, thus leading to expensive energy requirements. In contrast, biologically plausible credit assignment is compatible with practically any learning condition and is energy-efficient. As a result, it accommodates hardware and scientific modeling, e.g. learning with physical systems and non-differentiable behavior. Furthermore, it can lead to the development of real-time, adaptive neuromorphic processing systems. In addressing this problem, an interdisciplinary branch of artificial intelligence research that lies at the intersection of neuroscience, cognitive science, and machine learning has emerged. In this paper, we survey several vital algorithms that model bio-plausible rules of credit assignment in artificial neural networks, discussing the solutions they provide for different scientific fields as well as their advantages on CPUs, GPUs, and novel implementations of neuromorphic hardware. We conclude by discussing the future challenges that will need to be addressed in order to make such algorithms more useful in practical applications., Comment: 13 Pages, 1 figure

An autoassociative memory model is a function that, given a set of data points, takes as input an arbitrary vector and outputs the most similar data point from the memorized set. However, popular memory models fail to retrieve images even when the corruption is mild and easy to detect for a human evaluator. This is because similarities are evaluated in the raw pixel space, which does not contain any semantic information about the images. This problem can be easily solved by computing \emph{similarities} in an embedding space instead of the pixel space. We show that an effective way of computing such embeddings is via a network pretrained with a contrastive loss. As the dimension of embedding spaces is often significantly smaller than the pixel space, we also have a faster computation of similarity scores. We test this method on complex datasets such as CIFAR10 and STL10. An additional drawback of current models is the need of storing the whole dataset in the pixel space, which is often extremely large. We relax this condition and propose a class of memory models that only stores low-dimensional semantic embeddings, and uses them to retrieve similar, but not identical, memories. We demonstrate a proof of concept of this method on a simple task on the MNIST dataset., Comment: 8 Pages, 4 Figures, accepted for publication at ECAI 2023

The efficiency of supported catalysts depends on the porous microstructure of their solid support, which regulates mass transfer, exposure to the active phase, and mechanical strength. Here, we focus on the manufacturing of a specific type of catalytic support, $\gamma$-alumina extrudates, by a kneading-extrusion process. In this process, a paste is initially formed by blending and subsequently kneading a boehmite powder, an aluminum oxide hydroxide precursor of $\gamma$-alumina, with various liquids before undergoing extrusion. The crucial step in this process is the kneading step, which allows control over the textural and mechanical properties of the extrudate. The present experimental study aims to measure the impact of the kneading step on boehmite pastes prepared using a pilot kneader. The pastes are obtained by mixing boehmite powder with an acid and a basic solution in a two-step process known as peptization and neutralization. In this study, kneading is conducted at various mixing speeds and durations while monitoring the torque exerted by the boehmite paste on the kneader blades. Moreover, samples are extracted from the pilot kneader at various stages of the kneading process, and their textural properties are determined by both nitrogen sorption and mercury intrusion porosimetry. Our findings show that, at fixed composition, the textural properties of boehmite pastes are controlled by the overall deformation accumulated during kneading. Conversely, for a fixed accumulated deformation, the pH sets the textural properties. Finally, we identify an empirical control parameter that captures the combined effects of pH and accumulated deformation on the key textural attributes of boehmite pastes. These results set the stage for a systematic design approach of boehmite-based catalyst supports., Comment: 14 pages, 13 figures

Recently a new formulation for scattering amplitudes in Tr($\Phi^3$) theory has been given based on simple combinatorial ideas in the space of kinematic data. This allows all-loop integrated amplitudes to be expressed as ''curve integrals'' defined using tropical building blocks - the ''headlight functions''. This paper shows how the formulation extends to the amplitudes of more general Lagrangians. We will present a number of different ways of introducing tropical ''numerator functions'' that allow us to describe general Lagrangian interactions. The simplest family of these ''tropical numerators'' computes the amplitudes of interesting Lagrangians with infinitely many interactions. We also describe methods for tropically formulating the amplitudes for general Lagrangians. One uses a variant of ''Wick contraction'' to glue together numerator factors for general interaction vertices. Another uses a natural characterization of polygons on surfaces to give a novel combinatorial description of all possible diagrams associated with arbitrary valence interactions., Comment: 35 pages, 13 figures

We consider the open unit disk $\mathbb{D}$ equipped with the hyperbolic metric and the associated hyperbolic Laplacian $\mathfrak{L}$. For $\lambda \in \mathbb{C}$ and $n \in \mathbb{N}$, a $\lambda$-polyharmonic function of order $n$ is a function $f: \mathbb{D} \to \mathbb{C}$ such that $(\mathfrak{L}- \lambda \, I)^n f = 0$. If $n =1$, one gets $\lambda$-harmonic functions. Based on a Theorem of Helgason on the latter functions, we prove a boundary integral representation theorem for $\lambda$-polyharmonic functions. For this purpose, we first determine $n^{\text{th}}$-order $\lambda$-Poisson kernels. Subsequently, we introduce the $\lambda$-polyspherical functions and determine their asymptotics at the boundary $\partial \mathbb{D}$, i.e., the unit circle. In particular, this proves that, for eigenvalues not in the interior of the $L^2$-spectrum, the zeroes of these functions do not accumulate at the boundary circle. Hence the polyspherical functions can be used to normalise the $n^{\text{th}}$-order Poisson kernels. By this tool, we extend to this setting several classical results of potential theory: namely, we study the boundary behaviour of $\lambda$-polyharmonic functions, starting with Dirichlet and Riquier type problems and then proceeding to Fatou type admissible boundary limits.

Recent advances in theoretical biology suggest that basal cognition and sentient behaviour are emergent properties of in vitro cell cultures and neuronal networks, respectively. Such neuronal networks spontaneously learn structured behaviours in the absence of reward or reinforcement. In this paper, we characterise this kind of self-organisation through the lens of the free energy principle, i.e., as self-evidencing. We do this by first discussing the definitions of reactive and sentient behaviour in the setting of active inference, which describes the behaviour of agents that model the consequences of their actions. We then introduce a formal account of intentional behaviour, that describes agents as driven by a preferred endpoint or goal in latent state-spaces. We then investigate these forms of (reactive, sentient, and intentional) behaviour using simulations. First, we simulate the aforementioned in vitro experiments, in which neuronal cultures spontaneously learn to play Pong, by implementing nested, free energy minimising processes. The simulations are then used to deconstruct the ensuing predictive behaviour, leading to the distinction between merely reactive, sentient, and intentional behaviour, with the latter formalised in terms of inductive planning. This distinction is further studied using simple machine learning benchmarks (navigation in a grid world and the Tower of Hanoi problem), that show how quickly and efficiently adaptive behaviour emerges under an inductive form of active inference., Comment: 33 pages, 9 figures

This paper concerns structure learning or discovery of discrete generative models. It focuses on Bayesian model selection and the assimilation of training data or content, with a special emphasis on the order in which data are ingested. A key move - in the ensuing schemes - is to place priors on the selection of models, based upon expected free energy. In this setting, expected free energy reduces to a constrained mutual information, where the constraints inherit from priors over outcomes (i.e., preferred outcomes). The resulting scheme is first used to perform image classification on the MNIST dataset to illustrate the basic idea, and then tested on a more challenging problem of discovering models with dynamics, using a simple sprite-based visual disentanglement paradigm and the Tower of Hanoi (cf., blocks world) problem. In these examples, generative models are constructed autodidactically to recover (i.e., disentangle) the factorial structure of latent states - and their characteristic paths or dynamics.

This is part of a series of papers describing the new curve integral formalism for scattering amplitudes of the colored scalar tr$\phi^3$ theory. We show that the curve integral manifests a very surprising fact about these amplitudes: the dependence on the number of particles, $n$, and the loop order, $L$, is effectively decoupled. We derive the curve integrals at tree-level for all $n$. We then show that, for higher loop-order, it suffices to study the curve integrals for $L$-loop tadpole-like amplitudes, which have just one particle per color trace-factor. By combining these tadpole-like formulas with the the tree-level result, we find formulas for the all $n$ amplitudes at $L$ loops. We illustrate this result by giving explicit curve integrals for all the amplitudes in the theory, including the non-planar amplitudes, through to two loops, for all $n$.

This is the first in a series of papers presenting a new understanding of scattering amplitudes based on fundamentally combinatorial ideas in the kinematic space of the scattering data. We study the simplest theory of colored scalar particles with cubic interactions, at all loop orders and to all orders in the topological 't Hooft expansion. We find a novel formula for loop-integrated amplitudes, with no trace of the conventional sum over Feynman diagrams, but instead determined by a beautifully simple counting problem attached to any order of the topological expansion. These results represent a significant step forward in the decade-long quest to formulate the fundamental physics of the real world in a radically new language, where the rules of spacetime and quantum mechanics, as reflected in the principles of locality and unitarity, are seen to emerge from deeper mathematical structures.

Artificial intelligence (AI) is rapidly becoming one of the key technologies of this century. The majority of results in AI thus far have been achieved using deep neural networks trained with the error backpropagation learning algorithm. However, the ubiquitous adoption of this approach has highlighted some important limitations such as substantial computational cost, difficulty in quantifying uncertainty, lack of robustness, unreliability, and biological implausibility. It is possible that addressing these limitations may require schemes that are inspired and guided by neuroscience theories. One such theory, called predictive coding (PC), has shown promising performance in machine intelligence tasks, exhibiting exciting properties that make it potentially valuable for the machine learning community: PC can model information processing in different brain areas, can be used in cognitive control and robotics, and has a solid mathematical grounding in variational inference, offering a powerful inversion scheme for a specific class of continuous-state generative models. With the hope of foregrounding research in this direction, we survey the literature that has contributed to this perspective, highlighting the many ways that PC might play a role in the future of machine learning and computational intelligence at large., Comment: 37 Pages, 9 Figures

Recently, there has been extensive research on the capabilities of biologically plausible algorithms. In this work, we show how one of such algorithms, called predictive coding, is able to perform causal inference tasks. First, we show how a simple change in the inference process of predictive coding enables to compute interventions without the need to mutilate or redefine a causal graph. Then, we explore applications in cases where the graph is unknown, and has to be inferred from observational data. Empirically, we show how such findings can be used to improve the performance of predictive coding in image classification tasks, and conclude that such models are able to perform simple end-to-end causal inference tasks., Comment: 44 Pages, 24 Figures. Changed title and abstract, following the ICML accepted version

All industries are trying to leverage Artificial Intelligence (AI) based on their existing big data which is available in so called tabular form, where each record is composed of a number of heterogeneous continuous and categorical columns also known as features. Deep Learning (DL) has constituted a major breakthrough for AI in fields related to human skills like natural language processing, but its applicability to tabular data has been more challenging. More classical Machine Learning (ML) models like tree-based ensemble ones usually perform better. This paper presents a novel DL model using Graph Neural Network (GNN) more specifically Interaction Network (IN), for contextual embedding and modelling interactions among tabular features. Its results outperform those of a recently published survey with DL benchmark based on five public datasets, also achieving competitive results when compared to boosted-tree solutions.

We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!, Comment: Added further evaluations on another ChatGPT version and on GPT-4. The GHOSTS and miniGHOSTS datasets are available at https://github.com/xyfrieder/science-GHOSTS

An investigation on the correlation between the strong alcoholic drinking and SARS-Cov-2 uninfection rate is carried out. The investigation is done through a simple survey in China based on the social media software Weixin and the survey mini program Wenjuanxin, during 15:00 Jan.1, 2023 to 12:35 Jan.3, 2023. From the $211$ survey questionnaires collected, we find a significant positive correlation between the frequent (no less than three times a week) strong liquor (higher than $40\%$ alcoholic content in volume) drinking and the higher SARS-Cov-2 uninfection rate ($38.6\%$ compared to the value $25.1\%$ for general population). The p-value of this statistical result shows that the correlation is significant., Comment: 16 pages, 4 figures, 1 table

Predictive coding is a message-passing framework initially developed to model information processing in the brain, and now also topic of research in machine learning due to some interesting properties. One of such properties is the natural ability of generative models to learn robust representations thanks to their peculiar credit assignment rule, that allows neural activities to converge to a solution before updating the synaptic weights. Graph neural networks are also message-passing models, which have recently shown outstanding results in diverse types of tasks in machine learning, providing interdisciplinary state-of-the-art performance on structured data. However, they are vulnerable to imperceptible adversarial attacks, and unfit for out-of-distribution generalization. In this work, we address this by building models that have the same structure of popular graph neural network architectures, but rely on the message-passing rule of predictive coding. Through an extensive set of experiments, we show that the proposed models are (i) comparable to standard ones in terms of performance in both inductive and transductive tasks, (ii) better calibrated, and (iii) robust against multiple kinds of adversarial attacks., Comment: 27 Pages, 31 Figures

Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. Training such models, however, is quite inefficient and unstable. In this work, we show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one, and has theoretical guarantees in terms of convergence. The proposed algorithm, that we call incremental predictive coding (iPC) is also more biologically plausible than the original one, as it it fully automatic. In an extensive set of experiments, we show that iPC constantly performs better than the original formulation on a large number of benchmarks for image classification, as well as for the training of both conditional and masked language models, in terms of test accuracy, efficiency, and convergence with respect to a large set of hyperparameters., Comment: Change of title and abstract, that now reflect the version accepted for publication. One co-author also added, that performed the additional experiments

A large amount of recent research has the far-reaching goal of finding training methods for deep neural networks that can serve as alternatives to backpropagation (BP). A prominent example is predictive coding (PC), which is a neuroscience-inspired method that performs inference on hierarchical Gaussian generative models. These methods, however, fail to keep up with modern neural networks, as they are unable to replicate the dynamics of complex layers and activation functions. In this work, we solve this problem by generalizing PC to arbitrary probability distributions, enabling the training of architectures, such as transformers, that are hard to approximate with only Gaussian assumptions. We perform three experimental analyses. First, we study the gap between our method and the standard formulation of PC on multiple toy examples. Second, we test the reconstruction quality on variational autoencoders, where our method reaches the same reconstruction quality as BP. Third, we show that our method allows us to train transformer networks and achieve a performance comparable with BP on conditional language models. More broadly, this method allows neuroscience-inspired learning to be applied to multiple domains, since the internal distributions can be flexibly adapted to the data, tasks, and architectures used.