Search

One of the open challenges in Reinforcement Learning is the hard exploration problem in sparse reward environments. Various types of intrinsic rewards have been proposed to address this challenge by pushing towards diversity. This diversity might be imposed at different levels, favouring the agent to explore different states, policies or behaviours (State, Policy and Skill level diversity, respectively). However, the impact of diversity on the agent's behaviour remains unclear. In this work, we aim to fill this gap by studying the effect of different levels of diversity imposed by intrinsic rewards on the exploration patterns of RL agents. We select four intrinsic rewards (State Count, Intrinsic Curiosity Module (ICM), Maximum Entropy, and Diversity is all you need (DIAYN)), each pushing for a different diversity level. We conduct an empirical study on MiniGrid environment to compare their impact on exploration considering various metrics related to the agent's exploration, namely: episodic return, observation coverage, agent's position coverage, policy entropy, and timeframes to reach the sparse reward. The main outcome of the study is that State Count leads to the best exploration performance in the case of low-dimensional observations. However, in the case of RGB observations, the performance of State Count is highly degraded mostly due to representation learning challenges. Conversely, Maximum Entropy is less impacted, resulting in a more robust exploration, despite being not always optimal. Lastly, our empirical study revealed that learning diverse skills with DIAYN, often linked to improved robustness and generalisation, does not promote exploration in MiniGrid environments. This is because: i) learning the skill space itself can be challenging, and ii) exploration within the skill space prioritises differentiating between behaviours rather than achieving uniform state visitation., Comment: 45 pages, 17 figures. Submitted to Neural Computing and Applications Journal

In this paper, extending some ideas of Fano, we study the birational geometry of the Hilbert scheme of 0-dimensional subschemes of length 2 of a rational normal scroll. This fourfold has three elementary contractions associated to the three faces of its nef cone. We study natural projective realizations of these contractions. In particular, given a smooth rational normal scroll $S_{a,b}$ of degree $r$ in ${\mathbb P}^{r+1}$ with $1 \leq a \leq b$ and a+b=r, i.e., $S_{a,b}$ is the relative Proj of the vector bundle $O_{{\mathbb P}^1}(a)\oplus O_{{\mathbb P}^1}(b)$ embedded in ${\mathbb P}^{r+1}$ with its O(1) line bundle (from an abstract viewpoint $S_{a,b}\cong {\mathbb F}_{b-a}$), we consider the subvariety $X_{a,b}$ of the Grassmannian $G(1,r+1)$ described by all lines that are secant or tangent to $S_{a,b}$. The variety $X_{a,b}$ is the image of some of the aforementioned contractions, it is smooth if a>1, and it is singular at a unique point if a=1. We compute the degree of $X_{a,b}$ and the local structure of the singularity of $X_{a,b}$ when a=1. Finally we discuss in some detail the case r=4, originally considered by Fano, because the smooth hyperplane sections of $X_{2,2}$ and $X_{1,3}$ are the Fano 3-folds that appear as number 16 in the Mori-Mukai list of Fano 3-folds with Picard number 2. We prove that any smooth hyperplane section of $X_{2,2}$ is also a hyperplane section of $X_{1,3}$, and we discuss the GIT-stability of the smooth hyperplane sections of $X_{1,3}$ where $G$ is the subgroup of the projective automorphisms of $X_{1,3}$ coming from the ones of $S_{1,3}.$, Comment: 18 pages

Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities; however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies-areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community., Comment: Preprint, under review

This short note is the extended abstract of a seminar I have delivered on several occasions over the past few months on canonical threefolds whose canonical volume is "close" to the lower bound 4/3p_g - 10/3. This is a project in collaboration with S. Coughlan, Y. Hu, and T. Zhang., Comment: 3 pages

In this paper, we completely classify the canonical threefolds on the Noether line with geometric genus $p_g \ge 11$ by studying their moduli spaces. For every such moduli space, we establish an explicit stratification, estimate the number of its irreducible components and prove the dimension formula. A new and unexpected phenomenon is that the number of irreducible components grows linearly with the geometric genus, while the moduli space of canonical surfaces on the Noether line with any prescribed geometric genus has at most two irreducible components. The key idea in the proof is to relate the canonical threefolds on the Noether line to the simple fibrations in $(1, 2)$-surfaces by proving a conjecture stated by two of the authors in [CP]., Comment: Comments are welcome

The temporal credit assignment problem is a central challenge in Reinforcement Learning (RL), concerned with attributing the appropriate influence to each actions in a trajectory for their ability to achieve a goal. However, when feedback is delayed and sparse, the learning signal is poor, and action evaluation becomes harder. Canonical solutions, such as reward shaping and options, require extensive domain knowledge and manual intervention, limiting their scalability and applicability. In this work, we lay the foundations for Credit Assignment with Language Models (CALM), a novel approach that leverages Large Language Models (LLMs) to automate credit assignment via reward shaping and options discovery. CALM uses LLMs to decompose a task into elementary subgoals and assess the achievement of these subgoals in state-action transitions. Every time an option terminates, a subgoal is achieved, and CALM provides an auxiliary reward. This additional reward signal can enhance the learning process when the task reward is sparse and delayed without the need for human-designed rewards. We provide a preliminary evaluation of CALM using a dataset of human-annotated demonstrations from MiniHack, suggesting that LLMs can be effective in assigning credit in zero-shot settings, without examples or LLM fine-tuning. Our preliminary results indicate that the knowledge of LLMs is a promising prior for credit assignment in RL, facilitating the transfer of human knowledge into value functions., Comment: 9 pages

As Deep Reinforcement Learning (Deep RL) research moves towards solving large-scale worlds, efficient environment simulations become crucial for rapid experimentation. However, most existing environments struggle to scale to high throughput, setting back meaningful progress. Interactions are typically computed on the CPU, limiting training speed and throughput, due to slower computation and communication overhead when distributing the task across multiple machines. Ultimately, Deep RL training is CPU-bound, and developing batched, fast, and scalable environments has become a frontier for progress. Among the most used Reinforcement Learning (RL) environments, MiniGrid is at the foundation of several studies on exploration, curriculum learning, representation learning, diversity, meta-learning, credit assignment, and language-conditioned RL, and still suffers from the limitations described above. In this work, we introduce NAVIX, a re-implementation of MiniGrid in JAX. NAVIX achieves over 200 000x speed improvements in batch mode, supporting up to 2048 agents in parallel on a single Nvidia A100 80 GB. This reduces experiment times from one week to 15 minutes, promoting faster design iterations and more scalable RL model development.

We compute the automorphism group of all the elements of a family of surfaces of general type with $p_g=q=2$ and $K^2=7$, originally constructed by C. Rito. We discuss the consequences of our results towards the Mumford-Tate conjecture., Comment: 14 pages, 1 figure. To appear in Rendiconti del Seminario Matematico di Torino

A re-calculation of a known family of formulas of PI is carried out, revisiting the old Archimedes' algorithm. This allows to identify a general family equation and three new simple formulas of Pi in terms of the golden ratio PHI in the form of infinite nested square roots, with some geometrical properties that enhance the link between the circle and the golden ratio. Applying the same criteria, a fourth formula is given, that brings to the known Dixon's squaring the circle approximation, thus an easier approach to this problem is suggested, by a rectangle with both sides proportional to the golden ratio PHI., Comment: 10 pages, 5 figures - review process for publishing in JAMP Journal of Applied Mathematics and Physics

Temperature is a major source of inaccuracy in high-sensitivity accelerometers and gravimeters. Active thermal control systems require power and may not be ideal in some contexts such as airborne or spaceborne applications. We propose a solution that relies on multiple thermometers placed within the accelerometer to measure temperature and thermal gradient variations. Machine Learning algorithms are used to relate the temperatures to their effect on the accelerometer readings. However, obtaining labeled data for training these algorithms can be difficult. Therefore, we also developed a training platform capable of replicating temperature variations in a laboratory setting. Our experiments revealed that thermal gradients had a significant effect on accelerometer readings, emphasizing the importance of multiple thermometers. The proposed method was experimentally tested and revealed a great potential to be extended to other sources of inaccuracy, such as rotations, as well as to other types of measuring systems, such as magnetometers or gyroscopes.

The Credit Assignment Problem (CAP) refers to the longstanding challenge of Reinforcement Learning (RL) agents to associate actions with their long-term consequences. Solving the CAP is a crucial step towards the successful deployment of RL in the real world since most decision problems provide feedback that is noisy, delayed, and with little or no information about the causes. These conditions make it hard to distinguish serendipitous outcomes from those caused by informed decision-making. However, the mathematical nature of credit and the CAP remains poorly understood and defined. In this survey, we review the state of the art of Temporal Credit Assignment (CA) in deep RL. We propose a unifying formalism for credit that enables equitable comparisons of state-of-the-art algorithms and improves our understanding of the trade-offs between the various methods. We cast the CAP as the problem of learning the influence of an action over an outcome from a finite amount of experience. We discuss the challenges posed by delayed effects, transpositions, and a lack of action influence, and analyse how existing methods aim to address them. Finally, we survey the protocols to evaluate a credit assignment method and suggest ways to diagnose the sources of struggle for different methods. Overall, this survey provides an overview of the field for new-entry practitioners and researchers, it offers a coherent perspective for scholars looking to expedite the starting stages of a new study on the CAP, and it suggests potential directions for future research., Comment: 56 pages, 2 figures, 4 tables

In this work we classify all smooth surfaces with geometric genus equal to three and an action of a group G isomorphic to (Z/2)^k such that the quotient is a plane. We find 11 families. We compute the canonical map of all of them, finding in particular a family of surfaces with canonical map of degree 16 that we could not find in the literature. We discuss the quotients by all subgroups of G finding several K3 surfaces with symplectic involutions. In particular we show that six families are families of triple K3 burgers in the sense of Laterveer., Comment: 30 pages. v2: added a comment on the dependence of the three involutions that determine the structure of triple K3 burgers. Final version to appear on Rend. Mat. Appl. (7)

Atrial Fibrillation (AF) is characterized by disorganised electrical activity in the atria and is known to be sustained by the presence of regions of fibrosis (scars) or functional cellular remodeling, both of which may lead to areas of slow conduction. Estimating the effective conductivity of the myocardium and identifying regions of abnormal propagation is therefore crucial for the effective treatment of AF. We hypothesise that the spatial distribution of tissue conductivity can be directly inferred from an array of concurrently acquired contact electrograms (EGMs). We generate a dataset of simulated cardiac AP propagation using randomised scar distributions and a phenomenological cardiac model and calculate contact EGMs at various positions on the field. EGMs are enriched with noise extracted from biological data acquired in the lab. A deep neural network, based on a modified U-net architecture, is trained to estimate the location of the scar and quantify conductivity of the tissue with a Jaccard index of 91%. We adapt a wavelet-based surrogate testing analysis to confirm that the inferred conductivity distribution is an accurate representation of the ground truth input to the model. We find that the root mean square error (RMSE) between the ground truth and our predictions is significantly smaller ($p_{val}<0.01$) than the RMSE between the ground truth and surrogate samples., Comment: 16 pages, 15 figures

In 1949 Fano published his last paper on $3$-folds with canonical sectional curves. There he constructed and described a $3$-fold of the type $X^{22}_3$ in ${\mathbb P}^{13}$ with canonical curve section, which we like to call Fano's last Fano. We report on Fano's construction, providing various (in our opinion missing) proofs, in modern language and trying to use results and techniques available at that time. Then we construct Fano's with modern tools, in particular via the Hilbert scheme of zero cycles on a rational surface; as a consequence we easily point out the corresponding example in the Mori-Mukai classification., Comment: v1: 18 pages; v2: 22 pages, final version to appear on Atti Accad. Naz. Lincei Cl. Sci. Fis. Mat. Natur. Rend. Lincei (9) Mat. Appl

We introduce the notion of a simple fibration in $(1,2)$-surfaces. That is, a hypersurface inside a certain weighted projective space bundle over a curve such that the general fibre is a minimal surface of general type with $p_g=2$ and $K^2=1$. We prove that almost all Gorenstein simple fibrations over the projective line with at worst canonical singularities are canonical threefolds "on the Noether line" with $K^3=\frac43 p_g-\frac{10}3$, and we classify them. Among them, we find all the canonical threefolds on the Noether line that have previously appeared in the literature. The Gorenstein simple fibrations over $\mathbb{P}^1$ are Cartier divisors in a toric $4$-fold. This allows to us to show among other things, that the previously known canonical threefolds on the Noether line form an open subset of the moduli space of canonical threefolds, that the general element of this component is a Mori Dream Space, and that there is a second component when the geometric genus is congruent to $6$ modulo $8$; the threefolds in this component are new., Comment: 38 pages. Minor changes, to appear in Forum of Mathematics, Sigma

We describe an algorithm that constructs a list of all topological types of holomorphic actions of a finite group on a compact Riemann surface $C$ of genus at least $g \geq 2$ with $C/G \cong \mathbb{P}^1$., Comment: v2: corrected definition of Hurwitz equivalence using the braid group

Poly(vinylidene fluoride-trifluoroethylene, P(VDF-TrFE)) and P(VDF-TrFE)/barium titanate nanoparticle (BTNP) films are prepared and tested as substrates for neuronal stimulation through direct piezoelectric effect. Films are characterized in terms of surface, mechanical, and piezoelectric features before in vitro testing on SH-SY5Y cells. In particular, BTNPs significantly improve piezoelectric properties of the films (4.5-fold increased d31). Both kinds of films support good SH-SY5Y viability and differentiation. Ultrasound (US) stimulation is proven to elicit Ca2+ transients and to enhance differentiation in cells grown on the piezoelectric substrates. For the first time in the literature, this study demonstrates the suitability of polymer/ceramic composite films and US for neuronal stimulation through direct piezoelectric effect.

In this short note we construct unbounded families of minimal surfaces of general type with canonical map of degree 4 such that the limits of the slopes assume countably many different values among 6+2/3 and 8., Comment: 8 pages,v2: added a reference, v3: minor changes, final version to appear on Port. Math

The Coleman-Oort conjecture says that for large $g$ there are no positive-dimensional Shimura subvarieties of $\mathsf{A}_g$ generically contained in the Jacobian locus. Counterexamples are known for $g\leq 7$. They can all be constructed using families of Galois coverings of curves satisfying a numerical condition. These families are already classified in cases where: a) the Galois group is cyclic, b) it is abelian and the family is 1-dimensional, and c) $g\leq 9$. By means of carefully designed computations and theoretical arguments excluding a large number of cases we are able to prove that for $g\leq 100$ there are no other families than those already known., Comment: Accepted for publication on Revista de la Real Academia de Ciencias Exactas, F\'isicas y Naturales. Serie A. Matem\'aticas

We study a family of surfaces of general type with $p_g=q=2$ and $K^2=7$, originally constructed by C. Rito. We provide an alternative construction of these surfaces, that allows us to describe their Albanese map and the corresponding locus $\mathcal{M}$ in the moduli space of the surfaces of general type. In particular we prove that $\mathcal{M}$ is an open subset, and it has three connected components, two dimensional, irreducible and generically smooth., Comment: 26 pages, 7 figures. In this version we have corrected a miscalculation of the previous version regarding the number of components

We investigate the performance of fully convolutional networks to simulate the motion and interaction of surface waves in open and closed complex geometries. We focus on a U-Net architecture and analyse how well it generalises to geometric configurations not seen during training. We demonstrate that a modified U-Net architecture is capable of accurately predicting the height distribution of waves on a liquid surface within curved and multi-faceted open and closed geometries, when only simple box and right-angled corner geometries were seen during training. We also consider a separate and independent 3D CNN for performing time-interpolation on the predictions produced by our U-Net. This allows generating simulations with a smaller time-step size than the one the U-Net has been trained for.

We produce an example of a rigid, but not infinitesimally rigid smooth compact complex surface with ample canonical bundle using results about arrangements of lines inspired by work of Hirzebruch, Kapovich and Millson, Manetti and Vakil., Comment: 28 pages, 1 figure. For associated M2 files visit https://www.math.uni-hamburg.de/home/bothmer/M2/rigidNotInf/ To be published in the special issue of Bollettino dell'Unione Matematica Italiana dedicated to Fabrizio Catanese

Dynamical systems can be modelled by partial differential equations and numerical computations are used everywhere in science and engineering. In this work, we investigate the performance of recurrent and convolutional deep neural network architectures to predict the surface waves. The system is governed by the Saint-Venant equations. We improve on the long-term prediction over previous methods while keeping the inference time at a fraction of numerical simulations. We also show that convolutional networks perform at least as well as recurrent networks in this task. Finally, we assess the generalisation capability of each network by extrapolating in longer time-frames and in different physical settings.