Search

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,700 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai., Comment: 27 pages, 6 figures

Energy is no doubt an intuitive concept. Following a previous analysis on the nature of elementary particles and associated elementary quantum fields, the peculiar status and role of energy is scrutinised further at elementary and larger scales. Energy physical characterisation shows that it is a primordial component of reality highlighting the quantum fields natural tendencies to interact, the elementary particles natural tendency to constitute complex bodies and every material thing natural tendency to actualise and be active. Energy therefore is a primordial notion in need of a proper assessment., Comment: 30 pages, 5 figures

In this paper, we extend the port-Hamiltonian framework by introducing the concept of Stokes-Lagrange structure, which enables the implicit definition of a Hamiltonian over an $N$-dimensional domain and incorporates energy ports into the system. This new framework parallels the existing Dirac and Stokes-Dirac structures. We propose the Stokes-Lagrange structure as a specific case where the subspace is explicitly defined via differential operators that satisfy an integration by parts formula. By examining various examples through the lens of the Stokes-Lagrange structure, we demonstrate the existence of multiple equivalent system representations. These representations provide significant advantages for both numerical simulation and control design, offering additional tools for the modelling and control of port-Hamiltonian systems., Comment: 50 pages

The human brain receives nutrients and oxygen through an intricate network of blood vessels. Pathology affecting small vessels, at the mesoscopic scale, represents a critical vulnerability within the cerebral blood supply and can lead to severe conditions, such as Cerebral Small Vessel Diseases. The advent of 7 Tesla MRI systems has enabled the acquisition of higher spatial resolution images, making it possible to visualise such vessels in the brain. However, the lack of publicly available annotated datasets has impeded the development of robust, machine learning-driven segmentation algorithms. To address this, the SMILE-UHURA challenge was organised. This challenge, held in conjunction with the ISBI 2023, in Cartagena de Indias, Colombia, aimed to provide a platform for researchers working on related topics. The SMILE-UHURA challenge addresses the gap in publicly available annotated datasets by providing an annotated dataset of Time-of-Flight angiography acquired with 7T MRI. This dataset was created through a combination of automated pre-segmentation and extensive manual refinement. In this manuscript, sixteen submitted methods and two baseline methods are compared both quantitatively and qualitatively on two different datasets: held-out test MRAs from the same dataset as the training data (with labels kept secret) and a separate 7T ToF MRA dataset where both input volumes and labels are kept secret. The results demonstrate that most of the submitted deep learning methods, trained on the provided training dataset, achieved reliable segmentation performance. Dice scores reached up to 0.838 $\pm$ 0.066 and 0.716 $\pm$ 0.125 on the respective datasets, with an average performance of up to 0.804 $\pm$ 0.15.

Phenotyping consists in applying algorithms to identify individuals associated with a specific, potentially complex, trait or condition, typically out of a collection of Electronic Health Records (EHRs). Because a lot of the clinical information of EHRs are lying in texts, phenotyping from text takes an important role in studies that rely on the secondary use of EHRs. However, the heterogeneity and highly specialized aspect of both the content and form of clinical texts makes this task particularly tedious, and is the source of time and cost constraints in observational studies. To facilitate the development, evaluation and reproductibility of phenotyping pipelines, we developed an open-source Python library named medkit. It enables composing data processing pipelines made of easy-to-reuse software bricks, named medkit operations. In addition to the core of the library, we share the operations and pipelines we already developed and invite the phenotyping community for their reuse and enrichment. medkit is available at https://github.com/medkit-lib/medkit

Experimental continuation encompasses a set of methods that combine control and continuation to obtain the full bifurcation diagram of a nonlinear system experimentally, including responses that would be unstable in the system without feedback control. Such control-based methods thus allow the experimenter to directly and exhaustively explore the dynamics of the system without the need for a good mathematical model. The objective of this paper is twofold, namely (i) to review and present the state-of-the-art methods in a unified manner and (ii) to introduce a novel experimental derivative-free arclength continuation procedure, termed arclength control-based continuation. These methods are also demonstrated and compared on an electronic Duffing oscillator and a clamped thin plate featuring geometrical nonlinearity. Finally, the current state of the art is reflected upon, and the challenges lying ahead for this growing field are discussed.

We show that causal contextuality scenarios are equivalent to a subset of the formerly published spacetime games, which generalize game theory to decisions arbitrarily located in Minkowski spacetime. This insight leads to certain constructs and proofs being shorter, simpler, and more intuitive when expressed in the spacetime game framework than in the causal contextuality scenario framework. This shows that the insights of both frameworks taken together can contribute positively to advancing the field of quantum foundations., Comment: Updated technical report, 32 pages. Talk given at the IQSA 2024 conference in Brussels

We conceptualize in the paper the linear degenerate symplectic flag varieties as symmetric degenerations within the framework of type $A$ equioriented quivers. First, in the larger context of symmetric degenerations, we give a self-contained proof of the equivalence of different degeneration orders. Furthermore, we investigate the PBW locus: geometric properties of the degenerate varieties in this locus are proved by realizing them from different perspectives., Comment: 23 pages

Cerulli Irelli and Lanini have shown that PBW degenerations of flag varieties in type A and C are actually Schubert varieties of higher rank. We introduce Dynkin cones to parameterise specific abelianisations of classical Lie algebras. Within this framework, we generalise their result to all degenerations of flag varieties defined by degree vectors originating from a Dynkin cone. This framework allows us to determine the extent to which a flag variety can be degenerate while still naturally being a Schubert variety of the same Lie type. Furthermore, we compute the defining relations for the corresponding degenerate simple modules in all classical types., Comment: 27 pages, this is a preliminary version

The performance of Large Language Models (LLMs) relies heavily on the quality of prompts, which are often manually engineered and task-specific, making them costly and non-scalable. We propose a novel approach, Supervisory Prompt Training (SPT). SPT automates the generation of highly effective prompts using a dual LLM system. In this system, one LLM, the generator, performs a task while the other, the corrector, provides feedback and generates improved prompts. In contrast to earlier techniques, both the generator and corrector collaboratively and continuously improve their prompts over time. We also introduce the concept of \textit{impact scores} to measure the sentence-level effectiveness of the prompts. Our method was tested on four benchmarks, testing the level of hallucinations in LLMs. Notably, we were able to increase the accuracy of GPT-4 on GSM8K from 65.8\% to 94.1\% (28.3\% increase). SPT advances LLMs by refining prompts to enhance performance and reduce hallucinations, offering an efficient and scalable alternative to traditional model fine-tuning.

We present a corpus of 100 documents, OBSINFOX, selected from 17 sources of French press considered unreliable by expert agencies, annotated using 11 labels by 8 annotators. By collecting more labels than usual, by more annotators than is typically done, we can identify features that humans consider as characteristic of fake news, and compare them to the predictions of automated classifiers. We present a topic and genre analysis using Gate Cloud, indicative of the prevalence of satire-like text in the corpus. We then use the subjectivity analyzer VAGO, and a neural version of it, to clarify the link between ascriptions of the label Subjective and ascriptions of the label Fake News. The annotated dataset is available online at the following url: https://github.com/obs-info/obsinfox Keywords: Fake News, Multi-Labels, Subjectivity, Vagueness, Detail, Opinion, Exaggeration, French Press, Comment: Paper to appear in the Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

In this survey, we present a detailed guide on using the computer algebra system OSCAR to compute monomial bases for simple, finite-dimensional modules of simple, complex Lie algebras. We will also demonstrate how to determine monomial bases for the homogeneous coordinate ring of a (partial) flag variety, depending on a chosen birational sequence and a monomial order. This survey will be updated to reflect any advancements in OSCAR's capabilities in these areas., Comment: 16 pages, Submitted as a bookchapter for the upcoming Oscar book

First, two examples of 1D distributed port-Hamiltonian systems with dissipation, given in explicit (descriptor) form, are considered: the Dzekster model for the seepage of underground water and a nanorod model with non-local viscous damping. Implicit representations in Stokes-Lagrange subspaces are formulated. These formulations lead to modified Hamiltonian functions with spatial differential operators. The associated power balance equations are derived, together with the new boundary ports. Second, the port-Hamiltonian formulations for the Timoshenko and the Euler-Bernoulli beams are recalled, the latter being a flow-constrained version of the former. Implicit representations of these models in Stokes-Lagrange subspaces and corresponding power balance equations are derived. Bijective transformations are proposed between the explicit and implicit representations. It is proven these transformations commute with the flow-constraint projection operator., Comment: 14 pages, submitted to MTNS 2024

In this paper, we establish that FFLV polytopes, which describe monomial bases compatible with the PBW filtration on finite-dimensional simple modules for $\lie{sl}_n$ and $\lie{sp}_n$, are actually string polytopes as described by Littelmann and Berenstein-Zelevinsky for Demazure modules of higher-rank Lie algebras.

We explicitly provide minimal Gr\"obner bases for simple, finite-dimensional modules of complex Lie algebras of types A and C, using a homogeneous ordering that is compatible with the PBW filtration on the universal enveloping algebras., Comment: 6 pages, clearified the monomial ordering

The release of large datasets and developments in AI have led to dramatic improvements in decoding methods that reconstruct seen images from human brain activity. We evaluate the prospect of further improving recent decoding methods by optimizing for consistency between reconstructions and brain activity during inference. We sample seed reconstructions from a base decoding method, then iteratively refine these reconstructions using a brain-optimized encoding model that maps images to brain activity. At each iteration, we sample a small library of images from an image distribution (a diffusion model) conditioned on a seed reconstruction from the previous iteration. We select those that best approximate the measured brain activity when passed through our encoding model, and use these images for structural guidance during the generation of the small library in the next iteration. We reduce the stochasticity of the image distribution at each iteration, and stop when a criterion on the "width" of the image distribution is met. We show that when this process is applied to recent decoding methods, it outperforms the base decoding method as measured by human raters, a variety of image feature metrics, and alignment to brain activity. These results demonstrate that reconstruction quality can be significantly improved by explicitly aligning decoding distributions to brain activity distributions, even when the seed reconstruction is output from a state-of-the-art decoding algorithm. Interestingly, the rate of refinement varies systematically across visual cortex, with earlier visual areas generally converging more slowly and preferring narrower image distributions, relative to higher-level brain areas. Brain-optimized inference thus offers a succinct and novel method for improving reconstructions and exploring the diversity of representations across visual brain areas., Comment: 7 pages, 8 figures, submitted to the 2023 AAAI Workshop on Brain Encoding and Decoding. arXiv admin note: text overlap with arXiv:2306.00927

Deep learning has made significant advances in creating efficient representations of time series data by automatically identifying complex patterns. However, these approaches lack interpretability, as the time series is transformed into a latent vector that is not easily interpretable. On the other hand, Symbolic Aggregate approximation (SAX) methods allow the creation of symbolic representations that can be interpreted but do not capture complex patterns effectively. In this work, we propose a set of requirements for a neural representation of univariate time series to be interpretable. We propose a new unsupervised neural architecture that meets these requirements. The proposed model produces consistent, discrete, interpretable, and visualizable representations. The model is learned independently of any downstream tasks in an unsupervised setting to ensure robustness. As a demonstration of the effectiveness of the proposed model, we propose experiments on classification tasks using UCR archive datasets. The obtained results are extensively compared to other interpretable models and state-of-the-art neural representation learning models. The experiments show that the proposed model yields, on average better results than other interpretable approaches on multiple datasets. We also present qualitative experiments to asses the interpretability of the approach., Comment: International Conference on Data Science and Advanced Analytics (DSAA) 2023

We present a hybrid approach to the automated measurement of vagueness and subjectivity in texts. We first introduce the expert system VAGO, we illustrate it on a small benchmark of fact vs. opinion sentences, and then test it on the larger French press corpus FreSaDa to confirm the higher prevalence of subjective markers in satirical vs. regular texts. We then build a neural clone of VAGO, based on a BERT-like architecture, trained on the symbolic VAGO scores obtained on FreSaDa. Using explainability tools (LIME), we show the interest of this neural version for the enrichment of the lexicons of the symbolic version, and for the production of versions in other languages., Comment: Paper to appear in the Proceedings of the 2023 IEEE International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT)

The order and chain polytopes, introduced by Richard P. Stanley, form a pair of Ehrhart equivalent polytopes associated to a given finite poset. A conjecture by Takayuki Hibi and Nan Li states that the $f$-vector of the chain polytope dominates the $f$-vector of the order polytope. In this paper we prove a stronger form of that conjecture for a special class of posets. More precisely, we show that the $f$-vectors increase monotonically over an admissible family of chain-order polytopes for such posets., Comment: 16 pages

Single-cell technologies offer insights into molecular feature distributions, but comparing them poses challenges. We propose a kernel-testing framework for non-linear cell-wise distribution comparison, analyzing gene expression and epigenomic modifications. Our method allows feature-wise and global transcriptome/epigenome comparisons, revealing cell population heterogeneities. Using a classifier based on embedding variability, we identify transitions in cell states, overcoming limitations of traditional single-cell analysis. Applied to single-cell ChIP-Seq data, our approach identifies untreated breast cancer cells with an epigenomic profile resembling persister cells. This demonstrates the effectiveness of kernel testing in uncovering subtle population variations that might be missed by other methods.