Search

ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged. We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.

Though large language models (LLMs) have demonstrated exceptional performance across numerous problems, their application to predictive tasks in relational databases remains largely unexplored. In this work, we address the notion that LLMs cannot yield satisfactory results on relational databases due to their interconnected tables, complex relationships, and heterogeneous data types. Using the recently introduced RelBench benchmark, we demonstrate that even a straightforward application of LLMs achieves competitive performance on these tasks. These findings establish LLMs as a promising new baseline for ML on relational databases and encourage further research in this direction.

We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure. The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.

The vast portion of workloads employing LLMs involves answering questions grounded on PDF or scan content. We introduce the Arctic-TILT achieving accuracy on par with models 1000$\times$ its size on these use cases. It can be fine-tuned and deployed on a single 24GB GPU, lowering operational costs while processing Visually Rich Documents with up to 400k tokens. The model establishes state-of-the-art results on seven diverse Document Understanding benchmarks, as well as provides reliable confidence scores and quick inference, which are essential for processing files in large-scale or time-sensitive enterprise environments.

We perform a missing, reproducible evaluation of all publicly available GPT-4 family models concerning the Document Understanding field, where it is frequently required to comprehend text spacial arrangement and visual clues in addition to textual semantics. Benchmark results indicate that though it is hard to achieve satisfactory results with text-only models, GPT-4 Vision Turbo performs well when one provides both text recognized by an external OCR engine and document images on the input. Evaluation is followed by analyses that suggest possible contamination of textual GPT-4 models and indicate the significant performance drop for lengthy documents.

We call on the Document AI (DocAI) community to reevaluate current methodologies and embrace the challenge of creating more practically-oriented benchmarks. Document Understanding Dataset and Evaluation (DUDE) seeks to remediate the halted research progress in understanding visually-rich documents (VRDs). We present a new dataset with novelties related to types of questions, answers, and document layouts based on multi-industry, multi-domain, and multi-page VRDs of various origins, and dates. Moreover, we are pushing the boundaries of current methods by creating multi-task and multi-domain evaluation setups that more accurately simulate real-world situations where powerful generalization and adaptation under low-resource settings are desired. DUDE aims to set a new standard as a more practical, long-standing benchmark for the community, and we hope that it will lead to future extensions and contributions that address real-world challenges. Finally, our work illustrates the importance of finding more efficient ways to model language, images, and layout in DocAI., Comment: Accepted at ICCV 2023

The output structure of database-like tables, consisting of values structured in horizontal rows and vertical columns identifiable by name, can cover a wide range of NLP tasks. Following this constatation, we propose a framework for text-to-table neural models applicable to problems such as extraction of line items, joint entity and relation extraction, or knowledge base population. The permutation-based decoder of our proposal is a generalized sequential method that comprehends information from all cells in the table. The training maximizes the expected log-likelihood for a table's content across all random permutations of the factorization order. During the content inference, we exploit the model's ability to generate cells in any order by searching over possible orderings to maximize the model's confidence and avoid substantial error accumulation, which other sequential models are prone to. Experiments demonstrate a high practical value of the framework, which establishes state-of-the-art results on several challenging datasets, outperforming previous solutions by up to 15%.

We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model., Comment: Accepted at ICDAR 2021

This paper investigates various Transformer architectures on the WikiReading Information Extraction and Machine Reading Comprehension dataset. The proposed dual-source model outperforms the current state-of-the-art by a large margin. Next, we introduce WikiReading Recycled-a newly developed public dataset and the task of multiple property extraction. It uses the same data as WikiReading but does not inherit its predecessor's identified disadvantages. In addition, we provide a human-annotated test set with diagnostic subsets for a detailed analysis of model performance., Comment: Accepted at CoNLL 2020; this article supersedes arXiv: 2006.08281

The paper presents a novel method of finding a fragment in a long temporal sequence similar to the set of shorter sequences. We are the first to propose an algorithm for such a search that does not rely on computing the average sequence from query examples. Instead, we use query examples as is, utilizing all of them simultaneously. The introduced method based on the Dynamic Time Warping (DTW) technique is suited explicitly for few-shot query-by-example retrieval tasks. We evaluate it on two different few-shot problems from the field of Natural Language Processing. The results show it either outperforms baselines and previous approaches or achieves comparable results when a low number of examples is available.

We propose a differentiable successive halving method of relaxing the top-k operator, rendering gradient-based optimization possible. The need to perform softmax iteratively on the entire vector of scores is avoided by using a tournament-style selection. As a result, a much better approximation of top-k with lower computational cost is achieved compared to the previous approach., Comment: Work in progress

We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations during the training process, thus focusing on the task-specific parts of an input. A reduction of quadratic time and memory complexity to sublinear was achieved due to a robust trainable top-$k$ operator. Our experiments on a challenging long document summarization task show that even our simple baseline performs comparably to the current SOTA, and with trainable pooling, we can retain its top quality, while being $1.8\times$ faster during training, $4.5\times$ faster during inference, and up to $13\times$ more computationally efficient in the decoder., Comment: Accepted at ACL 2022

In this paper, we investigate the Dual-source Transformer architecture on the WikiReading information extraction and machine reading comprehension dataset. The proposed model outperforms the current state-of-the-art by a large margin. Next, we introduce WikiReading Recycled - a newly developed public dataset, supporting the task of multiple property extraction. It keeps the spirit of the original WikiReading but does not inherit the identified disadvantages of its predecessor., Comment: 5 pages

This paper presents the winning system for the propaganda Technique Classification (TC) task and the second-placed system for the propaganda Span Identification (SI) task. The purpose of TC task was to identify an applied propaganda technique given propaganda text fragment. The goal of SI task was to find specific text fragments which contain at least one propaganda technique. Both of the developed solutions used semi-supervised learning technique of self-training. Interestingly, although CRF is barely used with transformer-based language models, the SI task was approached with RoBERTa-CRF architecture. An ensemble of RoBERTa-based models was proposed for the TC task, with one of them making use of Span CLS layers we introduce in the present paper. In addition to describing the submitted systems, an impact of architectural decisions and training schemes is investigated along with remarks regarding training models of the same or better quality with lower computational budget. Finally, the results of error analysis are presented.

We propose a new shared task of semantic retrieval from legal texts, in which a so-called contract discovery is to be performed, where legal clauses are extracted from documents, given a few examples of similar clauses from other legal acts. The task differs substantially from conventional NLI and shared tasks on legal information extraction (e.g., one has to identify text span instead of a single document, page, or paragraph). The specification of the proposed task is followed by an evaluation of multiple solutions within the unified framework proposed for this branch of methods. It is shown that state-of-the-art pretrained encoders fail to provide satisfactory results on the task proposed. In contrast, Language Model-based solutions perform better, especially when unsupervised fine-tuning is applied. Besides the ablation studies, we addressed questions regarding detection accuracy for relevant text fragments depending on the number of examples available. In addition to the dataset and reference results, LMs specialized in the legal domain were made publicly available., Comment: Submitted to Findings of EMNLP

We propose an algorithm for learning the Horn envelope of an arbitrary domain using an expert, or an oracle, capable of answering certain types of queries about this domain. Attribute exploration from formal concept analysis is a procedure that solves this problem, but the number of queries it may ask is exponential in the size of the resulting Horn formula in the worst case. We recall a well-known polynomial-time algorithm for learning Horn formulas with membership and equivalence queries and modify it to obtain a polynomial-time probably approximately correct algorithm for learning the Horn envelope of an arbitrary domain., Comment: 21 pages, 1 figure

We develop an effective surface theory for the surface states of a Weyl semimetal. This theory includes the peculiar Fermi arc states on the surface as well as leakage of the states from the surface to the bulk. Subjecting the model to a magnetic field perpendicular to the surface results in quantum oscillations. The oscillations are different from the usual ones since they do not involve a closed Fermi surface cross section. It has been shown previously that the Quantum oscillations can be understood semiclassically as resulting from motion of electrons on the surface Fermi arcs as well as tunneling through chiral Landau levels associated with the bulk. In this work we develop an effective surface theory and use it to analyze the quantum oscillation in the semiclassical regime and beyond. Specifically, we show that when a pair of Weyl points are close to each other the surface quantum oscillations acquire a phase offset which originates from the bulk. While the surface states are responsible for a large part of the electron motion, tunneling through the bulk is necessary for completing the orbit. This tunneling makes use of the bulk, zero energy, chiral Landau level in each Weyl node. When the nodes are close in momentum space their chiral levels overlap and a gap at zero energy is formed. This gap causes the phase offset in the surface quantum oscillations., Comment: 10 pages, 5 figures

We revisit the notion of probably approximately correct implication bases from the literature and present a first formulation in the language of formal concept analysis, with the goal to investigate whether such bases represent a suitable substitute for exact implication bases in practical use-cases. To this end, we quantitatively examine the behavior of probably approximately correct implication bases on artificial and real-world data sets and compare their precision and recall with respect to their corresponding exact implication bases. Using a small example, we also provide qualitative insight that implications from probably approximately correct bases can still represent meaningful knowledge from a given data set., Comment: 17 pages, 8 figures; typos added, corrected x-label on graphs

We study a model of two dimensional, topological superconductivity on a square lattice. The model contains hopping, spin orbit coupling and a time reversal symmetry breaking Zeeman term. This term, together with the chemical potential act as knobs that induce transitions between trivial and topological superconductivity. As previously found numerically, the transitions are seen in the entanglement entropy as cusps as a function of model parameters. In this work we study the entanglement entropy analytically by keeping only its most important components. Our study is based on the intuition that the number of Fermi surfaces in the system controls the topological invariant. With our approximate expression for the entanglement entropy we are able to extract the divergent entanglement entropy derivative close to the phase transition., Comment: 6 pages, 3 figures

Attribute exploration has been investigated in several studies, with particular emphasis on the algorithmic aspects of this knowledge acquisition method. In its basic version the method itself is rather simple and transparent. But when background knowledge and partially described counter-examples are admitted, it gets more difficult. Here we discuss this case in an abstract, somewhat "axiomatic" setting, providing a terminology that clarifies the abstract strategy of the method rather than its algorithmic implementation.

In this paper we study the phase diagram of a disordered, spin-orbit coupled superconductor with $s$-wave or $d+id$-wave pairing symmetry in symmetry class $D$. We analyze the topological phase transitions by applying three different methods which include a disorder averaged entanglement entropy, a disorder averaged real-space Chern number, and an evaluation of the momentum space Chern number in a disorder averaged effective model. We find evidence for a disorder-induced topological state. While in the clean limit there is a single phase transition from a trivial phase with a Chern number $C=4$ to a topological phase with $C=1$, in the disordered system there is an intermediate phase with $C=3$. The phase transition from the trivial $C=4$ phase into the intermediate phase with $C=3$ is seen in the real-space calculation of the Chern number. In spite of this, this phase transition is not detectable in the entanglement entropy. A second phase transition from the disorder induced $C=3$ into the $C=1$ phase is seen in all three quantities., Comment: 5 pages, 6 figures, v3:adjusted to published version

We introduce the notion of topological entropy of a formal languages as the topological entropy of the minimal topological automaton accepting it. Using a characterization of this notion in terms of approximations of the Myhill-Nerode congruence relation, we are able to compute the topological entropies of certain example languages. Those examples suggest that the notion of a "simple" formal language coincides with the language having zero entropy.

The classification of electron systems according to their topology has been at the forefront of condensed matter research in recent years. It has been found that systems of the same symmetry, previously thought of as equivalent, may in fact be distinguished by their topological properties. Moreover, the non-trivial topology found in some insulators and superconductors has profound physical implications that can be observed experimentally and can potentially be used for applications. However, characterizing a system's topology is not always a simple task, even for a theoretical model. When translation and other symmetries are present in a quadratic model the topological invariants are readily defined and easily calculated in a variety of symmetry classes. However, once interactions or disorder come into play the task becomes difficult, and in many cases prohibitively so. The goal of this paper is to suggest alternatives to the topological invariants which are based on the entanglement spectrum and entanglement entropy. Using quadratic models of superconductors we demonstrate that these entanglement properties are sensitive to changes in topology. We choose quadratic models since the topological phase diagram can be mapped using the topological invariants and then compared to the entanglement entropy/spectrum features. This work sets the stage for learning about topology in interacting and disordered systems through their entanglement properties., Comment: 12 pages, 9 figures