Search

Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is assessing the validity of the benchmarks themselves. This is most commonly done via Benchmark Agreement Testing (BAT), where new benchmarks are validated against established ones using some agreement metric (e.g., rank correlation). Despite the crucial role of BAT for benchmark builders and consumers, there are no standardized procedures for such agreement testing. This deficiency can lead to invalid conclusions, fostering mistrust in benchmarks and upending the ability to properly choose the appropriate benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how some overlooked methodological choices can significantly influence BAT results, potentially undermining the validity of conclusions. To address these inconsistencies, we propose a set of best practices for BAT and demonstrate how utilizing these methodologies greatly improves BAT robustness and validity. To foster adoption and facilitate future research,, we introduce BenchBench, a python package for BAT, and release the BenchBench-leaderboard, a meta-benchmark designed to evaluate benchmarks using their peers. Our findings underscore the necessity for standardized BAT, ensuring the robustness and validity of benchmark evaluations in the evolving landscape of language model research. BenchBench Package: https://github.com/IBM/BenchBench Leaderboard: https://huggingface.co/spaces/per/BenchBench, Comment: Under Review

Quantitative Susceptibility Mapping (QSM) is a technique for measuring magnetic susceptibility of tissues, aiding in the detection of pathologies like traumatic brain injury and multiple sclerosis by analyzing variations in substances such as iron and calcium. Despite its clinical value, achieving high-resolution QSM (voxel sizes < 1 mm3) reduces signal-to-noise ratio (SNR), compromising diagnostic quality. To mitigate this, we applied the Marchenko-Pastur Principal Component Analysis (MP-PCA) denoising technique on T2* weighted data, to enhance the quality of R2*, T2*, and QSM maps. Denoising was tested on a numerical phantom, healthy subjects, and patients with brain metastases and sickle cell disease, demonstrating effective and robust improvements across different scan settings. Further analysis examined noise propagation in R2* and T2* values, revealing lower noise-related variations in R2* values compared to T2* values which tended to be overestimated due to noise. Reduced variability was observed in QSM values post denoising, demonstrating MP-PCA's potential to improve the, Comment: 9 pages, 8 figures, 3 tables. It was accepted to be presented in a peer-reviewed annual ISMRM meeting, which will be held in Singapore in May 2024

Quantum computers are expected to revolutionize our ability to process information. The advancement from classical to quantum computing is a product of our advancement from classical to quantum physics -- the more our understanding of the universe grows, so does our ability to use it for computation. A natural question that arises is, what will physics allow in the future? Can more advanced theories of physics increase our computational power, beyond quantum computing? An active field of research in physics studies theoretical phenomena outside the scope of explainable quantum mechanics, that form when attempting to combine Quantum Mechanics (QM) with General Relativity (GR) into a unified theory of Quantum Gravity (QG). QG is known to present the possibility of a quantum superposition of causal structure and event orderings. In the literature of quantum information theory, this translates to a superposition of unitary evolution orders. In this work we show a first example of a natural computational model based on QG, that provides an exponential speedup over standard quantum computation (under standard hardness assumptions). We define a model and complexity measure for a quantum computer that has the ability to generate a superposition of unitary evolution orders, and show that such computer is able to solve in polynomial time two of the fundamental problems in computer science: The Graph Isomorphism Problem ($\mathsf{GI}$) and the Gap Closest Vector Problem ($\mathsf{GapCVP}$), with gap $O\left( n \sqrt{n} \right)$. These problems are believed by experts to be hard to solve for a regular quantum computer. Interestingly, our model does not seem overpowered, and we found no obvious way to solve entire complexity classes that are considered hard in computer science, like the classes $\mathbf{NP}$ and $\mathbf{SZK}$.

In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt!, Comment: Submitted to NAACL demo track

The increasing versatility of language models (LMs) has given rise to a new class of benchmarks that comprehensively assess a broad range of capabilities. Such benchmarks are associated with massive computational costs, extending to thousands of GPU hours per model. However, the efficiency aspect of these evaluation efforts had raised little discussion in the literature. In this work, we present the problem of Efficient Benchmarking, namely, intelligently reducing the computation costs of LM evaluation without compromising reliability. Using the HELM benchmark as a test case, we investigate how different benchmark design choices affect the computation-reliability trade-off. We propose to evaluate the reliability of such decisions, by using a new measure -- Decision Impact on Reliability, DIoR for short. We find, for example, that a benchmark leader may change by merely removing a low-ranked model from the benchmark, and observe that a correct benchmark ranking can be obtained by considering only a fraction of the evaluation examples. Based on our findings, we outline a set of concrete recommendations for efficient benchmark design and utilization practices. To take a step further, we use our findings to propose an evaluation algorithm, that, when applied to the HELM benchmark, leads to dramatic cost savings with minimal loss of benchmark reliability, often reducing computation by x100 or more., Comment: Accepted to NAACL main track

Purpose: Disease progression of retinal atrophy associated with AMD requires the accurate quantification of the retinal atrophy changes on longitudinal OCT studies. It is based on finding, comparing, and delineating subtle atrophy changes on consecutive pairs (prior and current) of unregistered OCT scans. Methods: We present a fully automatic end-to-end pipeline for the simultaneous detection and quantification of time-related atrophy changes associated with dry AMD in pairs of OCT scans of a patient. It uses a novel simultaneous multi-channel column-based deep learning model trained on registered pairs of OCT scans that concurrently detects and segments retinal atrophy segments in consecutive OCT scans by classifying light scattering patterns in matched pairs of vertical pixel-wide columns (A-scans) in registered prior and current OCT slices (B-scans). Results: Experimental results on 4,040 OCT slices with 5.2M columns from 40 scans pairs of 18 patients (66% training/validation, 33% testing) with 24.13+-14.0 months apart in which Complete RPE and Outer Retinal Atrophy (cRORA) was identified in 1,998 OCT slices (735 atrophy lesions from 3,732 segments, 0.45M columns) yield a mean atrophy segments detection precision, recall of 0.90+-0.09, 0.95+-0.06 and 0.74+-0.18, 0.94+-0.12 for atrophy lesions with AUC=0.897, all above observer variability. Simultaneous classification outperforms standalone classification precision and recall by 30+-62% and 27+-0% for atrophy segments and lesions. Conclusions: simultaneous column-based detection and quantification of retinal atrophy changes associated with AMD is accurate and outperforms standalone classification methods. Translational relevance: an automatic and efficient way to detect and quantify retinal atrophy changes associated with AMD., Comment: 34 pages, 5 figures and 2 tables

This article provides recommendations for implementing quantitative susceptibility mapping (QSM) for clinical brain research. It is a consensus of the ISMRM Electro-Magnetic Tissue Properties Study Group. While QSM technical development continues to advance rapidly, the current QSM methods have been demonstrated to be repeatable and reproducible for generating quantitative tissue magnetic susceptibility maps in the brain. However, the many QSM approaches available give rise to the need in the neuroimaging community for guidelines on implementation. This article describes relevant considerations and provides specific implementation recommendations for all steps in QSM data acquisition, processing, analysis, and presentation in scientific publications. We recommend that data be acquired using a monopolar 3D multi-echo GRE sequence, that phase images be saved and exported in DICOM format and unwrapped using an exact unwrapping approach. Multi-echo images should be combined before background removal, and a brain mask created using a brain extraction tool with the incorporation of phase-quality-based masking. Background fields should be removed within the brain mask using a technique based on SHARP or PDF, and the optimization approach to dipole inversion should be employed with a sparsity-based regularization. Susceptibility values should be measured relative to a specified reference, including the common reference region of whole brain as a region of interest in the analysis, and QSM results should be reported with - as a minimum - the acquisition and processing specifications listed in the last section of the article. These recommendations should facilitate clinical QSM research and lead to increased harmonization in data acquisition, analysis, and reporting., Comment: Authors are listed in alphabetical order. * indicates corresponding authors

The field of Natural Language Generation (NLG) suffers from a severe shortage of labeled data due to the extremely expensive and time-consuming process involved in manual annotation. A natural approach for coping with this problem is active learning (AL), a well-known machine learning technique for improving annotation efficiency by selectively choosing the most informative examples to label. However, while AL has been well-researched in the context of text classification, its application to NLG remains largely unexplored. In this paper, we present a first systematic study of active learning for NLG, considering a diverse set of tasks and multiple leading selection strategies, and harnessing a strong instruction-tuned model. Our results indicate that the performance of existing AL strategies is inconsistent, surpassing the baseline of random example selection in some cases but not in others. We highlight some notable differences between the classification and generation scenarios, and analyze the selection behaviors of existing AL strategies. Our findings motivate exploring novel approaches for applying AL to generation tasks., Comment: Accepted to EMNLP2023 as a long paper

Massive neutrinos modify the expansion history of the universe and suppress the structure formation below their free streaming scale. Cosmic microwave background (CMB) observations at small angular scales can be used to constrain the total mass $\Sigma m_\nu$ of the three neutrino flavors. However, at these scales, the CMB-measured $\Sigma m_\nu$ is degenerate with $\tau$, the optical depth to reionization, which quantifies the damping of CMB anisotropies due to the scattering of CMB photons with free electrons along the line of sight. Here we revisit the idea to use 21-cm power spectrum observations to provide direct estimates for $\tau$. A joint analysis of CMB and 21-cm data can alleviate the $\tau-\Sigma m_\nu$ degeneracy, making it possible to measure $\Sigma m_\nu$ with unprecedented precision. Forecasting for the upcoming Hydrogen Epoch of Reionization Array (HERA), we find that a $\lesssim\mathcal{O}(10\%)$ measurement of $\tau$ is achievable, which would enable a $\gtrsim 5\sigma$ measurement of $\Sigma m_\nu=60\,[{\rm meV}]$, for any astrophysics model that we considered. Precise estimates of $\tau$ also help reduce uncertainties in other cosmological parameters, such as $A_s$, the amplitude of the primordial scalar fluctuations power spectrum., Comment: 17 pages, 4 figures, 9 tables

Algorithms used by organizations increasingly wield power in society as they decide the allocation of key resources and basic goods. In order to promote fairer, juster, and more transparent uses of such decision-making power, explainable artificial intelligence (XAI) aims to provide insights into the logic of algorithmic decision-making. Despite much research on the topic, consumer-facing applications of XAI remain rare. A central reason may be that a viable platform-based monetization strategy for this new technology has yet to be found. We introduce and describe a novel monetization strategy for fusing algorithmic explanations with programmatic advertising via an explanation platform. We claim the explanation platform represents a new, socially-impactful, and profitable form of human-algorithm interaction and estimate its potential for revenue generation in the high-risk domains of finance, hiring, and education. We then consider possible undesirable and unintended effects of monetizing XAI and simulate these scenarios using real-world credit lending data. Ultimately, we argue that monetizing XAI may be a double-edged sword: while monetization may incentivize industry adoption of XAI in a variety of consumer applications, it may also conflict with the original legal and ethical justifications for developing XAI. We conclude by discussing whether there may be ways to responsibly and democratically harness the potential of monetized XAI to provide greater consumer access to algorithmic explanations.

We study the random \'etale algebra generated by a random polynomial with i.i.d. coefficients distributed according to Haar measure normalized on $\mathbb{Z}_p$. We determine the probability that this random algebra is an unramified field, explicitly. In addition, we prove a private case of a conjecture made by Bhargava, Cremona, Fisher, and Gajovi\'c. More precisely, we show that this probability is a rational function of $p$ that is invariant under replacing $p$ by $1/p$.

We present nBIIG, a neural Business Intelligence (BI) Insights Generation system. Given a table, our system applies various analyses to create corresponding RDF representations, and then uses a neural model to generate fluent textual insights out of these representations. The generated insights can be used by an analyst, via a human-in-the-loop paradigm, to enhance the task of creating compelling table reports. The underlying generative neural model is trained over large and carefully distilled data, curated from multiple BI domains. Thus, the system can generate faithful and fluent insights over open-domain tables, making it practical and useful., Comment: Accepted to AAAI-23

In response to growing recognition of the social impact of new AI-based technologies, major AI and ML conferences and journals now encourage or require papers to include ethics impact statements and undergo ethics reviews. This move has sparked heated debate concerning the role of ethics in AI research, at times devolving into name-calling and threats of "cancellation." We diagnose this conflict as one between atomist and holist ideologies. Among other things, atomists believe facts are and should be kept separate from values, while holists believe facts and values are and should be inextricable from one another. With the goal of reducing disciplinary polarization, we draw on numerous philosophical and historical sources to describe each ideology's core beliefs and assumptions. Finally, we call on atomists and holists within the ever-expanding data science community to exhibit greater empathy during ethical disagreements and propose four targeted strategies to ensure AI research benefits society., Comment: 9 pages, 1 figure, 2 tables. To be published in Patterns by Cell Press

Generating natural language statements to convey logical inferences from tabular data (i.e., Logical NLG) is a process with one input and a variety of valid outputs. This characteristic underscores the need for a method to produce a diverse set of valid outputs, presenting different perspectives of the input data. We propose a simple yet effective diversity-enhancing scheme that builds upon an inherent property of the statements, their logic-types, by using a type-controlled table-to-text generation model. We demonstrate, through extensive automatic and human evaluations over the two publicly available Logical NLG datasets, that our proposed method both facilitates the ability to effectively control the generated statement type, and produces results superior to the strongest baselines in terms of quality and factuality-diversity trade-off., Comment: 4 pages, 4 figures

We study the roots of a random polynomial over the field of p-adic numbers. For a random monic polynomial with coefficients in $\mathbb{Z}_p$, we obtain an asymptotic formula for the factorial moments of the number of roots of this polynomial. In addition, we show the probability that a random polynomial of degree $n$ has more than $\log n$ roots is $O\big(n^{-K}\big)$ for some $K > 0$.

Paraphrase generation has been widely used in various downstream tasks. Most tasks benefit mainly from high quality paraphrases, namely those that are semantically similar to, yet linguistically diverse from, the original sentence. Generating high-quality paraphrases is challenging as it becomes increasingly hard to preserve meaning as linguistic diversity increases. Recent works achieve nice results by controlling specific aspects of the paraphrase, such as its syntactic tree. However, they do not allow to directly control the quality of the generated paraphrase, and suffer from low flexibility and scalability. Here we propose $QCPG$, a quality-guided controlled paraphrase generation model, that allows directly controlling the quality dimensions. Furthermore, we suggest a method that given a sentence, identifies points in the quality control space that are expected to yield optimal generated paraphrases. We show that our method is able to generate paraphrases which maintain the original meaning while achieving higher diversity than the uncontrolled baseline. The models, the code, and the data can be found in https://github.com/IBM/quality-controlled-paraphrase-generation., Comment: Accepted as a long paper at ACL 2022