Search

After the introduction of Large Language Models (LLMs), there have been substantial improvements in the performance of Natural Language Generation (NLG) tasks, including Text Summarization and Machine Translation. However, LLMs still produce outputs containing hallucinations, that is, content not grounded in factual information. Therefore, developing methods to assess the factuality of LLMs has become urgent. Indeed, resources for factuality evaluation have recently emerged. Although challenging, these resources face one or more of the following limitations: (i) they are tailored to a specific task or domain; (ii) they are limited in size, thereby preventing the training of new factuality evaluators; (iii) they are designed for simpler verification tasks, such as claim verification. To address these issues, we introduce LLM-Oasis, to the best of our knowledge the largest resource for training end-to-end factuality evaluators. LLM-Oasis is constructed by extracting claims from Wikipedia, falsifying a subset of these claims, and generating pairs of factual and unfactual texts. We then rely on human annotators to both validate the quality of our dataset and to create a gold standard test set for benchmarking factuality evaluation systems. Our experiments demonstrate that LLM-Oasis presents a significant challenge for state-of-the-art LLMs, with GPT-4o achieving up to 60% accuracy in our proposed end-to-end factuality evaluation task, highlighting its potential to drive future research in the field., Comment: 15 pages. To be submitted to CL journal

Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics, ranking them according to their correlation with human judgments. Their results guide researchers toward enhancing the next generation of metrics and MT systems. With the recent introduction of neural metrics, the field has witnessed notable advancements. Nevertheless, the inherent opacity of these metrics has posed substantial challenges to the meta-evaluation process. This work highlights two issues with the meta-evaluation framework currently employed in WMT, and assesses their impact on the metrics rankings. To do this, we introduce the concept of sentinel metrics, which are designed explicitly to scrutinize the meta-evaluation process's accuracy, robustness, and fairness. By employing sentinel metrics, we aim to validate our findings, and shed light on and monitor the potential biases or inconsistencies in the rankings. We discover that the present meta-evaluation framework favors two categories of metrics: i) those explicitly trained to mimic human quality assessments, and ii) continuous metrics. Finally, we raise concerns regarding the evaluation capabilities of state-of-the-art metrics, emphasizing that they might be basing their assessments on spurious correlations found in their training data., Comment: Presented at ACL 2024 Main Conference. 29 pages

The Son Of X-Shooter (SOXS) is a single object spectrograph, built by an international consortium for the 3.58-m ESO New Technology Telescope at the La Silla Observatory [1]. It offers a simultaneous spectral coverage over 350-2000 nm, with two separate spectrographs. In this paper we present the status of the Near InfraRed (NIR) cryogenic echelle cross-dispersed spectrograph [1], in the range 0.80-2.00 {\mu}m with 15 orders, equipped with an 2k x 2k Hawaii H2RG IR array from Teledyne, working at 40K, that is currently assembled and tested on the SOXS instrument, in the premises of INAF in Padova. We describe the different tests and results of the cryo, vacuum, opto-mechanics and detector subsystems that finally will be part of the PAE by ESO., Comment: 8 Pages, 7 Figures, Astronomical Telescopes and Instrumentation, SPIE Proceedings 2024

Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLM-based metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization. In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our factuality annotations of long-form summarization at https://github.com/Babelscape/FENICE., Comment: ACL 2024 camera ready. Code and data at https://github.com/Babelscape/FENICE

Observatories need to measure and evaluate the scientific output and overall impact of their facilities. An observatory bibliography consists of the papers published using that observatory's data, typically gathered by searching the major journals for relevant keywords. Recently, the volume of literature and methods by which the publications pool is evaluated has increased. Efficient and standardized procedures are necessary to assign meaningful metadata; enable user-friendly retrieval; and provide the opportunity to derive reports, statistics, and visualizations to impart a deeper understanding of the research output. In 2021, a group of observatory bibliographers from around the world convened online to continue the discussions presented in Lagerstrom (2015). We worked to extract general guidelines from our experiences, techniques, and lessons learnt. The paper explores the development, application, and current status of telescope bibliographies and future trends. This paper briefly describes the methodologies employed in constructing databases, along with the various bibliometric techniques used to analyze and interpret them. We explain reasons for non-standardization and why it is essential for each observatory to identify metadata and metrics that are meaningful for them; caution the (over-)use of comparisons among facilities that are, ultimately, not comparable through bibliometrics; and highlight the benefits of telescope bibliographies, both for researchers within the astronomical community and for stakeholders beyond the specific observatories. There is tremendous diversity in the ways bibliographers track publications and maintain databases, due to parameters such as resources, type of observatory, historical practices, and reporting requirements to funders and outside agencies. However, there are also common sets of Best Practices., Comment: 19 pages, 3 appendices

This work concerns the long-term dynamics of a spatiotemporal many-body deterministic model that exhibits emergence and self-organization, and which has been recently proposed as a new paradigm for Artificial Life. Collective structures emerge in the form of dynamic networks, created by bursts of spatiotemporal activity (avalanches) at the edge of a synchronization phase transition. The spatiotemporal dynamics is portraited by a motion picture and quantified by time varying collective parameters, which revealed that the dynamic networks undergo a "life cycle", made of self-creation, self-regulation, and self-destruction. The power spectra of the collective parameters show 1/f power-law tails, and the statistical properties of the avalanches, evaluated in terms of their size and durations, show power laws with characteristic exponents in agreement with those values found in the literature concerning neural networks. The mechanism underlying avalanches is discussed in terms of local-to-collective excitability. Finally, the connections that link the present work to self-organized criticality, neural networks and artificial life are discussed.

This work concerns a many-body deterministic model that displays life-like properties as emergence, complexity, self-organization, spontaneous compartmentalization, and self-regulation. The model portraits the dynamics of an ensemble of locally coupled polar phase oscillators, moving in a two-dimensional space, that in certain conditions exhibit emergent superstructures. Those superstructures are self-organized dynamic networks, resulting from a synchronization process of many units, over length scales much greater than the interaction length. Such networks compartmentalize the two-dimensional space with no a priori constraints, due to the formation of porous transport walls, and represent a highly complex and novel non-linear behavior. The analysis is numerically carried out as a function of a control parameter showing distinct regimes: static, stable dynamic networks, intermittency, and chaos. A statistical analysis is drawn to determine the control parameter ranges for the various behaviors to appear., Comment: 14 pages, 8 figures and 4 movies

In recent years, research in text summarization has mainly focused on the news domain, where texts are typically short and have strong layout features. The task of full-book summarization presents additional challenges which are hard to tackle with current resources, due to their limited size and availability in English only. To overcome these limitations, we present "Echoes from Alexandria", or in shortened form, "Echoes", a large resource for multilingual book summarization. Echoes features three novel datasets: i) Echo-Wiki, for multilingual book summarization, ii) Echo-XSum, for extremely-compressive multilingual book summarization, and iii) Echo-FairySum, for extractive book summarization. To the best of our knowledge, Echoes, with its thousands of books and summaries, is the largest resource, and the first to be multilingual, featuring 5 languages and 25 language pairs. In addition to Echoes, we also introduce a new extractive-then-abstractive baseline, and, supported by our experimental results and manual analysis of the summaries generated, we argue that this baseline is more suitable for book summarization than purely-abstractive approaches. We release our resource and software at https://github.com/Babelscape/echoes-from-alexandria in the hope of fostering innovative research in multilingual book summarization., Comment: 9 pages, long paper at ACL 2023

One of the common traits of past and present approaches for Semantic Role Labeling (SRL) is that they rely upon discrete labels drawn from a predefined linguistic inventory to classify predicate senses and their arguments. However, we argue this need not be the case. In this paper, we present an approach that leverages Definition Modeling to introduce a generalized formulation of SRL as the task of describing predicate-argument structures using natural language definitions instead of discrete labels. Our novel formulation takes a first step towards placing interpretability and flexibility foremost, and yet our experiments and analyses on PropBank-style and FrameNet-style, dependency-based and span-based SRL also demonstrate that a flexible model with an interpretable output does not necessarily come at the expense of performance. We release our software for research purposes at https://github.com/SapienzaNLP/dsrl.

SOXS (Son Of X-Shooter) is a single object spectrograph built by an international consortium for the ESO NTT telescope. SOXS is based on the heritage of the X-Shooter at the ESO-VLT with two arms (UV-VIS and NIR) working in parallel, with a Resolution-Slit product of about 4500, capable of simultaneously observing over the entire band the complete spectral range from the U- to the H-band. SOXS will carry out rapid and long-term Target of Opportunity requests on a variety of astronomical objects. The SOXS vacuum and cryogenic control system has been designed to evacuate, cool down and maintain the UV-VIS detector and the entire NIR spectrograph to their operating temperatures. The design chosen allows the two arms to be operated independently. This paper describes the final design of the cryo-vacuum control system, its functionalities and the tests performed in the integration laboratories., Comment: 12 pages, 6 figures, SPIE proceedings of the conference Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation IV

We present statistics on the number of refereed astronomy journal articles that used data from NASA's Spitzer Space Telescope through the end of the calendar year 2020. We discuss the various types of science programs and science categories that were used to collect data during the mission and discuss how operational changes brought on by the depletion of cryogen in May 2009, including the resulting budget cuts, impacted the publication rate. The post-cryogenic (warm) mission produced fewer papers than the cryogenic mission, but the percentage of the exposure time published did not appreciably change between the warm and cryogenic missions. This was mostly because in the warm mission the length of observations increased, so that each warm paper on average uses more data than the cryogenic papers. We also discuss the speed of publication, archival usage, and the tremendous efficacy of the Legacy and Exploration Science programs (large, coherent investigations), including the value of having well-advertised enhanced data products hosted in centralized archives. We also identify the observations that have been published the largest number of times, and sort them by a variety of metrics (including program type, instrument used, and observation length). Data that have the highest reuse rates in publications were taken early in the Spitzer mission, or belong to one of the large surveys (large either in number of objects, in number of hours observed, or in area covered on the sky). We also assess how often authors have cited the Spitzer fundamental papers or have correctly referenced the Spitzer data they used, finding that as many as 40% of papers have failed to cite the papers, and 15% have made it impossible to identify the data they used., Comment: Accepted by PASP

The simultaneous detection of organic molecules of the form C$_2$H$_{\text{n}}$O, such as ketene (CH$_2$CO), acetaldehyde (CH$_3$CHO), and ethanol (CH$_3$CH$_2$OH), toward early star-forming regions offers hints of shared chemical history. Several reaction routes have been proposed and experimentally verified under various interstellar conditions to explain the formation pathways involved. Most noticeably, the non-energetic processing of C$_2$H$_2$ ice with OH-radicals and H-atoms was shown to provide formation routes to ketene, acetaldehyde, ethanol, and vinyl alcohol (CH$_2$CHOH) along the H$_2$O formation sequence on grain surfaces. In this work, the non-energetic formation scheme is extended with laboratory measurements focusing on the energetic counterpart, induced by cosmic rays penetrating the H$_2$O-rich ice mantle. The focus here is on the H$^+$ radiolysis of interstellar C$_2$H$_2$:H$_2$O ice analogs at 17 K. Ultra-high vacuum experiments were performed to investigate the 200 keV H$^+$ radiolysis chemistry of predeposited C$_2$H$_2$:H$_2$O ices, both as mixed and layered geometries. Fourier-transform infrared spectroscopy was used to monitor in situ newly formed species as a function of the accumulated energy dose (or H$^+$ fluence). The infrared (IR) spectral assignments are further confirmed in isotope labeling experiments using H$_2$$^{18}$O. The energetic processing of C$_2$H$_2$:H$_2$O ice not only results in the formation of (semi-) saturated hydrocarbons (C$_2$H$_4$ and C$_2$H$_6$) and polyynes as well as cumulenes (C$_4$H$_2$ and C$_4$H$_4$), but it also efficiently forms O-bearing COMs, including vinyl alcohol, ketene, acetaldehyde, and ethanol, for which the reaction cross-section and product composition are derived. A clear composition transition of the product, from H-poor to H-rich species, is observed as a function of the accumulated energy dose., Comment: 14 pages, 7 figures, 2 tables

This work deals with a mesoscopic and deterministic theory for the collective dynamics of Water Coherence Domains, and it represents a continuation of a previous work published in Plos ONE https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0188753 This approach qualitatively reproduces some features of the experimental phenomenology, and it could sustain an evolutionary theory based on layered synchronization processes. Its relevance for the description of the processes behind the emergence of life is discussed., Comment: The manuscrips was rejected, and we rethought the whole work in another way

Context: Reliable, directly measured optical properties of astrophysical ice analogs in the infrared (IR) and terahertz (THz) range are missing. These parameters are of great importance to model the dust continuum radiative transfer in dense and cold regions, here thick ice mantles are present, and are necessary for the interpretation of future observations planned in the far-IR region. Aims: Coherent THz radiation allows direct measurement of the complex dielectric function (refractive index) of astrophysically relevant ice species in the THz range. Methods: The time-domain waveforms and the frequency-domain spectra of reference samples of CO ice, deposited at a temperature of 28.5 K and annealed to 33 K at different thicknesses, have been recorded. A new algorithm is developed to reconstruct the real and imaginary parts of the refractive index from the time-domain THz data. Results: The complex refractive index in the wavelength range of 1 mm - 150 ${\mu}$m (0.3 - 2.0 THz) has been determined for the studied ice samples, and compared with available data found in the literature. Conclusions: The developed algorithm of reconstructing the real and imaginary parts of the refractive index from the time-domain THz data enables, for the first time, the determination of optical properties of astrophysical ice analogs without using the Kramers-Kronig relations. The obtained data provide a benchmark to interpret the observational data from current ground based facilities as well as future space telescope missions, and have been used to estimate the opacities of the dust grains in presence of CO ice mantles.

We present a bibliographic analysis of Chandra, Hubble, and Spitzer publications. We find (a) archival data are used in >60% of the publication output and (b) archives for these missions enable a much broader set of institutions and countries to scientifically use data from these missions. Specifically, we find that authors from institutions that have published few papers from a given mission publish 2/3 archival publications, while those with many publications typically have 1/3 archival publications. We also show that countries with lower GDP per capita overwhelmingly produce archival publications, while countries with higher GDP per capital produce guest observer and archival publications in equal amounts. We argue that robust archives are thus not only critical for the scientific productivity of mission data, but also the scientific accessibility of mission data. We argue that the astronomical community should support archives to maximize the overall scientific societal impact of astronomy, and represent an excellent investment in astronomy's future., Comment: White Paper submitted to the NAS call for Astro2020 Decadal Survey APC papers

We demonstrate attacks on the boot ROMs of the Nintendo 3DS in order to exfiltrate secret information from normally protected areas of memory and gain persistent early code execution on devices which have not previously been compromised. The attack utilizes flaws in the RSA signature verification implementation of one of the boot ROMs in order to overflow ASN.1 length fields and cause invalid firmware images to appear valid to the signature parser. This is then used to load a custom firmware image which overwrites the data-abort vector with a custom data abort handler, then induces a data-abort exception in order to reliably redirect boot ROM code flow at boot time. This executes a payload which, due to its reliable early execution by a privileged processor, is able to function as a persistent exploit of the system in order to exfiltrate secret information (such as encryption keys) from normally protected areas of memory., Comment: 5 pages, 2 figures; corrected typos, corrected authorship error