Search

Human evaluation is a critical component in machine translation system development and has received much attention in text translation research. However, little prior work exists on the topic of human evaluation for speech translation, which adds additional challenges such as noisy data and segmentation mismatches. We take first steps to fill this gap by conducting a comprehensive human evaluation of the results of several shared tasks from the last International Workshop on Spoken Language Translation (IWSLT 2023). We propose an effective evaluation strategy based on automatic resegmentation and direct assessment with segment context. Our analysis revealed that: 1) the proposed evaluation strategy is robust and scores well-correlated with other types of human judgements; 2) automatic metrics are usually, but not always, well-correlated with direct assessment scores; and 3) COMET as a slightly stronger automatic metric than chrF, despite the segmentation noise introduced by the resegmentation step systems. We release the collected human-annotated data in order to encourage further investigation., Comment: LREC-COLING2024 publication (with corrections for Table 3)

Many NLP tasks require to automatically identify the most significant words in a text. In this work, we derive word significance from models trained to solve semantic task: Natural Language Inference and Paraphrase Identification. Using an attribution method aimed to explain the predictions of these models, we derive importance scores for each input token. We evaluate their relevance using a so-called cross-task evaluation: Analyzing the performance of one model on an input masked according to the other model's weight, we show that our method is robust with respect to the choice of the initial task. Additionally, we investigate the scores from the syntax point of view and observe interesting patterns, e.g. words closer to the root of a syntactic tree receive higher importance scores. Altogether, these observations suggest that our method can be used to identify important words in sentences without any explicit word importance labeling in training., Comment: Published in the Findings of ACL 2023

We present a novel approach to generating scripts by using agents with different personality types. To manage character interaction in the script, we employ simulated dramatic networks. Automatic and human evaluation on multiple criteria shows that our approach outperforms a vanilla-GPT2-based baseline. We further introduce a new metric to evaluate dialogue consistency based on natural language inference and demonstrate its validity., Comment: Non-archival paper at the 4th Workshop on Narrative Understanding (WNU 2022)

In simultaneous speech translation, one can vary the size of the output window, system latency and sometimes the allowed level of rewriting. The effect of these properties on readability and comprehensibility has not been tested with modern neural translation systems. In this work, we propose an evaluation method and investigate the effects on comprehension and user preferences. It is a pilot study with 14 users on 2 hours of German documentaries or speeches with online translations into Czech. We collect continuous feedback and answers on factual questions. Our results show that the subtitling layout or flicker have a little effect on comprehension, in contrast to machine translation itself and individual competence. Other results show that users with a limited knowledge of the source language have different preferences to stability and latency than the users with zero knowledge. The results are statistically insignificant, however, we show that our method works and can be reproduced in larger volume.

CeCuAl3 and CeAuAl3, crystallizing in the non-centrosymmetric BaNiSn3 tetragonal structure, are known mainly for their unusual neutron scattering spectra involving additional excitations ascribed to vibron quasi-bound quantum state in CeCuAl3 and anti-crossing of phonon and crystal field excitations in CeAuAl3. In this work, we present results of nuclear magnetic resonance experiments on their lanthanum analogues - LaCuAl3 and LaAuAl3. The character of nuclear magnetic resonance spectra of 139La, 27Al, and 65Cu measured in LaAuAl3 and LaCuAl3 is dominated by electric quadrupole interaction. The spectral parameters acquired from experimental data are confronted with values obtained from the electronic structure calculations. The results show remarkable diffrences for the two compounds. The 139La spectrum in LaAuAl3 can be interpreted by a single spectral component corresponding to uniform environment of La atoms in the crystal structure, whereas for LaCuAl3 the spectrum decomposition yields a wide distribution of spectral parameters, which is not possible to explain by a single La environment, and multiple non-equivalent La positions in the crystal structure are required to interpret the spectrum.

The magnetic phase diagram of the tetragonal Ho$_2$RhIn$_8$ compound has similar features to many related systems, revealing a zero magnetic field AF1 and a field-induced AF2 phases. Details of the magnetic order in the AF2 phase were not reported yet for any of the related compounds. In addition, only the Ho$_2$RhIn$_8$ phase diagram contains a small region of the incommensurate zero-field AF3 phase. We have performed a number of neutron diffraction experiments on single crystals of Ho$_2$RhIn$_8$ using several diffractometers including experiments in both horizontal and vertical magnetic fields up to 4 T. We present details of the magnetic structures in all magnetic phases of the rich phase diagram of Ho$_2$RhIn$_8$. The Ho magnetic moments point along the tetragonal $c$ axis in every phase. The ground-state AF1 phase is characterized by propagation vector $\textbf{k}$ = (1/2, 0, 0). The more complex ferrimagnetic AF2 phase is described by four propagation vectors $\textbf{k}_{0}$ = (0, 0, 0), $\textbf{k}_{1}$ = (1/2, 0, 0), $\textbf{k}_{2}$ = (0, 1/2, 1/2), $\textbf{k}_{3}$ = (1/2, 1/2, 1/2). The magnetic structure in the AF3 phase is incommensurate with $\textbf{k}_{AF3}$ = (0.5, $\delta$, 0). Our results are consistent with theoretical calculations based on crystal field theory., Comment: submitted to PRB

R2RhIn8 compounds (space group P4/mmm, R is a rare-earth element) belong to a large group of structurally related tetragonal materials which involves several heavy-fermion superconductors based on Ce. We have succeeded to grow single crystals of compounds with Nd, Dy and Er and following our previous bulk measurements, we performed neutron-diffraction studies to determine their magnetic structures. The Laue diffraction experiment showed that the antiferromagnetic order below the N\'eel temperature is in all three compounds characterized by the propagation vector k = (1/2, 1/2, 1/2). The amplitude and direction of the magnetic moments, as well as the invariance symmetry of the magnetic structure, were determined by subsequent experiments using two- and four-circle diffractometers. The critical exponents were determined from the temperature dependence of the intensities below TN., Comment: 10 pages, 7 figures, submitted to PRB

Growth of atomic indium chains - 1D islands - on the Si(100)-2x1 surface was observed by scanning tunneling microscopy (STM) at room temperature and simulated by means of a kinetic Monte Carlo method. Density of indium islands and island size distribution were obtained for various deposition rates and coverage. STM observation of growth during deposition of indium provided information on growth kinetics and relaxation of grown layers. Important role of C-type defects at adsorption of metal atoms was observed. Measured growth characteristics were simulated using a microscopic model with anisotropic surface diffusion and forbidden zones along the metal chains. An analysis of experimental and simulation data shows that detachment of indium adatoms from the chains substantially influences a growth scenario and results in monotonously decreasing chain length distribution function at low coverage. Diffusion barriers determined from the simulations correspond to almost isotropic diffusion of indium adatoms on the surface. The results are discussed with respect to data reported in earlier papers for other metals., Comment: 9 pages, 9 figures, 2 tables; submitted to Physical Review B

Experiments with resonant photons at the U M4 edge have been performed on a sample of USb_{0.9}Te_{0.1}, which has an incommensurate magnetic structure with k = 0.596(2) reciprocal lattice units. The reflections of the form , as observed previously in a commensurate k = 1/2 system [N. Bernhoeft et al., Phys. Rev. B 69 174415 (2004)] are observed, removing any doubt that these occur because of multiple scattering or high-order contamination of the incident photon beam. They are clearly connected with the presence of a 3k configuration. Measurements of the reflections from the sample UAs_{0.8}Se_{0.2} in a magnetic field show that the transition at T* ~ 50 K is between a low-temperature 2k and high-temperature 3k state and that this transition is sensitive to an applied magnetic field. These experiments stress the need for quantitative theory to explain the intensities of these reflections., Comment: submitted to Phys. Rev. B

We have used element specific X-ray resonant scattering to investigate the M edge resonances in a single crystal of \unpo. Earlier neutron diffraction and M\"{o}ssbauer studies had shown the presence of long-range AF order below T$_{N}$ = 19 K, with sizeable magnetic moment both on the U and the Np ions. RXS results confirm the presence of an ordered dipole magnetic moment on the Np ions, but also reveal the presence of an anti-ferro arrangement of the electric quadrupole moments on both U and Np ions, with the same propagation vector that defines the magnetic structure. From the azimuthal dependence of the intensities we are able to determine the exact configuration of the quadrupolar ordering. The intensities of Bragg peaks associated with magnetic dipole and electric quadrupole order have different temperature dependences. On cooling below T$_{N}$, the magnetic dipole order develops faster on the uranium ions, with magnetic order on the Np ions increasing at a lower rate. At the same temperature, quadrupolar order on both the U and Np ions occurs along with an internal Jahn-Teller lattice distortion., Comment: Submitted to Physical Review B

Using resonant x-ray scattering to perform diffraction experiments at the U M$_{4}$ edge novel reflections of the generic form $$ have been observed in UAs$_{0.8}$Se$_{0.2}$ where $\vec{k} = < k00>$, with $k = {1/2}$ reciprocal lattice units, is the wave vector of the primary (magnetic) order parameter. The $$ reflections, with $10^{-4}$ of the $$ magnetic intensities, cannot be explained on the basis of the primary order parameter within standard scattering theory. A full experimental characterisation of these reflections is presented including their energy, azimuthal and temperature dependencies. On this basis we establish that the reflections most likely arise from the electric dipole operator involving transitions between the core 3d and partially filled $5f$ states. The temperature dependence couples the $$ peak to the triple-$\vec{k}$ region of the phase diagram: Below $\sim 50$ K, where previous studies have suggested a transition to a double-$\vec{k}$ state, the intensity of the $$ is dramatically reduced. Whilst we are unable to give a definite explanation of how these novel reflections appear, this paper concludes with a discussion of possible ideas for these reflections in terms of the coherent superposition of the 3 primary (magnetic) order parameters.