Search

Streaming multi-talker speech translation is a task that involves not only generating accurate and fluent translations with low latency but also recognizing when a speaker change occurs and what the speaker's gender is. Speaker change information can be used to create audio prompts for a zero-shot text-to-speech system, and gender can help to select speaker profiles in a conventional text-to-speech model. We propose to tackle streaming speaker change detection and gender classification by incorporating speaker embeddings into a transducer-based streaming end-to-end speech translation model. Our experiments demonstrate that the proposed methods can achieve high accuracy for both speaker change detection and gender classification.

Molecular gas, as the fuel for star formation, and its relationship with atomic gas are crucial for understanding how galaxies regulate their star forming (SF) activities. We conducted IRAM 30m observations of 23 nearby spiral galaxies from the CHANG-ES project to investigatet the distribution of molecular gas and the Kennicutt-Schmidt law. Combining these results with atomic gas masses from previous studies, we aim to investigate the scaling relations that connect the molecular and atomic gas masses with stellar masses and the baryonic Tully-Fisher relation. Based on spatially resolved observations of the three CO lines, we calculated the total molecular gas masses, the ratios between different CO lines, and derived physical parameters such as temperature and optical depth. The median line ratios for nuclear/disk regions are 8.6/6.1 (^{12}\mathrm{CO}/^{13}\mathrm{CO}\ J=1{-}0) and 0.53/0.39 (^{12}\mathrm{CO}\ J=2{-}1/J=1{-}0). Molecular gas mass derived from ^{13}\mathrm{CO} is correlated but systematically lower than that from ^{12}\mathrm{CO}. Most galaxies follow the spatially resolved SF scaling relation with a median gas depletion timescale of approximately 1 Gyr, while a few exhibit shorter timescales of approximately 0.1 Gyr. The molecular-to-atomic gas mass ratio correlates strongly with stellar mass, consistent with previous studies. Galaxies with lower stellar masses show an excess of atomic gas, indicating less efficient conversion to molecular gas. Most galaxies tightly follow the baryonic Tully-Fisher relation, but NGC 2992 and NGC 4594 deviate from the relation due to different physical factors. We find that the ratio of the cold gas (comprising molecular and atomic gas) to the total baryon mass decreases with the gravitational potential of the galaxy, as traced by rotation velocity, which could be due to gas consumption in SF or being heated to the hot phase., Comment: 15 pages, 11 figures (Figure 3 shows only a single galaxy example from the sample; all other galaxies in the sample are available online only.), 4 tables (full version of Table 2 and Table 3 is only available online). Accepted for publication in A&A

We present the results from calibrating the data of the Commensal Radio Astronomy FAST Survey (CRAFTS) for \HI intensity mapping by the Five-hundred-meter Aperture Spherical Radio Telescope (FAST). Using 70 hours of drift-scan observation with the L-band (1.05-1.45GHz) 19-beam receiver, we obtain the data covering $270\,\rm deg^2$ sky area. We employ both the pulsar backend and the spectrum backend to calibrate the spectral time-ordered-data (TOD) before projecting them onto HEALPix maps. We produce calibrated TOD with frequency resolution of 30 kHz and time resolution of 1 s and the map data-cube with frequency resolution of 30kHz and spatial resolution of $2.95\,\rm arcmin^2$. We carefully examine the pointing errors, noise overflow, RFI contamination and their effect on the data quality. The resulting noise level is $\sim$ 5.7 mJy for the calibrated TOD and 1.6 mJy for the map, which is consistent with the theoretical predictions within 5\% at RFI-free channels. We also validate the data by Principal Components Analysis (PCA) and find most foreground components are concentrated in the first 30 modes. We identify 447 isolated bright continuum sources in our data matching the NRAO-VLA Sky Survey (NVSS) catalog, with relative flux error of 8.3\% for TOD and 11.9\% for the map-level. We also measure the \HI emission of 90 galaxies with redshift $z<0.07$ and compare with \HI-MaNGA spectra from the Green Bank Telescope (GBT), yielding an overall relative error of the \HI integral flux of 16.7\%. Our results confirm the feasibility of conducting cosmological \HI signal detection with CRAFTS., Comment: 30 pages, 30 figures, and 3 tables

We present the analysis of a comprehensive sample of 352 early-type galaxies using public data, to investigate the correlations between CO luminosities and mid-infrared luminosities observed by \textit{Wide-field Infrared Survey Explorer} (\textit{WISE}). We find strong correlations between both CO (1-0) and CO (2-1) luminosities and 12 \micron\ luminosity, boasting a correlation coefficient greater than 0.9 and an intrinsic scatter smaller than 0.1 dex. The consistent slopes observed for the relationships of CO (1-0) and CO (2-1) suggest that the line ratio R21 lacks correlation with mid-infrared emission in early-type galaxies, which is significantly different from star-forming galaxies. Moreover, the slopes of $L_{\rm CO (1-0)}$--$L_{\mbox{12\micron}}$ and $L_{\rm CO (2-1)}$--$L_{\mbox{12\micron}}$ relations in early-type galaxies are steeper than those observed in star-forming galaxies. Given the absence of correlation with color, morphology or sSFR, the correlation between deviations and the molecular gas mass surface density could be eliminated by correcting the possible 12 \micron\ emission from old stars or adopting a systematically different $\alpha_{\rm CO}$. The latter, on average, is equivalent to adding an constant CO brightness density, specifically ${2.8{_{-0.6}}\!\!\!\!\!\!\!\!\!^{+0.8}}~[\mathrm{K~km~s^{-1}}]$ and ${4.4{_{-1.4}}\!\!\!\!\!\!\!\!\!^{+2.2}}~[\mathrm{K~km~s^{-1}}]$ for CO (1-0) and (2-1) respectively. These explorations will serve as useful tools for estimating the molecular gas content in gas-poor galaxies and understanding associated quenching processes., Comment: 20 pages, 6 figures, accepted for publication in ApJ

Utilizing temporal information to improve the performance of 3D detection has made great progress recently in the field of autonomous driving. Traditional transformer-based temporal fusion methods suffer from quadratic computational cost and information decay as the length of the frame sequence increases. In this paper, we propose a novel method called MambaDETR, whose main idea is to implement temporal fusion in the efficient state space. Moreover, we design a Motion Elimination module to remove the relatively static objects for temporal fusion. On the standard nuScenes benchmark, our proposed MambaDETR achieves remarkable result in the 3D object detection task, exhibiting state-of-the-art performance among existing temporal fusion methods.

End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation. The evaluation on the Zh-En test set of CoVoST 2, demonstrates that the proposed Isochrony-Controlled ST achieves 0.92 speech overlap and 8.9 BLEU, which has only a 1.4 BLEU drop compared to the ST baseline.

Generative Error Correction (GEC) has emerged as a powerful post-processing method to enhance the performance of Automatic Speech Recognition (ASR) systems. However, we show that GEC models struggle to generalize beyond the specific types of errors encountered during training, limiting their ability to correct new, unseen errors at test time, particularly in out-of-domain (OOD) scenarios. This phenomenon amplifies with named entities (NEs), where, in addition to insufficient contextual information or knowledge about the NEs, novel NEs keep emerging. To address these issues, we propose DARAG (Data- and Retrieval-Augmented Generative Error Correction), a novel approach designed to improve GEC for ASR in in-domain (ID) and OOD scenarios. We augment the GEC training dataset with synthetic data generated by prompting LLMs and text-to-speech models, thereby simulating additional errors from which the model can learn. For OOD scenarios, we simulate test-time errors from new domains similarly and in an unsupervised fashion. Additionally, to better handle named entities, we introduce retrieval-augmented correction by augmenting the input with entities retrieved from a database. Our approach is simple, scalable, and both domain- and language-agnostic. We experiment on multiple datasets and settings, showing that DARAG outperforms all our baselines, achieving 8\% -- 30\% relative WER improvements in ID and 10\% -- 33\% improvements in OOD settings., Comment: Preprint. Under Review

This study explores a relationship between the CO luminosity-full width at half-maximum linewidth linear relation (i.e. the CO LFR) and mean galaxy property of the local star-forming galaxy sample in the xCOLDGASS data base, via a mathematical statement. The whole data base galaxies were separated into two subsamples based on their stellar masses and redshifts, being a help to examine the dependence issue of the CO LFR. Selecting the galaxy data with a stringent requirement was also implemented in order to assure the validly of the CO LFR. An algorithm of the linear regression was conducted with the data of the subsample. An assessment about the linear correlation manifested a valid CO LFR occurs in the selected galaxy of the subsample, and the intercept of the CO LFR may be related with the mean galaxy properties such as the molecular gas fraction and galaxy size. For the finding on the intercept of the CO LFR, we aligned that intercept with those galaxy properties via the involvement of a $\psi$ parameter. On evaluating the $\psi$ value with our local star-forming galaxy sample, we numerically determined a relationship between the statistical result and the galaxy property in a different stellar mass range. It also shows a possible method on estimating galaxy property., Comment: 10 pages, 3 tables, 2 figures; we sincerely appreciate the suggestion of the referee and the acceptance of the MNRAS. Y-HW is greatly grateful to all the co-authors for their works on his articles

Facial Action Units (AUs) are of great significance in the realm of affective computing. In this paper, we propose AU-LLaVA, the first unified AU recognition framework based on the Large Language Model (LLM). AU-LLaVA consists of a visual encoder, a linear projector layer, and a pre-trained LLM. We meticulously craft the text descriptions and fine-tune the model on various AU datasets, allowing it to generate different formats of AU recognition results for the same input image. On the BP4D and DISFA datasets, AU-LLaVA delivers the most accurate recognition results for nearly half of the AUs. Our model achieves improvements of F1-score up to 11.4% in specific AU recognition compared to previous benchmark results. On the FEAFA dataset, our method achieves significant improvements over all 24 AUs compared to previous benchmark results. AU-LLaVA demonstrates exceptional performance and versatility in AU recognition.

This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges., Comment: project page: https://jamesjg.github.io/MVLLaVA_homepage/

We used the Five-hundred-meter Aperture Spherical radio Telescope (FAST) to search for the molecular emissions in the L-band between 1.0 and 1.5 GHz toward four comets, C/2020 F3 (NEOWISE), C/2020 R4 (ATLAS), C/2021 A1 (Leonard), and 67P/Churyumov-Gerasimenko during or after their perihelion passages. Thousands of molecular transition lines fall in this low-frequency range, many attributed to complex organic or prebiotic molecules. We conducted a blind search for the possible molecular lines in this frequency range in those comets and could not identify clear signals of molecular emissions in the data. Although several molecules have been detected at high frequencies of great than 100 GHz in comets, our results confirm that it is challenging to detect molecular transitions in the L-band frequency ranges. The non-detection of L-band molecular lines in the cometary environment could rule out the possibility of unusually strong lines, which could be caused by the masers or non-LTE effects. Although the line strengths are predicted to be weak, for FAST, using the ultra-wide bandwidth receiver and improving the radio frequency interference environments would enhance the detectability of those molecular transitions at low frequencies in the future., Comment: 14 pages, 6 figures, 6 tables, accepted for publication in RAA

Facial expression recognition (FER) is a critical task in multimedia with significant implications across various domains. However, analyzing the causes of facial expressions is essential for accurately recognizing them. Current approaches, such as those based on facial action units (AUs), typically provide AU names and intensities but lack insight into the interactions and relationships between AUs and the overall expression. In this paper, we propose a novel method called ExpLLM, which leverages large language models to generate an accurate chain of thought (CoT) for facial expression recognition. Specifically, we have designed the CoT mechanism from three key perspectives: key observations, overall emotional interpretation, and conclusion. The key observations describe the AU's name, intensity, and associated emotions. The overall emotional interpretation provides an analysis based on multiple AUs and their interactions, identifying the dominant emotions and their relationships. Finally, the conclusion presents the final expression label derived from the preceding analysis. Furthermore, we also introduce the Exp-CoT Engine, designed to construct this expression CoT and generate instruction-description data for training our ExpLLM. Extensive experiments on the RAF-DB and AffectNet datasets demonstrate that ExpLLM outperforms current state-of-the-art FER methods. ExpLLM also surpasses the latest GPT-4o in expression CoT generation, particularly in recognizing micro-expressions where GPT-4o frequently fails., Comment: project page: https://starhiking.github.io/ExpLLM_Page/

High-redshift radio(-loud) galaxies (H$z$RGs) are massive galaxies with powerful radio-loud active galactic nuclei (AGNs) and serve as beacons for protocluster identification. However, the interplay between H$z$RGs and the large-scale environment remains unclear. To understand the connection between H$z$RGs and the surrounding obscured star formation, we investigated the overdensity and spatial distribution of submillimeter-bright galaxies (SMGs) in the field of 4C\,23.56, a well-known H$z$RG at $z=2.48$. We used SCUBA-2 data ($\sigma\,{\sim}\,0.6$\,mJy) to estimate the $850\,{\rm \mu m}$ source number counts and examine the radial and azimuthal overdensities of the $850\,{\rm \mu m}$ sources in the vicinity of the H$z$RG. The angular distribution of SMGs is inhomogeneous around the H$z$RG 4C\,23.56, with fewer sources oriented along the radio jet. We also find a significant overdensity of bright SMGs (${\rm S}_{850\rm\,\mu m}\geq5\,$mJy). Faint and bright SMGs exhibit different spatial distributions. The former are concentrated in the core region, while the latter prefer the outskirts of the H$z$RG field. High-resolution observations show that the seven brightest SMGs in our sample are intrinsically bright, suggesting that the overdensity of bright SMGs is less likely due to the source multiplicity., Comment: 19 pages, 17 figures, 5 tables, accepted to A&A

Measuring the environments of massive galaxies at high redshift is crucial to understanding galaxy evolution and the conditions that gave rise to the distribution of matter we see in the Universe today. While high-$z$ radio galaxies (H$z$RGs) and quasars tend to reside in protocluster-like systems, the environments of their radio-quiet counterparts are relatively unexplored, particularly in the submillimetre, which traces dust-obscured star formation. In this study we search for 850 $\mu$m-selected submillimetre galaxies in the environments of massive ($M_{\star} > 10^{11} M_{\odot}$), radio-quiet ($L_{500 {\rm MHz}} \lesssim 10^{25}$ W Hz$^{-1}$) galaxies at $z \sim 1\text{--}3$ using S2COSMOS data. By constructing number counts in circular regions of radius 1--6 arcmin and comparing with blank-field measurements, we find no significant overdensities of SMGs around massive radio-quiet galaxies at any of these scales, despite being sensitive down to overdensities of $\delta \sim 0.4$. To probe deeper than the catalogue we also examine the distribution of peaks in the SCUBA-2 SNR map, which reveals only tentative signs of any difference in the SMG densities of the radio-quiet galaxy environments compared to the blank field, and only on smaller scales (1$^{\prime}$ radii, corresponding to $\sim0.5$ Mpc) and higher SNR thresholds. We conclude that massive, radio-quiet galaxies at cosmic noon are typically in environments with $\delta\lesssim0.4$, which are either consistent with the blank field or contain only weak overdensities spanning sub-Mpc scales. The contrast between our results and studies of H$z$RGs with similar stellar masses and redshifts implies an intrinsic link between the wide-field environment and radio AGN luminosity at high redshift., Comment: 13 pages, 5 figures. Published in MNRAS

Language-agnostic many-to-one end-to-end speech translation models can convert audio signals from different source languages into text in a target language. These models do not need source language identification, which improves user experience. In some cases, the input language can be given or estimated. Our goal is to use this additional language information while preserving the quality of the other languages. We accomplish this by introducing a simple and effective linear input network. The linear input network is initialized as an identity matrix, which ensures that the model can perform as well as, or better than, the original model. Experimental results show that the proposed method can successfully enhance the specified language, while keeping the language-agnostic ability of the many-to-one ST models.

We present a study of the molecular gas in early-mid stage major-mergers, with a sample of 43 major-merger galaxy pairs selected from the Mapping Nearby Galaxies at Apache Point Observatory (MaNGA) survey and a control sample of 195 isolated galaxies selected from the xCOLD GASS survey. Adopting kinematic asymmetry as a new effective indicator to describe the merger stage, we aim to study the role of molecular gas in the merger-induced star formation enhancement along the merger sequence of galaxy pairs. We obtain the molecular gas properties from CO observations with the James Clerk Maxwell Telescope (JCMT), Institut de Radioastronomie Milimetrique (IRAM) 30-m telescope, and the MASCOT survey. Using these data, we investigate the differences in molecular gas fraction ($f_{\rm H_{2}}$), star formation rate (SFR), star formation efficiency (SFE), molecular-to-atomic gas ratio ($M_{\rm H_{2}}/M_{\rm HI}$), total gas fraction ($f_{\rm gas}$), and the star formation efficiency of total gas (${\rm SFE_{gas}}$) between the pair and control samples. In the full pair sample, our results suggest the $f_{\rm H_{2}}$ of paired galaxies is significantly enhanced, while the SFE is comparable to that of isolated galaxies. We detect significantly increased $f_{\rm H_{2}}$ and $M_{\rm H_{2}}/M_{\rm HI}$ in paired galaxies at the pericenter stage, indicating an accelerated transition from atomic gas to molecular gas due to interactions. Our results indicate that the elevation of $f_{\rm H_{2}}$ plays a major role in the enhancement of global SFR in paired galaxies at the pericenter stage, while the contribution of enhanced SFE in specific regions requires further explorations through spatially resolved observations of a larger sample spanning a wide range of merger stages., Comment: 25 pages, 12 figures, 5 tables, accepted for publication in ApJS

Astrochemistry has been widely developed as a power tool to probe physical properties of the interstellar medium (ISM) in various conditions of the Milky Way (MW) Galaxy, and in near and distant galaxies. Most current studies conventionally apply linear scaling to all elemental abundances based on the gas-phase metallicity. However, these elements, including carbon and oxygen, are enriched differentially by stellar nucleosynthesis and the overall galactic chemical evolution, evident from $\alpha$-enhancement in multiple galactic observations such as starbursts, high-redshift star-forming galaxies, and low-metallicity dwarfs. We perform astrochemical modeling to simulate the impact of an $\alpha$-enhanced ISM gas cloud on the abundances of the three phases of carbon (C$^+$, C, CO) dubbed as `the carbon cycle'. The ISM environmental parameters considered include two cosmic-ray ionization rates ($\zeta_{\rm CR}=10^{-17}$ and $10^{-15}\,{\rm s}^{-1}$), two isotropic FUV radiation field strengths ($\chi/\chi_0=1$ and $10^2$), and (sub-)linear dust-to-gas relations against metallicity, mimicking the ISM conditions of different galaxy types. In galaxies with [C/O] $<$ 0, CO, C and C$^+$ all decrease in both abundances and emission, though with differential biases. The low-$J$ CO emission is found to be the most stable tracer for the molecular gas, while C and C$^+$ trace H$_2$ gas only under limited conditions, in line with recent discoveries of [CI]-dark galaxies. We call for caution when using [CII]~$158\mu$m and [CI](1-0) as alternative H$_2$-gas tracers for both diffuse and dense gas with non-zero [C/O] ratios., Comment: MNRAS accepted. 21 pages, 15 figures, comments welcome!

The growing need for instant spoken language transcription and translation is driven by increased global communication and cross-lingual interactions. This has made offering translations in multiple languages essential for user applications. Traditional approaches to automatic speech recognition (ASR) and speech translation (ST) have often relied on separate systems, leading to inefficiencies in computational resources, and increased synchronization complexity in real time. In this paper, we propose a streaming Transformer-Transducer (T-T) model able to jointly produce many-to-one and one-to-many transcription and translation using a single decoder. We introduce a novel method for joint token-level serialized output training based on timestamp information to effectively produce ASR and ST outputs in the streaming setting. Experiments on {it,es,de}->en prove the effectiveness of our approach, enabling the generation of one-to-many joint outputs with a single decoder for the first time., Comment: \c{opyright} 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

The intrinsic anomalous Hall effect (AHE) has been reported in numerous ferromagnetic (FM) Weyl semimetals. However, AHE in the antiferromagnetic (AFM) or paramagnetic (PM) state of Weyl semimetals has been rarely observed experimentally, and only in centrosymmetric materials. Different mechanisms have been proposed to establish the connection between the AHE and the type of magnetic order. In this paper, we report AHE in both the AFM and PM states of non-centrosymmetric compound SmAlSi. To account for the AHE in non-centrosymmetric Weyl semimetals without FM, we introduce a new mechanism based on magnetic field-induced Weyl nodes evolution. Angle-dependent quantum oscillations in SmAlSi provide evidence for the Weyl points and large AHE in both the PM and the AFM states. The proposed mechanism qualitatively explains the temperature dependence of the anomalous Hall conductivity (AHC), which displays unconventional power law behavior of the AHC in both AFM and PM states of SmAlSi.

Simultaneous Speech-to-Text translation serves a critical role in real-time crosslingual communication. Despite the advancements in recent years, challenges remain in achieving stability in the translation process, a concern primarily manifested in the flickering of partial results. In this paper, we propose a novel revision-controllable method designed to address this issue. Our method introduces an allowed revision window within the beam search pruning process to screen out candidate translations likely to cause extensive revisions, leading to a substantial reduction in flickering and, crucially, providing the capability to completely eliminate flickering. The experiments demonstrate the proposed method can significantly improve the decoding stability without compromising substantially on the translation quality., Comment: accepted by ASRU 2023

End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code., Comment: Accepted to ICASSP 2024

In this work, we investigate the spectroscopy of higher $B_c$ mesons, with a special focus on the consideration of the unquenched effects. To account for such effects, we employ the modified Godfrey-Isgur model and introduce a screening potential. The resulting mass spectrum of the concerned higher $B_c$ states is then presented, showing significant deviations after considering the unquenched effects. This emphasizes the importance of considering the unquenched effects when studying of the higher $B_c$ mesons. Furthermore, we determine the corresponding spatial wave functions of these $B_c$ mesons, which have practical applications in subsequent studies of their decays. These decays include two-body Okuba-Zweig-Iizuka allowed strong decays, dipion transitions between $B_c$ mesons, radiative decays, and some typical weak decays. With the ongoing high-luminosity upgrade of the Large Hadron Collider, we expect the discovery of additional $B_c$ states in the near future. The knowledge gained from the mass spectrum and the different decay modes will undoubtedly provide valuable insights for future experimental explorations of these higher $B_c$ mesons., Comment: 30 pages, 7 figures and 17 tables

In this paper, we explore the zero-shot capability of the Segment Anything Model (SAM) for food image segmentation. To address the lack of class-specific information in SAM-generated masks, we propose a novel framework, called FoodSAM. This innovative approach integrates the coarse semantic mask with SAM-generated masks to enhance semantic segmentation quality. Besides, we recognize that the ingredients in food can be supposed as independent individuals, which motivated us to perform instance segmentation on food images. Furthermore, FoodSAM extends its zero-shot capability to encompass panoptic segmentation by incorporating an object detector, which renders FoodSAM to effectively capture non-food object information. Drawing inspiration from the recent success of promptable segmentation, we also extend FoodSAM to promptable segmentation, supporting various prompt variants. Consequently, FoodSAM emerges as an all-encompassing solution capable of segmenting food items at multiple levels of granularity. Remarkably, this pioneering framework stands as the first-ever work to achieve instance, panoptic, and promptable segmentation on food images. Extensive experiments demonstrate the feasibility and impressing performance of FoodSAM, validating SAM's potential as a prominent and influential tool within the domain of food image segmentation. We release our code at https://github.com/jamesjg/FoodSAM., Comment: Code is available at https://github.com/jamesjg/FoodSAM

In end-to-end automatic speech recognition system, one of the difficulties for language expansion is the limited paired speech and text training data. In this paper, we propose a novel method to generate augmented samples with unpaired speech feature segments and text data for model pre-training, which has the advantage of low cost without using additional speech data. When mixing 20,000 hours augmented speech data generated by our method with 12,500 hours original transcribed speech data for Italian Transformer transducer model pre-training, we achieve 8.7% relative word error rate reduction. The pre-trained model achieves similar performance as the model pre-trained with multilingual transcribed 75,000 hours raw speech data. When merging the augmented speech data with the multilingual data to pre-train a new model, we achieve even more relative word error rate reduction of 12.2% over the baseline, which further verifies the effectiveness of our method for speech data augmentation.

In real-world applications, users often require both translations and transcriptions of speech to enhance their comprehension, particularly in streaming scenarios where incremental generation is necessary. This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. To produce ASR and ST content effectively with minimal latency, we propose a joint token-level serialized output training method that interleaves source and target words by leveraging an off-the-shelf textual aligner. Experiments in monolingual (it-en) and multilingual (\{de,es,it\}-en) settings demonstrate that our approach achieves the best quality-latency balance. With an average ASR latency of 1s and ST latency of 1.3s, our model shows no degradation or even improves output quality compared to separate ASR and ST models, yielding an average improvement of 1.1 WER and 0.4 BLEU in the multilingual case., Comment: Accepted at ASRU 2023

Symmetry is a crucial factor in determining the topological properties of materials. In nonmagnetic chiral crystals, the existence of the Kramers Weyl fermions reveals the topological nature of the Kramers degeneracy at time-reversal-invariant momenta (TRIMs). However, it is not clear whether Weyl nodes can also be pinned at points of symmetry in magnetic materials where the time-reversal is spontaneously broken. In this study, we introduce a new type of Weyl fermions, called Heesch Weyl fermions (HWFs), which are stabilized and pinned at points of symmetry by the Heesch groups in inadmissible chiral antiferromagnets. The emergence of HWFs is fundamentally different from that of Kramers Weyl fermions, as it does not rely on any anti-unitary symmetry $\mathcal{A}$ that satisfies $\mathcal{A}^2=-1$. Importantly, the emergence of HWFs is closely related to the antiferromagnetic order, as they are generally obscured by nodal lines in the parent nonmagnetic state. Using group theory analysis, we classify all the magnetic little co-groups of momenta where Heesch Weyl nodes are enforced and pinned by symmetry. With the guidance of this classification and first-principles calculations, we identify antiferromagnetic (AFM) materials such as YMnO$_3$ and Mn$_3$IrGe as candidate hosts for the AFM-order-induced HWFs.We also explore novel properties of Heesch Weyl antiferromagnets, such as nonlinear anomalous Hall effects and axial movement of Heesch Weyl nodes. Our findings shed new light on the role of symmetry in determining and stabilizing topological properties in magnetic materials, and open up new avenues for the design and exploration of topological materials.

Berry curvature multipoles appearing in topological quantum materials have recently attracted much attention. Their presence can manifest in novel phenomena, such as nonlinear anomalous Hall effects (NLAHE). The notion of Berry curvature multipoles extends our understanding of Berry curvature effects on the material properties. Hence, research on this subject is of fundamental importance and may also enable future applications in energy harvesting and high-frequency technology. It was shown that a Berry curvature dipole can give rise to a 2nd order NLAHE in materials of low crystalline symmetry. Here, we demonstrate a fundamentally new mechanism for Berry curvature multipoles in antiferromagnets that are supported by the underlying magnetic symmetries. Carrying out electric transport measurements on the kagome antiferromagnet FeSn, we observe a 3rd order NLAHE, which appears as a transverse voltage response at the 3rd harmonic frequency when a longitudinal a.c. current drive is applied. Interestingly, this NLAHE is strongest at and above room temperature. We combine these measurements with a scaling law analysis, a symmetry analysis, model calculations, first-principle calculations, and magnetic Monte-Carlo simulations to show that the observed NLAHE is induced by a Berry curvature quadrupole appearing in the spin-canted state of FeSn. At a practical level, our study establishes NLAHE as a sensitive probe of antiferromagnetic phase transitions in other materials, such as moir\'e superlattices, two-dimensional van der Waal magnets, and quantum spin liquid candidates, that remain poorly understood to date. More broadly, Berry curvature multipole effects are predicted to exist for 90 magnetic point groups. Hence, our work opens a new research area to study a variety of topological magnetic materials through nonlinear measurement protocols.

We propose gated language experts and curriculum training to enhance multilingual transformer transducer models without requiring language identification (LID) input from users during inference. Our method incorporates a gating mechanism and LID loss, enabling transformer experts to learn language-specific information. By combining gated transformer experts with shared transformer layers, we construct multilingual transformer blocks and utilize linear experts to effectively regularize the joint network. The curriculum training scheme leverages LID to guide the gated experts in improving their respective language performance. Experimental results on a bilingual task involving English and Spanish demonstrate significant improvements, with average relative word error reductions of 12.5% and 7.3% compared to the baseline bilingual and monolingual models, respectively. Notably, our method achieves performance comparable to the upper-bound model trained and inferred with oracle LID. Extending our approach to trilingual, quadrilingual, and pentalingual models reveals similar advantages to those observed in the bilingual models, highlighting its ease of extension to multiple languages.

This paper proposes a novel application system for the generation of three-dimensional (3D) character animation driven by markerless human body motion capturing. The entire pipeline of the system consists of five stages: 1) the capturing of motion data using multiple cameras, 2) detection of the two-dimensional (2D) human body joints, 3) estimation of the 3D joints, 4) calculation of bone transformation matrices, and 5) generation of character animation. The main objective of this study is to generate a 3D skeleton and animation for 3D characters using multi-view images captured by ordinary cameras. The computational complexity of the 3D skeleton reconstruction based on 3D vision has been reduced as needed to achieve frame-by-frame motion capturing. The experimental results reveal that our system can effectively and efficiently capture human actions and use them to animate 3D cartoon characters in real-time.

Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability. However, it is challenging to adapt it with text-only data. Factorized neural transducer (FNT) model was proposed to mitigate this problem. The improved adaptation ability of FNT on text-only adaptation data came at the cost of lowered accuracy compared to the standard neural transducer model. We propose several methods to improve the performance of the FNT model. They are: adding CTC criterion during training, adding KL divergence loss during adaptation, using a pre-trained language model to seed the vocabulary predictor, and an efficient adaptation approach by interpolating the vocabulary predictor with the n-gram language model. A combination of these approaches results in a relative word-error-rate reduction of 9.48\% from the standard FNT model. Furthermore, n-gram interpolation with the vocabulary predictor improves the adaptation speed hugely with satisfactory adaptation performance.

Automatic Speech Recognition (ASR) systems typically yield output in lexical form. However, humans prefer a written form output. To bridge this gap, ASR systems usually employ Inverse Text Normalization (ITN). In previous works, Weighted Finite State Transducers (WFST) have been employed to do ITN. WFSTs are nicely suited to this task but their size and run-time costs can make deployment on embedded applications challenging. In this paper, we describe the development of an on-device ITN system that is streaming, lightweight & accurate. At the core of our system is a streaming transformer tagger, that tags lexical tokens from ASR. The tag informs which ITN category might be applied, if at all. Following that, we apply an ITN-category-specific WFST, only on the tagged text, to reliably perform the ITN conversion. We show that the proposed ITN solution performs equivalent to strong baselines, while being significantly smaller in size and retaining customization capabilities., Comment: 8 pages. 6 page paper 2 page references

Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. It is thus possible to use a single transducer model to perform both tasks. In real-world applications, such joint ASR and ST models may need to be streaming and do not require source language identification (i.e. language-agnostic). In this paper, we propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers. Based on the transducer model structure, we propose four methods, a unified joint and prediction network for multilingual output, a clustered multilingual encoder, target language identification for encoder, and connectionist temporal classification regularization. Experimental results show that LAMASSU not only drastically reduces the model size but also reaches the performances of monolingual ASR and bilingual ST models., Comment: INTERSPEECH 2023

In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language. The backbone of SM2 is Transformer Transducer, which has high streaming capability. Instead of human labeled speech translation (ST) data, SM2 models are trained using weakly supervised data generated by converting the transcriptions in speech recognition corpora with a machine translation service. With 351 thousand hours of anonymized speech training data from 25 languages, SM2 models achieve comparable or even better ST quality than some recent popular large-scale non-streaming speech models. More importantly, we show that SM2 has the truly zero-shot capability when expanding to new target languages, yielding high quality ST results for {source-speech, target-text} pairs that are not seen during training.

Kramers nodal lines (KNLs) have recently been proposed theoretically as a special type of Weyl line degeneracy connecting time-reversal invariant momenta. KNLs are robust to spin orbit coupling and are inherent to all non-centrosymmetric achiral crystal structures, leading to unusual spin, magneto-electric, and optical properties. However, their existence in in real quantum materials has not been experimentally established. Here we gather the experimental evidence pointing at the presence of KNLs in SmAlSi, a non-centrosymmetric metal that develops incommensurate spin density wave order at low temperature. Using angle-resolved photoemission spectroscopy, density functional theory calculations, and magneto-transport methods, we provide evidence suggesting the presence of KNLs, together with observing Weyl fermions under the broken inversion symmetry in the paramagnetic phase of SmAlSi. We discuss the nesting possibilities regarding the emergent magnetic orders in SmAlSi. Our results provide a solid basis of experimental observations for exploring correlated topology in SmAlSi., Comment: 40 pages, 5 figures, 1 table