Search

With an extremely high dimensionality, the spatial degree of freedom of entangled photons is a key tool for quantum foundation and applied quantum techniques. To fully utilize the feature, the essential task is to experimentally characterize the multiphoton spatial wave function including the entangled amplitude and phase information at different evolutionary stages. However, there is no effective method to measure it. Quantum state tomography is costly, and quantum holography requires additional references. Here we introduce quantum Shack-Hartmann wavefront sensing to perform efficient and reference-free measurement of the biphoton spatial wave function. The joint probability distribution of photon pairs at the back focal plane of a microlens array is measured and used for amplitude extraction and phase reconstruction. In the experiment, we observe that the biphoton amplitude correlation becomes weak while phase correlation shows up during free-space propagation. Our work is a crucial step in quantum physical and adaptive optics and paves the way for characterizing quantum optical fields with high-order correlations or topological patterns., Comment: Main text: 6 pages, 4 figures; Supplemental Material: 13 pages, 11 figures. Accepted by Physical Review Letters: https://journals.aps.org/prl/accepted/da075Yd1Ae71fc8532571485c61018959de0ae8f7

End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code., Comment: Accepted to ICASSP 2024

This study is focused on understanding and quantifying the change in phoneme and prosody information encoded in the Self-Supervised Learning (SSL) model, brought by an accent identification (AID) fine-tuning task. This problem is addressed based on model probing. Specifically, we conduct a systematic layer-wise analysis of the representations of the Transformer layers on a phoneme correlation task, and a novel word-level prosody prediction task. We compare the probing performance of the pre-trained and fine-tuned SSL models. Results show that the AID fine-tuning task steers the top 2 layers to learn richer phoneme and prosody representation. These changes share some similarities with the effects of fine-tuning with an Automatic Speech Recognition task. In addition, we observe strong accent-specific phoneme representations in layer 9. To sum up, this study provides insights into the understanding of SSL features and their interactions with fine-tuning tasks., Comment: Accepted by Interspeech 2023

Einstein-Podolsky-Rosen (EPR) steering, a fundamental concept of quantum nonlocality, describes one observer's capability to remotely affect another distant observer's state by local measurements. Unlike quantum entanglement and Bell nonlocality, both associated with the symmetric quantum correlation, EPR steering depicts the unique asymmetric property of quantum nonlocality. With the local filter operation in which some system components are discarded, quantum nonlocality can be distilled to enhance the nonlocal correlation, and even the hidden nonlocality can be activated. However, asymmetric quantum nonlocality in the filter operation still lacks a well-rounded investigation, especially considering the discarded parts where quantum nonlocal correlations may still exist with probabilities. Here, in both theory and experiment, we investigate the effect of reusing the discarded particles from local filter. We observe all configurations of EPR steering simultaneously and other intriguing evolution of asymmetric quantum nonlocality, such as reversing the direction of one-way EPR steering. This work provides a perspective to answer "What is the essential role of utilizing quantum steering as a resource?", and demonstrates a practical toolbox for manipulating asymmetric quantum systems with significant potential applications in quantum information tasks., Comment: 11pages, 6figures

The quantum wave function of multiple particles provides additional information which is inaccessible to detectors working alone. Here, we introduce the coincidence wavefront sensing (CWS) method to reconstruct the phase of the multiphoton transverse spatial wave function. The spatially resolved coincidence photon counting is involved. Numerical simulations of two-photon cases using the weak measurement wavefront sensor are performed to test its correctness, and the phase information hidden in the correlation are revealed. Our work provides a direct spatial way to characterize multipartite quantum systems, and leads to fundamental studies like experimental Bohmian mechanics and applications in quantum optical technologies., Comment: 9 pages, 3 figures. Note for the third version (v3): After publication, we found the author names of the 22nd reference was misspelled, and the experimental setup of another referring letter was misinterpreted in the main text. So we corrected the names of the 22nd reference and removed the 46th reference along with its description in the main text. (c) 2023 American Physical Society

Photons play essential roles in fundamental physics and practical technologies. They have become one of the attractive informaiton carriers for quantum computation and quantum simulation. Recently, various photonic degrees of freedom supported by optical resonant cavities form photonic synthetic dimensions, which contribute to all-optical platforms for simulating novel topological materials. The photonic discrete or continuous degrees of freedom are mapped to the lattices or momenta of the simulated topological matter, and the couplings between optical modes are equivalent to the interactions among quasi-particles. Mature optical modulations enable flexible engineering of the simulated Hamiltonian. Meanwhile, the resonant detection methods provide direct approaches to obtaining the corresponding energy band structures, particle distributions and dynamical evolutions. In this Review, we give an overview of the synthetic dimensions in optical cavities, including frequency, orbital angular momentum, time-multiplexed lattice, and independent parameters. Abundant higher-dimensional topological models have been demonstrated in lower dimensional synthetic systems. We further discuss the potential development of photonic synthetic dimensions in the future.

Exceptional points (EPs), at which more than one eigenvalue and eigenvector coalesce, are unique spectral features of Non-Hermiticity (NH) systems. They exist widely in open systems with complex energy spectra. We experimentally demonstrate the appearance of paired EPs in a periodical driven degenerate optical cavity along the synthetic orbital angular momentum (OAM) dimension with a tunable parameter. The complex-energy band structures and the key features of EPs, i.e. their Fermi arcs, parity-time symmetry breaking transition, energy swapping, and half-integer band windings are directly observed by detecting the cavity's transmission spectrum. Our results advance the fundamental understanding of NH physics and demonstrate the flexibility of using the photonic synthetic dimensions to implement NH systems.

Neural network pruning compresses automatic speech recognition (ASR) models effectively. However, in multilingual ASR, language-agnostic pruning may lead to severe performance drops on some languages because language-agnostic pruning masks may not fit all languages and discard important language-specific parameters. In this work, we present ASR pathways, a sparse multilingual ASR model that activates language-specific sub-networks ("pathways"), such that the parameters for each language are learned explicitly. With the overlapping sub-networks, the shared parameters can also enable knowledge transfer for lower-resource languages via joint multilingual training. We propose a novel algorithm to learn ASR pathways, and evaluate the proposed method on 4 languages with a streaming RNN-T model. Our proposed ASR pathways outperform both dense models and a language-agnostically pruned model, and provide better performance on low-resource languages compared to the monolingual sparse models., Comment: Accepted by ICASSP 2023

The weak measurement wavefront sensor detects the phase gradient of light like the Shack-Hartmann sensor does. However, the use of one thin birefringent crystal to displace light beams results in a wavelength-dependent phase difference between the two polarization components, which limits the practical application. Using a Savart plate which consists of two such crystals can compensate for the phase difference and realize achromatic wavefront sensing when combined with an achromatic retarder. We discuss the spatial resolution of the sensor and experimentally reconstruct a wavefront modulated by a pattern. Then we obtain the Zernike coefficients with three different wavelengths before and after modulation. Our work makes this new wavefront sensor more applicable to actual tasks like biomedical imaging., Comment: 4 pages, 4 figures. \copyright 2022 Optica Publishing Group. Users may use, reuse, and build upon the article, or use the article for text or data mining, so long as such uses are for non-commercial purposes and appropriate attribution is maintained. All other rights are reserved

The task of wavefront sensing is to measure the phase of the optical field. Here, we demonstrate that the widely used Shack-Hartmann wavefront sensor detects the weak value of transverse momentum, usually achieved by the method of quantum weak measurement. We extend its input states to partially coherent states and compare it with the weak measurement wavefront sensor, which has a higher spatial resolution but a smaller dynamic range. Since weak values are commonly used in investigating fundamental quantum physics and quantum metrology, our work would find essential applications in these fields., Comment: 3 pages, 2 figures

Current leading mispronunciation detection and diagnosis (MDD) systems achieve promising performance via end-to-end phoneme recognition. One challenge of such end-to-end solutions is the scarcity of human-annotated phonemes on natural L2 speech. In this work, we leverage unlabeled L2 speech via a pseudo-labeling (PL) procedure and extend the fine-tuning approach based on pre-trained self-supervised learning (SSL) models. Specifically, we use Wav2vec 2.0 as our SSL model, and fine-tune it using original labeled L2 speech samples plus the created pseudo-labeled L2 speech samples. Our pseudo labels are dynamic and are produced by an ensemble of the online model on-the-fly, which ensures that our model is robust to pseudo label noise. We show that fine-tuning with pseudo labels achieves a 5.35% phoneme error rate reduction and 2.48% MDD F1 score improvement over a labeled-samples-only fine-tuning baseline. The proposed PL method is also shown to outperform conventional offline PL methods. Compared to the state-of-the-art MDD systems, our MDD solution produces a more accurate and consistent phonetic error diagnosis. In addition, we conduct an open test on a separate UTD-4Accents dataset, where our system recognition outputs show a strong correlation with human perception, based on accentedness and intelligibility., Comment: Accepted to Interspeech 2022

Synthetic dimensions based on particles' internal degrees of freedom, such as frequency, spatial modes and arrival time, have attracted significant attention. They offer ideal large-scale lattices to simulate nontrivial topological phenomena. Exploring more synthetic dimensions is one of the paths toward higher dimensional physics. In this work, we design and experimentally control the coupling among synthetic dimensions consisting of the intrinsic photonic orbital angular momentum and spin angular momentum degrees of freedom in a degenerate optical resonant cavity, which generates a periodically driven spin-orbital coupling system. We directly characterize the system's properties, including the density of states, energy band structures and topological windings, through the transmission intensity measurements. Our work demonstrates a novel mechanism for exploring the spatial modes of twisted photons as the synthetic dimension, which paves the way to design rich topological physics in a highly compact platform.

Many social media users prefer consuming content in the form of videos rather than text. However, in order for content creators to produce videos with a high click-through rate, much editing is needed to match the footage to the music. This posts additional challenges for more amateur video makers. Therefore, we propose a novel attention-based model VMT (Video-Music Transformer) that automatically generates piano scores from video frames. Using music generated from models also prevent potential copyright infringements that often come with using existing music. To the best of our knowledge, there is no work besides the proposed VMT that aims to compose music for video. Additionally, there lacks a dataset with aligned video and symbolic music. We release a new dataset composed of over 7 hours of piano scores with fine alignment between pop music videos and MIDI files. We conduct experiments with human evaluation on VMT, SeqSeq model (our baseline), and the original piano version soundtrack. VMT achieves consistent improvements over the baseline on music smoothness and video relevance. In particular, with the relevance scores and our case study, our model has shown the capability of multimodality on frame-level actors' movement for music generation. Our VMT model, along with the new dataset, presents a promising research direction toward composing the matching soundtrack for videos. We have released our code at https://github.com/linchintung/VMT, Comment: Rejected by ISMIR 2020

This work presents a lifelong learning approach to train a multilingual Text-To-Speech (TTS) system, where each language was seen as an individual task and was learned sequentially and continually. It does not require pooled data from all languages altogether, and thus alleviates the storage and computation burden. One of the challenges of lifelong learning methods is "catastrophic forgetting": in TTS scenario it means that model performance quickly degrades on previous languages when adapted to a new language. We approach this problem via a data-replay-based lifelong learning method. We formulate the replay process as a supervised learning problem, and propose a simple yet effective dual-sampler framework to tackle the heavily language-imbalanced training samples. Through objective and subjective evaluations, we show that this supervised learning formulation outperforms other gradient-based and regularization-based lifelong learning methods, achieving 43% Mel-Cepstral Distortion reduction compared to a fine-tuning baseline., Comment: Accepted to ICASSP 2022. Camera-ready