Search

Recently, self-supervised pretraining has achieved impressive results in end-to-end (E2E) automatic speech recognition (ASR). However, the dominant sequence-to-sequence (S2S) E2E model is still hard to fully utilize the self-supervised pre-training methods because its decoder is conditioned on acoustic representation thus cannot be pretrained separately. In this paper, we propose a pretrained Transformer (Preformer) S2S ASR architecture based on hybrid CTC/attention E2E models to fully utilize the pretrained acoustic models (AMs) and language models (LMs). In our framework, the encoder is initialized with a pretrained AM (wav2vec2.0). The Preformer leverages CTC as an auxiliary task during training and inference. Furthermore, we design a one-cross decoder (OCD), which relaxes the dependence on acoustic representations so that it can be initialized with pretrained LM (DistilGPT2). Experiments are conducted on the AISHELL-1 corpus and achieve a $4.6\%$ character error rate (CER) on the test set. Compared with our vanilla hybrid CTC/attention Transformer baseline, our proposed CTC/attention-based Preformer yields $27\%$ relative CER reduction. To the best of our knowledge, this is the first work to utilize both pretrained AM and LM in a S2S ASR system., Comment: ASRU2021

Enhancing the quality of low-light images plays a very important role in many image processing and multimedia applications. In recent years, a variety of deep learning techniques have been developed to address this challenging task. A typical framework is to simultaneously estimate the illumination and reflectance, but they disregard the scene-level contextual information encapsulated in feature spaces, causing many unfavorable outcomes, e.g., details loss, color unsaturation, artifacts, and so on. To address these issues, we develop a new context-sensitive decomposition network architecture to exploit the scene-level contextual dependencies on spatial scales. More concretely, we build a two-stream estimation mechanism including reflectance and illumination estimation network. We design a novel context-sensitive decomposition connection to bridge the two-stream mechanism by incorporating the physical principle. The spatially-varying illumination guidance is further constructed for achieving the edge-aware smoothness property of the illumination component. According to different training patterns, we construct CSDNet (paired supervision) and CSDGAN (unpaired supervision) to fully evaluate our designed architecture. We test our method on seven testing benchmarks to conduct plenty of analytical and evaluated experiments. Thanks to our designed context-sensitive decomposition connection, we successfully realized excellent enhanced results, which fully indicates our superiority against existing state-of-the-art approaches. Finally, considering the practical needs for high-efficiency, we develop a lightweight CSDNet (named LiteCSDNet) by reducing the number of channels. Further, by sharing an encoder for these two components, we obtain a more lightweight version (SLiteCSDNet for short). SLiteCSDNet just contains 0.0301M parameters but achieves the almost same performance as CSDNet., Comment: Accepted by IEEE TNNLS. Code is available at https://github.com/KarelZhang/CSDNet-CSDGAN

Images captured from low-light scenes often suffer from severe degradations, including low visibility, color cast and intensive noises, etc. These factors not only affect image qualities, but also degrade the performance of downstream Low-Light Vision (LLV) applications. A variety of deep learning methods have been proposed to enhance the visual quality of low-light images. However, these approaches mostly rely on significant architecture engineering to obtain proper low-light models and often suffer from high computational burden. Furthermore, it is still challenging to extend these enhancement techniques to handle other LLVs. To partially address above issues, we establish Retinex-inspired Unrolling with Architecture Search (RUAS), a general learning framework, which not only can address low-light enhancement task, but also has the flexibility to handle other more challenging downstream vision applications. Specifically, we first establish a nested optimization formulation, together with an unrolling strategy, to explore underlying principles of a series of LLV tasks. Furthermore, we construct a differentiable strategy to cooperatively search specific scene and task architectures for RUAS. Last but not least, we demonstrate how to apply RUAS for both low- and high-level LLV applications (e.g., enhancement, detection and segmentation). Extensive experiments verify the flexibility, effectiveness, and efficiency of RUAS., Comment: Submitted to IEEE TPAMI. Code is available at https://github.com/vis-opt-group/RUAS

Recently, self-supervised pre-training has gained success in automatic speech recognition (ASR). However, considering the difference between speech accents in real scenarios, how to identify accents and use accent features to improve ASR is still challenging. In this paper, we employ the self-supervised pre-training method for both accent identification and accented speech recognition tasks. For the former task, a standard deviation constraint loss (SDC-loss) based end-to-end (E2E) architecture is proposed to identify accents under the same language. As for accented speech recognition task, we design an accent-dependent ASR system, which can utilize additional accent input features. Furthermore, we propose a frame-level accent feature, which is extracted based on the proposed accent identification model and can be dynamically adjusted. We pre-train our models using 960 hours unlabeled LibriSpeech dataset and fine-tune them on AESRC2020 speech dataset. The experimental results show that our proposed accent-dependent ASR system is significantly ahead of the AESRC2020 baseline and achieves $6.5\%$ relative word error rate (WER) reduction compared with our accent-independent ASR system., Comment: INTERSPEECH2021

Recently self-supervised learning has emerged as an effective approach to improve the performance of automatic speech recognition (ASR). Under such a framework, the neural network is usually pre-trained with massive unlabeled data and then fine-tuned with limited labeled data. However, the non-streaming architecture like bidirectional transformer is usually adopted by the neural network to achieve competitive results, which can not be used in streaming scenarios. In this paper, we mainly focus on improving the performance of streaming transformer under the self-supervised learning framework. Specifically, we propose a novel two-stage training method during fine-tuning, which combines knowledge distilling and self-training. The proposed training method achieves 16.3% relative word error rate (WER) reduction on Librispeech noisy test set. Finally, by only using the 100h clean subset of Librispeech as the labeled data and the rest (860h) as the unlabeled data, our streaming transformer based model obtains competitive WERs 3.5/8.7 on Librispeech clean/noisy test sets., Comment: INTERSPEECH2021

Despite many studies on the transmission mechanism of the Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), it remains still challenging to efficiently reduce mortality. In this work, we apply a two-population Susceptible-Infected-Removed (SIR) model to investigate the COVID-19 spreading when contacts between elderly and non-elderly individuals are reduced due to the high mortality risk of elderly people. We discover that the reduction of connections between two populations can delay the death curve but cannot well reduce the final mortality. We propose a merged SIR model, which advises elderly individuals to interact less with their non-elderly connections at the initial stage but interact more with their non-elderly relationships later, to reduce the final mortality. Finally, immunizing elderly hub individuals can also significantly decrease mortality., Comment: 16 pages, 12 figures

We report the local probe investigation of a frustrated antiferromagnet Pr$_3$BWO$_9$ with the distorted kagome lattice. Absence of magnetic order or spin freezing is indicated by the spectral analysis down to 0.3 K and specific heat measurements down to 0.09 K. The Knight shifts show an upturn behavior with the sample cooling down, which is further suppressed by external field. For the spin dynamics, gapped spin excitation is observed from the temperature dependence of spin-lattice relaxation rates, with the gap size proportional to the applied magnetic field intensity. Comparatively, an unexpected sharp peak is observed in the nuclear spin-spin relaxation rate data at $T^*\sim 4-5$ K. These results indicate an unconventional persistent fluctuating paramagnetic ground state with antiferromagnetic collective spin excitations in the strongly frustrated spin system., Comment: 6 pages, 4 figures

Nowadays voice search for points of interest (POI) is becoming increasingly popular. However, speech recognition for local POI has remained to be a challenge due to multi-dialect and massive POI. This paper improves speech recognition accuracy for local POI from two aspects. Firstly, a geographic acoustic model (Geo-AM) is proposed. The Geo-AM deals with multi-dialect problem using dialect-specific input feature and dialect-specific top layer. Secondly, a group of geo-specific language models (Geo-LMs) are integrated into our speech recognition system to improve recognition accuracy of long tail and homophone POI. During decoding, specific language models are selected on demand according to users' geographic location. Experiments show that the proposed Geo-AM achieves 6.5%$\sim$10.1% relative character error rate (CER) reduction on an accent testset and the proposed Geo-AM and Geo-LM totally achieve over 18.7% relative CER reduction on Tencent Map task., Comment: Accepted by SLT 2021

End-to-end models are favored in automatic speech recognition (ASR) because of their simplified system structure and superior performance. Among these models, Transformer and Conformer have achieved state-of-the-art recognition accuracy in which self-attention plays a vital role in capturing important global information. However, the time and memory complexity of self-attention increases squarely with the length of the sentence. In this paper, a prob-sparse self-attention mechanism is introduced into Conformer to sparse the computing process of self-attention in order to accelerate inference speed and reduce space consumption. Specifically, we adopt a Kullback-Leibler divergence based sparsity measurement for each query to decide whether we compute the attention function on this query. By using the prob-sparse attention mechanism, we achieve impressively 8% to 45% inference speed-up and 15% to 45% memory usage reduction of the self-attention module of Conformer Transducer while maintaining the same level of error rate.

High-performance osmotic energy conversion (OEC) with perm-selective porous membrane requires both high ionic selectivity and permeability simultaneously. Here, hydrodynamic slip is considered on surfaces of nanopores to break the tradeoff between ionic selectivity and permeability, because it decreases the viscous friction at solid-liquid interfaces which can promote ionic diffusion during OEC. Taking advantage of simulations, influences from individual slipping surfaces on the OEC performance have been investigated, i.e. the slipping inner surface (surfaceinner) and exterior surfaces on the low- and high-concentration sides (surfaceL and surfaceH). Results show that the slipping surfaceL is crucial for high-performance OEC. For nanopores with various lengths, the slipping surfaceL simultaneously increases both ionic permeability and selectivity of nanopores, which results in both significantly enhanced electric power and energy conversion efficiency. While for nanopores longer than 30 nm, the slipping surfaceinner plays a dominant role in the increase of electric power, which induces a considerable decrease in energy conversion efficiency due to enhanced transport of both cations and anions. Considering the difficulty in hydrodynamic slip modification to the surfaceinner of nanopores, the surface modification to the surfaceL may be a better choice to achieve high-performance OEC. Our results provide feasible guidance to the design of porous membranes for high-performance osmotic energy harvesting., Comment: 25 pages, 7 figures

High-performance osmotic energy conversion (OEC) requires both high ionic selectivity and permeability in nanopores. Here, through systematical explorations of influences from individual charged nanopore surfaces on the performance of OEC, we find that the charged exterior surface on the low-concentration side (surfaceL) is essential to achieve high-performance osmotic power generation, which can significantly improve the ionic selectivity and permeability simultaneously. Detailed investigation of ionic transport indicates that electric double layers near charged surfaces provide high-speed passages for counterions. The charged surfaceL enhances cation diffusion through enlarging the effective diffusive area, and inhibits anion transport by electrostatic repulsion. Different areas of charged exterior surfaces have been considered to mimic membranes with different porosities in practical applications. Through adjusting the width of the charged ring region on the surfaceL, electric power in single nanopores increases from 0.3 to 3.4 pW with a plateau at the width of ~200 nm. The power density increases from 4200 to 4900 W/m2 and then decreases monotonously that reaches the commercial benchmark at the charged width of ~480 nm. While, energy conversion efficiency can be promoted from 4% to 26%. Our results provide useful guide in the design of nanoporous membranes for high-performance osmotic energy harvesting., Comment: 30 pages and 7 figures

We explore parton collisional effects on the conversion of geometry eccentricities into azimuthal anisotropies in Pb+Pb collisions at $\sqrt{s_{NN}}$ = 5.02 TeV using a multi-phase transport model. The initial eccentricity $\varepsilon_{n}$ (n = 2,3) and flow harmonics $v_{n}$ (n = 2,3) are investigated as a function of the number of parton collisions ($N_{coll}$) during the source evolution of partonic phase. It is found that partonic collisions leads to generate elliptic flow $v_{2}$ and triangular flow $v_{3}$ in Pb+Pb collisions. On the other hand, partonic collisions also result in an evolution of the eccentricity of geometry. The collisional effect on the flow conversion efficiency is therefore studied. We find that the partons with larger $N_{coll}$ show a lower flow conversion efficiency, which reflect differential behaviors with respect to $N_{coll}$. It provides an additional insight into the dynamics of the space-momentum transformation during the QGP evolution from a transport model point of view., Comment: 11 pages, 12 figues

This paper proposes an extremely lightweight phone-based transducer model with a tiny decoding graph on edge devices. First, a phone synchronous decoding (PSD) algorithm based on blank label skipping is first used to speed up the transducer decoding process. Then, to decrease the deletion errors introduced by the high blank score, a blank label deweighting approach is proposed. To reduce parameters and computation, deep feedforward sequential memory network (DFSMN) layers are used in the transducer encoder, and a CNN-based stateless predictor is adopted. SVD technology compresses the model further. WFST-based decoding graph takes the context-independent (CI) phone posteriors as input and allows us to flexibly bias user-specific information. Finally, with only 0.9M parameters after SVD, our system could give a relative 9.1% - 20.5% improvement compared with a bigger conventional hybrid system on edge devices., Comment: Accepted by ICASSP 2021

Low-light image enhancement plays very important roles in low-level vision field. Recent works have built a large variety of deep learning models to address this task. However, these approaches mostly rely on significant architecture engineering and suffer from high computational burden. In this paper, we propose a new method, named Retinex-inspired Unrolling with Architecture Search (RUAS), to construct lightweight yet effective enhancement network for low-light images in real-world scenario. Specifically, building upon Retinex rule, RUAS first establishes models to characterize the intrinsic underexposed structure of low-light images and unroll their optimization processes to construct our holistic propagation structure. Then by designing a cooperative reference-free learning strategy to discover low-light prior architectures from a compact search space, RUAS is able to obtain a top-performing image enhancement network, which is with fast speed and requires few computational resources. Extensive experiments verify the superiority of our RUAS framework against recently proposed state-of-the-art methods.

Nanopores that exhibit ionic current rectification (ICR) behave like diodes, such that they transport ions more efficiently in one direction than the other. Conical nanopores have been shown to rectify ionic current, but only those with at least 500 nm in length exhibit significant ICR. Here, through the finite element method, we show how ICR of conical nanopores with length below 200 nm can be tuned by controlling individual charged surfaces i.e. inner pore surface (surface_inner), and exterior pore surfaces on the tip and base side (surface_tip and surface_base). The charged surface_inner and surface_tip can induce obvious ICR individually, while the effects of the charged surface_base on ICR can be ignored. The fully charged surface_inner alone could render the nanopore counterion-selective and induces significant ion concentration polarization in the tip region, which causes reverse ICR compared to nanopores with all surface charged. In addition, the direction and degree of rectification can be further tuned by the depth of the charged surface_inner. When considering the exterior membrane surface only, the charged surface_tip causes intra-pore ionic enrichment and depletion under opposite biases which results in significant ICR. Its effective region is within ~40 nm beyond the tip orifice. We also found that individual charged parts of the pore system contributed to ICR in an additive way due to the additive effect on the ion concentration regulation along the pore axis. With various combinations of fully/partially charged surface_inner and surface_tip, diverse ICR ratios from ~2 to ~170 can be achieved. Our findings shed light on the mechanism of ionic current rectification in ultra-short conical nanopores, and provide a useful guide to the design and modification of ultra-short conical nanopores in ionic circuits and nanofluidic sensors., Comment: 25 pages, 8 figures

We report our nuclear magnetic resonance (NMR) study on the structurally spin chain compound Ni$_2$NbBO$_6$ with complex magnetic coupling. The antiferromagnetic transition is monitored by the line splitting resulting from the staggered internal hyperfine field. The magnetic coupling configuration proposed by the first-principle density functional theory (DFT) is supported by our NMR spectral analysis. For the spin dynamics, a prominent peak at $T\sim35$ K well above the N\'{e}el temperature ($T_N\sim20$ K at $\mu_0H=10$ T) is observed from the spin-lattice relaxation data. As compared with the dc-susceptibility, this behavior indicates a antiferromagnetic coupling with the typical energy scale of $\sim3$ meV. Thus, the Ni$_2$NbBO$_6$ compound can be viewed as strongly ferromagnetically coupled armchair spin chains along the crystalline $b$-axis. These facts place strong constraints to the theoretical model for this compound., Comment: 6 pages, 4 figures

The attention mechanism of the Listen, Attend and Spell (LAS) model requires the whole input sequence to calculate the attention context and thus is not suitable for online speech recognition. To deal with this problem, we propose multi-head monotonic chunk-wise attention (MTH-MoChA), an improved version of MoChA. MTH-MoChA splits the input sequence into small chunks and computes multi-head attentions over the chunks. We also explore useful training strategies such as LSTM pooling, minimum world error rate training and SpecAugment to further improve the performance of MTH-MoChA. Experiments on AISHELL-1 data show that the proposed model, along with the training strategies, improve the character error rate (CER) of MoChA from 8.96% to 7.68% on test set. On another 18000 hours in-car speech data set, MTH-MoChA obtains 7.28% CER, which is significantly better than a state-of-the-art hybrid system.

The squeezed state is important in quantum metrology and quantum information. The most effective generation tool known is the optical parametric oscillator (OPO). Currently, only the squeezed states of lower-order spatial modes can be generated by an OPO. However, the squeezed states of higher-order complex spatial modes are more useful for applications such as quantum metrology, quantum imaging and quantum information. A major challenge for future applications is efficient generation. Here, we use cascaded phase-only spatial light modulators to modulate the amplitude and phase of the incident fundamental mode squeezed state. This efficiently generates a series of squeezed higher-order Hermite-Gauss modes and a squeezed arbitrary complex amplitude distributed mode. The method may yield new applications in biophotonics, quantum metrology and quantum information processing., Comment: 11 pages, 6 figures