Search

This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. Recently, conditional flow matching (CFM) generative models have been successfully adopted for waveform generation tasks, leveraging a single vector field estimation objective for training. Although these models can generate high-fidelity waveform signals, they require significantly more ODE steps compared to GAN-based models, which only need a single generation step. Additionally, the generated samples often lack high-frequency information due to noisy vector field estimation, which fails to ensure high-frequency reproduction. To address this limitation, we enhance pre-trained CFM-based generative models by incorporating a fixed-step generator modification. We utilized reconstruction losses and adversarial feedback to accelerate high-fidelity waveform generation. Through adversarial flow matching optimization, it only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. Moreover, we significantly reduce inference speed from 16 steps to 2 or 4 steps. Additionally, by scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance, with a perceptual evaluation of speech quality (PESQ) score of 4.454 on the LibriTTS dataset. Audio samples, source code and checkpoints will be available at https://github.com/sh-lee-prml/PeriodWave., Comment: 9 pages, 9 tables, 1 figure

Recently, universal waveform generation tasks have been investigated conditioned on various out-of-distribution scenarios. Although GAN-based methods have shown their strength in fast waveform generation, they are vulnerable to train-inference mismatch scenarios such as two-stage text-to-speech. Meanwhile, diffusion-based models have shown their powerful generative performance in other domains; however, they stay out of the limelight due to slow inference speed in waveform generation tasks. Above all, there is no generator architecture that can explicitly disentangle the natural periodic features of high-resolution waveform signals. In this paper, we propose PeriodWave, a novel universal waveform generation model. First, we introduce a period-aware flow matching estimator that can capture the periodic features of the waveform signal when estimating the vector fields. Additionally, we utilize a multi-period estimator that avoids overlaps to capture different periodic features of waveform signals. Although increasing the number of periods can improve the performance significantly, this requires more computational costs. To reduce this issue, we also propose a single period-conditional universal estimator that can feed-forward parallel by period-wise batch inference. Additionally, we utilize discrete wavelet transform to losslessly disentangle the frequency information of waveform signals for high-frequency modeling, and introduce FreeU to reduce the high-frequency noise for waveform generation. The experimental results demonstrated that our model outperforms the previous models both in Mel-spectrogram reconstruction and text-to-speech tasks. All source code will be available at \url{https://github.com/sh-lee-prml/PeriodWave}., Comment: 24 pages, 16 tables, 4 figures

Networks are ubiquitous in various fields, representing systems where nodes and their interconnections constitute their intricate structures. We introduce a network decomposition scheme to reveal multiscale core-periphery structures lurking inside, using the concept of locally defined nodal hub centrality and edge-pruning techniques built upon it. We demonstrate that the hub-centrality-based edge pruning reveals a series of breaking points in network decomposition, which effectively separates a network into its backbone and shell structures. Our local-edge decomposition method iteratively identifies and removes locally least important nodes, and uncovers an onion-like hierarchical structure as a result. Compared with the conventional $k$-core decomposition method, our method based on relative information residing in local structures exhibits a clear advantage in terms of discovering locally crucial substructures. Furthermore, we introduce the core-periphery score to properly separate the core and periphery with our decomposition scheme. By extending the method combined with the network community structure, we successfully detect multiple core-periphery structures by decomposition inside each community. Moreover, the application of our decomposition to supernode networks defined from the communities reveals the intricate relation between the two representative mesoscale structures., Comment: 10 pages, 8 figures, 1 table

We report on the temporal changes in undergraduate-level physics programs at Korean universities from 1915 to 2023 by analyzing data on physics-related departments and their students using basic statistics and the scaling theory of statistical physics. Our analysis reveals that the number of departments peaked around the turn of the 21st century, and it has been steadily decreasing ever since, with particularly severe declines in private universities located outside the capital region. Besides the change in the overall numbers, we also show the change in the self-identity of physics-related departments reflected in department names, which reveals a recent trend of emphasizing more application-side such as semiconductors and data. As a sophisticated measure to quantify regional imbalances relative to the population eligible for higher education, we present scaling exponents from the scaling theory, which shows a shift from sublinear to linear for departments and a shift from linear to superlinear for students. The result indicates the exacerbation of the regional imbalance of university-level physics education in Korea., Comment: 11 pages, in Korean language, 8 figures, 1 table, in Korean

Despite rapid advances in the field of emotional text-to-speech (TTS), recent studies primarily focus on mimicking the average style of a particular emotion. As a result, the ability to manipulate speech emotion remains constrained to several predefined labels, compromising the ability to reflect the nuanced variations of emotion. In this paper, we propose EmoSphere-TTS, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech. Without any human annotation, we use the arousal, valence, and dominance pseudo-labels to model the complex nature of emotion via a Cartesian-spherical transformation. Furthermore, we propose a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics. The experimental results demonstrate the model ability to control emotional style and intensity with high-quality expressive speech., Comment: Accepted at INTERSPEECH 2024

Understanding the behaviors of ecological systems is challenging given their multi-faceted complexity. To proceed, theoretical models such as Lotka-Volterra dynamics with random interactions have been investigated by the dynamical mean-field theory to provide insights into underlying principles such as how biodiversity and stability depend on the randomness in interaction strength. Yet the fully-connected structure assumed in these previous studies is not realistic as revealed by a vast amount of empirical data. We derive a generic formula for the abundance distribution under an arbitrary distribution of degree, the number of interacting neighbors, which leads to degree-dependent abundance patterns of species. Notably, in contrast to the well-mixed system, the number of surviving species can be reduced as the community becomes cooperative in heterogeneous interaction structures. Our study, therefore, demonstrates that properly taking into account heterogeneity in the interspecific interaction structure is indispensable to understanding the diversity in large ecosystems, and our general theoretical framework can apply to a much wider range of interacting many-body systems.

While renormalization groups are fundamental in physics, renormalization of complex networks remains vague in its conceptual definition and methodology. Here, we propose a novel strategy to renormalize complex networks. Rather than resorting to handling the bare structure of a network, we overlay it with a readily renormalizable physical model, which reflects real-world scenarios with a broad generality. From the renormalization of the overlying system, we extract a rigorous and simple renormalization group transformation of arbitrary networks. In this way, we obtain a transparent, model-dependent physical meaning of the network renormalization, which in our case is a scale transformation preserving the transition dynamics of low-density particles. We define the strength of a node in accordance with the physical model and trace the change of its distribution under our renormalization process. This analysis demonstrates that the strength distributions of scale-free networks remain scale-invariant, whereas those of homogeneous random networks do not., Comment: 5 pages, 4 figures

Although there has been significant advancement in the field of speech-to-speech translation, conventional models still require language-parallel speech data between the source and target languages for training. In this paper, we introduce TranSentence, a novel speech-to-speech translation without language-parallel speech data. To achieve this, we first adopt a language-agnostic sentence-level speech encoding that captures the semantic information of speech, irrespective of language. We then train our model to generate speech based on the encoded embedding obtained from a language-agnostic sentence-level speech encoder that is pre-trained with various languages. With this method, despite training exclusively on the target language's monolingual data, we can generate target language speech in the inference stage using language-agnostic speech embedding from the source language speech. Furthermore, we extend TranSentence to multilingual speech-to-speech translation. The experimental results demonstrate that TranSentence is superior to other models., Comment: Accepted by ICASSP 2024

Emotional voice conversion involves modifying the pitch, spectral envelope, and other acoustic characteristics of speech to match a desired emotional state while maintaining the speaker's identity. Recent advances in EVC involve simultaneously modeling pitch and duration by exploiting the potential of sequence-to-sequence models. In this study, we focus on parallel speech generation to increase the reliability and efficiency of conversion. We introduce a duration-flexible EVC (DurFlex-EVC) that integrates a style autoencoder and a unit aligner. The previous variable-duration parallel generation model required text-to-speech alignment. We consider self-supervised model representation and discrete speech units to be the core of our parallel generation. The style autoencoder promotes content style disentanglement by separating the source style of the input features and applying them with the target style. The unit aligner encodes unit-level features by modeling emotional context. Furthermore, we enhance the style of the features with a hierarchical stylize encoder and generate high-quality Mel-spectrograms with a diffusion-based generator. The effectiveness of the approach has been validated through subjective and objective evaluations and has been demonstrated to be superior to baseline models., Comment: 14 pages, 11 figures, 12 tables

Large language models (LLM)-based speech synthesis has been widely adopted in zero-shot speech synthesis. However, they require a large-scale data and possess the same limitations as previous autoregressive speech models, including slow inference speed and lack of robustness. This paper proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis frameworks could significantly improve the robustness and expressiveness of the synthetic speech. Furthermore, we significantly improve the naturalness and speaker similarity of synthetic speech even in zero-shot speech synthesis scenarios. For text-to-speech, we adopt the text-to-vec framework, which generates a self-supervised speech representation and an F0 representation based on text representations and prosody prompts. Then, HierSpeech++ generates speech from the generated vector, F0, and voice prompt. We further introduce a high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The experimental results demonstrated that the hierarchical variational autoencoder could be a strong zero-shot speech synthesizer given that it outperforms LLM-based and diffusion-based models. Moreover, we achieved the first human-level quality zero-shot speech synthesis. Audio samples and source code are available at https://github.com/sh-lee-prml/HierSpeechpp., Comment: 16 pages, 9 figures, 12 tables

Although voice conversion (VC) systems have shown a remarkable ability to transfer voice style, existing methods still have an inaccurate pitch and low speaker adaptation quality. To address these challenges, we introduce Diff-HierVC, a hierarchical VC system based on two diffusion models. We first introduce DiffPitch, which can effectively generate F0 with the target voice style. Subsequently, the generated F0 is fed to DiffVoice to convert the speech with a target voice style. Furthermore, using the source-filter encoder, we disentangle the speech and use the converted Mel-spectrogram as a data-driven prior in DiffVoice to improve the voice style transfer capacity. Finally, by using the masked prior in diffusion models, our model can improve the speaker adaptation quality. Experimental results verify the superiority of our model in pitch generation and voice style transfer performance, and our model also achieves a CER of 0.83% and EER of 3.29% in zero-shot VC scenarios., Comment: INTERSPEECH 2023 (Oral)

We address the relatively less known facts on the equivalence and technical realizations surrounding two network models showing the "small-world" property, namely the Newman-Watts and the Harary models. We provide the most accurate (in terms of faithfulness to the original literature) versions of these models to clarify the deviation from them existing in their variants adopted in one of the most popular network analysis packages. The difference in technical realizations of those models could be conceived as minor details, but we discover significantly notable changes caused by the possibly inadvertent modification. For the Harary model, the stochasticity in the original formulation allows a much wider range of the clustering coefficient and the average shortest path length. For the Newman-Watts model, due to the drastically different degree distributions, the clustering coefficient can also be affected, which is verified by our higher-order analytic derivation. During the process, we discover the equivalence of the Newman-Watts (better known in the network science or physics community) and the Harary (better known in the graph theory or mathematics community) models under a specific condition of restricted parity in variables, which would bridge the two relatively independently developed models in different fields. Our result highlights the importance of each detailed step in constructing network models and the possibility of deeply related models, even if they might initially appear distinct in terms of the time period or the academic disciplines from which they emerged., Comment: 11 pages, 5 figures, 1 table

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to predict the quantized prosody vector; however, it suffers from the issues of long-term dependency and slow inference. This study proposes a novel approach called DiffProsody in which expressive speech is synthesized using a diffusion-based latent prosody generator and prosody conditional adversarial training. Our findings confirm the effectiveness of our prosody generator in generating a prosody vector. Furthermore, our prosody conditional discriminator significantly improves the quality of the generated speech by accurately emulating prosody. We use denoising diffusion generative adversarial networks to improve the prosody generation speed. Consequently, DiffProsody is capable of generating prosody 16 times faster than the conventional diffusion model. The superior performance of our proposed method has been demonstrated via experiments., Comment: 10 pages, 8 figures, 5 tables, under review

Despite rapid progress in the voice style transfer (VST) field, recent zero-shot VST systems still lack the ability to transfer the voice style of a novel speaker. In this paper, we present HierVST, a hierarchical adaptive end-to-end zero-shot VST model. Without any text transcripts, we only use the speech dataset to train the model by utilizing hierarchical variational inference and self-supervised representation. In addition, we adopt a hierarchical adaptive generator that generates the pitch representation and waveform audio sequentially. Moreover, we utilize unconditional generation to improve the speaker-relative acoustic capacity in the acoustic representation. With a hierarchical adaptive structure, the model can adapt to a novel voice style and convert speech progressively. The experimental results demonstrate that our method outperforms other VST models in zero-shot VST scenarios. Audio samples are available at \url{https://hiervst.github.io/}., Comment: INTERSPEECH 2023 (Oral)

Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https://jisang93.github.io/pausespeech-demo/., Comment: 13 pages, 4 figures, 3 tables, under reivew

Recently, denoising diffusion models have demonstrated remarkable performance among generative models in various domains. However, in the speech domain, the application of diffusion models for synthesizing time-varying audio faces limitations in terms of complexity and controllability, as speech synthesis requires very high-dimensional samples with long-term acoustic features. To alleviate the challenges posed by model complexity in singing voice synthesis, we propose HiddenSinger, a high-quality singing voice synthesis system using a neural audio codec and latent diffusion models. To ensure high-fidelity audio, we introduce an audio autoencoder that can encode audio into an audio codec as a compressed representation and reconstruct the high-fidelity audio from the low-dimensional compressed latent vector. Subsequently, we use the latent diffusion models to sample a latent representation from a musical score. In addition, our proposed model is extended to an unsupervised singing voice learning framework, HiddenSinger-U, to train the model using an unlabeled singing voice dataset. Experimental results demonstrate that our model outperforms previous models in terms of audio quality. Furthermore, the HiddenSinger-U can synthesize high-quality singing voices of speakers trained solely on unlabeled data., Comment: 11 pages, 5 figures, 5 tables, under review

Diffusion-based generative models have exhibited powerful generative performance in recent years. However, as many attributes exist in the data distribution and owing to several limitations of sharing the model parameters across all levels of the generation process, it remains challenging to control specific styles for each attribute. To address the above problem, this paper presents decoupled denoising diffusion models (DDDMs) with disentangled representations, which can control the style for each attribute in generative models. We apply DDDMs to voice conversion (VC) tasks to address the challenges of disentangling and controlling each speech attribute (e.g., linguistic information, intonation, and timbre). First, we use a self-supervised representation to disentangle the speech representation. Subsequently, the DDDMs are applied to resynthesize the speech from the disentangled representations for denoising with respect to each attribute. Moreover, we also propose the prior mixup for robust voice style transfer, which uses the converted representation of the mixed style as a prior distribution for the diffusion models. The experimental results reveal that our method outperforms publicly available VC models. Furthermore, we show that our method provides robust generative performance regardless of the model size. Audio samples are available https://hayeong0.github.io/DDDM-VC-demo/., Comment: 23 pages, 10 figures, 17 tables, under review

Researchers developed chromosome capture methods such as Hi-C to better understand DNA's 3D folding in nuclei. The Hi-C method captures contact frequencies between DNA segment pairs across the genome. When analyzing Hi-C data sets, it is common to group these pairs using standard bioinformatics methods (e.g., PCA). Other approaches handle Hi-C data as weighted networks, where connected node represent DNA segments in 3D proximity. In this representation, one can leverage community detection techniques developed in complex network theory to group nodes into mesoscale communities containing similar connection patterns. While there are several successful attempts to analyze Hi-C data in this way, it is common to report and study the most typical community structure. But in reality, there are often several valid candidates. Therefore, depending on algorithm design, different community detection methods focusing on slightly different connectivity features may have differing views on the ideal node groupings. In fact, even the same community detection method may yield different results if using a stochastic algorithm. This ambiguity is fundamental to community detection and shared by most complex networks whenever interactions span all scales in the network. This is known as community inconsistency. This paper explores this inconsistency of 3D communities in Hi-C data for all human chromosomes. We base our analysis on two inconsistency metrics, one local and one global, and quantify the network scales where the community separation is most variable. For example, we find that TADs are less reliable than A/B compartments and that nodes with highly variable node-community memberships are associated with open chromatin. Overall, our study provides a helpful framework for data-driven researchers and increases awareness of some inherent challenges when clustering Hi-C data into 3D communities., Comment: 10 pages, 7 figures

Perturbations made to networked systems may result in partial structural loss, such as a blackout in a power-grid system. Investigating the resultant disturbance in network properties is quintessential to understand real networks in action. The removal of nodes is a representative disturbance, but previous studies are seemingly contrasting about its effect on arguably the most fundamental network statistic, the degree distribution. The key question is about the functional form of the degree distributions that can be altered during node removal or sampling, which is decisive in the remaining subnetwork's static and dynamical properties. In this work, we clarify the situation by utilizing the relative entropies with respect to the reference distributions in the Poisson and power-law form. Introducing general sequential node removal processes with continuously different levels of hub protection to encompass a series of scenarios including random removal and preferred or protective removal of the hub, we classify the altered degree distributions starting from various power-law forms by comparing two relative entropy values. From the extensive investigation in various scenarios based on direct node-removal simulations and by solving the rate equation of degree distributions, we discover in the parameter space two distinct regimes, one where the degree distribution is closer to the power-law reference distribution and the other closer to the Poisson distribution., Comment: 12 pages, 7 figures

Using {\it ab initio} approaches for extended Hubbard interactions coupled to phonons, we reveal that the intersite Coulomb interaction plays important roles in determining various distinctive phases of the paradigmatic charge ordered materials of Ba$_{1-x}$K$_x A$O$_3$ ($A=$ Bi and Sb). We demonstrated that all their salient doping dependent experiment features such as breathing instabilities, anomalous phonon dispersions, and transition between charge-density wave and superconducting states can be accounted very well if self-consistently obtained nearest neighbor Hubbard interaction are included, thus establishing a minimal criterion for reliable descriptions of spontaneous charge orders in solids., Comment: 4 pages, 2 additional pages for references and 4 pages supplementary materials, title and abstract are modified

Uncovering structural properties of ecological networks is a crucial starting point of studying the system's stability in response to various types of perturbations. We analyze pollination and seed disposal networks, which are representative examples of mutualistic networks in ecosystems, in various scales. In particular, we examine mesoscale properties such as the nested structure, the core-periphery structure, and the community structure by statistically investigating their interrelationships with real network data. As a result of community detection in different scales, we find the absence of meaningful hierarchy between networks, and the negative correlation between the modularity and the two other structures (nestedness and core-periphery-ness), which themselves are highly positively correlated. In addition, no characteristic scale of communities is perceivable from the community-inconsistency analysis. Therefore, the community structures, which are most widely studied mesoscale structures of networks, are not in fact adequate to characterize the mutualistic networks of this scale in ecosystems., Comment: 9 pages, 4 figures, in Korean

Bipartite networks composed of dichotomous node sets are ubiquitous in nature and society. Partly for simplicity's sake, many studies have focused on their projection onto their unipartite versions where one only needs to care about a single type of node. When it comes to mesoscale structures such as communities, however, properly incorporating a priori structural restrictions such as bipartivity is ever more important. In this paper, as a case study, we take the community structure of bipartite networks in various scales to examine the amount of information of bipartivity encoded in the community detection procedure. In particular, we report the robustness in reliability of detected community based on consistency by comparing the detection algorithm with or without the consideration of bipartivity. From the analysis with model networks embedding prescribed communities and real networks, we find that the community detection tailored to take the bipartivity into account clearly yields more robust community structures than the one without such structural information. This demonstrates the necessity for customizing the community detection algorithm by encoding whatever information is known about networks of interest and, at the same time, raises an interesting question on the possibility of estimating the quantitative amount of information from such a customization., Comment: 8 pages, 7 figures