Search

The DIarization of SPeaker and LAnguage in Conversational Environments (DISPLACE) 2024 challenge is the second in the series of DISPLACE challenges, which involves tasks of speaker diarization (SD) and language diarization (LD) on a challenging multilingual conversational speech dataset. In the DISPLACE 2024 challenge, we also introduced the task of automatic speech recognition (ASR) on this dataset. The dataset containing 158 hours of speech, consisting of both supervised and unsupervised mono-channel far-field recordings, was released for LD and SD tracks. Further, 12 hours of close-field mono-channel recordings were provided for the ASR track conducted on 5 Indian languages. The details of the dataset, baseline systems and the leader board results are highlighted in this paper. We have also compared our baseline models and the team's performances on evaluation data of DISPLACE-2023 to emphasize the advancements made in this second version of the challenge., Comment: 5 pages, 3 figures, Interspeech 2024

In a code-switched (CS) scenario, the use of spoken language diarization (LD) as a pre-possessing system is essential. Further, the use of implicit frameworks is preferable over the explicit framework, as it can be easily adapted to deal with low/zero resource languages. Inspired by speaker diarization (SD) literature, three frameworks based on (1) fixed segmentation, (2) change point-based segmentation and (3) E2E are proposed to perform LD. The initial exploration with synthetic TTSF-LD dataset shows, using x-vector as implicit language representation with appropriate analysis window length ($N$) can able to achieve at per performance with explicit LD. The best implicit LD performance of $6.38$ in terms of Jaccard error rate (JER) is achieved by using the E2E framework. However, considering the E2E framework the performance of implicit LD degrades to $60.4$ while using with practical Microsoft CS (MSCS) dataset. The difference in performance is mostly due to the distributional difference between the monolingual segment duration of secondary language in the MSCS and TTSF-LD datasets. Moreover, to avoid segment smoothing, the smaller duration of the monolingual segment suggests the use of a small value of $N$. At the same time with small $N$, the x-vector representation is unable to capture the required language discrimination due to the acoustic similarity, as the same speaker is speaking both languages. Therefore, to resolve the issue a self-supervised implicit language representation is proposed in this study. In comparison with the x-vector representation, the proposed representation provides a relative improvement of $63.9\%$ and achieved a JER of $21.8$ using the E2E framework., Comment: Planning to Submit in IEEE-JSTSP

Spoken language diarization (LD) and related tasks are mostly explored using the phonotactic approach. Phonotactic approaches mostly use explicit way of language modeling, hence requiring intermediate phoneme modeling and transcribed data. Alternatively, the ability of deep learning approaches to model temporal dynamics may help for the implicit modeling of language information through deep embedding vectors. Hence this work initially explores the available speaker diarization frameworks that capture speaker information implicitly to perform LD tasks. The performance of the LD system on synthetic code-switch data using the end-to-end x-vector approach is 6.78% and 7.06%, and for practical data is 22.50% and 60.38%, in terms of diarization error rate and Jaccard error rate (JER), respectively. The performance degradation is due to the data imbalance and resolved to some extent by using pre-trained wave2vec embeddings that provide a relative improvement of 30.74% in terms of JER.

Speaker Verification (SV) is a task to verify the claimed identity of the claimant using his/her voice sample. Though there exists an ample amount of research in SV technologies, the development concerning a multilingual conversation is limited. In a country like India, almost all the speakers are polyglot in nature. Consequently, the development of a Multilingual SV (MSV) system on the data collected in the Indian scenario is more challenging. With this motivation, the Indic- Multilingual Speaker Verification (I-MSV) Challenge 2022 has been designed for understanding and comparing the state-of-the-art SV techniques. For the challenge, approximately $100$ hours of data spoken by $100$ speakers has been collected using $5$ different sensors in $13$ Indian languages. The data is divided into development, training, and testing sets and has been made publicly available for further research. The goal of this challenge is to make the SV system robust to language and sensor variations between enrollment and testing. In the challenge, participants were asked to develop the SV system in two scenarios, viz. constrained and unconstrained. The best system in the constrained and unconstrained scenario achieved a performance of $2.12\%$ and $0.26\%$ in terms of Equal Error Rate (EER), respectively.

Spoken language change detection (LCD) refers to identifying the language transitions in a code-switched utterance. Similarly, identifying the speaker transitions in a multispeaker utterance is known as speaker change detection (SCD). Since tasks-wise both are similar, the architecture/framework developed for the SCD task may be suitable for the LCD task. Hence, the aim of the present work is to develop LCD systems inspired by SCD. Initially, both LCD and SCD are performed by humans. The study suggests humans require (a) a larger duration around the change point and (b) language-specific prior exposure, for performing LCD as compared to SCD. The larger duration requirement is incorporated by increasing the analysis window length of the unsupervised distance-based approach. This leads to a relative performance improvement of 29.1% and 2.4%, and a priori language knowledge provides a relative improvement of 31.63% and 14.27% on the synthetic and practical codeswitched datasets, respectively. The performance difference between the practical and synthetic datasets is mostly due to differences in the distribution of the monolingual segment duration.

Spoken language change detection (LCD) refers to detecting language switching points in a multilingual speech signal. Speaker change detection (SCD) refers to locating the speaker change points in a multispeaker speech signal. The objective of this work is to understand the challenges in LCD task by comparing it with SCD task. Human subjective study for change detection is performed for LCD and SCD. This study demonstrates that LCD requires larger duration spectro-temporal information around the change point compared to SCD. Based on this, the work explores automatic distance based and model based LCD approaches. The model based ones include Gaussian mixture model and universal background model (GMM-UBM), attention, and Generative adversarial network (GAN) based approaches. Both the human and automatic LCD tasks infer that the performance of the LCD task improves by incorporating more and more spectro-temporal duration., Comment: The work is substantially modified. The new version of the same will be submitted soon

The automatic recognition of pathological speech, particularly from children with any articulatory impairment, is a challenging task due to various reasons. The lack of available domain specific data is one such obstacle that hinders its usage for different speech-based applications targeting pathological speakers. In line with the challenge, in this work, we investigate a few data augmentation techniques to simulate training data for improving the children speech recognition considering the case of cleft lip and palate (CLP) speech. The augmentation techniques explored in this study, include vocal tract length perturbation (VTLP), reverberation, speaking rate, pitch modification, and speech feature modification using cycle consistent adversarial networks (CycleGAN). Our study finds that the data augmentation methods significantly improve the CLP speech recognition performance, which is more evident when we used feature modification using CycleGAN, VTLP and reverberation based methods. More specifically, the results from this study show that our systems produce an improved phone error rate compared to the systems without data augmentation.

The cleft lip and palate (CLP) speech intelligibility is distorted due to the deformation in their articulatory system. For addressing the same, a few previous works perform phoneme specific modification in CLP speech. In CLP speech, both the articulation error and the nasalization distorts the intelligibility of a word. Consequently, modification of a specific phoneme may not always yield in enhanced entire word-level intelligibility. For such cases, it is important to identify and isolate the phoneme specific error based on the knowledge of acoustic events. Accordingly, the phoneme specific error modification algorithms can be exploited for transforming the specified errors and enhance the word-level intelligibility. Motivated by that, in this work, we combine some of salient phoneme specific enhancement approaches and demonstrate their effectiveness in improving the word-level intelligibility of CLP speech. The enhanced speech samples are evaluated using subjective and objective evaluation metrics.

Smartphones have been employed with biometric-based verification systems to provide security in highly sensitive applications. Audio-visual biometrics are getting popular due to their usability, and also it will be challenging to spoof because of their multimodal nature. In this work, we present an audio-visual smartphone dataset captured in five different recent smartphones. This new dataset contains 103 subjects captured in three different sessions considering the different real-world scenarios. Three different languages are acquired in this dataset to include the problem of language dependency of the speaker recognition systems. These unique characteristics of this dataset will pave the way to implement novel state-of-the-art unimodal or audio-visual speaker recognition systems. We also report the performance of the bench-marked biometric verification systems on our dataset. The robustness of biometric algorithms is evaluated towards multiple dependencies like signal noise, device, language and presentation attacks like replay and synthesized signals with extensive experiments. The obtained results raised many concerns about the generalization properties of state-of-the-art biometrics methods in smartphones.

Sonorant sounds are characterized by regions with prominent formant structure, high energy and high degree of periodicity. In this work, the vocal-tract system, excitation source and suprasegmental features derived from the speech signal are analyzed to measure the sonority information present in each of them. Vocal-tract system information is extracted from the Hilbert envelope of numerator of group delay function. It is derived from zero time windowed speech signal that provides better resolution of the formants. A five-dimensional feature set is computed from the estimated formants to measure the prominence of the spectral peaks. A feature representing strength of excitation is derived from the Hilbert envelope of linear prediction residual, which represents the source information. Correlation of speech over ten consecutive pitch periods is used as the suprasegmental feature representing periodicity information. The combination of evidences from the three different aspects of speech provides better discrimination among different sonorant classes, compared to the baseline MFCC features. The usefulness of the proposed sonority feature is demonstrated in the tasks of phoneme recognition and sonorant classification.

Cleft lip and palate (CLP) refer to a congenital craniofacial condition that causes various speech-related disorders. As a result of structural and functional deformities, the affected subjects' speech intelligibility is significantly degraded, limiting the accessibility and usability of speech-controlled devices. Towards addressing this problem, it is desirable to improve the CLP speech intelligibility. Moreover, it would be useful during speech therapy. In this study, the cycle-consistent adversarial network (CycleGAN) method is exploited for improving CLP speech intelligibility. The model is trained on native Kannada-speaking childrens' speech data. The effectiveness of the proposed approach is also measured using automatic speech recognition performance. Further, subjective evaluation is performed, and those results also confirm the intelligibility improvement in the enhanced speech over the original., Comment: 8 pages, 4 figures, IEEE spoken language and technology workshop

Biometric recognition is a trending technology that uses unique characteristics data to identify or verify/authenticate security applications. Amidst the classically used biometrics, voice and face attributes are the most propitious for prevalent applications in day-to-day life because they are easy to obtain through restrained and user-friendly procedures. The pervasiveness of low-cost audio and face capture sensors in smartphones, laptops, and tablets has made the advantage of voice and face biometrics more exceptional when compared to other biometrics. For many years, acoustic information alone has been a great success in automatic speaker verification applications. Meantime, the last decade or two has also witnessed a remarkable ascent in face recognition technologies. Nonetheless, in adverse unconstrained environments, neither of these techniques achieves optimal performance. Since audio-visual information carries correlated and complementary information, integrating them into one recognition system can increase the system's performance. The vulnerability of biometrics towards presentation attacks and audio-visual data usage for the detection of such attacks is also a hot topic of research. This paper made a comprehensive survey on existing state-of-the-art audio-visual recognition techniques, publicly available databases for benchmarking, and Presentation Attack Detection (PAD) algorithms. Further, a detailed discussion on challenges and open problems is presented in this field of biometrics.

Video captioning is a popular task that challenges models to describe events in videos using natural language. In this work, we investigate the ability of various visual feature representations derived from state-of-the-art convolutional neural networks to capture high-level semantic context. We introduce the Weighted Additive Fusion Transformer with Memory Augmented Encoders (WAFTM), a captioning model that incorporates memory in a transformer encoder and uses a novel method, to fuse features, that ensures due importance is given to more significant representations. We illustrate a gain in performance realized by applying Word-Piece Tokenization and a popular REINFORCE algorithm. Finally, we benchmark our model on two datasets and obtain a CIDEr of 92.4 on MSVD and a METEOR of 0.091 on the ActivityNet Captions Dataset., Comment: 6 pages

The present study focuses on the application of a picosecond (ps) TALIF technique in krypton (Kr) at variable pressure (0.1-10 mbar). The laser intensity (I, units W.cm$^{-2}$) is tuned between 1 and 480 MW.cm$^{-2}$, and the depletion of the density of the Kr 5p'[3/2]$_2$ fluorescing state through photoionization (PIN) and amplified stimulated emission (ASE) is investigated. This is done by combining TALIF experiments with a simple 0D numerical model. We demonstrate that for a gas pressure of 3 mbar and 15 < $\le$ 480 MW.cm$^{-2}$, a saturated fluorescence signal is obtained, which is largely attributed to PIN, ASE being negligible. Also, a broadening of the two-photon absorption line (i.e. 4p$^6$ $^1$S0$\to$5p'[3/2]$_2$) is recorded due to the production of charged species through PIN, inducing a Stark effect. For I$\le$15 MW.cm$^{-2}$, though, PIN is significantly limited, the absorption line is noticeably narrowed, and the quadratic dependence of the TALIF signal intensity versus the laser energy is obtained. Thus, in this case, the investigated Kr TALIF scheme, using the 5p'[3/2]$_2\to$5s[3/2]$_1$ fluorescence channel, can be used for calibration purposes in ps-TALIF experiments. These results are of interest for fundamental research since most ps-TALIF studies performed in Kr do not investigate in detail the role of PIN and ASE on the depletion of the Kr 5p'[3/2]$_2$ state density. Moreover, this work contributes to the development of ps-TALIF for determining absolute densities and quenching coefficients of H and N atoms in plasmas. This is useful in numerous plasma-based applications (e.g. thin film synthesis, biomedical treatments, plasma-assisted combustion, ...), for which the knowledge of the density/kinetics of reactive atoms is essential.

Distinct striation patterns are observed in the spectrograms of speech and music. This motivated us to propose three novel time-frequency features for speech-music classification. These features are extracted in two stages. First, a preset number of prominent spectral peak locations are identified from the spectra of each frame. These important peak locations obtained from each frame are used to form Spectral peak sequences (SPS) for an audio interval. In second stage, these SPS are treated as time series data of frequency locations. The proposed features are extracted as periodicity, average frequency and statistical attributes of these spectral peak sequences. Speech-music categorization is performed by learning binary classifiers on these features. We have experimented with Gaussian mixture models, support vector machine and random forest classifiers. Our proposal is validated on four datasets and benchmarked against three baseline approaches. Experimental results establish the validity of our proposal., Comment: 4 pages, 16 figures

Pumping fluids without an aid of an external power source are desirable in a number of applications ranging from a cooling of microelectronic circuits to Micro Total Analysis Systems (micro-TAS). Although, several microfluidic pumps exist, yet passive micropumps demonstrate better energy efficiency while providing a better control over a pumping rate and its operation. The fluid pumping rate and their easy maneuverability are critical in some applications; therefore, in the current work, we have developed a leaf-mimicking micropump that demonstrated ~6 fold increase in a volumetric pumping rate as compared to the micropumps having a single capillary fluid delivery system. We have discussed a simple, scalable, yet inexpensive method to design and fabricate these leaf mimicking micopump. The microstructure of the micropumps were characterised through scanning electron microscopy and its pumping performance (volumetric pumping rate and pressure head sustainence) were assessed experimentally. The working principle of the proposed micropump is attributed to its structural elements; where branched-shaped microchannels deliver the fluid acting like veins of leaves while the connected microporous support resembles mesophyll cells matrix that instantaneously transfers the delivered fluid by a capillary action to multiple pores mimicking the stomata for evaporation. Such design of micropumps will enable an efficient delivery of the desired volume of a fluid to any 2D/3D micro/nanofluidic devices used in an engineering and biological applications., Comment: 19 pages including 6 figures and 1 table. The The work was presented as an oral presentation in the National conference on Convergence of Pharmaceutical Sciences and Biomedical technology, 2018, National Institute of Pharmaceutical Education and Research, Ahmadabad, India ( 26th March - 28th March, 2018)

Fish gills are one of the most primitive gas/solute exchange organs, having the highest ventilation volume, present in nature. Such performance is attributed to a functional unit of gill - secondary lamella - that can extract oxygen from an ambience even at a very low partial pressure. For centuries, gills have stood as one of the simplest but an elegant gas/solute exchange organs. Although the role of various morphometric factors of fish gills on gas/solute exchange capabilities have been reported, there has been limited understanding on what makes fish gills as an excellent gas/solute exchange system. Therefore, in the current study, we have theoretically studied the variation of few structural and parametric ratios, which were known to have role in gas/solute exchange, with respect to the weight of fishes. Thereafter, modelling and simulation of convection-diffusion transport through a two dimensional model of secondary lamella were carried out to study different factors affecting the performance of gills. The results obtained from both the studies (theoretical and computational) were in good agreement with each other. Thus, our study suggested that fish gills have optimized parametric ratios, at multiple length scales, throughout an evolution to arrive at an organ with enhanced mass transport capabilities. Further, our study also highlighted the role of length of primary and secondary lamella, surface area of secondary lamellae and inter-lamellar distance on gas/solute exchange capabilities of fish gills. Thus, these defined morphological parameters and parametric ratios could be exploited in future to design and develop efficient gas/solute exchange microdevices., Comment: 18 pages, 7 figures, A part of results published as Kumar P, P S Gandhi, M Majumder, Design and fabrication of thin microvascularised polymer matrices inspired from secondary lamellae of fish gills,""Bioinspiration, Biomimetics, and Bioreplication VI", an international conference, SPIE/NDE, (Las Vegas, USA), 9797-40, 2016

Integrated micro-nanochannel networks in fluidic devices are desirable in a number of applications ranging from self-healing/cooling materials to bioengineering. The conventional micro-manufacturing techniques are capable of either producing microchannel or nanochannel networks for a fluidic application but lack proficiency in the production of an integrated micro-nanochannel network with a smooth transition from micro-to-nano scale dimension. In addition, these techniques possess limitations such as heavy initial investment, sophistication in operation and scale-up capabilities. Therefore, the current paper demonstrates the combination of micro/nanotechnologies to design and develop a biomimetic 3-D integrated micro-nanochannel network in PDMS device for solute exchange. We have used 3-D printer, a scalable technology, to design and manufacture micro-mold having fractal-shaped features. Further, electrospinning was used to deposit nanofibrous network on the fractal mold. Subsequent micro-molding with PDMS was used to obtain fractal-shaped microchannels integrated with embedded nanofibers. Henceforth, solvent etching of nanofibers followed by bonding of thin PDMS membrane generated by spin coating to open end of channels leads to the formation of functional microdevices. These PDMS devices mimic the natural vasculature of a living system, where fractal-shaped microchannels will assist in efficient fluid flow and the site of nanovascular network participates in heat/mass transport operations. Further, dye flow propounds the functionality of such devices. Our study hence proposes a simple and scalable hybrid microtechnolgy to fabricate fluidic devices having multiscale architecture. This will also facilitate the rapid fabrication of microfluidic devices for biomedical, diagnostics, sensors and micro-TAS applications., Comment: 5 pages, 5 figures, conference proceedings in 11th International Conference on MicroManufacturing, UC Irvine, 2016

Control of compliant mechanical systems is increasingly being researched for several applications including flexible link robots and ultra-precision positioning systems. The control problem in these systems is challenging, especially with gravity coupling and large deformations, because of inherent underactuation and the combination of lumped and distributed parameters of a nonlinear system. In this paper we consider an ultra-flexible inverted pendulum on a cart and propose a new nonlinear energy shaping controller to keep the pendulum at the upward position with the cart stopped at a desired location. The design is based on a model, obtained via the constrained Lagrange formulation, which previously has been validated experimentally. The controller design consists of a partial feedback linearization step followed by a standard PID controller acting on two passive outputs. Boundedness of all signals and (local) asymptotic stability of the desired equilibrium is theoretically established. Simulations and experimental evidence assess the performance of the proposed controller., Comment: 11 pages, 7 figures, extended version of the NOLCOS 2016 paper