Search

Perceiving and understanding non-speech sounds and non-verbal speech is essential to making decisions that help us interact with our surroundings. In this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We build GAMA by integrating an LLM with multiple types of audio representations, including features from a custom Audio Q-Former, a multi-layer aggregator that aggregates features from multiple layers of an audio encoder. We fine-tune GAMA on a large-scale audio-language dataset, which augments it with audio understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for Complex Audio Reasoning), a synthetically generated instruction-tuning (IT) dataset with instructions that require the model to perform complex reasoning on the input audio. We instruction-tune GAMA with CompA-R to endow it with complex reasoning abilities, where we further add a soft prompt as input with high-level semantic evidence by leveraging event tags of the input audio. Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for evaluating the capabilities of LALMs on open-ended audio question-answering that requires complex reasoning. Through automated and expert human evaluations, we show that GAMA outperforms all other LALMs in literature on diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on CompA-R proves to be superior in its complex reasoning and instruction following capabilities., Comment: Project Website: https://sreyan88.github.io/gamaaudio/

Cultural variation exists between nations (e.g., the United States vs. China), but also within regions (e.g., California vs. Texas, Los Angeles vs. San Francisco). Measuring this regional cultural variation can illuminate how and why people think and behave differently. Historically, it has been difficult to computationally model cultural variation due to a lack of training data and scalability constraints. In this work, we introduce a new research problem for the NLP community: How do we measure variation in cultural constructs across regions using language? We then provide a scalable solution: building knowledge-guided lexica to model cultural variation, encouraging future work at the intersection of NLP and cultural understanding. We also highlight modern LLMs' failure to measure cultural variation or generate culturally varied language., Comment: Accepted at NAACL 2024

A site-specific radio channel representation (SSCR) takes the surroundings of the communication system into account by considering the environment geometry, including buildings, vegetation, and mobile objects with their material and surface properties. We present methods for an SSCR that is spatially consistent, such that mobile transmitter and receiver cause a correlated time-varying channel impulse response and closely spaced antennas are correctly correlated. An SSCR is composed of a dynamically varying number of multipath components solely defined by the environment geometry and the material of the environmental objects. Hence, the environment geometry is the only natural scenario parameterization and specific calibration procedures shall be avoided. 5G and 6G physical layer technologies are increasingly able to exploit the properties of a wide range of environments from dense urban areas to railways, road transportation, industrial automation, and unmanned aerial vehicles. The channel impulse response in this wide range of scenarios has generally non-stationary statistical properties, i.e., the Doppler spectrum, power delay profile, K-factor and spatial correlation are all spatially variant (or time-variant for mobile receivers). SSCRs will enable research and development of emerging 5G and 6G technologies such as distributed multiple-input multiple-output systems, reconfigurable intelligent surfaces, multi-band communication, and joint communication and sensing. We highlight the state of the art and summarize research directions for future work towards an SSCR., Comment: 9 pages, 6 figures, to appear in IEEE Communication Magazine

The Facial Action Coding System (FACS) for studying facial expressions is manual and requires significant effort and expertise. This paper explores the use of automated techniques to generate Action Units (AUs) for studying facial expressions. We propose an unsupervised approach based on Principal Component Analysis (PCA) and facial keypoint tracking to generate data-driven AUs called PCA AUs using the publicly available DISFA dataset. The PCA AUs comply with the direction of facial muscle movements and are capable of explaining over 92.83 percent of the variance in other public test datasets (BP4D-Spontaneous and CK+), indicating their capability to generalize facial expressions. The PCA AUs are also comparable to a keypoint-based equivalence of FACS AUs in terms of variance explained on the test datasets. In conclusion, our research demonstrates the potential of automated techniques to be an alternative to manual FACS labeling which could lead to efficient real-time analysis of facial expressions in psychology and related fields. To promote further research, we have made code repository publicly available., Comment: This preprint has not undergone peer review or any post-submission improvements or corrections. The Version of Record of this contribution is published in [LNCS,volume 14301], and is available online at https://doi.org/10.1007/978-3-031-45170-6_85

Let $G$ be a simple graph with associated diagonal matrix of vertex degrees $D(G)$, adjacency matrix $A(G)$, Laplacian matrix $L(G)$ and signless Laplacian matrix $Q(G)$. Recently, Nikiforov proposed the family of matrices $A_\alpha(G)$ defined for any real $\alpha\in [0,1]$ as $A_\alpha(G):=\alpha\,D(G)+(1-\alpha)\,A(G)$, and also mentioned that the matrices $A_\alpha(G)$ can underpin a unified theory of $A(G)$ and $Q(G)$. Inspired from the above definition, we introduce the $B_\alpha$-matrix of $G$, $B_\alpha(G):=\alpha A(G)+(1-\alpha)L(G)$ for $\alpha\in [0,1]$. Note that $ L(G)=B_0(G), D(G)=2B_{\frac{1}{2}}(G), Q(G)=3B_{\frac{2}{3}}(G), A(G)=B_1(G)$. In this article, we study several spectral properties of $ B_\alpha $-matrices to unify the theories of adjacency, Laplacian, and signless Laplacian matrices of graphs. In particular, we prove that each eigenvalue of $ B_\alpha(G) $ is continuous on $ \alpha $. Using this, we characterize positive semidefinite $ B_\alpha $-matrices in terms of $\alpha$. As a consequence, we provide an upper bound of the independence number of $ G $. Besides, we establish some bounds for the largest and the smallest eigenvalues of $B_\alpha(G)$. As a result, we obtain a bound for the chromatic number of $G$ and deduce several known results. In addition, we present a Sachs-type result for the characteristic polynomial of a $ B_\alpha $-matrix., Comment: The final version of the article to be appear in Ars Mathematica Contemporanea

Monolayer molybdenum disulfide ($\mathrm{MoS_2}$) under strain has many interesting properties and possible applications in technology. A recent experimental study examined the effect of strain on the bandgap of monolayer $\mathrm{MoS_2}$ on a mildly curved graphite surface, reporting that under biaxial strain with a Poisson's ratio of 0.44, the bandgap decreases at a rate of 400 meV/\% strain. In this work, we performed density functional theory (DFT) calculations for a free-standing $\mathrm{MoS_2}$ monolayer, using the generalized gradient approximation (GGA) PBE, the hybrid functional HSE06, and many-body perturbation theory with the GW approximation using PBE wavefunctions (G0W0@PBE). For the unstrained monolayer, we found a standard level of agreement for the bandgap between theory and experiment. For biaxial strain at the experimental Poisson's ratio, we found that the bandgap decreases at rates of 63 meV/\% strain (PBE), 73 meV/\% strain (HSE06), and 43 meV/\% strain (G0W0@PBE), which are significantly smaller than the experimental rate. We also found that PBE predicts a similarly smaller rate (90 meV/\% strain) for a different Poisson's ratio of 0.25. Spin-orbit correction (SOC) has little effect on the gap or its strain dependence. The strong disagreement between theory and experiment may reflect an unexpectedly strong effect of the substrate on the strain dependence of the gap. Additionally, we observed a transition from a direct to an indirect bandgap under strain, and (under an equal biaxial strain of 10\%) a semiconductor-to-metal transition, consistent with previous theoretical work.

While coreference resolution is a well-established research area in Natural Language Processing (NLP), research focusing on Thai language remains limited due to the lack of large annotated corpora. In this work, we introduce ThaiCoref, a dataset for Thai coreference resolution. Our dataset comprises 777,271 tokens, 44,082 mentions and 10,429 entities across four text genres: university essays, newspapers, speeches, and Wikipedia. Our annotation scheme is built upon the OntoNotes benchmark with adjustments to address Thai-specific phenomena. Utilizing ThaiCoref, we train models employing a multilingual encoder and cross-lingual transfer techniques, achieving a best F1 score of 67.88\% on the test set. Error analysis reveals challenges posed by Thai's unique linguistic features. To benefit the NLP community, we make the dataset and the model publicly available at http://www.github.com/nlp-chula/thai-coref .

The development of existing facial coding systems, such as the Facial Action Coding System (FACS), relied on manual examination of facial expression videos for defining Action Units (AUs). To overcome the labor-intensive nature of this process, we propose the unsupervised learning of an automated facial coding system by leveraging computer-vision-based facial keypoint tracking. In this novel facial coding system called the Data-driven Facial Expression Coding System (DFECS), the AUs are estimated by applying dimensionality reduction to facial keypoint movements from a neutral frame through a proposed Full Face Model (FFM). FFM employs a two-level decomposition using advanced dimensionality reduction techniques such as dictionary learning (DL) and non-negative matrix factorization (NMF). These techniques enhance the interpretability of AUs by introducing constraints such as sparsity and positivity to the encoding matrix. Results show that DFECS AUs estimated from the DISFA dataset can account for an average variance of up to 91.29 percent in test datasets (CK+ and BP4D-Spontaneous) and also surpass the variance explained by keypoint-based equivalents of FACS AUs in these datasets. Additionally, 87.5 percent of DFECS AUs are interpretable, i.e., align with the direction of facial muscle movements. In summary, advancements in automated facial coding systems can accelerate facial expression analysis across diverse fields such as security, healthcare, and entertainment. These advancements offer numerous benefits, including enhanced detection of abnormal behavior, improved pain analysis in healthcare settings, and enriched emotion-driven interactions. To facilitate further research, the code repository of DFECS has been made publicly accessible.

In this work, we present a heretofore unseen application of Ising machines to perform trust region-based optimisation with box constraints. This is done by considering a specific form of opto-electronic oscillator-based coherent Ising machines with clipped transfer functions, and proposing appropriate modifications to facilitate trust-region optimisation. The enhancements include the inclusion of non-symmetric coupling and linear terms, modulation of noise, and compatibility with convex-projections to improve its convergence. The convergence of the modified Ising machine has been shown under the reasonable assumptions of convexity or invexity. The mathematical structures of the modified Ising machine and trust-region methods have been exploited to design a new trust-region method to effectively solve unconstrained optimisation problems in many scenarios, such as machine learning and optimisation of parameters in variational quantum algorithms. Hence, the proposition is useful for both classical and quantum-classical hybrid scenarios. Finally, the convergence of the Ising machine-based trust-region method, has also been proven analytically, establishing the feasibility of the technique., Comment: This is a first draft; proofs of the lemmas, theorems, and corollaries herein will be included in the next version, along with experimental results. Reviews, comments, and discussions are welcome

Voice conversion (VC) research traditionally depends on scripted or acted speech, which lacks the natural spontaneity of real-life conversations. While natural speech data is limited for VC, our study focuses on filling in this gap. We introduce a novel data-sourcing pipeline that makes the release of a natural speech dataset for VC, named NaturalVoices. The pipeline extracts rich information in speech such as emotion and signal-to-noise ratio (SNR) from raw podcast data, utilizing recent deep learning methods and providing flexibility and ease of use. NaturalVoices marks a large-scale, spontaneous, expressive, and emotional speech dataset, comprising over 3,800 hours speech sourced from the original podcasts in the MSP-Podcast dataset. Objective and subjective evaluations demonstrate the effectiveness of using our pipeline for providing natural and expressive data for VC, suggesting the potential of NaturalVoices for broader speech generation tasks.

Community and core-periphery are two widely studied graph structures, with their coexistence observed in real-world graphs (Rombach, Porter, Fowler \& Mucha [SIAM J. App. Math. 2014, SIAM Review 2017]). However, the nature of this coexistence is not well understood and has been pointed out as an open problem (Yanchenko \& Sengupta [Statistics Surveys, 2023]). Especially, the impact of inferring the core-periphery structure of a graph on understanding its community structure is not well utilized. In this direction, we introduce a novel quantification for graphs with ground truth communities, where each community has a densely connected part (the core), and the rest is more sparse (the periphery), with inter-community edges more frequent between the peripheries. Built on this structure, we propose a new algorithmic concept that we call relative centrality to detect the cores. We observe that core-detection algorithms based on popular centrality measures such as PageRank and degree centrality can show some bias in their outcome by selecting very few vertices from some cores. We show that relative centrality solves this bias issue and provide theoretical and simulation support, as well as experiments on real-world graphs. Core detection is known to have important applications with respect to core-periphery structures. In our model, we show a new application: relative-centrality-based algorithms can select a subset of the vertices such that it contains sufficient vertices from all communities, and points in this subset are better separable into their respective communities. We apply the methods to 11 biological datasets, with our methods resulting in a more balanced selection of vertices from all communities such that clustering algorithms have better performance on this set.

$ $Large language models (LLMs) have become an increasingly important workload for cloud providers catering to both enterprise and consumer applications. LLM inference requests from these applications have end-to-end latency SLOs that must be adhered to in production settings. However, existing LLM serving systems focus on optimization objectives such as request serving throughput or request execution latency rather than the end-to-end latency SLOs. Achieving end-to-end SLOs for latency-sensitive requests is challenging due to head-of-line (HOL) blocking in the request queue, which results from bursty arrival rates and insufficient resources. To address the above challenge, we propose QLM, a multi-model queue management framework for LLM serving. QLM uses stochastic programming to orchestrate the actions of multiple LLM Serving Operations (LSOs) to reduce HOL blocking and maximize SLO attainment. Specifically, QLM uses the following LSOs: model swapping, request eviction, GPU-CPU state swapping, load balancing, and warm model start. Evaluation on heterogeneous GPU devices and models with real-world LLM serving dataset shows that QLM improves SLO attainment by 40-90% and throughput by 20-400% while maintaining or improving device utilization compared to other state-of-the-art LLM serving systems.

The morphology of the 21-cm signal emitted by the neutral hydrogen present in the intergalactic medium (IGM) during the Epoch of Reionization (EoR) depends both on the properties of the sources of ionizing radiation and on the underlying physical processes within the IGM. Variation in the morphology of the IGM 21-cm signal due to the different sources of the EoR is expected to have a significant impact on the 21-cm bispectrum, which is one of the crucial observable statistics that can evaluate the non-Gaussianity present in the signal and which can be estimated from radio interferometric observations of the EoR. Here we present the 21-cm bispectrum for different reionization scenarios assuming different simulated models for the sources of reionization. We also demonstrate how well the 21-cm bispectrum can distinguish between different IGM 21-cm signal morphologies, arising due to the differences in the reionization scenarios, which will help us shed light on the nature of the sources of ionizing photons. Our estimated large-scale bispectrum for all unique $k$-triangle shapes shows a significant difference in the magnitude and sign across different reionization scenarios. Additionally, our focused analysis of bispectrum for a few specific $k$-triangle shapes (e.g. squeezed-limit, linear, and shapes in the vicinity of the squeezed-limit) shows that the large scale 21-cm bispectrum can distinguish between reionization scenarios that show inside-out, outside-in and a combination of inside-out and outside-in morphologies. These results highlight the potential of using the 21-cm bispectrum for constraining different reionization scenarios., Comment: 30 pages, 7 figures, comments are welcome, accepted in JCAP

Recent advances in style transfer text-to-speech (TTS) have improved the expressiveness of synthesized speech. However, encoding stylistic information (e.g., timbre, emotion, and prosody) from diverse and unseen reference speech remains a challenge. This paper introduces StyleMoE, an approach that addresses the issue of learning averaged style representations in the style encoder by creating style experts that learn from subsets of data. The proposed method replaces the style encoder in a TTS framework with a Mixture of Experts (MoE) layer. The style experts specialize by learning from subsets of reference speech routed to them by the gating network, enabling them to handle different aspects of the style space. As a result, StyleMoE improves the style coverage of the style encoder for style transfer TTS. Our experiments, both objective and subjective, demonstrate improved style transfer for diverse and unseen reference speech. The proposed method enhances the performance of existing state-of-the-art style transfer TTS models and represents the first study of style MoE in TTS., Comment: Published in Audio Imagination: NeurIPS 2024 Workshop

Spatiotemporal data is prevalent in a wide range of edge devices, such as those used in personal communication and financial transactions. Recent advancements have sparked a growing interest in integrating spatiotemporal analysis with large-scale language models. However, spatiotemporal data often contains sensitive information, making it unsuitable for open third-party access. To address this challenge, we propose a Graph-GAN-based model for generating privacy-protected spatiotemporal data. Our approach incorporates spatial and temporal attention blocks in the discriminator and a spatiotemporal deconvolution structure in the generator. These enhancements enable efficient training under Gaussian noise to achieve differential privacy. Extensive experiments conducted on three real-world spatiotemporal datasets validate the efficacy of our model. Our method provides a privacy guarantee while maintaining the data utility. The prediction model trained on our generated data maintains a competitive performance compared to the model trained on the original data.

Modular data centers (MDCs) that can be placed right at the energy farms and powered mostly by renewable energy, are proven to be a flexible and effective approach to lowering the carbon footprint of data centers. However, the main challenge of using renewable energy is the high variability of power produced, which implies large volatility in powering computing resources at MDCs, and degraded application performance due to the task evictions and migrations. This causes challenges for platform operators to decide the MDC deployment. To this end, we present SkyBox, a framework that employs a holistic and learning-based approach for platform operators to explore the efficient use of renewable energy with MDC deployment across geographical regions. SkyBox is driven by the insights based on our study of real-world power traces from a variety of renewable energy farms -- the predictable production of renewable energy and the complementary nature of energy production patterns across different renewable energy sources and locations. With these insights, SkyBox first uses the coefficient of variation metric to select the qualified renewable farms, and proposes a subgraph identification algorithm to identify a set of farms with complementary energy production patterns. After that, SkyBox enables smart workload placement and migrations to further tolerate the power variability. Our experiments with real power traces and datacenter workloads show that SkyBox has the lowest carbon emissions in comparison with current MDC deployment approaches. SkyBox also minimizes the impact of the power variability on cloud virtual machines, enabling rMDCs a practical solution of efficiently using renewable energy.

We study the importance of precise modelling of the photometric redshift error distributions on the estimation of parameters from cross correlation measurements and present a working example of the scattering matrix formalism to correct for the redshift bin mismatch of objects in tomographic cross correlation analysis. We measured the angular galaxy auto-power spectrum and cross-power spectrum in four tomographic bins with redshift intervals $z = [0.0,0.3,0.45,0.6,0.8]$ from the cross correlation of Planck Cosmic Microwave Background lensing potential and photometric galaxy catalogue from the Dark Energy Spectroscopic Instrument Legacy Imaging Survey Data Release 8. We estimated galaxy linear bias and amplitude of cross correlation using maximum likelihood estimation to put constraints on the $\sigma_{8}$ parameter. We show that the modified Lorentzian function used to fit the photometric redshift error distribution performs well only near the peaks of the distribution. We adopt a sum of Gaussians model to capture the broad tails of the error distribution. Our sum of Gaussians model yields $\sim 2-5\,\sigma$ smaller values of cross correlation amplitude compared to the $\Lambda$CDM expectations. We compute the $\sigma_{8}$ parameter after correcting for the redshift bin mismatch of objects following the scattering matrix approach. The $\sigma_{8}$ parameter becomes consistent with $\Lambda$CDM model in the last tomographic bin but shows $\sim 1-3\,\sigma$ tension in other redshift bins., Comment: 20 pages, 17 figures, 4 tables, published in Astronomy and Astrophysics

During epithelial-mesenchymal transition (EMT), epithelial cells change their morphology, disperse, and gain mesenchymal-like characteristics. Usually, cells are categorized into discrete cell types or states based on gene expression and other cellular features. Subsequently, EMT is investigated as a dynamical process where cells jump from one discrete state to another. In the current work, we moved away from this idea of discrete state transition and investigated EMT dynamics in a continuous phenotypic space. We used morphology to define the phenotype of a cell. We used the data from quantitative image analysis of MDA-MB-468 cells undergoing EGF-induced EMT. We defined the morphological state space or 'morphospace' using the morphological features extracted through image analysis. During EMT, as the morphology changed, the distribution of cells in the morphospace also changed. However, this morphospace had a very high dimension. We reduced it to a 2-dimensional "reduced morphospace" and investigated the temporal change in the spatial distribution of cells in this reduced space. We used proper orthogonal decomposition to find dominant dynamical features of this spatio-temporal data. The modal analysis detected key features of EMT in this experimental system - reversible transition, distinct paths of phenotypic transition during induction and reversal of EMT, and enhanced diversity of cells during reversal of EMT. We also provide some intuitive physical meaning of the spatial modes and connect them to the key molecular event during EMT., Comment: 33 pages, 8 figures

We show with Gaia XP spectroscopy that extremely metal-rich stars in the Milky Way (EMR; $[M/H]_{XP} > 0.5$) - but only those - are largely confined to a tight "knot" at the center of the Galaxy. This EMR knot is round in projection, has a fairly abrupt edge near $\sim 1.5$kpc, and is a dynamically hot system. This central knot also contains very metal-rich (VMR; $+0.2\le [M/H]_{XP} \le +0.4$) stars. However, in contrast to EMR stars, the bulk of VMR stars form an extended, highly flattened distribution in the inner Galaxy ($R_{\mathrm{GC}}\lesssim 5$ kpc). We draw on TNG50 simulations of Milky Way analogs for context and find that compact, metal-rich knots confined to $<1.5$kpc are a universal feature. In typical simulated analogs, the top 5-10% most metal-rich stars are confined to a central knot; however, in our Milky Way data this fraction is only 0.1%. Dust-penetrating wide-area near-infrared spectroscopy, such as SDSS-V, will be needed for a rigorous estimate of the fraction of stars in the Galactic EMR knot. Why in our Milky Way only EMR giants are confined to such a central knot remains to be explained. Remarkably, the central few kiloparsecs of the Milky Way harbor both the highest concentration of metal-poor stars (the `poor old heart') and almost all EMR stars. This highlights the stellar population diversity at the bottom of galactic potential wells., Comment: 11 pages, 7 figures, submitted to ApJ

The infall of the Large Magellanic Cloud (LMC) is predicted to displace the inner Milky Way (MW), imprinting an apparent 'reflex motion' on the observed velocities of distant halo stars. We construct the largest all-sky spectroscopic dataset of luminous red giant stars from $50-160$ kpc, including a new survey of the southern celestial hemisphere. We fit the full 6D kinematics of our data to measure the amplitude and direction of the inner MW's motion towards the outer halo. The observed velocity grows with distance such that, relative to halo stars at $100$ kpc, the inner MW is lurching at $\approx 40$ km s$^{-1}$ towards a recent location along the LMC's past orbit. Our measurements align with N-body simulations of the halo's response to a $1.8 \times 10^{11} M_\odot$ LMC on first infall, suggesting that the LMC is at least 15% as massive as the MW. Our findings highlight the dramatic disequilibrium of the MW outskirts, and will enable more accurate measurements of the total mass of our Galaxy., Comment: 25 pages, 15 figures. Submitted to ApJ

In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field., Comment: This version has been accepted and published at NeurIPS 2024 Track Datasets and Benchmarks (Spotlight)

Today's era is the digitized era. Managing such generated big data is an important factor for data scientists. Day by day, it increases the demand for big data storage systems. Different organizations are involved in providing storage-related services. They follow the different architectures or storage models for storing big data. In this survey paper, our target is to highlight such storage architectures which provided by different renowned storage service providers. On an architectural basis, we divide the big data storage systems into five parts, Distributed file systems (DFS), Clustered File Systems (CFS), Cloud Storage, Archive Storage, and Object Storage Systems (OSS). Also, we reveal a detailed architectural view of the big data storage systems provided by the different organizations under these parts., Comment: 32 Pages