Search

Multimodal Emotion Recognition (MER) is an important research topic. This paper advocates for a transformative paradigm in MER. The rationale behind our work is that current approaches often rely on a limited set of basic emotion labels, which do not adequately represent the rich spectrum of human emotions. These traditional and overly simplistic emotion categories fail to capture the inherent complexity and subtlety of human emotional experiences, leading to limited generalizability and practicality. Therefore, we propose a new MER paradigm called Open-vocabulary MER (OV-MER), which encompasses a broader range of emotion labels to reflect the richness of human emotions. This paradigm relaxes the label space, allowing for the prediction of arbitrary numbers and categories of emotions. To support this transition, we provide a comprehensive solution that includes a newly constructed database based on LLM and human collaborative annotations, along with corresponding metrics and a series of benchmarks. We hope this work advances emotion recognition from basic emotions to more nuanced emotions, contributing to the development of emotional AI.

This study examines fairness within the rideshare industry, focusing on both drivers' wages and riders' trip fares. Through quantitative analysis, we found that drivers' hourly wages are significantly influenced by factors such as race/ethnicity, health insurance status, tenure to the platform, and working hours. Despite platforms' policies not intentionally embedding biases, disparities persist based on these characteristics. For ride fares, we propose a method to audit the pricing policy of a proprietary algorithm by replicating it; we conduct a hypothesis test to determine if the predicted rideshare fare is greater than the taxi fare, taking into account the approximation error in the replicated model. Challenges in accessing data and transparency hinder our ability to isolate discrimination from other factors, underscoring the need for collaboration with rideshare platforms and drivers to enhance fairness in algorithmic wage determination and pricing., Comment: Accepted to the Humans, Algorithmic Decision-Making and Society: Modeling Interactions and Impact, co-located with the International Conference on Machine Learning, Vienna, Austria

The development of sustainable transportation and communication systems requires an increase in both energy density and capacity retention of Li-batteries. Using substrates forming a solid solution with body centered cubic Li enhances the cycle stability of anode-less batteries. However, it remains unclear how the substrate microstructure affects the lithiation behavior. Here, we deploy a correlative, near-atomic scale probing approach through combined ion- and electron-microscopy to examine the distribution of Li in Li-Ag diffusion couples as model system. We reveal that Li regions with over 93.8% at.% nucleate within Ag at random high angle grain boundaries, whereas grain interiors are not lithiated. We evidence the role of kinetics and mechanical constraint from the microstructure over equilibrium thermodynamics in dictating the lithiation process. The findings suggest that grain size and grain boundary character are critical to enhance the electrochemical performance of interlayers/electrodes, particularly for improving lithiation kinetics and hence reducing dendrite formation.

We explore the bound neutrons decay into invisible particles (e.g., $n\rightarrow 3 \nu$ or $nn \rightarrow 2 \nu$) in the JUNO liquid scintillator detector. The invisible decay includes two decay modes: $ n \rightarrow { inv} $ and $ nn \rightarrow { inv} $. The invisible decays of $s$-shell neutrons in $^{12}{\rm C}$ will leave a highly excited residual nucleus. Subsequently, some de-excitation modes of the excited residual nuclei can produce a time- and space-correlated triple coincidence signal in the JUNO detector. Based on a full Monte Carlo simulation informed with the latest available data, we estimate all backgrounds, including inverse beta decay events of the reactor antineutrino $\bar{\nu}_e$, natural radioactivity, cosmogenic isotopes and neutral current interactions of atmospheric neutrinos. Pulse shape discrimination and multivariate analysis techniques are employed to further suppress backgrounds. With two years of exposure, JUNO is expected to give an order of magnitude improvement compared to the current best limits. After 10 years of data taking, the JUNO expected sensitivities at a 90% confidence level are $\tau/B( n \rightarrow { inv} ) > 5.0 \times 10^{31} \, {\rm yr}$ and $\tau/B( nn \rightarrow { inv} ) > 1.4 \times 10^{32} \, {\rm yr}$., Comment: 28 pages, 7 figures, 4 tables

Multimodal emotion recognition is an important research topic in artificial intelligence. Over the past few decades, researchers have made remarkable progress by increasing the dataset size and building more effective algorithms. However, due to problems such as complex environments and inaccurate annotations, current systems are hard to meet the demands of practical applications. Therefore, we organize the MER series of competitions to promote the development of this field. Last year, we launched MER2023, focusing on three interesting topics: multi-label learning, noise robustness, and semi-supervised learning. In this year's MER2024, besides expanding the dataset size, we further introduce a new track around open-vocabulary emotion recognition. The main purpose of this track is that existing datasets usually fix the label space and use majority voting to enhance the annotator consistency. However, this process may lead to inaccurate annotations, such as ignoring non-majority or non-candidate labels. In this track, we encourage participants to generate any number of labels in any category, aiming to describe emotional states as accurately as possible. Our baseline code relies on MERTools and is available at: https://github.com/zeroQiaoba/MERTools/tree/master/MER2024.

Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on downstream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require independent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts, including 1) Feature Consistency Alignment: which imposes constraints to the prompted feature changes to maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which encourages the prompted images to contain richer task-specific semantics with language guidance. We validate the effectiveness of TVP through extensive experiments with 6 modern MLLMs on a wide variety of tasks ranging from object recognition and counting to multimodal reasoning and hallucination correction., Comment: Accepted in CVPR 2024 as Poster (Highlight)

Multimodal fusion is a significant method for most multimodal tasks. With the recent surge in the number of large pre-trained models, combining both multimodal fusion methods and pre-trained model features can achieve outstanding performance in many multimodal tasks. In this paper, we present our approach, which leverages both advantages for addressing the task of Expression (Expr) Recognition and Valence-Arousal (VA) Estimation. We evaluate the Aff-Wild2 database using pre-trained models, then extract the final hidden layers of the models as features. Following preprocessing and interpolation or convolution to align the extracted features, different models are employed for modal fusion. Our code is available at GitHub - FulgenceWen/ABAW6th.

In this paper, we concentrate on decentralized optimization problems with nonconvex and nonsmooth objective functions, especially on the decentralized training of nonsmooth neural networks. We introduce a unified framework to analyze the global convergence of decentralized stochastic subgradient-based methods. We prove the global convergence of our proposed framework under mild conditions, by establishing that the generated sequence asymptotically approximates the trajectories of its associated differential inclusion. Furthermore, we establish that our proposed framework covers a wide range of existing efficient decentralized subgradient-based methods, including decentralized stochastic subgradient descent (DSGD), DSGD with gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGD-M). In addition, we introduce the sign map to regularize the update directions in DSGD-M, and show it is enclosed in our proposed framework. Consequently, our convergence results establish, for the first time, global convergence of these methods when applied to nonsmooth nonconvex objectives. Preliminary numerical experiments demonstrate that our proposed framework yields highly efficient decentralized subgradient-based methods with convergence guarantees in the training of nonsmooth neural networks., Comment: 22 pages

A 20-kiloton liquid scintillator detector is designed in the Jiangmen Underground Neutrino Observatory (JUNO) for multiple physics purposes, including the determination of the neutrino mass ordering through reactor neutrinos, as well as measuring supernova neutrinos, solar neutrinos, and atmosphere neutrinos to explore different physics topics. Efficient reconstruction algorithms are needed to achieve these physics goals in a wide energy range from MeV to GeV. In this paper, we present a novel method for reconstructing the energy of events using hit information from 25600 3-inch photomultiplier tubes (PMTs) and the OCCUPANCY method. Our algorithm exhibits good performance in accurate energy reconstruction, validated with electron Monte Carlo samples spanning kinetic energies from 10 MeV to 1 GeV.

Graph computing has become increasingly crucial in processing large-scale graph data, with numerous systems developed for this purpose. Two years ago, we introduced GraphScope as a system addressing a wide array of graph computing needs, including graph traversal, analytics, and learning in one system. Since its inception, GraphScope has achieved significant technological advancements and gained widespread adoption across various industries. However, one key lesson from this journey has been understanding the limitations of a "one-size-fits-all" approach, especially when dealing with the diversity of programming interfaces, applications, and data storage formats in graph computing. In response to these challenges, we present GraphScope Flex, the next iteration of GraphScope. GraphScope Flex is designed to be both resource-efficient and cost-effective, while also providing flexibility and user-friendliness through its LEGO-like modularity. This paper explores the architectural innovations and fundamental design principles of GraphScope Flex, all of which are direct outcomes of the lessons learned during our ongoing development process. We validate the adaptability and efficiency of GraphScope Flex with extensive evaluations on synthetic and real-world datasets. The results show that GraphScope Flex achieves 2.4X throughput and up to 55.7X speedup over other systems on the LDBC Social Network and Graphalytics benchmarks, respectively. Furthermore, GraphScope Flex accomplishes up to a 2,400X performance gain in real-world applications, demonstrating its proficiency across a wide range of graph computing scenarios with increased effectiveness.

The mechanical performance of metallic metamaterials with 3-dimensional solid frames is typically a combination of the geometrical effect ("architecture") and the characteristic size effects of the base material ("microstructure"). In this study, for the first time, the temperature- and rate-dependent mechanical response of copper microlattices has been investigated. The microlattices were fabricated via a localized electrodeposition in liquid (LEL) process which enables high-precision additive manufacturing of metal at the micro-scale. The metal microlattices possess a unique microstructure with micron sized grains that are rich with randomly oriented growth twins and near-ideal nodal connectivity. Importantly, copper microlattices exhibited unique temperature (-150 and 25 degree C) and strain rate (0.001~100 s-1) dependent deformation behavior during in situ micromechanical testing. Systematic compression tests of fully dense copper micropillars, equivalent in diameter and length to the struts of the microlattice at comparable extreme loading conditions, allow us to investigate the intrinsic deformation mechanism of copper. Combined with the post-mortem microstructural analysis, substantial shifts in deformation mechanisms depending on the temperature and strain rate were revealed. On the one hand, at room temperature (25 degree C), dislocation slip based plastic deformation occurs and leads to a localized deformation of the micropillars. On the other hand, at cryogenic temperature (-150 degree C), mechanical twinning occurs and leads to relatively homogeneous deformation of the micropillars. Based on the intrinsic deformation mechanisms of copper, the temperature and strain rate dependent deformation behavior of microlattices could be explained.

MoS2 nanostructures are promising catalysts for proton-exchange-membrane (PEM) electrolyzers to replace expensive noble metals. Their broadscale application demands high activity for the hydrogen evolution reaction (HER) as well as robust durability. Doping is commonly applied to enhance the HER activity of MoS2-based nanocatalysts, but the effect of dopants in the electrochemical and structural stability is yet to be discussed. Herein, we correlate operando electrochemical measurements to the structural evolution of the materials down to the nanometric scale by identical location electron microscopy and spectroscopy. The range of stable operation for MoS2 nanocatalysts with and without rhenium doping is experimentally defined. The responsible degradation mechanisms at first electrolyte contact, open circuit stabilization and HER conditions are experimentally identified and confirmed with the calculated Pourbaix diagram of Re-doped MoS2. Doping MoS2-based nanocatalysts is validated as a promising strategy for the continuous improvement of high performance and durable PEM electrolyzers.

The core-collapse supernova (CCSN) is considered one of the most energetic astrophysical events in the universe. The early and prompt detection of neutrinos before (pre-SN) and during the supernova (SN) burst presents a unique opportunity for multi-messenger observations of CCSN events. In this study, we describe the monitoring concept and present the sensitivity of the system to pre-SN and SN neutrinos at the Jiangmen Underground Neutrino Observatory (JUNO), a 20 kton liquid scintillator detector currently under construction in South China. The real-time monitoring system is designed to ensure both prompt alert speed and comprehensive coverage of progenitor stars. It incorporates prompt monitors on the electronic board as well as online monitors at the data acquisition stage. Assuming a false alert rate of 1 per year, this monitoring system exhibits sensitivity to pre-SN neutrinos up to a distance of approximately 1.6 (0.9) kiloparsecs and SN neutrinos up to about 370 (360) kiloparsecs for a progenitor mass of 30 solar masses, considering both normal and inverted mass ordering scenarios. The pointing ability of the CCSN is evaluated by analyzing the accumulated event anisotropy of inverse beta decay interactions from pre-SN or SN neutrinos. This, along with the early alert, can play a crucial role in facilitating follow-up multi-messenger observations of the next galactic or nearby extragalactic CCSN., Comment: 24 pages, 9 figures, accepted for the publication at JCAP

Multimodal emotion recognition is an important research topic in artificial intelligence, whose main goal is to integrate multimodal clues to identify human emotional states. Current works generally assume accurate labels for benchmark datasets and focus on developing more effective architectures. However, emotion annotation relies on subjective judgment. To obtain more reliable labels, existing datasets usually restrict the label space to some basic categories, then hire plenty of annotators and use majority voting to select the most likely label. However, this process may result in some correct but non-candidate or non-majority labels being ignored. To ensure reliability without ignoring subtle emotions, we propose a new task called ``Explainable Multimodal Emotion Recognition (EMER)''. Unlike traditional emotion recognition, EMER takes a step further by providing explanations for these predictions. Through this task, we can extract relatively reliable labels since each label has a certain basis. Meanwhile, we borrow large language models (LLMs) to disambiguate unimodal clues and generate more complete multimodal explanations. From them, we can extract richer emotions in an open-vocabulary manner. This paper presents our initial attempt at this task, including introducing a new dataset, establishing baselines, and defining evaluation metrics. In addition, EMER can serve as a benchmark task to evaluate the audio-video-text understanding performance of multimodal LLMs.

In solids, chemical short-range order (CSRO) refers to the self-organisation of atoms of certain species occupying specific crystal sites. CSRO is increasingly being envisaged as a lever to tailor the mechanical and functional properties of materials. Yet quantitative relationships between properties and the morphology, number density, and atomic configurations of CSRO domains remain elusive. Herein, we showcase how machine learning-enhanced atom probe tomography (APT) can mine the near-atomically resolved APT data and jointly exploit the technique's high elemental sensitivity to provide a 3D quantitative analysis of CSRO in a CoCrNi medium-entropy alloy. We reveal multiple CSRO configurations, with their formation supported by state-of-the-art Monte-Carlo simulations. Quantitative analysis of these CSROs allows us to establish relationships between processing parameters and physical properties. The unambiguous characterization of CSRO will help refine strategies for designing advanced materials by manipulating atomic-scale architectures.

Phase transformations and crystallographic defects are two essential tools to drive innovations in materials. Bulk materials design via tuning chemical compositions has been systematized using phase diagrams. We show here that the same thermodynamic concept can be applied to understand the chemistry at defects. We present a combined experimental and modelling approach to scope and build phase diagrams for defects. The discovery was enabled by triggering phase transformations of individual defects through local alloying, and sequentially imaging the structural and chemical changes using atomic-resolution scanning transmission electron microscopy. By observing atomic-scale phase transformations of a Mg grain boundary through Ga alloying, we exemplified the method to construct a grain boundary phase diagram using ab initio simulations and thermodynamic principles. The methodology enables a systematic development of defect phase diagrams to propel a new paradigm for materials design utilizing chemical complexity and phase transformations at defects.

The electrochemical corrosion rate of Magnesium (Mg) and Mg alloys depends on the stability of the formed surface layer. Based on the Mg substrate, the oxide structure comprises a dense MgO/Mg(OH)2 mixture underneath a porous plate-like Mg(OH)2 layer. While the kinetics of the anodic partial reaction has been mainly attributed to MgO, recent studies showed an effect of the Mg(OH)2 layer thickness on the cathodic partial reaction. A thinner Mg(OH)2 layer has been associated with a higher kinetics of the oxygen reduction rate. In the present study, the proposed mechanism has been further investigated via in situ respirometric measurements with Mg-Al-Ca solid solution in electrolytes with different pH values (pH = 8-13). The results indicate an additional effect based on the structure of the surface layer in the passive state of Mg corrosion. Furthermore, two different Al enriched interlayers at the Mg/MgO- and MgO/Mg(OH)2 interfaces were observed and discussed in terms of their thermodynamic stability under alkaline immersion conditions.

In this study, the structure and composition of the native oxide forming on the basal plane (0001) of Mg-2Al-0.1Ca is investigated by a correlative approach, combining scanning transmission electron microscopy (STEM) and atom probe tomography (APT). Atom probe specimens were prepared conventionally in a Ga focused ion beam (FIB) as well as a Xe plasma FIB in a cryogenic setup and subsequently cleaned in the atom probe to remove surface contamination before oxidation. While thermal energy input from the laser and longer atmospheric exposure time increased the measured hydrogen content in the specimen's apex region, cryo preparation revealed, that the hydrogen uptake in magnesium is independent of the employment of conventional or cryogenic FIB preparation. TEM measurements demonstrated the growth of a (111) MgO oxide layer with 3-4 nm thickness on the basal (0001) plane of the Mg atom probe specimen. APT data further revealed the formation of an aluminum-rich region between bulk Mg and the native oxide. The aluminum enrichment of up to ~20 at.% at the interface is consistent with an inward growth of the oxide scale.

During the last decade, automatic data analysis methods concerning different aspects of crystal analysis have been developed, e.g., unsupervised primitive unit cell extraction and automated crystal distortion and defects detection. However, an automatic, unsupervised motif extraction method is still not widely available yet. Here, we propose and demonstrate a novel method for the automatic motif extraction in real space from crystalline images based on a variational approach involving the unit cell projection operator. Due to the non-convex nature of the resulting minimization problem, a multi-stage algorithm is used. First, we determine the primitive unit cell in form of two lattice vectors. Second, a motif image is estimated using the unit cell information. Finally, the motif is determined in terms of atom positions inside the unit cell. The method was tested on various synthetic and experimental HAADF STEM images. The results are a representation of the motif in form of an image, atomic positions, primitive unit cell vectors, and a denoised and a modeled reconstruction of the input image. The method was applied to extract the primitive cells of complex $\mu$-phase structures Nb$_\text{6.4}$Co$_\text{6.6}$ and Nb$_\text{7}$Co$_\text{6}$, where subtle differences between their interplanar spacings were determined.