Search

Mixture of Experts (MoE) models are highly effective in scaling model capacity while preserving computational efficiency, with the gating network, or router, playing a central role by directing inputs to the appropriate experts. In this paper, we establish a novel connection between MoE frameworks and attention mechanisms, demonstrating how quadratic gating can serve as a more expressive and efficient alternative. Motivated by this insight, we explore the implementation of quadratic gating within MoE models, identifying a connection between the self-attention mechanism and the quadratic gating. We conduct a comprehensive theoretical analysis of the quadratic softmax gating MoE framework, showing improved sample efficiency in expert and parameter estimation. Our analysis provides key insights into optimal designs for quadratic gating and expert functions, further elucidating the principles behind widely used attention mechanisms. Through extensive evaluations, we demonstrate that the quadratic gating MoE outperforms the traditional linear gating MoE. Moreover, our theoretical insights have guided the development of a novel attention mechanism, which we validated through extensive experiments. The results demonstrate its favorable performance over conventional models across various tasks., Comment: Pedram Akbarian, Huy Nguyen, and Xing Han made equal contributions to this work

With the growing prominence of the Mixture of Experts (MoE) architecture in developing large-scale foundation models, we investigate the Hierarchical Mixture of Experts (HMoE), a specialized variant of MoE that excels in handling complex inputs and improving performance on targeted tasks. Our investigation highlights the advantages of using varied gating functions, moving beyond softmax gating within HMoE frameworks. We theoretically demonstrate that applying tailored gating functions to each expert group allows HMoE to achieve robust results, even when optimal gating functions are applied only at select hierarchical levels. Empirical validation across diverse scenarios supports these theoretical claims. This includes large-scale multimodal tasks, image classification, and latent domain discovery and prediction tasks, where our modified HMoE models show great performance improvements., Comment: 58 pages

With the growing demand for offline PDF chatbots in automotive industrial production environments, optimizing the deployment of large language models (LLMs) in local, low-performance settings has become increasingly important. This study focuses on enhancing Retrieval-Augmented Generation (RAG) techniques for processing complex automotive industry documents using locally deployed Ollama models. Based on the Langchain framework, we propose a multi-dimensional optimization approach for Ollama's local RAG implementation. Our method addresses key challenges in automotive document processing, including multi-column layouts and technical specifications. We introduce improvements in PDF processing, retrieval mechanisms, and context compression, tailored to the unique characteristics of automotive industry documents. Additionally, we design custom classes supporting embedding pipelines and an agent supporting self-RAG based on LangGraph best practices. To evaluate our approach, we constructed a proprietary dataset comprising typical automotive industry documents, including technical reports and corporate regulations. We compared our optimized RAG model and self-RAG agent against a naive RAG baseline across three datasets: our automotive industry dataset, QReCC, and CoQA. Results demonstrate significant improvements in context precision, context recall, answer relevancy, and faithfulness, with particularly notable performance on the automotive industry dataset. Our optimization scheme provides an effective solution for deploying local RAG systems in the automotive sector, addressing the specific needs of PDF chatbots in industrial production environments. This research has important implications for advancing information processing and intelligent production in the automotive industry.

Binary verification plays a pivotal role in software security, yet building a verification service that is both open and trustworthy poses a formidable challenge. In this paper, we introduce a novel binary verification service, AGORA, scrupulously designed to overcome the challenge. At the heart of this approach lies a strategic insight: certain tasks can be delegated to untrusted entities, while the corresponding validators are securely housed within the trusted computing base (TCB). AGORA can validate untrusted assertions generated for versatile policies. Through a novel blockchain-based bounty task manager, it also utilizes crowdsourcing to remove trust in theorem provers. These synergistic techniques successfully ameliorate the TCB size burden associated with two procedures: binary analysis and theorem proving. The design of AGORA allows untrusted parties to participate in these complex processes. Moreover, based on running the optimized TCB within trusted execution environments and recording the verification process on a blockchain, the public can audit the correctness of verification results. By implementing verification workflows for software-based fault isolation policy and side-channel mitigation, our evaluation demonstrates the efficacy of AGORA.

This paper investigates the neutrino transition magnetic moment in the $U(1)_X$SSM. $U(1)_X$SSM is the $U(1)$ extension of Minimal Supersymmetric Standard Model (MSSM) and its local gauge group is extended to $SU(3)_C\times SU(2)_L \times U(1)_Y\times U(1)_X$. To obtain this model, three singlet new Higgs superfields and right-handed neutrinos are added to the MSSM, which can explain the results of neutrino oscillation experiments. The neutrino transition magnetic moment is induced by electroweak radiative corrections. By applying effective Lagrangian method and on-shell scheme, we study the associated Feynman diagrams and the transition magnetic moment of neutrinos in the model. We fit experimental data for neutrino mass variances and mixing angle. Based on the range of data selection, the influences of different sensitive parameters on the results are analysed. The numerical analysis shows that many parameters have an effect on the neutrino transition moment, such as $g_X$, $M_2$, $\lambda_H$ and $g_{YX}$. For our numerical results, the order of magnitude of $\mu_{ij}^M/\mu_B$ is around $10^{-20}$ $\sim$ $10^{-19}$.

Achieving fairness across diverse clients in Federated Learning (FL) remains a significant challenge due to the heterogeneity of the data and the inaccessibility of sensitive attributes from clients' private datasets. This study addresses this issue by introducing \texttt{EquiFL}, a novel approach designed to enhance both local and global fairness in federated learning environments. \texttt{EquiFL} incorporates a fairness term into the local optimization objective, effectively balancing local performance and fairness. The proposed coordination mechanism also prevents bias from propagating across clients during the collaboration phase. Through extensive experiments across multiple benchmarks, we demonstrate that \texttt{EquiFL} not only strikes a better balance between accuracy and fairness locally at each client but also achieves global fairness. The results also indicate that \texttt{EquiFL} ensures uniform performance distribution among clients, thus contributing to performance fairness. Furthermore, we showcase the benefits of \texttt{EquiFL} in a real-world distributed dataset from a healthcare application, specifically in predicting the effects of treatments on patients across various hospital locations.

In the $U(1)_X$SSM, we delve into the flavor violation of $h \rightarrow bs$, where $h$ is identified with the SM-like Higgs boson discovered at the LHC. As the U(1) extension of the minimal supersymmetric standard model (MSSM), the U(1)XSSM has new super fields such as right-handed neutrinos and three Higgs singlets. We conduct a thorough analysis of the underlying mechanisms and parameter dependencies of $h \rightarrow bs$ in the $U(1)_X$SSM. In the $U(1)_X$SSM, we discover that the Br$(h \rightarrow bs)$ for the Higgs decay to $bs$ could significantly differ from the expectation in the standard model (SM), depending on the values of the new parameters introduced in the model. Our research not only contributes to a deeper understanding of Higgs physics within the $U(1)_X$SSM, but also provides valuable guidance for new physics (NP)., Comment: 24 pages,8 figures

It is well known that the muon magnetic dipole moment (MDM) has close relation with the new physics (NP) in the development of the Standard Model (SM). Combined with the Fermilab National Accelerator Laboratory (FNAL) and the Brookhaven National Laboratory (BNL) E821 result, the departure from the SM prediction is about 5.0 $\sigma$. We study the electroweak corrections from several type two-loop SUSY diagrams and the virtual SUSY particles include chargino, neutralino, scalar lepton and scalar neutrino. Based on the latest experimental constraints, we study the muon MDM under the next to the minimal supersymmetric extension of the SM with local B-L gauge symmetry (N-B-LSSM). The abundant numerical results verify that $\tan{\beta},~T_e,~M^2_L,~M^2_e,~M_{BB'}$ play an important role in muon MDM. $M^2_e,~\tan{\beta}$ and $T_e$ are sensitive parameters to muon MDM. From the data obtained in all the figures of the numerical results, most of the values of $a_{\mu}^{NBL}$ are in 2$\sigma$ interval, which can compensate the departure between the experiment data and the SM prediction.

Originating from semantic bugs, Entity-Inconsistency Bugs (EIBs) involve misuse of syntactically valid yet incorrect program entities, such as variable identifiers and function names, which often have security implications. Unlike straightforward syntactic vulnerabilities, EIBs are subtle and can remain undetected for years. Traditional detection methods, such as static analysis and dynamic testing, often fall short due to the versatile and context-dependent nature of EIBs. However, with advancements in Large Language Models (LLMs) like GPT-4, we believe LLM-powered automatic EIB detection becomes increasingly feasible through these models' semantics understanding abilities. This research first undertakes a systematic measurement of LLMs' capabilities in detecting EIBs, revealing that GPT-4, while promising, shows limited recall and precision that hinder its practical application. The primary problem lies in the model's tendency to focus on irrelevant code snippets devoid of EIBs. To address this, we introduce a novel, cascaded EIB detection system named WitheredLeaf, which leverages smaller, code-specific language models to filter out most negative cases and mitigate the problem, thereby significantly enhancing the overall precision and recall. We evaluated WitheredLeaf on 154 Python and C GitHub repositories, each with over 1,000 stars, identifying 123 new flaws, 45% of which can be exploited to disrupt the program's normal operations. Out of 69 submitted fixes, 27 have been successfully merged.

In real-world graph data, distribution shifts can manifest in various ways, such as the emergence of new categories and changes in the relative proportions of existing categories. It is often important to detect nodes of novel categories under such distribution shifts for safety or insight discovery purposes. We introduce a new approach, Recall-Constrained Optimization with Selective Link Prediction (RECO-SLIP), to detect nodes belonging to novel categories in attributed graphs under subpopulation shifts. By integrating a recall-constrained learning framework with a sample-efficient link prediction mechanism, RECO-SLIP addresses the dual challenges of resilience against subpopulation shifts and the effective exploitation of graph structure. Our extensive empirical evaluation across multiple graph datasets demonstrates the superior performance of RECO-SLIP over existing methods. The experimental code is available at https://github.com/hsinghuan/novel-node-category-detection., Comment: Accepted to ECML-PKDD 2024

Accurate traffic forecasting is a fundamental problem in intelligent transportation systems and learning long-range traffic representations with key information through spatiotemporal graph neural networks (STGNNs) is a basic assumption of current traffic flow prediction models. However, due to structural limitations, existing STGNNs can only utilize short-range traffic flow data; therefore, the models cannot adequately learn the complex trends and periodic features in traffic flow. Besides, it is challenging to extract the key temporal information from the long historical traffic series and obtain a compact representation. To solve the above problems, we propose a novel LSTTN (Long-Short Term Transformer-based Network) framework comprehensively considering the long- and short-term features in historical traffic flow. First, we employ a masked subseries Transformer to infer the content of masked subseries from a small portion of unmasked subseries and their temporal context in a pretraining manner, forcing the model to efficiently learn compressed and contextual subseries temporal representations from long historical series. Then, based on the learned representations, long-term trend is extracted by using stacked 1D dilated convolution layers, and periodic features are extracted by dynamic graph convolution layers. For the difficulties in making time-step level prediction, LSTTN adopts a short-term trend extractor to learn fine-grained short-term temporal features. Finally, LSTTN fuses the long-term trend, periodic features and short-term features to obtain the prediction results. Experiments on four real-world datasets show that in 60-minute-ahead long-term forecasting, the LSTTN model achieves a minimum improvement of 5.63\% and a maximum improvement of 16.78\% over baseline models. The source code is available at https://github.com/GeoX-Lab/LSTTN., Comment: 15 pages, 10 figures, 6 tables

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks., Comment: 47 pages, 9 tables, 12 figures

In the present study, we investigate the decays of the top quark $t\rightarrow c\gamma$, $t\rightarrow cg$, $t\rightarrow cZ$ and $t\rightarrow ch$. They are extremely rare processes in the Standard Model (SM). As the $U(1)$ extension of the minimal supersymmetric standard model (MSSM), the $U(1)_X$SSM features new superfields such as the right-handed neutrinos and three Higgs singlets. We analyze the effects of different sensitive parameters on the results and make reasonable theorecial predictions, which provides a useful reference for future experimental development. Considering the constraint of the updated experimental data, the numerical results show that the branching ratios of all four processes $t\rightarrow c\gamma,~cg,~cZ,~ch$ can reach the same order of magnitude as their experimental upper limits. Among them, $\tan\beta$ has the most obvious effect on each process and is the main parameter. $g_X$, $g_{YX}$, $\mu$, $M_2$, $\lambda_H$, $M_{U23}^2$ and $M_{Q23}^2$ are important parameters for the processes, and have effects on the numerical results., Comment: pages 25, figures 27

$U(1)_X$SSM is a non-universal Abelian extension of the Minimal Supersymmetric Standard Model (MSSM) and its local gauge group is extended to $SU(3)_C\times SU(2)_L \times U(1)_Y\times U(1)_X$. Based on the latest data of neutral meson mixing and experimental limitations, we investigate the process of $B^{0}-\bar{B^{0}}$ mixing in $U(1)_X$SSM. Using the effective Hamiltonian method, the Wilson coefficients and mass difference $\triangle m_{B}$ are derived. The abundant numerical results verify that $~v_S,~M^2_D,~\lambda_C,~{\mu},~M_2,~\tan{\beta},~g_{YX},~M_1$ and $~\lambda_H$ are sensitive parameters to the process of $B^{0}-\bar{B^{0}}$ mixing. With further measurement in the experiment, the parameter space of the $U(1)_X$SSM will be further constrained during the mixing process of $B^{0}-\bar{B^{0}}$.

We study lepton flavor violating (LFV) decays $B^0\rightarrow{{l_i}^{\pm}{l_j}^{\mp}}$ ($B^0\rightarrow e{\mu}$, $B^0\rightarrow e{\tau}$ and $B^0\rightarrow {\mu}{\tau}$) in the $U(1)_X$SSM (SSM is acronym for the supersymmetric standard model), which is the $U(1)_X$ extension of the minimal supersymmetric standard model (MSSM). The local gauge group of $U(1)_X$SSM is $SU(3)_C\times SU(2)_L \times U(1)_Y \times U(1)_X$. These processes ($B^0\rightarrow e{\mu}$, $B^0\rightarrow e{\tau}$ and $B^0\rightarrow {\mu}{\tau}$) are strictly forbidden in the standard model (SM), but these LFV decays are a signal of new physics (NP). We use the Mass Insertion Approximation (MIA) to find sensitive parameters that directly influence the result of the branching ratio of LFV decay $B_d\rightarrow{{l_i}^{\pm}{l_j}^{\mp}}$. Combined with the latest experimental results, we analyze the relationship between different sensitive parameters and the branching ratios of the three processes. According to the numerical analysis, these elements are very restricted from the non observation of CLFV.

Gravitational wave high-energy Electromagnetic Counterpart All-sky Monitor (GECAM) is a pair of microsatellites (i.e. GECAM-A and GECAM-B) dedicated to monitoring gamma-ray transients including gravitational waves high-energy electromagnetic counterparts, Gamma-ray Bursts, Soft Gamma-ray Repeaters, Solar Flares and Terrestrial Gamma-ray Flashes. Since launch in December 2020, GECAM-B has detected hundreds of astronomical and terrestrial events. For these bursts, localization is the key for burst identification and classification as well as follow-up observations in multi-wavelength. Here, we propose a Bayesian localization method with Poisson data with Gaussian background profile likelihood to localize GECAM bursts based on the burst counts distribution in detectors with different orientations. We demonstrate that this method can work well for all kinds of bursts, especially for extremely short ones. In addition, we propose a new method to estimate the systematic error of localization based on a confidence level test, which can overcome some problems of the existing method in literature. We validate this method by Monte Carlo simulations, and then apply it to a burst sample with accurate location and find that the mean value of the systematic error of GECAM-B localization is $\sim 2.5^{\circ}$. By considering this systematic error, we can obtain a reliable localization probability map for GECAM bursts. Our methods can be applied to other gamma-ray monitors., Comment: The paper has been accepted by Astrophysical Journal Supplement Series

Recent advances in Transformer architectures have empowered their empirical success in a variety of tasks across different domains. However, existing works mainly focus on predictive accuracy and computational cost, without considering other practical issues, such as robustness to contaminated samples. Recent work by Nguyen et al., (2022) has shown that the self-attention mechanism, which is the center of the Transformer architecture, can be viewed as a non-parametric estimator based on kernel density estimation (KDE). This motivates us to leverage a set of robust kernel density estimation methods for alleviating the issue of data contamination. Specifically, we introduce a series of self-attention mechanisms that can be incorporated into different Transformer architectures and discuss the special properties of each method. We then perform extensive empirical studies on language modeling and image classification tasks. Our methods demonstrate robust performance in multiple scenarios while maintaining competitive results on clean datasets., Comment: Accepted by NeurIPS 2023 as a poster; 23 pages, 5 figures, 11 tables

Conformal prediction is a simple and powerful tool that can quantify uncertainty without any distributional assumptions. Many existing methods only address the average coverage guarantee, which is not ideal compared to the stronger conditional coverage guarantee. Existing methods of approximating conditional coverage require additional models or time effort, which makes them not easy to scale. In this paper, we propose a modified non-conformity score by leveraging the local approximation of the conditional distribution using kernel density estimation. The modified score inherits the spirit of split conformal methods, which is simple and efficient and can scale to high dimensional settings. We also proposed a unified framework that brings together our method and several state-of-the-art. We perform extensive empirical evaluations: results measured by both average and conditional coverage confirm the advantage of our method., Comment: 21 pages, 7 figures, 8 tables

We propose a novel approach to the problem of clustering hierarchically aggregated time-series data, which has remained an understudied problem though it has several commercial applications. We first group time series at each aggregated level, while simultaneously leveraging local and global information. The proposed method can cluster hierarchical time series (HTS) with different lengths and structures. For common two-level hierarchies, we employ a combined objective for local and global clustering over spaces of discrete probability measures, using Wasserstein distance coupled with Soft-DTW divergence. For multi-level hierarchies, we present a bottom-up procedure that progressively leverages lower-level information for higher-level clustering. Our final goal is to improve both the accuracy and speed of forecasts for a larger number of HTS needed for a real-world application. To attain this goal, each time series is first assigned the forecast for its cluster representative, which can be considered as a "shrinkage prior" for the set of time series it represents. Then this base forecast can be quickly fine-tuned to adjust to the specifics of that time series. We empirically show that our method substantially improves performance in terms of both speed and accuracy for large-scale forecasting tasks involving much HTS., Comment: 17 pages, 3 figures, 4 tables

With growing concerns regarding data privacy and rapid increase in data volume, Federated Learning(FL) has become an important learning paradigm. However, jointly learning a deep neural network model in a FL setting proves to be a non-trivial task because of the complexities associated with the neural networks, such as varied architectures across clients, permutation invariance of the neurons, and presence of non-linear transformations in each layer. This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework that allows each client to build a personalised model without enforcing a common architecture across clients. This allows each client to optimize with respect to local data and compute constraints, while still benefiting from the learnings of other (potentially more powerful) clients. The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client. The extensive experimental results demonstrate that the FedHeNN framework is capable of learning better performing models on clients in both the settings of homogeneous and heterogeneous architectures across clients.