Search

Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost., Comment: Accepted for ICLR: https://openreview.net/forum?id=zl3pfz4VCV

Large Language Models (LLMs) are prone to generating plausible yet incorrect responses, known as hallucinations. Effectively detecting hallucinations is therefore crucial for the safe deployment of LLMs. Recent research has linked hallucinations to model uncertainty, suggesting that hallucinations can be detected by measuring dispersion over answer distributions obtained from a set of samples drawn from a model. While drawing from the distribution over tokens defined by the model is a natural way to obtain samples, in this work, we argue that it is sub-optimal for the purpose of detecting hallucinations. We show that detection can be improved significantly by taking into account model uncertainty in the Bayesian sense. To this end, we propose a very simple and efficient approach that perturbs an appropriate subset of model parameters, or equivalently hidden unit activations, during sampling. We demonstrate its effectiveness across a wide range of datasets and model architectures.

Autism Spectrum Disorder (ASD) is a chronic neurodevelopmental disorder symptoms of which includes repetitive behaviour and lack of social and communication skills. Even though these symptoms can be seen very clearly in social but a large number of individuals with ASD remain undiagnosed. In this paper, we worked on a methodology for the detection of ASD from a 3-dimensional walking video dataset, utilizing supervised machine learning (ML) classification algorithms and nature-inspired optimization algorithms for feature extraction from the dataset. The proposed methodology involves the classification of ASD using a supervised ML classification algorithm and extracting important and relevant features from the dataset using nature-inspired optimization algorithms. We also included the ranking coefficients to find the initial leading particle. This selection of particle significantly reduces the computation time and hence, improves the total efficiency and accuracy for ASD detection. To evaluate the efficiency of the proposed methodology, we deployed various combinationsalgorithms of classification algorithm and nature-inspired algorithms resulting in an outstanding classification accuracy of $100\%$ using the random forest classification algorithm and gravitational search algorithm for feature selection. The application of the proposed methodology with different datasets would enhance the robustness and generalizability of the proposed methodology. Due to high accuracy and less total computation time, the proposed methodology will offer a significant contribution to the medical and academic fields, providing a foundation for future research and advancements in ASD diagnosis.

Nonlinear mathematical models introduce the relation between various physical and biological interactions present in nature. One of the most famous models is the Lotka-Volterra model which defined the interaction between predator and prey species present in nature. However, predators, scavengers, and prey populations coexist in a natural system where scavengers can additionally rely on the dead bodies of predators present in the system. Keeping this in mind, the formulation and simulation of the predator prey scavenger model is introduced in this paper. For the predation response, respective prey species are assumed to have Holling's functional response of type III. The proposed model is tested for various simulations and is found to be showing satisfactory results in different scenarios. After simulations, the American forest dataset is taken for parameter estimation which imitates the real-world case. For parameter estimation, a physics-informed deep neural network is used with the Adam backpropagation method which prevents the avalanche effect in trainable parameters updation. For neural networks, mean square error and physics-informed informed error are considered. After the neural network, the hence-found parameters are fine-tuned using the Broyden-Fletcher-Goldfarb-Shanno algorithm. Finally, the hence-found parameters using a natural dataset are tested for stability using Jacobian stability analysis. Future research work includes minimization of error induced by parameters, bifurcation analysis, and sensitivity analysis of the parameters.

The anomalous magnetic moment of the muon ($\,a_{\mu}\,$) provides a stringent test of the quantum nature of the Standard Model (SM) and its extensions. To probe beyond the SM physics, one needs to be able to subtract the SM contributions, which consists of a non-perturbative part, namely, the hadronic vacuum polarization (HVP) of the photon. The state of the art is to predominantly use two different methods to extract this HVP: lattice computation, and dispersion relation-based, data-driven method. Thus one can construct different forms of the ``$a_{\mu}$ test" which compares the precise measurement of $a_{\mu}$ to its theory prediction. Additionally, this opens the possibility for another subtle test, where these two ``theory" predictions themselves are compared against each other, which is denoted as the ``HVP-test". This test is particularly sensitive to hadronic scale new physics. Therefore, in this work, we consider a SM extension consisting of a generic, light $\sim(100~{\rm MeV}-1~{\rm GeV})$ vector boson and study its impact on both tests. We develop a comprehensive formalism for this purpose. We find that in the case of data-driven HVP being used in the $a_{\mu}$ test, the new physics contributions effectively cancels for a flavor-universal vector boson. As an illustration of these general results, we consider two benchmark models: i)~the dark photon ($\,A'\,$) and ii)~a gauge boson coupled to baryon-number ($\,B\,$). Using a combination of these tests, we are able to constrain the parameter space of $B$ and $A'$, complementarily to the existing limits. As a spin-off, our preliminary analysis of the spectrum of invariant mass of $3\pi$ in events with ISR at the $B-$ factories (BaBar, Belle) manifests the value of such a study in searching for $B\to 3\pi$ decay, thus motivating a dedicated search by experimental collaborations., Comment: 21 pages + 3 appendices, 10 figures + 17 appendix figures

Personalized driving refers to an autonomous vehicle's ability to adapt its driving behavior or control strategies to match individual users' preferences and driving styles while maintaining safety and comfort standards. However, existing works either fail to capture every individual preference precisely or become computationally inefficient as the user base expands. Vision-Language Models (VLMs) offer promising solutions to this front through their natural language understanding and scene reasoning capabilities. In this work, we propose a lightweight yet effective on-board VLM framework that provides low-latency personalized driving performance while maintaining strong reasoning capabilities. Our solution incorporates a Retrieval-Augmented Generation (RAG)-based memory module that enables continuous learning of individual driving preferences through human feedback. Through comprehensive real-world vehicle deployment and experiments, our system has demonstrated the ability to provide safe, comfortable, and personalized driving experiences across various scenarios and significantly reduce takeover rates by up to 76.9%. To the best of our knowledge, this work represents the first end-to-end VLM-based motion control system in real-world autonomous vehicles., Comment: arXiv admin note: substantial text overlap with arXiv:2410.15281

Spherical proportional counters (SPCs) are gaseous particle detectors sensitive to single ionization electrons in their target media, with large detector volumes and low background rates. The $\mbox{NEWS-G}$ collaboration employs this technology to search for low-mass dark matter, having previously performed searches with detectors at the Laboratoire Souterrain de Modane (LSM), including a recent campaign with a 135 cm diameter SPC filled with methane. While in situ calibrations of the detector response were carried out at the LSM, measurements of the mean ionization yield and fluctuations of methane gas in SPCs were performed using a 30 cm diameter detector. The results of multiple measurements taken at different operating voltages are presented. A UV laser system was used to measure the mean gas gain of the SPC, along with $\mathrm{^{37}Ar}$ and aluminum-fluorescence calibration sources. These measurements will inform the energy response model of future operating detectors., Comment: 11 pages, 8 figures

With the broader usage and highly successful development of Large Language Models (LLMs), there has been a growth of interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning ability, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to language interaction and decision-making. In this paper, we first introduce the novel concept of designing LLMs for autonomous driving (LLM4AD). Then, we propose a comprehensive benchmark for evaluating the instruction-following abilities of LLM4AD in simulation. Furthermore, we conduct a series of experiments on real-world vehicle platforms, thoroughly evaluating the performance and potential of our LLM4AD systems. Finally, we envision the main challenges of LLM4AD, including latency, deployment, security and privacy, safety, trust and transparency, and personalization. Our research highlights the significant potential of LLMs to enhance various aspects of autonomous vehicle technology, from perception and scene understanding to language interaction and decision-making.

The goal of this paper is to improve (upcycle) an existing large language model without the prohibitive requirements of continued pre-training of the full-model. The idea is to split the pre-training data into semantically relevant groups and train an expert on each subset. An expert takes the form of a lightweight adapter added on the top of a frozen base model. During inference, an incoming query is first routed to the most relevant expert which is then loaded onto the base model for the forward pass. Unlike typical Mixture of Experts (MoE) models, the experts in our method do not work with other experts for a single query. Hence, we dub them "introvert" experts. Freezing the base model and keeping the experts as lightweight adapters allows extreme parallelism during training and inference. Training of all experts can be done in parallel without any communication channels between them. Similarly, the inference can also be heavily parallelized by distributing experts on different GPUs and routing each request to the GPU containing its relevant expert. We implement a proof-of-concept version of this method and show the validity of our approach.

Despite their recent successes, Transformer-based large language models show surprising failure modes. A well-known example of such failure modes is their inability to length-generalize: solving problem instances at inference time that are longer than those seen during training. In this work, we further explore the root cause of this failure by performing a detailed analysis of model behaviors on the simple parity task. Our analysis suggests that length generalization failures are intricately related to a model's inability to perform random memory accesses within its context window. We present supporting evidence for this hypothesis by demonstrating the effectiveness of methodologies that circumvent the need for indexing or that enable random token access indirectly, through content-based addressing. We further show where and how the failure to perform random memory access manifests through attention map visualizations., Comment: Published as a conference paper at COLM 2024

Modulation of the polarization state of optical beams is crucial in advanced optical applications such as polarization microscopy, polarization rotation, and medical imaging. We demonstrate a continuous polarization rotator by utilizing the liquid crystals (LC) in theta-cell configuration comprising of circular and parallel alignment of LC on two aligning glass substrates. A rotation of plane of polarization up to 160 degree is observed using Stokes polarimetry and substantiated by the mathematical analysis using Mueller matrix formalism. Further, the device is demonstrated as a plane of polarization detector with high precision detection angle. The plane of polarization is analyzed through single shot imaging of the intensity distribution of transmitted light through the LC theta-cell.

The NEWS-G direct detection experiment uses spherical proportional counters to search for light dark matter candidates. New results from a 10 day physics run with a $135\,\mathrm{cm}$ in diameter spherical proportional counter at the Laboratoire Souterrain de Modane are reported. The target consists of $114\,\mathrm{g}$ of methane, providing sensitivity to dark matter spin-dependent coupling to protons. New constraints are presented in the mass range $0.17$ to $1.2\,\mathrm{GeV/c^2}$, with a 90% confidence level cross-section upper limit of $30.9\,\mathrm{pb}$ for a mass of $0.76\,\mathrm{GeV/c^2}$., Comment: 7 pages, 3 figures

Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time., Comment: Accepted to the 2024 NeurIPS Datasets and Benchmarks track; data and code are available at: https://www.qualcomm.com/developer/software/qevd-dataset and https://github.com/Qualcomm-AI-research/FitCoach

Using the mathematical theory of shape calculus as tool, we perform a rigorous investigation of the virtual work principle for the computation of electromagnetic forces in static settings. The main goal is to shed light on the widely held belief that the virtual work principle entails passively advecting the fields with the virtual displacement when computing deformation-dependent field energies gradients. The adjoint approach to differentiation of functionals under variational constraints provides a mathematical justification for this belief. However, it also shows that passively advecting the actual electromagnetic fields is sufficient only in the case of linear materials. In the general case, also fields arising as solutions of adjoint variational problems have to be taken into account.

Finetuning large language models (LLMs) in federated learning (FL) settings has become increasingly important as it allows resource-constrained devices to finetune a model using private data. However, finetuning LLMs using backpropagation requires excessive memory (especially from intermediate activations) for resource-constrained devices. While Forward-mode Auto-Differentiation (AD) can significantly reduce memory footprint from activations, we observe that directly applying it to LLM finetuning results in slow convergence and poor accuracy. In this paper, we introduce Spry, an FL algorithm that splits trainable weights of an LLM among participating clients, such that each client computes gradients using forward-mode AD that are closer estimations of the true gradients. Spry achieves a low memory footprint, high accuracy, and fast convergence. We formally prove that the global gradients in Spry are unbiased estimators of true global gradients for homogeneous data distributions across clients, while heterogeneity increases bias of the estimates. We also derive Spry's convergence rate, showing that the gradients decrease inversely proportional to the number of FL rounds, indicating the convergence up to the limits of heterogeneity. Empirically, Spry reduces the memory footprint during training by 1.4-7.1x in contrast to backpropagation, while reaching comparable accuracy, across a wide range of language tasks, models, and FL settings. Spry reduces the convergence time by 1.2-20.3x and achieves 5.2-13.5% higher accuracy against zero-order methods. When finetuning Llama2-7B with LoRA, compared to the peak memory consumption of 33.9GB of backpropagation, Spry only consumes 6.2GB of peak memory. For OPT13B, the reduction is from 76.5GB to 10.8GB. Spry makes feasible previously impossible FL deployments on commodity edge devices. Our source code is available at https://github.com/Astuary/Spry., Comment: Accepted to NeurIPS 2024

Generative Artificial Intelligence (GAI) models such as ChatGPT have experienced a surge in popularity, attracting 100 million active users in 2 months and generating an estimated 10 million daily queries. Despite this remarkable adoption, there remains a limited understanding to which extent this innovative technology influences higher education. This research paper investigates the impact of GAI on university students and Higher Education Institutions (HEIs). The study adopts a mixed-methods approach, combining a comprehensive survey with scenario analysis to explore potential benefits, drawbacks, and transformative changes the new technology brings. Using an online survey with 130 participants we assessed students' perspectives and attitudes concerning present ChatGPT usage in academics. Results show that students use the current technology for tasks like assignment writing and exam preparation and believe it to be a effective help in achieving academic goals. The scenario analysis afterwards projected potential future scenarios, providing valuable insights into the possibilities and challenges associated with incorporating GAI into higher education. The main motivation is to gain a tangible and precise understanding of the potential consequences for HEIs and to provide guidance responding to the evolving learning environment. The findings indicate that irresponsible and excessive use of the technology could result in significant challenges. Hence, HEIs must develop stringent policies, reevaluate learning objectives, upskill their lecturers, adjust the curriculum and reconsider examination approaches.