Search

This study introduces an approach to optimize Parameter Efficient Fine Tuning (PEFT) for Pretrained Language Models (PLMs) by implementing a Shared Low Rank Adaptation (ShareLoRA). By strategically deploying ShareLoRA across different layers and adapting it for the Query, Key, and Value components of self-attention layers, we achieve a substantial reduction in the number of training parameters and memory usage. Importantly, ShareLoRA not only maintains model performance but also exhibits robustness in both classification and generation tasks across a variety of models, including RoBERTa, GPT-2, LLaMA and LLaMA2. It demonstrates superior transfer learning capabilities compared to standard LoRA applications and mitigates overfitting by sharing weights across layers. Our findings affirm that ShareLoRA effectively boosts parameter efficiency while ensuring scalable and high-quality performance across different language model architectures., Comment: 16 pages, 5 figures

Deploying Large Language Models (LLMs) locally on mobile devices presents a significant challenge due to their extensive memory requirements. In this paper, we introduce LinguaLinked, a system for decentralized, distributed LLM inference on mobile devices. LinguaLinked enables collaborative execution of the inference task across multiple trusted devices. LinguaLinked ensures data privacy by processing information locally. LinguaLinked uses three key strategies. First, an optimized model assignment technique segments LLMs and uses linear optimization to align segments with each device's capabilities. Second, an optimized data transmission mechanism ensures efficient and structured data flow between model segments while also maintaining the integrity of the original model structure. Finally, LinguaLinked incorporates a runtime load balancer that actively monitors and redistributes tasks among mobile devices to prevent bottlenecks, enhancing the system's overall efficiency and responsiveness. We demonstrate that LinguaLinked facilitates efficient LLM inference while maintaining consistent throughput and minimal latency through extensive testing across various mobile devices, from high-end to low-end Android devices. In our evaluations, compared to the baseline, LinguaLinked achieves an inference performance acceleration of $1.11\times$ to $1.61\times$ in single-threaded settings, $1.73\times$ to $2.65\times$ with multi-threading. Additionally, runtime load balancing yields an overall inference acceleration of $1.29\times$ to $1.32\times$., Comment: 16 pages, 8 figures

Optically dense clouds in the interstellar medium composed predominantly of molecular hydrogen, known as molecular clouds, are sensitive to energy injection in the form of photon absorption, cosmic-ray scattering, and dark matter (DM) scattering. The ionization rates in dense molecular clouds are heavily constrained by observations of abundances of various molecular tracers. Recent studies have set constraints on the DM-electron scattering cross section using measurements of ionization rates in dense molecular clouds. Here we calculate the analogous bounds on the DM-proton cross section using the molecular Migdal effect, recently adapted from the neutron scattering literature to the DM context. These bounds may be the strongest limits on a strongly-coupled DM subfraction, and represent the first application of the Migdal effect to astrophysical systems., Comment: v2: Added discussion of scalar mediator model. 7 pages, 4 figures

Large Language Models' safety remains a critical concern due to their vulnerability to adversarial attacks, which can prompt these systems to produce harmful responses. In the heart of these systems lies a safety classifier, a computational model trained to discern and mitigate potentially harmful, offensive, or unethical outputs. However, contemporary safety classifiers, despite their potential, often fail when exposed to inputs infused with adversarial noise. In response, our study introduces the Adversarial Prompt Shield (APS), a lightweight model that excels in detection accuracy and demonstrates resilience against adversarial prompts. Additionally, we propose novel strategies for autonomously generating adversarial training datasets, named Bot Adversarial Noisy Dialogue (BAND) datasets. These datasets are designed to fortify the safety classifier's robustness, and we investigate the consequences of incorporating adversarial examples into the training process. Through evaluations involving Large Language Models, we demonstrate that our classifier has the potential to decrease the attack success rate resulting from adversarial attacks by up to 60%. This advancement paves the way for the next generation of more reliable and resilient conversational agents., Comment: 11 pages, 2 figures

Jailbreak vulnerabilities in Large Language Models (LLMs), which exploit meticulously crafted prompts to elicit content that violates service guidelines, have captured the attention of research communities. While model owners can defend against individual jailbreak prompts through safety training strategies, this relatively passive approach struggles to handle the broader category of similar jailbreaks. To tackle this issue, we introduce FuzzLLM, an automated fuzzing framework designed to proactively test and discover jailbreak vulnerabilities in LLMs. We utilize templates to capture the structural integrity of a prompt and isolate key features of a jailbreak class as constraints. By integrating different base classes into powerful combo attacks and varying the elements of constraints and prohibited questions, FuzzLLM enables efficient testing with reduced manual effort. Extensive experiments demonstrate FuzzLLM's effectiveness and comprehensiveness in vulnerability discovery across various LLMs., Comment: Publish by ICASSP 2024 on 3/18/2024; Extended Arxiv version

Nuclear scattering events with large momentum transfer in atomic, molecular, or solid-state systems may result in electronic excitations. In the context of atomic scattering by dark matter (DM), this is known as the Migdal effect, but the same effect has also been studied in molecules in the chemistry and neutron scattering literature. Here we present two distinct Migdal-like effects from DM scattering in molecules, which we collectively refer to as the molecular Migdal effect: a center-of-mass recoil, equivalent to the standard Migdal treatment, and a non-adiabatic coupling resulting from corrections to the Born-Oppenheimer approximation. The molecular bonds break spherical symmetry, leading to large daily modulation in the Migdal rate from anisotropies in the matrix elements. Our treatment reduces to the standard Migdal effect in atomic systems but does not rely on the impulse approximation or any semiclassical treatments of nuclear motion, and as such may be extended to models where DM scatters through a long-range force. We demonstrate all of these features in a few simple toy models of diatomic molecules, namely ${\rm H}_2^+$, N$_2$, and CO, and find total molecular Migdal rates competitive with those in semiconductors for the same target mass. We discuss how our results may be extended to more realistic targets comprised of larger molecules which could be deployed at the kilogram scale., Comment: v1: 15+2 pages, 7 figures

Pairwise debiasing is one of the most effective strategies in reducing position bias in learning-to-rank (LTR) models. However, limiting the scope of this strategy, are the underlying assumptions required by many pairwise debiasing approaches. In this paper, we develop an approach based on a minimalistic set of assumptions that can be applied to a much broader range of user browsing patterns and arbitrary presentation layouts. We implement the approach as a simplified version of the Unbiased LambdaMART and demonstrate that it retains the underlying unbiasedness property in a wider variety of settings than the original algorithm. Finally, using simulations with "golden" relevance labels, we will show that the simplified version compares favourably with the original Unbiased LambdaMART when the examination of different positions in a ranked list is not assumed to be independent., Comment: 10 pages