Search

The increasing threat of SMS spam, driven by evolving adversarial techniques and concept drift, calls for more robust and adaptive detection methods. In this paper, we evaluate the potential of large language models (LLMs), both open-source and commercial, for SMS spam detection, comparing their performance across zero-shot, few-shot, fine-tuning, and chain-of-thought prompting approaches. Using a comprehensive dataset of SMS messages, we assess the spam detection capabilities of prominent LLMs such as GPT-4, DeepSeek, LLAMA-2, and Mixtral. Our findings reveal that while zero-shot learning provides convenience, it is unreliable for effective spam detection. Few-shot learning, particularly with carefully selected examples, improves detection but exhibits variability across models. Fine-tuning emerges as the most effective strategy, with Mixtral achieving 98.6% accuracy and a balanced false positive and false negative rate below 2%, meeting the criteria for robust spam detection. Furthermore, we explore the resilience of these models to adversarial attacks, finding that fine-tuning significantly enhances robustness against both perceptible and imperceptible manipulations. Lastly, we investigate the impact of concept drift and demonstrate that fine-tuned LLMs, especially when combined with few-shot learning, can mitigate its effects, maintaining high performance even on evolving spam datasets. This study highlights the importance of fine-tuning and tailored learning strategies to deploy LLMs effectively for real-world SMS spam detection, Comment: 17 pages

This paper presents a novel approach to deal with the cooperative localization problem in wireless sensor networks based on received signal strength measurements. In cooperative scenarios, the cost function of the localization problem becomes increasingly nonlinear and nonconvex due to the heightened interaction between sensor nodes, making the estimation of the positions of the target nodes more challenging. Although most of existing cooperative localization algorithms assure acceptable localization accuracy, their computational complexity increases dramatically, which may restrict their applicability. To reduce the computational complexity and provide competitive localization accuracy at the same time, we propose a localization algorithm based on the differential evolution with multiple populations, opposite-based learning, redirection, and anchoring. In this work, the cooperative localization cost function is split into several simpler cost functions, each of which accounts only for one individual target node. Then, each cost function is solved by a dedicated population of the proposed algorithm. In addition, an enhanced version of the proposed algorithm which incorporates the population midpoint scheme for further improvement in the localization accuracy is devised. Simulation results demonstrate that the proposed algorithms provide comparative localization accuracy with much lower computational complexity compared with the state-of-the-art algorithms.

This technical report introduces a Named Clinical Entity Recognition Benchmark for evaluating language models in healthcare, addressing the crucial natural language processing (NLP) task of extracting structured information from clinical narratives to support applications like automated coding, clinical trial cohort identification, and clinical decision support. The leaderboard provides a standardized platform for assessing diverse language models, including encoder and decoder architectures, on their ability to identify and classify clinical entities across multiple medical domains. A curated collection of openly available clinical datasets is utilized, encompassing entities such as diseases, symptoms, medications, procedures, and laboratory measurements. Importantly, these entities are standardized according to the Observational Medical Outcomes Partnership (OMOP) Common Data Model, ensuring consistency and interoperability across different healthcare systems and datasets, and a comprehensive evaluation of model performance. Performance of models is primarily assessed using the F1-score, and it is complemented by various assessment modes to provide comprehensive insights into model performance. The report also includes a brief analysis of models evaluated to date, highlighting observed trends and limitations. By establishing this benchmarking framework, the leaderboard aims to promote transparency, facilitate comparative analyses, and drive innovation in clinical entity recognition tasks, addressing the need for robust evaluation methods in healthcare NLP., Comment: Technical Report

Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

Adversarial examples add imperceptible alterations to inputs with the objective to induce misclassification in machine learning models. They have been demonstrated to pose significant challenges in domains like image classification, with results showing that an adversarially perturbed image to evade detection against one classifier is most likely transferable to other classifiers. Adversarial examples have also been studied in malware analysis. Unlike images, program binaries cannot be arbitrarily perturbed without rendering them non-functional. Due to the difficulty of crafting adversarial program binaries, there is no consensus on the transferability of adversarially perturbed programs to different detectors. In this work, we explore the robustness of malware detectors against adversarially perturbed malware. We investigate the transferability of adversarial attacks developed against one detector, against other machine learning-based malware detectors, and code similarity techniques, specifically, locality sensitive hashing-based detectors. Our analysis reveals that adversarial program binaries crafted for one detector are generally less effective against others. We also evaluate an ensemble of detectors and show that they can potentially mitigate the impact of adversarial program binaries. Finally, we demonstrate that substantial program changes made to evade detection may result in the transformation technique being identified, implying that the adversary must make minimal changes to the program binary., Comment: This is the full version of the paper with the same title to appear in the proceedings of the 2024 Workshop on Security and Artificial Intelligence (SECAI 2024)

Scaling up interferometry to 8m collectors should smooth-out the optical piston perturbations and allow a slow fringe tracker to obtain high precision correction on faint targets. In practice, the GRAVITY fringe tracker still observes high frequency OPD components that limit the exposure time, its precision and limiting magnitude. Perturbations seem to come from mechanical vibrations in the train of mirrors. As part of the GRAVITY+ efforts, accelerometers were added to all the mirrors of the coud\'e train to compensate in real-time the optical path using the main delay lines. We show their effectiveness on vibrations peaks between 40 and 200Hz and outline prospects for the upgrade of the deformable mirrors and the beam-compressor differential delay lines., Comment: 11 pages, 12 figures, Proc. SPIE Astronomical Telescopes + Instrumentation 2024 (Yokohama Japan), Optical and Infrared Interferometry and Imaging VI

NOTT (formerly Hi-5) is the L'-band (3.5-4.0~microns) nulling interferometer of Asgard, an instrument suite in preparation for the VLTI visitor focus. The primary scientific objectives of NOTT include characterizing (i) young planetary systems near the snow line, a critical region for giant planet formation, and (ii) nearby main-sequence stars close to the habitable zone, with a focus on detecting exozodiacal dust that could obscure Earth-like planets. In 2023-2024, the final warm optics have been procured and assembled in a new laboratory at KU Leuven. First fringes and null measurements were obtained using a Gallium Lanthanum Sulfide (GLS) photonic chip that was also tested at cryogenic temperatures. In this paper, we present an overall update of the NOTT project with a particular focus on the cold mechanical design, the first results in the laboratory with the final NOTT warm optics, and the ongoing Asgard integration activities. We also report on other ongoing activities such as the characterization of the photonic chip (GLS, LiNbO3, SiO), the development of the exoplanet science case, the design of the dispersion control module, and the progress with the self-calibration data reduction software., Comment: 11 pages (incl. 5 figures); Proc. SPIE Astronomical Telescopes + Instrumentation 2024 (Yokohama; Japan), Optical and Infrared Interferometry and Imaging VI

This study presents a comprehensive analysis and comparison of two predominant fine-tuning methodologies - full-parameter fine-tuning and parameter-efficient tuning - within the context of medical Large Language Models (LLMs). We developed and refined a series of LLMs, based on the Llama-2 architecture, specifically designed to enhance medical knowledge retrieval, reasoning, and question-answering capabilities. Our experiments systematically evaluate the effectiveness of these tuning strategies across various well-known medical benchmarks. Notably, our medical LLM Med42 showed an accuracy level of 72% on the US Medical Licensing Examination (USMLE) datasets, setting a new standard in performance for openly available medical LLMs. Through this comparative analysis, we aim to identify the most effective and efficient method for fine-tuning LLMs in the medical domain, thereby contributing significantly to the advancement of AI-driven healthcare applications., Comment: Published at AAAI 2024 Spring Symposium - Clinical Foundation Models

Propaganda is a form of communication intended to influence the opinions and the mindset of the public to promote a particular agenda. With the rise of social media, propaganda has spread rapidly, leading to the need for automatic propaganda detection systems. Most work on propaganda detection has focused on high-resource languages, such as English, and little effort has been made to detect propaganda for low-resource languages. Yet, it is common to find a mix of multiple languages in social media communication, a phenomenon known as code-switching. Code-switching combines different languages within the same text, which poses a challenge for automatic systems. With this in mind, here we propose the novel task of detecting propaganda techniques in code-switched text. To support this task, we create a corpus of 1,030 texts code-switching between English and Roman Urdu, annotated with 20 propaganda techniques, which we make publicly available. We perform a number of experiments contrasting different experimental setups, and we find that it is important to model the multilinguality directly (rather than using translation) as well as to use the right fine-tuning strategy. The code and the dataset are publicly available at https://github.com/mbzuai-nlp/propaganda-codeswitched-text

Text simplification is one of the domains in Natural Language Processing (NLP) that offers an opportunity to understand the text in a simplified manner for exploration. However, it is always hard to understand and retrieve knowledge from unstructured text, which is usually in the form of compound and complex sentences. There are state-of-the-art neural network-based methods to simplify the sentences for improved readability while replacing words with plain English substitutes and summarising the sentences and paragraphs. In the Knowledge Graph (KG) creation process from unstructured text, summarising long sentences and substituting words is undesirable since this may lead to information loss. However, KG creation from text requires the extraction of all possible facts (triples) with the same mentions as in the text. In this work, we propose a controlled simplification based on the factual information in a sentence, i.e., triple. We present a classical syntactic dependency-based approach to split and rephrase a compound and complex sentence into a set of simplified sentences. This simplification process will retain the original wording with a simple structure of possible domain facts in each sentence, i.e., triples. The paper also introduces an algorithm to identify and measure a sentence's syntactic complexity (SC), followed by reduction through a controlled syntactic simplification process. Last, an experiment for a dataset re-annotation is also conducted through GPT3; we aim to publish this refined corpus as a resource. This work is accepted and presented in International workshop on Learning with Knowledge Graphs (IWLKG) at WSDM-2023 Conference. The code and data is available at www.github.com/sallmanm/SynSim., Comment: This work is accepted and presented in International workshop on Learning with Knowledge Graphs (IWLKG) at WSDM'2023 Conference

The GRAVITY instrument has been revolutionary for near-infrared interferometry by pushing sensitivity and precision to previously unknown limits. With the upgrade of GRAVITY and the Very Large Telescope Interferometer (VLTI) in GRAVITY+, these limits will be pushed even further, with vastly improved sky coverage, as well as faint-science and high-contrast capabilities. This upgrade includes the implementation of wide-field off-axis fringe-tracking, new adaptive optics systems on all Unit Telescopes, and laser guide stars in an upgraded facility. GRAVITY+ will open up the sky to the measurement of black hole masses across cosmic time in hundreds of active galactic nuclei, use the faint stars in the Galactic centre to probe General Relativity, and enable the characterisation of dozens of young exoplanets to study their formation, bearing the promise of another scientific revolution to come at the VLTI., Comment: Published in the ESO Messenger

The short message service (SMS) was introduced a generation ago to the mobile phone users. They make up the world's oldest large-scale network, with billions of users and therefore attracts a lot of fraud. Due to the convergence of mobile network with internet, SMS based scams can potentially compromise the security of internet services as well. In this study, we present a new SMS scam dataset consisting of 153,551 SMSes. This dataset that we will release publicly for research purposes represents the largest publicly-available SMS scam dataset. We evaluate and compare the performance achieved by several established machine learning methods on the new dataset, ranging from shallow machine learning approaches to deep neural networks to syntactic and semantic feature models. We then study the existing models from an adversarial viewpoint by assessing its robustness against different level of adversarial manipulation. This perspective consolidates the current state of the art in SMS Spam filtering, highlights the limitations and the opportunities to improve the existing approaches., Comment: arXiv admin note: text overlap with arXiv:1911.00953 by other authors

Hi-5 is the L'-band (3.5-4.0 $\mu$m) high-contrast imager of Asgard, an instrument suite in preparation for the visitor focus of the VLTI. The system is optimized for high-contrast and high-sensitivity imaging within the diffraction limit of a single UT/AT telescope. It is designed as a double-Bracewell nulling instrument producing spectrally-dispersed (R=20, 400, or 2000) complementary nulling outputs and simultaneous photometric outputs for self-calibration purposes. In this paper, we present an update of the project with a particular focus on the overall architecture, opto-mechanical design of the warm and cold optics, injection system, and development of the photonic beam combiner. The key science projects are to survey (i) nearby young planetary systems near the snow line, where most giant planets are expected to be formed, and (ii) nearby main sequence stars near the habitable zone where exozodiacal dust that may hinder the detection of Earth-like planets. We present an update of the expected instrumental performance based on full end-to-end simulations using the new GRAVITY+ specifications of the VLTI and the latest planet formation models., Comment: 16 pages, 9 figures, SPIE 2022 "Astronomical Telescopes and Instrumentation" manuscript 12183-16