Search

Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While large multimodal models have shown strong generalization across various tasks, they exhibit poor generalization to hateful meme detection due to the dynamic nature of memes tied to emerging social trends and breaking news. Recent work further highlights the limitations of conventional supervised fine-tuning for large multimodal models in this context. To address these challenges, we propose Large Multimodal Model Retrieval-Guided Contrastive Learning (LMM-RGCL), a novel two-stage fine-tuning framework designed to improve both in-domain accuracy and cross-domain generalization. Experimental results on six widely used meme classification datasets demonstrate that LMM-RGCL achieves state-of-the-art performance, outperforming agent-based systems such as VPD-PALI-X-55B. Furthermore, our method effectively generalizes to out-of-domain memes under low-resource settings, surpassing models like GPT-4o., Comment: Preprint. Under Review

We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. These findings motivate and enable the inclusion of tied pairs in preference optimization as opposed to simply discarding them., Comment: 24 pages

The Directed Acyclic Transformer is a fast non-autoregressive (NAR) model that performs well in Neural Machine Translation. Two issues prevent its application to general Natural Language Generation (NLG) tasks: frequent Out-Of-Vocabulary (OOV) errors and the inability to faithfully generate entity names. We introduce Control-DAG, a constrained decoding algorithm for our Directed Acyclic T5 (DA-T5) model which offers lexical, vocabulary and length control. We show that Control-DAG significantly enhances DA-T5 on the Schema Guided Dialogue and the DART datasets, establishing strong NAR results for Task-Oriented Dialogue and Data-to-Text NLG., Comment: 11 pages. NAACL 2024

Two approaches have emerged to input images into large language models (LLMs). The first is to caption images into natural language. The second is to map image feature embeddings into the domain of the LLM and pass the mapped embeddings directly to the LLM. The majority of recent few-shot multimodal work reports performance using architectures that employ variations of one of these two approaches. But they overlook an important comparison between them. We design a controlled and focused experiment to compare these two approaches to few-shot visual question answering (VQA) with LLMs. Our findings indicate that for Flan-T5 XL, a 3B parameter LLM, connecting visual embeddings directly to the LLM embedding space does not guarantee improved performance over using image captions. In the zero-shot regime, we find using textual image captions is better. In the few-shot regimes, how the in-context examples are selected determines which is better.

Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers., Comment: ACL 2024; Project page: https://preflmr.github.io/

The integration of a Verticle Cavity Surface-Emitting Laser (VCSEL) array and a driving Application-Specific Integrated Circuit (ASIC) in a custom optical array transmitter module (ATx) for operation in the detector front-end is constructed, assembled and tested. The ATx provides 12 parallel channels with each channel operating at 10 Gbps. The optical transmitter eye diagram passes the eye mask and the bit-error rate (BER) less than 1E-12 transmission is achieved at 10 Gbps/ch. The overall insertion loss including the radiation induced attenuation is sufficiently low to meet the proposed link budget requirement., Comment: 10 pages, 9 figures

We present the design and test results of two optical data transmission ASICs for the High-Luminosity LHC (HL-LHC) experiments. These ASICs include a two-channel serializer (LOCs2) and a single-channel Vertical Cavity Surface Emitting Laser (VCSEL) driver (LOCld1V2). Both ASICs are fabricated in a commercial 0.25-um Silicon-on-Sapphire (SoS) CMOS technology and operate at a data rate up to 8 Gbps per channel. The power consumption of LOCs2 and LOCld1V2 are 1.25 W and 0.27 W at 8-Gbps data rate, respectively. LOCld1V2 has been verified meeting the radiation-tolerance requirements for HL-LHC experiments., Comment: 9 pages, 12 figures

A prototype Liquid-argon Trigger Digitizer Board (LTDB), called the LTDB Demonstrator, has been developed to demonstrate the functions of the ATLAS Liquid Argon Calorimeter Phase-I trigger electronics upgrade. Forty Analog-to-Digital converters and four FPGAs with embedded multi-gigabit-transceivers on each Demonstrator need high quality clocks. A clock distribution system based on commercial components has been developed for the Demonstrator. The design of the clock distribution system is presented. The performance of the clock distribution system has been evaluated. The components used in the clock distribution system have been qualified to meet radiation tolerance requirements of the Demonstrator., Comment: 7 pages, 4 figures, Proceedings of the Topical Workshop on Electronics for Particle Physics 2014

A compact radiation-tolerant array optical transmitter module (ATx) is developed to provide data transmission up to 10Gbps per channel with 12 parallel channels for collider detector applications. The ATx integrates a Vertical Cavity Surface-Emitting Laser (VCSEL) array and driver circuitry for electrical to optical conversion, an edge warp substrate for the electrical interface and a micro-lens array for the optical interface. This paper reports the continuing development of the ATx custom package. A simple, high-accuracy and reliable active-alignment method for the optical coupling is introduced. The radiation-resistance of the optoelectronic components is evaluated and the inclusion of a custom-designed array driver is discussed., Comment: 8 pages, 8 figures

In this paper, a remote FPGA-configuration method based on JTAG extension over optical fibers is presented. The method takes advantage of commercial components and ready-to-use software such as iMPACT and does not require any hardware or software development. The method combines the advantages of the slow remote JTAG configuration and the fast local flash memory configuration. The method has been verified successfully and used in the Demonstrator of Liquid-Argon Trigger Digitization Board (LTDB) for the ATLAS liquid argon calorimeter Phase-I trigger upgrade. All components on the FPGA side are verified to meet the radiation tolerance requirements., Comment: 8 pages, 4 figures

Generative Artificial Intelligence (AI) holds immense potential in medical applications. Numerous studies have explored the efficacy of various generative AI models within healthcare contexts, but there is a lack of a comprehensive and systematic evaluation framework. Given that some studies evaluating the ability of generative AI for medical applications have deficiencies in their methodological design, standardized guidelines for their evaluation are also currently lacking. In response, our objective is to devise standardized assessment guidelines tailored for evaluating the performance of generative AI systems in medical contexts. To this end, we conducted a thorough literature review using the PubMed and Google Scholar databases, focusing on research that tests generative AI capabilities in medicine. Our multidisciplinary team, comprising experts in life sciences, clinical medicine, medical engineering, and generative AI users, conducted several discussion sessions and developed a checklist of 23 items. The checklist is designed to encompass the critical evaluation aspects of generative AI in medical applications comprehensively. This checklist, and the broader assessment framework it anchors, address several key dimensions, including question collection, querying methodologies, and assessment techniques. We aim to provide a holistic evaluation of AI systems. The checklist delineates a clear pathway from question gathering to result assessment, offering researchers guidance through potential challenges and pitfalls. Our framework furnishes a standardized, systematic approach for research involving the testing of generative AI's applicability in medicine. It enhances the quality of research reporting and aids in the evolution of generative AI in medicine and life sciences., Comment: 11 pages, 0 figure, 2 tables

Minimum Bayes Risk (MBR) decoding can significantly improve translation performance of Multilingual Large Language Models (MLLMs). However, MBR decoding is computationally expensive. We show how the recently developed Reinforcement Learning technique, Direct Preference Optimization (DPO), can fine-tune MLLMs to get the gains of MBR without any additional computation in inference. Our method uses only a small monolingual fine-tuning set and yields significantly improved performance on multiple NMT test sets compared to MLLMs without DPO., Comment: To appear at NAACL 2024

Hateful memes have emerged as a significant concern on the Internet. Detecting hateful memes requires the system to jointly understand the visual and textual modalities. Our investigation reveals that the embedding space of existing CLIP-based systems lacks sensitivity to subtle differences in memes that are vital for correct hatefulness classification. We propose constructing a hatefulness-aware embedding space through retrieval-guided contrastive training. Our approach achieves state-of-the-art performance on the HatefulMemes dataset with an AUROC of 87.0, outperforming much larger fine-tuned large multimodal models. We demonstrate a retrieval-based hateful memes detection system, which is capable of identifying hatefulness based on data unseen in training. This allows developers to update the hateful memes detection system by simply adding new examples without retraining, a desirable feature for real services in the constantly evolving landscape of hateful memes on the Internet., Comment: ACL 2024 Main. The code is available from: https://github.com/JingbiaoMei/RGCL

Knowledge-based Visual Question Answering (KB-VQA) requires VQA systems to utilize knowledge from external knowledge bases to answer visually-grounded questions. Retrieval-Augmented Visual Question Answering (RA-VQA), a strong framework to tackle KB-VQA, first retrieves related documents with Dense Passage Retrieval (DPR) and then uses them to answer questions. This paper proposes Fine-grained Late-interaction Multi-modal Retrieval (FLMR) which significantly improves knowledge retrieval in RA-VQA. FLMR addresses two major limitations in RA-VQA's retriever: (1) the image representations obtained via image-to-text transforms can be incomplete and inaccurate and (2) relevance scores between queries and documents are computed with one-dimensional embeddings, which can be insensitive to finer-grained relevance. FLMR overcomes these limitations by obtaining image representations that complement those from the image-to-text transforms using a vision model aligned with an existing text-based retriever through a simple alignment network. FLMR also encodes images and questions using multi-dimensional embeddings to capture finer-grained relevance between queries and documents. FLMR significantly improves the original RA-VQA retriever's PRRecall@5 by approximately 8\%. Finally, we equipped RA-VQA with two state-of-the-art large multi-modal/language models to achieve $\sim61\%$ VQA score in the OK-VQA dataset., Comment: To appear at NeurIPS 2023. This is the camera-ready version. We fixed some numbers and added more experiments to address reviewers' comments

Schema-guided dialogue state trackers can generalise to new domains without further training, yet they are sensitive to the writing style of the schemata. Augmenting the training set with human or synthetic schema paraphrases improves the model robustness to these variations but can be either costly or difficult to control. We propose to circumvent these issues by grounding the state tracking model in knowledge-seeking turns collected from the dialogue corpus as well as the schema. Including these turns in prompts during finetuning and inference leads to marked improvements in model robustness, as demonstrated by large average joint goal accuracy and schema sensitivity improvements on SGD and SGD-X., Comment: Best Long Paper of SIGDIAL 2023

Ensuring that generated utterances are faithful to dialogue actions is crucial for Task-Oriented Dialogue Response Generation. Slot Error Rate (SER) only partially measures generation quality in that it solely assesses utterances generated from non-categorical slots whose values are expected to be reproduced exactly. Utterances generated from categorical slots, which are more variable, are not assessed by SER. We propose Schema-Guided Semantic Accuracy (SGSAcc) to evaluate utterances generated from both categorical and non-categorical slots by recognizing textual entailment. We show that SGSAcc can be applied to evaluate utterances generated from a wide range of dialogue actions in the Schema Guided Dialogue (SGD) dataset with good agreement with human judgment. We also identify a previously overlooked weakness in generating faithful utterances from categorical slots in unseen domains. We show that prefix tuning applied to T5 generation can address this problem. We further build an ensemble of prefix-tuning and fine-tuning models that achieves the lowest SER reported and high SGSAcc on the SGD dataset., Comment: 8 pages, 1 figure

We present the design and test results of the Miniature optical Transmitter (MTx) and Transceiver (MTRx) for the high luminosity LHC (HL-LHC) experiments. MTx and MTRx are Transmitter Optical Subassembly (TOSA) and Receiver Optical Subassembly (ROSA) based. There are two major developments: the Vertical Cavity Surface Emitting Laser (VCSEL) driver ASIC LOCld and the mechanical latch that provides the connection to fibers. In this paper, we concentrate on the justification of this work, the design of the latch and the test results of these two modules with a Commercial Off-The-Shelf (COTS) VCSEL driver., Comment: 5 pages, 7 figures

We propose a line code that has fast resynchronization capability and low latency. Both the encoder and decoder have been implemented in FPGAs. The encoder has also been implemented in an ASIC. The latency of the whole optical link (not including the optical fiber) is estimated to be less than 73.9 ns. In the case of radiation-induced link synchronization loss, the decoder can recover the synchronization in 25 ns. The line code will be used in the ATLAS liquid argon calorimeter Phase-I trigger upgrade and can also be potentially used in other LHC experiments., Comment: 10 pages, 7 figures

This paper presents several component prototypes towards a low-latency, small-form-factor optical link designed for the ATLAS Liquid Argon Calorimeter Phase-I trigger upgrade. A prototype of the custom-made dual-channel optical transmitter module, the Miniature optical Transmitter (MTx), with separate transmitter optical sub-assemblies (TOSAs) has been demonstrated at data rates up to 8 Gbps per channel. A Vertical-Cavity Surface-Emitting Laser (VCSEL) driver ASIC has been developed and is used in the current MTx prototypes. A serializer ASIC prototype, operating at up to 8 Gbps per channel, has been designed and tested. A low-latency, low-overhead encoder ASIC prototype has been designed and tested. The latency of the whole link, including the transmitter latency and the receiver latency but not the latency of the fiber, is estimated to be less than 57.9 ns. The size of the MTx is 45 mm x 15 mm x 6 mm., Comment: 12 pages, 13 figures

In this paper, we present the design and test results of LOCx2, a transmitter ASIC for the ATLAS Liquid Argon Calorimeter trigger upgrade. LOCx2 consists of two channels and each channel encodes ADC data with an overhead of 14.3% and transmits serial data at 5.12 Gbps with a latency of less than 27.2 ns. LOCx2 is fabricated with a commercial 0.25-um Silicon-on-Sapphire CMOS technology and is packaged in a 100-pin QFN package. The power consumption of LOCx2 is about 843 mW.

A Liquid-argon Trigger Digitizer Board (LTDB) is being developed to upgrade the ATLAS Liquid Argon Calorimeter Phase-I trigger electronics. The LTDB located at the front end needs to obtain the clock signals and be configured and monitored remotely from the back end. A clock and control system is being developed for the LTDB and the major functions of the system have been evaluated. The design and evaluation of the clock and control system are presented in this paper.

We report the development of a mid-board, TOSA and ROSA based miniature dual channel optical transmitter (MTx) and a transceiver (MTRx). The design transmission data rate is 5.12 Gbps per channel and receiving data rate 4.8 Gbps. MTx and MTRx are only 6 mm tall and are electrically and optically pluggable. Although the fiber TOSA/ROSA coupling is through a custom latch, the fiber uses the standard LC ferrule, flange and spring. Light coupling is ensured by the TOSA and ROSA with the LC coupling mechanism. With the dual channel serializer LOCx2 sits under MTx, one achieves high data transmission with a small PCB footprint, and enjoys the reliability of the hermetically packaged TOSA. MTx and MTRx are designed for detector front-end readout of the ATLAS Liquid Argon Calorimeter (LAr) trigger upgrade.

In this paper, we present a dual-channel serializer ASIC, LOCx2, and its pin-compatible backup, LOCx2-130, for detector front-end readout. LOCx2 is fabricated in a 0.25-um Silicon-on-Sapphire CMOS process and each channel operates at 5.12 Gbps, while LOCx2-130 is fabricated in a 130-nm bulk CMOS process and each channel operates at 4.8 Gbps. The power consumption and the transmission latency are 900 mW and 27 ns for LOCx2 and the corresponding simulation result of LOCx2-130 are 386 mW and 38 ns, respectively., Comment: 8 pages, 10 figures