Search

Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However, preparing such benchmarks in high quality native language is often costly and therefore limits the representativeness of evaluation datasets. While recent efforts focused on building more inclusive MMLU benchmarks, these are conventionally built using machine translation from high-resource languages, which may introduce errors and fail to account for the linguistic and cultural intricacies of the target languages. In this paper, we address the lack of native language MMLU benchmark especially in the under-represented Turkic language family with distinct morphosyntactic and cultural characteristics. We propose two benchmarks for Turkic language MMLU: TUMLU is a comprehensive, multilingual, and natively developed language understanding benchmark specifically designed for Turkic languages. It consists of middle- and high-school level questions spanning 11 academic subjects in Azerbaijani, Crimean Tatar, Karakalpak, Kazakh, Tatar, Turkish, Uyghur, and Uzbek. We also present TUMLU-mini, a more concise, balanced, and manually verified subset of the dataset. Using this dataset, we systematically evaluate a diverse range of open and proprietary multilingual large language models (LLMs), including Claude, Gemini, GPT, and LLaMA, offering an in-depth analysis of their performance across different languages, subjects, and alphabets. To promote further research and development in multilingual language understanding, we release TUMLU-mini and all corresponding evaluation scripts.

Surgical simulation offers a promising addition to conventional surgical training. However, available simulation tools lack photorealism and rely on hardcoded behaviour. Denoising Diffusion Models are a promising alternative for high-fidelity image synthesis, but existing state-of-the-art conditioning methods fall short in providing precise control or interactivity over the generated scenes. We introduce SurGrID, a Scene Graph to Image Diffusion Model, allowing for controllable surgical scene synthesis by leveraging Scene Graphs. These graphs encode a surgical scene's components' spatial and semantic information, which are then translated into an intermediate representation using our novel pre-training step that explicitly captures local and global information. Our proposed method improves the fidelity of generated images and their coherence with the graph input over the state-of-the-art. Further, we demonstrate the simulation's realism and controllability in a user assessment study involving clinical experts. Scene Graphs can be effectively used for precise and interactive conditioning of Denoising Diffusion Models for simulating surgical scenes, enabling high fidelity and interactive control over the generated content.

Adaptation of behavior requires the brain to change goals in a changing environment. Synaptic learning has demonstrated its effectiveness in changing the probability of selecting actions based on their outcome. In the extreme case, it is vital not to repeat an action to a given goal that led to harmful punishment. The present model proposes a simple neural mechanism of gain modulation that makes possible immediate changes in the probability of selecting a goal after punishment of variable intensity. Results show how gain modulation determine the type of elementary navigation process within the state space of a network of neuronal populations of excitatory neurons regulated by inhibition. Immediately after punishment, the system can avoid the punished populations by going back or by jumping to unpunished populations. This does not require particular credit assignment at the `choice' population but only gain modulation of neurons active at the time of punishment. Gain modulation does not require statistical relearning that may lead to further errors, but can encode memories of past experiences without modification of synaptic efficacies. Therefore, gain modulation can complements synaptic plasticity.

As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated their origins. Therefore, this study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs, focusing on the Dolma dataset and the OLMo model. Using zero-shot prompting and token co-occurrence analyses, we explore how biases in training data influence model outputs. Our findings reveal that biases present in pre-training data are amplified in model outputs. The study also examines the effects of prompt types, hyperparameters, and instruction-tuning on bias expression, finding instruction-tuning partially alleviating representational bias while still maintaining overall stereotypical gender associations, whereas hyperparameters and prompting variation have a lesser effect on bias expression. Our research traces bias throughout the LLM development pipeline and underscores the importance of mitigating bias at the pretraining stage.

Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans., Comment: Accepted to NAACL 2025

Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets face serious challenges for low-resource languages due to their dependence on data annotation. This work introduces a novel method, Multilingual Reverse Instructions (MURI), which generates high-quality instruction tuning datasets for low-resource languages without requiring human annotators or pre-existing multilingual models. Utilizing reverse instructions and a translation pipeline, MURI produces instruction-output pairs from existing human-written texts in low-resource languages. This method ensures cultural relevance and diversity by sourcing texts from different native domains and applying filters to eliminate inappropriate content. Our dataset, MURI-IT, includes more than 2 million instruction-output pairs across 200 languages. Evaluation by native speakers and fine-tuning experiments with mT5 models demonstrate the approach's effectiveness for both NLU and open-ended generation. We publicly release datasets and models at https://github.com/akoksal/muri.

Surgical scenes convey crucial information about the quality of surgery. Pixel-wise localization of tools and anatomical structures is the first task towards deeper surgical analysis for microscopic or endoscopic surgical views. This is typically done via fully-supervised methods which are annotation greedy and in several cases, demanding medical expertise. Considering the profusion of surgical videos obtained through standardized surgical workflows, we propose an annotation-efficient framework for the semantic segmentation of surgical scenes. We employ image-based self-supervised object discovery to identify the most salient tools and anatomical structures in surgical videos. These proposals are further refined within a minimally supervised fine-tuning step. Our unsupervised setup reinforced with only 36 annotation labels indicates comparable localization performance with fully-supervised segmentation models. Further, leveraging surgical phase labels as weak labels can better guide model attention towards surgical tools, leading to $\sim 2\%$ improvement in tool localization. Extensive ablation studies on the CaDIS dataset validate the effectiveness of our proposed solution in discovering relevant surgical objects with minimal or no supervision., Comment: 9 pages, 4 figures, 2 tables

Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge. We propose Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method for generating synthetic datasets, given a small number of user-written few-shots that demonstrate the task to be performed. Given the few-shot examples, we use large-scale public web-crawled corpora and similarity-based document retrieval to find other relevant human-written documents. Lastly, instruction-tuned large language models (LLMs) augment the retrieved documents into custom-formatted task samples, which then can be used for fine-tuning. We demonstrate that CRAFT can efficiently generate large-scale task-specific training datasets for four diverse tasks: biology question-answering (QA), medicine QA and commonsense QA as well as summarization. Our experiments show that CRAFT-based models outperform or achieve comparable performance to general LLMs for QA tasks, while CRAFT-based summarization models outperform models trained on human-curated data by 46 preference points.

Traditional benchmarking in NLP typically involves using static held-out test sets. However, this approach often results in an overestimation of performance and lacks the ability to offer comprehensive, interpretable, and dynamic assessments of NLP models. Recently, works like DynaBench (Kiela et al., 2021) and CheckList (Ribeiro et al., 2020) have addressed these limitations through behavioral testing of NLP models with test types generated by a multistep human-annotated pipeline. Unfortunately, manually creating a variety of test types requires much human labor, often at prohibitive cost. In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models. SYNTHEVAL first generates sentences via LLMs using controlled generation, and then identifies challenging examples by comparing the predictions made by LLMs with task-specific NLP models. In the last stage, human experts investigate the challenging examples, manually design templates, and identify the types of failures the taskspecific models consistently exhibit. We apply SYNTHEVAL to two classification tasks, sentiment analysis and toxic language detection, and show that our framework is effective in identifying weaknesses of strong models on these tasks. We share our code in https://github.com/Loreley99/SynthEval_CheckList., Comment: EMNLP 2024 - Findings

The anomalous magnetic ($\tilde{a}_\tau$) and electric dipole ($\tilde{d}_\tau$) moment of tau lepton described in $\tau\bar{\tau}\gamma$ vertices are studied via $\mu^{+}\mu^{-} \rightarrow \tau^+\bar{\tau}^-$ process at the Muon colliders designed with the proposed center-of-mass-energy / integrated luminosity configurations of 3 TeV/ 1 ab$^{-1}$ and 10 TeV/ 10 ab$^{-1}$.We obtained the $95\%$ confidence level limits on the $\tilde{a}_\tau$ and $\tilde{d}_\tau$ parameters without and with systematic uncertainty of $10\%$ and compared with the experimental results.The most stringent limits on the anomalous couplings without systematic uncertainty are $-2.63\times10^{-4}< \tilde{a}_\tau <2. 65\times10^{-4}$ and $\mid \tilde{d}_\tau \mid \leq 1.47\times10^{-18} e cm$ at $\sqrt s =10$ TeV and $L_{int}$=10 ab$^{-1}$ option of the Muon collider. Our results show that the Muon colliders lead to a remarkable improvement on the current experimental limits of the anomalous magnetic and electric dipole moments of the tau lepton., Comment: 12 pages, 3 figures

Graph-based holistic scene representations facilitate surgical workflow understanding and have recently demonstrated significant success. However, this task is often hindered by the limited availability of densely annotated surgical scene data. In this work, we introduce an end-to-end framework for the generation and optimization of surgical scene graphs on a downstream task. Our approach leverages the flexibility of graph-based spectral clustering and the generalization capability of foundation models to generate unsupervised scene graphs with learnable properties. We reinforce the initial spatial graph with sparse temporal connections using local matches between consecutive frames to predict temporally consistent clusters across a temporal neighborhood. By jointly optimizing the spatiotemporal relations and node features of the dynamic scene graph with the downstream task of phase segmentation, we address the costly and annotation-burdensome task of semantic scene comprehension and scene graph generation in surgical videos using only weak surgical phase labels. Further, by incorporating effective intermediate scene representation disentanglement steps within the pipeline, our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow recognition, Comment: 9 pages, 3 figures, 3 tables, MICCAI GRAIL Workshop paper

Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: https://github.com/ArdaYueksel/TurkishMMLU., Comment: EMNLP 2024 - Findings

Many datasets have been developed to train and evaluate document-level relation extraction (RE) models. Most of these are constructed using real-world data. It has been shown that RE models trained on real-world data suffer from factual biases. To evaluate and address this issue, we present CovEReD, a counterfactual data generation approach for document-level relation extraction datasets using entity replacement. We first demonstrate that models trained on factual data exhibit inconsistent behavior: while they accurately extract triples from factual data, they fail to extract the same triples after counterfactual modification. This inconsistency suggests that models trained on factual data rely on spurious signals such as specific entities and external knowledge $\unicode{x2013}$ rather than on the input context $\unicode{x2013}$ to extract triples. We show that by generating document-level counterfactual data with CovEReD and training models on them, consistency is maintained with minimal impact on RE performance. We release our CovEReD pipeline as well as Re-DocRED-CF, a dataset of counterfactual RE documents, to assist in evaluating and addressing inconsistency in document-level RE.

While current large language models (LLMs) perform well on many knowledge-related tasks, they are limited by relying on their parameters as an implicit storage mechanism. As a result, they struggle with memorizing rare events and with updating their memory as facts change over time. In addition, the uninterpretable nature of parametric memory makes it challenging to prevent hallucination. Model editing and augmenting LLMs with parameters specialized for memory are only partial solutions. In this paper, we introduce MemLLM, a novel method of enhancing LLMs by integrating a structured and explicit read-and-write memory module. MemLLM tackles the aforementioned challenges by enabling dynamic interaction with the memory and improving the LLM's capabilities in using stored knowledge. Our experiments indicate that MemLLM enhances the LLM's performance and interpretability, in language modeling in general and knowledge-intensive tasks in particular. We see MemLLM as an important step towards making LLMs more grounded and factual through memory augmentation.

We introduce, XoFTR, a cross-modal cross-view method for local feature matching between thermal infrared (TIR) and visible images. Unlike visible images, TIR images are less susceptible to adverse lighting and weather conditions but present difficulties in matching due to significant texture and intensity differences. Current hand-crafted and learning-based methods for visible-TIR matching fall short in handling viewpoint, scale, and texture diversities. To address this, XoFTR incorporates masked image modeling pre-training and fine-tuning with pseudo-thermal image augmentation to handle the modality differences. Additionally, we introduce a refined matching pipeline that adjusts for scale discrepancies and enhances match reliability through sub-pixel level refinement. To validate our approach, we collect a comprehensive visible-thermal dataset, and show that our method outperforms existing methods on many benchmarks., Comment: CVPR Image Matching Workshop, 2024. 12 pages, 7 figures, 5 tables. Codes and dataset are available at https://github.com/OnderT/XoFTR

Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact that ``sneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.

In the post-LHC era, the muon collider represents a frontier project capable of providing high-energy and high-luminosity leptonic collisions among future lepton-lepton particle accelerators. In addition, it provides significantly cleaner final states than those produced in hadron collisions. With this expectation in mind, in this article, we research the sensitivity of the anomalous neutral gauge boson couplings $Z\gamma\gamma\gamma$, $ZZ\gamma \gamma$, and $ZZZ\gamma$ defined by dimension-8 operators, through the $\mu^+\mu^- \to \mu^+\mu^-Z\gamma$ signal, with the $Z$-boson decaying to neutrino pair. The projections of new physics at the future muon collider with the center-of-mass energy of $\sqrt{s}=10$ TeV, integrated luminosity of ${\cal L}=10$ $\rm ab^{-1}$, and systematic uncertainties of $\delta_{sys}=0\%, 3\%, 5\%$, for extraction of expected sensitivity on the anomalous $f_ {T,j}/\Lambda^4$ couplings at $95\%$ confidence level, are of the order of ${\cal O}(10^{-4}- 10^{-3})$. Compared with the research of the ATLAS and CMS Collaborations on the anomalous quartic gauge couplings, we find that the high-luminosity future muon collider could have better sensitivity.

Attribution is a key concept in large language models (LLMs) as it enables control over information sources and enhances the factuality of LLMs. While existing approaches utilize open book question answering to improve attribution, factual datasets may reward language models to recall facts that they already know from their pretraining data, not attribution. In contrast, counterfactual open book QA datasets would further improve attribution because the answer could only be grounded in the given text. We propose Hallucination Augmented Recitations (HAR) for creating counterfactual datasets by utilizing hallucination in LLMs to improve attribution. For open book QA as a case study, we demonstrate that models finetuned with our counterfactual datasets improve text grounding, leading to better open book QA performance, with up to an 8.0% increase in F1 score. Our counterfactual dataset leads to significantly better performance than using humanannotated factual datasets, even with 4x smaller datasets and 4x smaller models. We observe that improvements are consistent across various model sizes and datasets, including multi-hop, biomedical, and adversarial QA datasets.

In this study, we explore the non-standard $\nu\bar{\nu}\gamma \gamma$ couplings parametrized by dimension-seven operators via $e^{+}e^{-} \to \nu\bar{\nu}\gamma$ process at the FCC-ee/CEPC and $\mu^{+}\mu^{-}\to\nu\bar{\nu}\gamma$ process at the Muon Colliders. For the detailed Monte Carlo simulation, all signal and relevant background events are produced within the framework of Madgraph where non-standard $\nu\bar{\nu}\gamma \gamma$ couplings are implemented. After passing through Pythia for parton showering and hadronization, detector effects are included via tuned corresponding detector cards for each collider in Delphes. Projected sensitivities on $\nu\bar{\nu}\gamma \gamma$ couplings are obtained at a 5$\sigma$ confidence level without and with $5\%$ systematic uncertainties for the FCC-ee/CEPC and the Muon Colliders, showcasing the complementarity between lepton colliders. Our best limit on the anomalous $\nu\bar{\nu}\gamma \gamma$ couplings even with 5\% systematic uncertainties for muon collider with $\sqrt{s}=10$ TeV and $L_{int}=3$ ab$^{-1}$ are found to be thirteen orders of magnitude stronger than the upper bound obtained from rare decay $Z\to\gamma\gamma\nu\bar{\nu}$ analysis using LEP data., Comment: 17 pages, 9 figures

The presence of multi-boson self-interactions is implied by the non-Abelian gauge structure of the Standard Model (SM). Precise measurements of these interactions allow not only testing the nature of the SM but also new physics contribution arising from the beyond SM. The investigation of these interactions can be approached in a model-independent manner using an effective theory approach, which forms the main motivation of this study. In this paper, we examine the anomalous neutral quartic gauge couplings through the process $\gamma \gamma \rightarrow Z Z$ at the Compact Linear Collider (CLIC) with the center-of-mass energy of $\sqrt{s}=3$ TeV, integrated luminosities of ${\cal L}=5$ $\rm ab^{-1}$. The anomalous neutral quartic gauge couplings is implemented into FeynRules to generate a UFO module inserted into Madgraph to generate both background and signal events. These events are then passed through Pythia 8 for parton showering and Delphes to include realistic detector effects. We obtain that the sensitivities on the anomalous quartic neutral gauge couplings with $95\%$ Confidence Level are given as: $f_{T0}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$, $f_{T1}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$,$f_{T2}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$,$f_{T0}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$, $f_{T5}/\Lambda^{4}=[-4.08; 4.08]\times 10^{-4}$ ${\rm TeV^{-4}}$ and $f_{T8}/\Lambda^{4}=[-1.10; 1.10]\times10^{-4}$ ${\rm TeV^{-4}}$. Our results on the anomalous quartic neutral gauge couplings are set more stringent sensitivity with respect to the recent experimental limits., Comment: 21 pages, 6 figures