Search

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community., Comment: https://huggingface.co/spaces/CONDA-Workshop/Data-Contamination-Database

We tackle the novel class discovery in point cloud segmentation, which discovers novel classes based on the semantic knowledge of seen classes. Existing work proposes an online point-wise clustering method with a simplified equal class-size constraint on the novel classes to avoid degenerate solutions. However, the inherent imbalanced distribution of novel classes in point clouds typically violates the equal class-size constraint. Moreover, point-wise clustering ignores the rich spatial context information of objects, which results in less expressive representation for semantic segmentation. To address the above challenges, we propose a novel self-labeling strategy that adaptively generates high-quality pseudo-labels for imbalanced classes during model training. In addition, we develop a dual-level representation that incorporates regional consistency into the point-level classifier learning, reducing noise in generated segmentation. Finally, we conduct extensive experiments on two widely used datasets, SemanticKITTI and SemanticPOSS, and the results show our method outperforms the state of the art by a large margin., Comment: Accepted by ECCV2024

The evolution of Artificial Intelligence (AI) has been significantly accelerated by advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning abilities in problem-solving and scientific discovery (i.e., AI4Science) once exclusive to human intellect. To comprehensively evaluate current models' performance in cognitive reasoning abilities, we introduce OlympicArena, which includes 11,163 bilingual problems across both text-only and interleaved text-image modalities. These challenges encompass a wide range of disciplines spanning seven fields and 62 international Olympic competitions, rigorously examined for data leakage. We argue that the challenges in Olympic competition problems are ideal for evaluating AI's cognitive reasoning due to their complexity and interdisciplinary nature, which are essential for tackling complex scientific challenges and facilitating discoveries. Beyond evaluating performance across various disciplines using answer-only criteria, we conduct detailed experiments and analyses from multiple perspectives. We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions. Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39.97% overall accuracy, illustrating current AI limitations in complex reasoning and multimodal integration. Through the OlympicArena, we aim to advance AI towards superintelligence, equipping it to address more complex challenges in science and beyond. We also provide a comprehensive set of resources to support AI research, including a benchmark dataset, an open-source annotation platform, a detailed evaluation tool, and a leaderboard with automatic submission features., Comment: 44 pages

Amid the expanding use of pre-training data, the phenomenon of benchmark dataset leakage has become increasingly prominent, exacerbated by opaque training processes and the often undisclosed inclusion of supervised data in contemporary Large Language Models (LLMs). This issue skews benchmark effectiveness and fosters potentially unfair comparisons, impeding the field's healthy development. To address this, we introduce a detection pipeline utilizing Perplexity and N-gram accuracy, two simple and scalable metrics that gauge a model's prediction precision on benchmark, to identify potential data leakages. By analyzing 31 LLMs under the context of mathematical reasoning, we reveal substantial instances of training even test set misuse, resulting in potentially unfair comparisons. These findings prompt us to offer several recommendations regarding model documentation, benchmark setup, and future evaluations. Notably, we propose the "Benchmark Transparency Card" to encourage clear documentation of benchmark utilization, promoting transparency and healthy developments of LLMs. we have made our leaderboard, pipeline implementation, and model predictions publicly available, fostering future research., Comment: 30 pages; Homepage: https://gair-nlp.github.io/benbench

Enumeration of quarter-plane lattice walks with small steps is a classical problem in combinatorics. An effective approach is the kernel method and the solution is derived by positive term extraction. Another approach is to reduce the lattice walk problem to a Carleman type Riemann boundary value problem (RBVP) and solve it by complex analysis. There are two parameter characterizing the solutions of a Carleman type Riemann Boundary value problem, the index $\chi$ and the conformal gluing function $w(x)$. In this paper, we propose a combinatorial insight to the RBVP approach. We show that the index can be treated as the canonical factorization in the kernel method and the conformal gluing function can be turned into a conformal mapping such that after this mapping, the positive degree terms and the negative degree terms can be naturally separated. The combinatorial insight of RBVP provide a connection between the kernel method, the RBVP approach and the Tutte's invariant method.

While the novel class discovery has recently made great progress, existing methods typically focus on improving algorithms on class-balanced benchmarks. However, in real-world recognition tasks, the class distributions of their corresponding datasets are often imbalanced, which leads to serious performance degeneration of those methods. In this paper, we consider a more realistic setting for novel class discovery where the distributions of novel and known classes are long-tailed. One main challenge of this new problem is to discover imbalanced novel classes with the help of long-tailed known classes. To tackle this problem, we propose an adaptive self-labeling strategy based on an equiangular prototype representation of classes. Our method infers high-quality pseudo-labels for the novel classes by solving a relaxed optimal transport problem and effectively mitigates the class biases in learning the known and novel classes. We perform extensive experiments on CIFAR100, ImageNet100, Herbarium19 and large-scale iNaturalist18 datasets, and the results demonstrate the superiority of our method. Our code is available at https://github.com/kleinzcy/NCDLR., Comment: TMLR2023, Final version

We tackle the problem of novel class discovery, which aims to learn novel classes without supervision based on labeled data from known classes. A key challenge lies in transferring the knowledge in the known-class data to the learning of novel classes. Previous methods mainly focus on building a shared representation space for knowledge transfer and often ignore modeling class relations. To address this, we introduce a class relation representation for the novel classes based on the predicted class distribution of a model trained on known classes. Empirically, we find that such class relation becomes less informative during typical discovery training. To prevent such information loss, we propose a novel knowledge distillation framework, which utilizes our class-relation representation to regularize the learning of novel classes. In addition, to enable a flexible knowledge distillation scheme for each data point in novel classes, we develop a learnable weighting function for the regularization, which adaptively promotes knowledge transfer based on the semantic similarity between the novel and known classes. To validate the effectiveness and generalization of our method, we conduct extensive experiments on multiple benchmarks, including CIFAR100, Stanford Cars, CUB, and FGVC-Aircraft datasets. Our results demonstrate that the proposed method outperforms the previous state-of-the-art methods by a significant margin on almost all benchmarks. Code is available at \href{https://github.com/kleinzcy/Cr-KD-NCD}{here}., Comment: ICCV2023

We tackle the novel class discovery problem, aiming to discover novel classes in unlabeled data based on labeled data from seen classes. The main challenge is to transfer knowledge contained in the seen classes to unseen ones. Previous methods mostly transfer knowledge through sharing representation space or joint label space. However, they tend to neglect the class relation between seen and unseen categories, and thus the learned representations are less effective for clustering unseen classes. In this paper, we propose a principle and general method to transfer semantic knowledge between seen and unseen classes. Our insight is to utilize mutual information to measure the relation between seen classes and unseen classes in a restricted label space and maximizing mutual information promotes transferring semantic knowledge. To validate the effectiveness and generalization of our method, we conduct extensive experiments both on novel class discovery and general novel class discovery settings. Our results show that the proposed method outperforms previous SOTA by a significant margin on several benchmarks., Comment: The derivation of Mutual Information in the manuscript is wrong

Background:Gene regulations often change over time rather than being constant. But many of gene regulatory networks extracted from databases are static. The tumor suppressor gene $P53$ is involved in the pathogenesis of many tumors, and its inhibition effects occur after a certain period. Therefore, it is of great significance to elucidate the regulation mechanism over time points. Result:A qualitative method for representing dynamic gene regulatory network is developed, called CdtGRN. It adopts the combination of convolutional neural networks(CNN) and fully connected networks(DNN) as the core mechanism of prediction. The ionizing radiation Affymetrix dataset (E-MEXP-549) was obtained at ArrayExpress, by microarray gene expression levels predicting relations between regulation. CdtGRN is tested against a time-delayed gene regulatory network with $22,284$ genes related to $P53$. The accuracy of CdtGRN reaches 92.07$\%$ on the classification of conservative verification set, and a kappa coefficient reaches $0.84$ and an average AUC accuracy is 94.25$\%$. This resulted in the construction of. Conclusion:The algorithm and program we developed in our study would be useful for identifying dynamic gene regulatory networks, and objectively analyze the delay of the regulatory relationship by analyzing the gene expression levels at different time points. The time-delayed gene regulatory network of $P53$ is also inferred and represented qualitatively, which is helpful to understand the pathological mechanism of tumors.

Visual anomaly detection is an important and challenging problem in the field of machine learning and computer vision. This problem has attracted a considerable amount of attention in relevant research communities. Especially in recent years, the development of deep learning has sparked an increasing interest in the visual anomaly detection problem and brought a great variety of novel methods. In this paper, we provide a comprehensive survey of the classical and deep learning-based approaches for visual anomaly detection in the literature. We group the relevant approaches in view of their underlying principles and discuss their assumptions, advantages, and disadvantages carefully. We aim to help the researchers to understand the common principles of visual anomaly detection approaches and identify promising research directions in this field.

Lattice paths in the quarter plane have led to a large and varied set of results in recent years. One major project has been the classification of step sets according to the properties of the corresponding generating functions, and this has involved a variety of techniques, some highly intricate and specialised. The famous Kreweras and reverse Kreweras walk models are two particularly interesting models, as they are among the only four cases which have algebraic generating functions. Here we investigate how the properties of the Kreweras and reverse Kreweras models change when boundary interactions are introduced. That is, we associate three real-valued weights $a,b,c$ with visits by the walks to the $x$-axis, the $y$-axis and the origin $(0,0)$ respectively. These models were partially solved in a recent paper by Beaton, Owczarek and Rechnitzer (2019). We apply the algebraic kernel method to completely solve these two models. We find that reverse Kreweras walks have an algebraic generating function for all $a,b,c$, regardless of whether the walks are restricted to end at the origin or on one of the axes, or may end anywhere at all. For Kreweras walks, the generating function for walks returning to the origin is algebraic, but the other cases are only D-finite. To our knowledge this is the first example of a quarter-plane model with this property., Comment: 21 pages, 2 figures; Mathematica notebooks with many of the calculations available on the first author's webpage