Search

Vision-language models (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspondence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing a single image and vice versa. We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16). ProLIP efficiently estimates uncertainty by an "uncertainty token" without extra parameters. We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demonstrate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g., shorter texts being more uncertain and more general inputs including specific ones. Utilizing text uncertainties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few-shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https://github.com/naver-ai/prolip, Comment: Code: https://github.com/naver-ai/prolip; 23 pages, 5.7 MB

Dense cores inherit turbulent motions from the interstellar medium in which they form. As a tool for comparison to both simulations and observations, it is valuable to construct theoretical core models that can relate their internal density and velocity structure while predicting their stability to gravitational collapse. To this end, we solve the angle-averaged equations of hydrodynamics under two assumptions: 1) the system is in a quasi-steady equilibrium; 2) the velocity field consists of radial bulk motion plus isotropic turbulence, with turbulent dispersion increasing as a power-law in the radius. The resulting turbulent equilibrium sphere (TES) solutions form a two-parameter family, characterized by the sonic radius $r_s$ and the power-law index $p$. The TES is equivalent to the Bonnor-Ebert (BE) sphere when $r_s\to \infty$. The density profile in outer regions of the TES is slightly shallower than the BE sphere, but is steeper than the logotropic model. Stability analysis shows that the TESs with size exceeding a certain critical radius are unstable to radial perturbations. The center-to-edge density contrast, mass, and radius of the marginally stable TES all increase with increasing average velocity dispersion. The FWHM of the column density profile is always smaller than the critical radius, by a larger factor at higher velocity dispersion, suggesting that observations need to probe beyond the FWHM to capture the full extent of turbulent cores. When applied to the highly turbulent regime typical of cluster-forming clumps, the critical mass and radius of the TES intriguingly resembles the typical mass and radius of observed star clusters., Comment: 17 pages, 11 figures, accepted for publication in ApJ. A python package that implements the model presented in this paper including a Jupyter notebook for reproducing the figures is available in https://github.com/sanghyukmoon/tesphere

Employing Large Language Models (LLMs) to assess the quality of generated responses, such as prompting instruct-tuned models or fine-tuning judge models, has become a widely adopted evaluation method. It is also known that such evaluators are vulnerable to biases, such as favoring longer responses. While it is important to overcome this problem, the specifics of these biases remain under-explored. In this work, we qualitatively identify six types of biases inherent in various judge models. We propose EvalBiasBench as a meta-evaluation collection of hand-crafted test cases for each bias type. Additionally, we present de-biasing dataset construction methods and the associated preference dataset OffsetBias. Experimental results demonstrate that fine-tuning on our dataset significantly enhances the robustness of judge models against biases and improves performance across most evaluation scenarios. We release our datasets and the fine-tuned judge model to public., Comment: EMNLP2024 Findings

Multimodal generative models have shown impressive advances with the help of powerful diffusion models. Despite the progress, generating sound solely from text poses challenges in ensuring comprehensive scene depiction and temporal alignment. Meanwhile, video-to-sound generation limits the flexibility to prioritize sound synthesis for specific objects within the scene. To tackle these challenges, we propose a novel video-and-text-to-sound generation method, called ReWaS, where video serves as a conditional control for a text-to-audio generation model. Our method estimates the structural information of audio (namely, energy) from the video while receiving key content cues from a user prompt. We employ a well-performing text-to-sound model to consolidate the video control, which is much more efficient for training multimodal diffusion models with massive triplet-paired (audio-video-text) data. In addition, by separating the generative components of audio, it becomes a more flexible system that allows users to freely adjust the energy, surrounding environment, and primary sound source according to their preferences. Experimental results demonstrate that our method shows superiority in terms of quality, controllability, and training efficiency. Our demo is available at https://naver-ai.github.io/rewas, Comment: Project page: https://naver-ai.github.io/rewas

Composed Image Retrieval (CIR) aims to retrieve a target image based on a reference image and conditioning text, enabling controllable searches. Due to the expensive dataset construction cost for CIR triplets, a zero-shot (ZS) CIR setting has been actively studied to eliminate the need for human-collected triplet datasets. The mainstream of ZS-CIR employs an efficient projection module that projects a CLIP image embedding to the CLIP text token embedding space, while fixing the CLIP encoders. Using the projected image embedding, these methods generate image-text composed features by using the pre-trained text encoder. However, their CLIP image and text encoders suffer from the task discrepancy between the pre-training task (text $\leftrightarrow$ image) and the target CIR task (image + text $\leftrightarrow$ image). Conceptually, we need expensive triplet samples to reduce the discrepancy, but we use cheap text triplets instead and update the text encoder. To that end, we introduce the Reducing Task Discrepancy of text encoders for Composed Image Retrieval (RTD), a plug-and-play training scheme for the text encoder that enhances its capability using a novel target-anchored text contrastive learning. We also propose two additional techniques to improve the proposed learning scheme: a hard negatives-based refined batch sampling strategy and a sophisticated concatenation scheme. Integrating RTD into the state-of-the-art projection-based ZS-CIR methods significantly improves performance across various datasets and backbones, demonstrating its efficiency and generalizability., Comment: 17 pages

NGC~613 is a nearby barred spiral galaxy with a nuclear ring. Exploiting high spatial resolution ($\approx20$ pc) Atacama Large Millimeter/sub-millimeter Array $^{12}$CO(1-0) observations, we study the giant molecular clouds (GMCs) in the nuclear ring and its vicinity, identifying $158$ spatially- and spectrally-resolved GMCs. The GMC sizes ($R_{\mathrm{c}}$) are comparable to those of the clouds in the Milky Way (MW) disc, but their gas masses, observed linewidths ($\sigma_{\mathrm{obs,los}}$) and gas mass surface densities are larger. The GMC size -- linewidth relation ($\sigma_{\mathrm{obs,los}}\propto R_{\mathrm{c}}^{0.77}$) is steeper than that of the clouds of the MW disc and centre, and the GMCs are on average only marginally gravitationally bound (with a mean virial parameter $\langle\alpha_{\mathrm{obs,vir}}\rangle\approx1.7$). We discuss the possible origins of the steep size -- linewidth relation and enhanced observed linewidths of the clouds and suggest that a combination of mechanisms such as stellar feedback, gas accretion and cloud-cloud collisions, as well as the gas inflows driven by the large-scale bar, may play a role., Comment: 15 pages, 8 figures, accepted for publication in MNRAS. arXiv admin note: text overlap with arXiv:2304.10471

Can web-based image processing and visualization tools easily integrate into existing websites without significant time and effort? Our Boostlet.js library addresses this challenge by providing an open-source, JavaScript-based web framework to enable additional image processing functionalities. Boostlet examples include kernel filtering, image captioning, data visualization, segmentation, and web-optimized machine-learning models. To achieve this, Boostlet.js uses a browser bookmark to inject a user-friendly plugin selection tool called PowerBoost into any host website. Boostlet also provides on-site access to a standard API independent of any visualization framework for pixel data and scene manipulation. Web-based Boostlets provide a modular architecture and client-side processing capabilities to apply advanced image-processing techniques using consumer-level hardware. The code is open-source and available., Comment: 5 pages, 5 figures

In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a novel methodology designed to meticulously extract modality-wise meaningful and well-aligned data from extensive, noisy image-text pair datasets. Our approach leverages hyperbolic embeddings and the concept of entailment cones to evaluate and filter out samples with meaningless or underspecified semantics, focusing on enhancing the specificity of each data sample. HYPE not only demonstrates a significant improvement in filtering efficiency but also sets a new state-of-the-art in the DataComp benchmark when combined with existing filtering techniques. This breakthrough showcases the potential of HYPE to refine the data selection process, thereby contributing to the development of more accurate and efficient self-supervised learning models. Additionally, the image specificity $\epsilon_{i}$ can be independently applied to induce an image-only dataset from an image-text or image-only data pool for training image-only self-supervised models and showed superior performance when compared to the dataset induced by CLIP score., Comment: ECCV 2024; 33pages, 4.5MB

Recent Vision-Language Pre-training (VLP) models have demonstrated significant advancements. Nevertheless, these models heavily rely on image-text pairs that capture only coarse and global information of an image, leading to a limitation in their regional understanding ability. In this work, we introduce \textbf{RegionVLM}, equipped with explicit regional modeling capabilities, allowing them to understand user-indicated image regions. To achieve this, we design a simple yet innovative architecture, requiring no modifications to the model architecture or objective function. Additionally, we leverage a dataset that contains a novel source of information, namely Localized Narratives, which has been overlooked in previous VLP research. Our experiments demonstrate that our single generalist model not only achieves an interactive dialogue system but also exhibits superior performance on various zero-shot region understanding tasks, without compromising its ability for global image understanding., Comment: NAACL 2024 Main Conference

Melanoma is the most aggressive form of skin cancer, and early detection can significantly increase survival rates and prevent cancer spread. However, developing reliable automated detection techniques is difficult due to the lack of standardized datasets and evaluation methods. This study introduces a unified melanoma classification approach that supports 54 combinations of 11 datasets and 24 state-of-the-art deep learning architectures. It enables a fair comparison of 1,296 experiments and results in a lightweight model deployable to the web-based MeshNet architecture named Mela-D. This approach can run up to 33x faster by reducing parameters 24x to yield an analogous 88.8\% accuracy comparable with ResNet50 on previously unseen images. This allows efficient and accurate melanoma detection in real-world settings that can run on consumer-level hardware., Comment: 10 pages, 9 figures

Composed image retrieval (CIR) task takes a composed query of image and text, aiming to search relative images for both conditions. Conventional CIR approaches need a training dataset composed of triplets of query image, query text, and target image, which is very expensive to collect. Several recent works have worked on the zero-shot (ZS) CIR paradigm to tackle the issue without using pre-collected triplets. However, the existing ZS-CIR methods show limited backbone scalability and generalizability due to the lack of diversity of the input texts during training. We propose a novel CIR framework, only using language for its training. Our LinCIR (Language-only training for CIR) can be trained only with text datasets by a novel self-supervision named self-masking projection (SMP). We project the text latent embedding to the token embedding space and construct a new text by replacing the keyword tokens of the original text. Then, we let the new and original texts have the same latent embedding vector. With this simple strategy, LinCIR is surprisingly efficient and highly effective; LinCIR with CLIP ViT-G backbone is trained in 48 minutes and shows the best ZS-CIR performances on four different CIR benchmarks, CIRCO, GeneCIS, FashionIQ, and CIRR, even outperforming supervised method on FashionIQ. Code is available at https://github.com/navervision/lincir, Comment: CVPR 2024 camera-ready; First two authors contributed equally; 17 pages, 3.1MB

Masked image modeling (MIM) has become a leading self-supervised learning strategy. MIMs such as Masked Autoencoder (MAE) learn strong representations by randomly masking input tokens for the encoder to process, with the decoder reconstructing the masked tokens to the input. However, MIM pre-trained encoders often exhibit a limited attention span, attributed to MIM's sole focus on regressing masked tokens only, which may impede the encoder's broader context learning. To tackle the limitation, we improve MIM by explicitly incorporating unmasked tokens into the training process. Specifically, our method enables the encoder to learn from broader context supervision, allowing unmasked tokens to experience broader contexts while the decoder reconstructs masked tokens. Thus, the encoded unmasked tokens are equipped with extensive contextual information, empowering masked tokens to leverage the enhanced unmasked tokens for MIM. As a result, our simple remedy trains more discriminative representations revealed by achieving 84.2% top-1 accuracy with ViT-B on ImageNet-1K with 0.6%p gain. We attribute the success to the enhanced pre-training method, as evidenced by the singular value spectrum and attention analyses. Finally, our models achieve significant performance gains at the downstream semantic segmentation and fine-grained visual classification tasks; and on diverse robust evaluation metrics. Code is available at https://github.com/naver-ai/lut, Comment: 24 pages, 6 figures, 10 tables. To be presented at ECCV'24

We study the Bochner-Riesz problem for the twisted Laplacian $\mathcal L$ on $\mathbb R^2$. For $p\in [1, \infty]\setminus\{2\}$, it has been conjectured that the Bochner-Riesz means $S_\lambda^\delta(\mathcal L) f$ of order $\delta$ converges in $L^p$ for every $f\in L^p$ if and only if $\delta> \max(0,|(p-2)/p|-1/2)$. We prove the conjecture by obtaining uniform $L^p$ bounds on $S_\lambda^\delta(\mathcal L)$ up to the sharp summability indices., Comment: 15 pages

In this note we show that the strong spherical maximal function in $\mathbb R^d$ is bounded on $L^p$ if $p>2(d+1)/(d-1)$ for $d\ge 3$., Comment: 13 pages

Image-Text Matching (ITM) task, a fundamental vision-language (VL) task, suffers from the inherent ambiguity arising from multiplicity and imperfect annotations. Deterministic functions are not sufficiently powerful to capture ambiguity, prompting the exploration of probabilistic embeddings to tackle the challenge. However, the existing probabilistic ITM approach encounters two key shortcomings; the burden of heavy computations due to the Monte Carlo approximation, and the loss saturation issue in the face of abundant false negatives. To overcome the issues, this paper presents an improved Probabilistic Cross-Modal Embeddings (named PCME++) by introducing a new probabilistic distance with a closed-form solution. In addition, two optimization techniques are proposed to enhance PCME++ further: first, the incorporation of pseudo-positives to prevent the negative effect under massive false negatives; second, mixed sample data augmentation for probabilistic matching. Experimental results on MS-COCO Caption and two extended benchmarks, CxC and ECCV Caption, demonstrate the effectiveness of PCME++ compared to state-of-the-art ITM methods. The robustness of PCME++ is also evaluated under noisy image-text correspondences. In addition, the potential applicability of PCME++ in automatic prompt-filtering for zero-shot classification is shown. The code is available at https://github.com/naver-ai/pcmepp, Comment: ICLR 2024 camera-ready; Code: https://github.com/naver-ai/pcmepp. Project page: https://naver-ai.github.io/pcmepp/. 30 pages, 2.2 MB

We study the elliptic maximal functions defined by averages over ellipses and rotated ellipses which are multi-parametric variants of the circular maximal function. We prove that those maximal functions are bounded on $L^p$ for some $p\neq \infty$. For this purpose, we obtain some sharp multi-parameter local smoothing estimates., Comment: to appear in Journal of Functional Analysis

Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), digital twin, Metaverse and other related digital technologies have attracted much attention in recent years. These new emerging technologies are changing the world significantly. This research introduces a fusion model, i.e. Fusion Universe (FU), where the virtual, physical, and cognitive worlds are merged together. Therefore, it is crucial to establish a set of principles for the fusion model that is compatible with our physical universe laws and principles. This paper investigates several aspects that could affect immersive and interactive experience; and proposes the fundamental principles for Fusion Universe that can integrate physical and virtual world seamlessly., Comment: 5 pages, 8 figures

Meta-learning performs adaptation through a limited amount of support set, which may cause a sample bias problem. To solve this problem, transductive meta-learning is getting more and more attention, going beyond the conventional inductive learning perspective. This paper proposes so-called task-adaptive pseudo labeling for transductive meta-learning. Specifically, pseudo labels for unlabeled query sets are generated from labeled support sets through label propagation. Pseudo labels enable to adopt the supervised setting as it is and also use the unlabeled query set in the adaptation process. As a result, the proposed method is able to deal with more examples in the adaptation process than inductive ones, which can result in better classification performance of the model. Note that the proposed method is the first approach of applying task adaptation to pseudo labeling. Experiments show that the proposed method outperforms the state-of-the-art (SOTA) technique in 5-way 1-shot few-shot classification.

With the extensive use of vision-language models in various downstream tasks, evaluating their robustness is crucial. In this paper, we propose a benchmark for assessing the robustness of vision-language models. We believe that a robust model should properly understand both linguistic and visual semantics and be resilient to explicit variations. In pursuit of this goal, we create new variants of texts and images in the MS-COCO test set and re-evaluate the state-of-the-art (SOTA) models with the new data. Specifically, we alter the meaning of text by replacing a word, and generate visually altered images that maintain some visual context while introducing noticeable pixel changes through image mixing techniques.Our evaluations on the proposed benchmark reveal substantial performance degradation in many SOTA models (e.g., Image-to-Text Recall@1: 81.9\% $\rightarrow$ 48.4\% in BLIP, 66.1\% $\rightarrow$ 37.6\% in VSE$\infty$), with the models often favoring the altered texts/images over the original ones. This indicates the current vision-language models struggle with subtle changes and often fail to understand the overall context of texts and images. Based on these findings, we propose semantic contrastive loss and visual contrastive loss to learn more robust embedding. Datasets and code are available at {\url{https://github.com/pseulki/rococo}}., Comment: Accepted to ECCV Synthetic Data for Computer Vision Workshop (Oral)

Recovering 3D human mesh in the wild is greatly challenging as in-the-wild (ITW) datasets provide only 2D pose ground truths (GTs). Recently, 3D pseudo-GTs have been widely used to train 3D human mesh estimation networks as the 3D pseudo-GTs enable 3D mesh supervision when training the networks on ITW datasets. However, despite the great potential of the 3D pseudo-GTs, there has been no extensive analysis that investigates which factors are important to make more beneficial 3D pseudo-GTs. In this paper, we provide three recipes to obtain highly beneficial 3D pseudo-GTs of ITW datasets. The main challenge is that only 2D-based weak supervision is allowed when obtaining the 3D pseudo-GTs. Each of our three recipes addresses the challenge in each aspect: depth ambiguity, sub-optimality of weak supervision, and implausible articulation. Experimental results show that simply re-training state-of-the-art networks with our new 3D pseudo-GTs elevates their performance to the next level without bells and whistles. The 3D pseudo-GT is publicly available in https://github.com/mks0601/NeuralAnnot_RELEASE., Comment: Published at CVPRW 2023

Recently, the idea of using FP8 as a number format for neural network training has been floating around the deep learning world. Given that most training is currently conducted with entire networks in FP32, or sometimes FP16 with mixed-precision, the step to having some parts of a network run in FP8 with 8-bit weights is an appealing potential speed-up for the generally costly and time-intensive training procedures in deep learning. A natural question arises regarding what this development means for efficient inference on edge devices. In the efficient inference device world, workloads are frequently executed in INT8. Sometimes going even as low as INT4 when efficiency calls for it. In this whitepaper, we compare the performance for both the FP8 and INT formats for efficient on-device inference. We theoretically show the difference between the INT and FP formats for neural networks and present a plethora of post-training quantization and quantization-aware-training results to show how this theory translates to practice. We also provide a hardware analysis showing that the FP formats are somewhere between 50-180% less efficient in terms of compute in dedicated hardware than the INT format. Based on our research and a read of the research field, we conclude that although the proposed FP8 format could be good for training, the results for inference do not warrant a dedicated implementation of FP8 in favor of INT8 for efficient inference. We show that our results are mostly consistent with previous findings but that important comparisons between the formats have thus far been lacking. Finally, we discuss what happens when FP8-trained networks are converted to INT8 and conclude with a brief discussion on the most efficient way for on-device deployment and an extensive suite of INT8 results for many models.

This paper proposes a novel diffusion-based model, CompoDiff, for solving zero-shot Composed Image Retrieval (ZS-CIR) with latent diffusion. This paper also introduces a new synthetic dataset, named SynthTriplets18M, with 18.8 million reference images, conditions, and corresponding target image triplets to train CIR models. CompoDiff and SynthTriplets18M tackle the shortages of the previous CIR approaches, such as poor generalizability due to the small dataset scale and the limited types of conditions. CompoDiff not only achieves a new state-of-the-art on four ZS-CIR benchmarks, including FashionIQ, CIRR, CIRCO, and GeneCIS, but also enables a more versatile and controllable CIR by accepting various conditions, such as negative text, and image mask conditions. CompoDiff also shows the controllability of the condition strength between text and image queries and the trade-off between inference speed and performance, which are unavailable with existing CIR methods. The code and dataset are available at https://github.com/navervision/CompoDiff, Comment: TMLR camera-ready; First two authors contributed equally; TMLR Expert Certification; 30 pages, 5.9MB

We need billion-scale images to achieve more generalizable and ground-breaking vision models, as well as massive dataset storage to ship the images (e.g., the LAION-4B dataset needs 240TB storage space). However, it has become challenging to deal with unlimited dataset storage with limited storage infrastructure. A number of storage-efficient training methods have been proposed to tackle the problem, but they are rarely scalable or suffer from severe damage to performance. In this paper, we propose a storage-efficient training strategy for vision classifiers for large-scale datasets (e.g., ImageNet) that only uses 1024 tokens per instance without using the raw level pixels; our token storage only needs <1% of the original JPEG-compressed raw pixels. We also propose token augmentations and a Stem-adaptor module to make our approach able to use the same architecture as pixel-based approaches with only minimal modifications on the stem layer and the carefully tuned optimization settings. Our experimental results on ImageNet-1k show that our method significantly outperforms other storage-efficient training methods with a large gap. We further show the effectiveness of our method in other practical scenarios, storage-efficient pre-training, and continual learning. Code is available at https://github.com/naver-ai/seit, Comment: ICCV 2023; First two authors contributed equally; code url: https://github.com/naver-ai/seit; 17 pages, 1.2MB

Nuclear rings at the centers of barred galaxies are known to be strongly magnetized. To explore the effects of magnetic fields on star formation in these rings and nuclear gas flows, we run magnetohydrodynamic simulations in which there is a temporally-constant magnetized inflow to the ring, representing a bar-driven inflow. The mass inflow rate is $1\,M_\odot\,\mathrm{yr}^{-1}$, and we explore models with a range of field strength in the inflow. We adopt the TIGRESS framework developed by Kim & Ostriker to handle radiative heating and cooling, star formation, and resulting supernova (SN) feedback. We find that magnetic fields are efficiently amplified in the ring due to rotational shear and SN feedback. Within a few $100\,\mathrm{Myr}$, the turbulent component $B_\mathrm{trb}$ in the ring saturates at $\sim 35\,\mu\mathrm{G}$ (in rough equipartition with the turbulent kinetic energy density), while the regular component $B_\mathrm{reg}$ exceeds $50\,\mu\mathrm{G}$. Expanding superbubbles created by clustered SN explosions vertically drag predominantly-toroidal fields from near the midplane to produce poloidal fields in high-altitude regions. The growth of magnetic fields greatly suppresses star formation at late times. Simultaneously, strong magnetic tension in the ring drives radially inward accretion flows from the ring to form a circumnuclear disk in the central region; this feature is absent in the unmagnetized model., Comment: 29 pages, 17 figures, accepted for publication in ApJ

Let $\mathcal L$ denote the twisted Laplacian in $\mathbb C^d$. We study almost everywhere convergence of the Bochner--Riesz mean $S^\delta_{t}(\mathcal L) f$ of $f\in L^p(\mathbb C^d)$ as $t\to \infty$, which is an expansion of $f$ in the special Hermite functions. For $2\le p\le \infty$, we obtain the sharp range of the summability indices $\delta$ for which the convergence of $S^\delta_{t}(\mathcal L) f$ holds for all $f\in L^p(\mathbb C^d)$., Comment: 23 pages

Many existing group fairness-aware training methods aim to achieve the group fairness by either re-weighting underrepresented groups based on certain rules or using weakly approximated surrogates for the fairness metrics in the objective as regularization terms. Although each of the learning schemes has its own strength in terms of applicability or performance, respectively, it is difficult for any method in the either category to be considered as a gold standard since their successful performances are typically limited to specific cases. To that end, we propose a principled method, dubbed as \ours, which unifies the two learning schemes by incorporating a well-justified group fairness metric into the training objective using a class wise distributionally robust optimization (DRO) framework. We then develop an iterative optimization algorithm that minimizes the resulting objective by automatically producing the correct re-weights for each group. Our experiments show that FairDRO is scalable and easily adaptable to diverse applications, and consistently achieves the state-of-the-art performance on several benchmark datasets in terms of the accuracy-fairness trade-off, compared to recent strong baselines.

Machine-learned force fields have generated significant interest in recent years as a tool for molecular dynamics (MD) simulations, with the aim of developing accurate and efficient models that can replace classical interatomic potentials. However, before these models can be confidently applied to materials simulations, they must be thoroughly tested and validated. The existing tests on the radial distribution function and mean-squared displacements are insufficient in assessing the transferability of these models. Here we present a more comprehensive set of benchmarking tests for evaluating the transferability of machine-learned force fields. We use a graph neural network (GNN)-based force field coupled with the OpenMM package to carry out MD simulations for Argon as a test case. Our tests include computational X-ray photon correlation spectroscopy (XPCS) signals, which capture the density fluctuation at various length scales in the liquid phase, as well as phonon density-of-state in the solid phase and the liquid-solid phase transition behavior. Our results show that the model can accurately capture the behavior of the solid phase only when the configurations from the solid phase are included in the training dataset. This underscores the importance of appropriately selecting the training data set when developing machine-learned force fields. The tests presented in this work provide a necessary foundation for the development and application of machine-learned force fields for materials simulations., Comment: 27 pages, 14 figures, under review

We study $L^{p}\times L^{q}\rightarrow L^{r}$-boundedness of (sub)bilinear maximal functions associated with degenerate hypersurfaces. First, we obtain the maximal bound on the sharp range of exponents $p,q,r$ (except some border line cases) for the bilinear maximal functions given by the model surface $\big\{(y,z)\in\mathbb{R}^{n}\times \mathbb{R}^{n}:|y|^{l_{1}}+|z|^{l_{2}}=1\big\}$, $(l_{1},l_{2})\in [1,\infty)^2$, $n\ge 2$. Our result manifests that nonvanishing Gaussian curvature is not good enough, in contrast with $L^p$-boundedness of the (sub)linear maximal operator associated to hypersurfaces, to characterize the best possible maximal boundedness. Secondly, we consider the bilinear maximal function associated to the finite type curve in $\mathbb R^2$ and obtain a complete characterization of the maximal bound. We also prove multilinear generalizations of the aforementioned results., Comment: 21 pages, 3 figures

Vision Transformer (ViT) extracts the final representation from either class token or an average of all patch tokens, following the architecture of Transformer in Natural Language Processing (NLP) or Convolutional Neural Networks (CNNs) in computer vision. However, studies for the best way of aggregating the patch tokens are still limited to average pooling, while widely-used pooling strategies, such as max and GeM pooling, can be considered. Despite their effectiveness, the existing pooling strategies do not consider the architecture of ViT and the channel-wise difference in the activation maps, aggregating the crucial and trivial channels with the same importance. In this paper, we present Group Generalized Mean (GGeM) pooling as a simple yet powerful pooling strategy for ViT. GGeM divides the channels into groups and computes GeM pooling with a shared pooling parameter per group. As ViT groups the channels via a multi-head attention mechanism, grouping the channels by GGeM leads to lower head-wise dependence while amplifying important channels on the activation maps. Exploiting GGeM shows 0.1%p to 0.7%p performance boosts compared to the baselines and achieves state-of-the-art performance for ViT-Base and ViT-Large models in ImageNet-1K classification task. Moreover, GGeM outperforms the existing pooling strategies on image retrieval and multi-modal representation learning tasks, demonstrating the superiority of GGeM for a variety of tasks. GGeM is a simple algorithm in that only a few lines of code are necessary for implementation.

We investigate $L^p$ boundedness of the maximal function defined by the averaging operator $f\to \mathcal{A}_t^s f$ over the two-parameter family of tori $\mathbb{T}_t^{s}:=\{ ( (t+s\cos\theta)\cos\phi,\,(t+s\cos\theta)\sin\phi,\, s\sin\theta ): \theta, \phi \in [0,2\pi) \}$ with $c_0t>s>0$ for some $c_0\in (0,1)$. We prove that the associated (two-parameter) maximal function is bounded on $L^p$ if and only if $p>2$. We also obtain $L^p$--$L^q$ estimates for the local maximal operator on a sharp range of $p,q$. Furthermore, the sharp smoothing estimates are proved including the sharp local smoothing estimates for the operators $f\to \mathcal A_t^s f$ and $f\to \mathcal A_t^{c_0t} f$. For the purpose, we make use of Bourgain--Demeter's decoupling inequality for the cone and Guth--Wang--Zhang's local smoothing estimates for the $2$ dimensional wave operator., Comment: 30 pages, 2 figures

In recent years, many deep neural architectures have been developed for image classification. Whether they are similar or dissimilar and what factors contribute to their (dis)similarities remains curious. To address this question, we aim to design a quantitative and scalable similarity measure between neural architectures. We propose Similarity by Attack Transferability (SAT) from the observation that adversarial attack transferability contains information related to input gradients and decision boundaries widely used to understand model behaviors. We conduct a large-scale analysis on 69 state-of-the-art ImageNet classifiers using our proposed similarity function to answer the question. Moreover, we observe neural architecture-related phenomena using model similarity that model diversity can lead to better performance on model ensembles and knowledge distillation under specific conditions. Our results provide insights into why developing diverse neural architectures with distinct components is necessary., Comment: ECCV 2024; 35pages, 2.56MB

We study $L^p$-$L^q$ bounds on the spectral projection operator $\Pi_\lambda$ associated to the Hermite operator $H=|x|^2-\Delta$ in $\mathbb R^d$. We are mainly concerned with a localized operator $\chi_E\Pi_\lambda\chi_E$ for a subset $E\subset\mathbb R^d$ and undertake the task of characterizing the sharp $L^p$--$L^q$ bounds. We obtain sharp bounds in extended ranges of $p,q$. First, we provide a complete characterization of the sharp $L^p$--$L^q$ bounds when $E$ is away from $\sqrt{\lambda}\mathbb S^{d-1}$. Secondly, we obtain the sharp bounds as the set $E$ gets close to $\sqrt\lambda\mathbb S^{d-1}$. Thirdly, we extend the range of $p,q$ for which the operator $\Pi_\lambda$ is uniformly bounded from $L^p(\mathbb R^d)$ to $L^q(\mathbb R^d)$., Comment: The paper is a modified version of a part of the paper Hermite spectral projection operator (arXiv:2006.11762v3). The previous paper will remain unpublished

We propose the first unified theoretical analysis of mixed sample data augmentation (MSDA), such as Mixup and CutMix. Our theoretical results show that regardless of the choice of the mixing strategy, MSDA behaves as a pixel-level regularization of the underlying training loss and a regularization of the first layer parameters. Similarly, our theoretical results support that the MSDA training strategy can improve adversarial robustness and generalization compared to the vanilla training strategy. Using the theoretical results, we provide a high-level understanding of how different design choices of MSDA work differently. For example, we show that the most popular MSDA methods, Mixup and CutMix, behave differently, e.g., CutMix regularizes the input gradients by pixel distances, while Mixup regularizes the input gradients regardless of pixel distances. Our theoretical results also show that the optimal MSDA strategy depends on tasks, datasets, or model parameters. From these observations, we propose generalized MSDAs, a Hybrid version of Mixup and CutMix (HMix) and Gaussian Mixup (GMix), simple extensions of Mixup and CutMix. Our implementation can leverage the advantages of Mixup and CutMix, while our implementation is very efficient, and the computation cost is almost neglectable as Mixup and CutMix. Our empirical study shows that our HMix and GMix outperform the previous state-of-the-art MSDA methods in CIFAR-100 and ImageNet classification tasks. Source code is available at https://github.com/naver-ai/hmix-gmix, Comment: First two authors contributed equally; 29 pages

We study an analogue of Marstrand's circle packing problem for curves in higher dimensions. We consider collections of curves which are generated by translation and dilation of a curve $\gamma$ in $\mathbb R^d$, i.e., $ x + t \gamma$, $(x,t) \in \mathbb R^d \times (0,\infty)$. For a Borel set $F \subset \mathbb R^d\times (0,\infty)$, we show the unions of curves $\bigcup_{(x,t) \in F} ( x+t\gamma )$ has Hausdorff dimension at least $\alpha+1$ whenever $F$ has Hausdorff dimension bigger than $\alpha$, $\alpha\in (0, d-1)$. We also obtain results for unions of curves generated by multi-parameter dilation of $\gamma$. One of the main ingredients is a local smoothing type estimate (for averages over curves) relative to fractal measures., Comment: 15 pages

We study pointwise convergence of the fractional Schr\"odinger means along sequences $t_n$ which converge to zero. Our main result is that bounds on the maximal function $\sup_{n} |e^{it_n(-\Delta)^{\alpha/2}} f| $ can be deduced from those on $\sup_{0

We obtain a complete characterization of $L^p-L^q$ Carleman estimates with weight $e^{v\cdot x}$ for the polyharmonic operators. Our result extends the Carleman inequalities for the Laplacian due to Kenig--Ruiz--Sogge. Consequently, we obtain new unique continuation properties of higher order Schr\"odinger equations relaxing the integrability assumption on the solution spaces.

We establish the optimal $L^p$, $p=2(d+3)/(d+1),$ eigenfunction bound for the Hermite operator $\mathcal H=-\Delta+|x|^2$ on $\mathbb R^d$. Let $\Pi_\lambda$ denote the projection operator to the vector space spanned by the eigenfunctions of $\mathcal H$ with eigenvalue $\lambda$. The optimal $L^2$--$L^p$ bounds on $\Pi_\lambda$, $2\le p\le \infty$, have been known by the works of Karadzhov and Koch-Tataru except $p=2(d+3)/(d+1)$. For $d\ge 3$, we prove the optimal bound for the missing endpoint case. Our result is built on a new phenomenon: improvement of the bound due to asymmetric localization near the sphere $\sqrt\lambda \mathbb S^{d-1}$., Comment: Final version, to appear in JEMS. The paper is an extended revision of a part of the paper Hermite spectral projection operator (arXiv:2006.11762). The earlier paper will remain unpublished permanently

Transformers have been widely used in numerous vision problems especially for visual recognition and detection. Detection transformers are the first fully end-to-end learning systems for object detection, while vision transformers are the first fully transformer-based architecture for image classification. In this paper, we integrate Vision and Detection Transformers (ViDT) to construct an effective and efficient object detector. ViDT introduces a reconfigured attention module to extend the recent Swin Transformer to be a standalone object detector, followed by a computationally efficient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load. In addition, we extend it to ViDT+ to support joint-task learning for object detection and instance segmentation. Specifically, we attach an efficient multi-scale feature fusion layer and utilize two more auxiliary training losses, IoU-aware loss and token labeling loss. Extensive evaluation results on the Microsoft COCO benchmark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer-based object detectors, and its extended ViDT+ achieves 53.2AP owing to its high scalability for large models. The source code and trained models are available at https://github.com/naver-ai/vidt., Comment: An extension of the ICLR paper, ViDT: An Efficient and Effective Fully Transformer-based Object Detector. arXiv admin note: substantial text overlap with arXiv:2110.03921

Image-Text matching (ITM) is a common task for evaluating the quality of Vision and Language (VL) models. However, existing ITM benchmarks have a significant limitation. They have many missing correspondences, originating from the data construction process itself. For example, a caption is only matched with one image although the caption can be matched with other similar images and vice versa. To correct the massive false negatives, we construct the Extended COCO Validation (ECCV) Caption dataset by supplying the missing associations with machine and human annotators. We employ five state-of-the-art ITM models with diverse properties for our annotation process. Our dataset provides x3.6 positive image-to-caption associations and x8.5 caption-to-image associations compared to the original MS-COCO. We also propose to use an informative ranking-based metric mAP@R, rather than the popular Recall@K (R@K). We re-evaluate the existing 25 VL models on existing and proposed benchmarks. Our findings are that the existing benchmarks, such as COCO 1K R@K, COCO 5K R@K, CxC R@1 are highly correlated with each other, while the rankings change when we shift to the ECCV mAP@R. Lastly, we delve into the effect of the bias introduced by the choice of machine annotator. Source code and dataset are available at https://github.com/naver-ai/eccv-caption, Comment: Published in ECCV 2022; 32 pages (2.3MB); Code and dataset: https://github.com/naver-ai/eccv-caption; v5 fixes errors in Table 4: the COCO 1K R@1 numbers were incorrect. All other tables and figures are correct. v5 also adds RSUM scores in Tab 4 and 5: RSUM has a high correlation with COCO 1K recalls; v4 fixes errors in v3 -- see the v4 comment for details

Domain generalization (DG) aims to learn a generalized model to an unseen target domain using only limited source domains. Previous attempts to DG fail to learn domain-invariant representations only from the source domains due to the significant domain shifts between training and test domains. Instead, we re-formulate the DG objective using mutual information with the oracle model, a model generalized to any possible domain. We derive a tractable variational lower bound via approximating the oracle model by a pre-trained model, called Mutual Information Regularization with Oracle (MIRO). Our extensive experiments show that MIRO significantly improves the out-of-distribution performance. Furthermore, our scaling experiments show that the larger the scale of the pre-trained model, the greater the performance improvement of MIRO. Source code is available at https://github.com/kakaobrain/miro., Comment: ECCV 2022 camera-ready

Recent studies have demonstrated that gradient matching-based dataset synthesis, or dataset condensation (DC), methods can achieve state-of-the-art performance when applied to data-efficient learning tasks. However, in this study, we prove that the existing DC methods can perform worse than the random selection method when task-irrelevant information forms a significant part of the training dataset. We attribute this to the lack of participation of the contrastive signals between the classes resulting from the class-wise gradient matching strategy. To address this problem, we propose Dataset Condensation with Contrastive signals (DCC) by modifying the loss function to enable the DC methods to effectively capture the differences between classes. In addition, we analyze the new loss function in terms of training dynamics by tracking the kernel velocity. Furthermore, we introduce a bi-level warm-up strategy to stabilize the optimization. Our experimental results indicate that while the existing methods are ineffective for fine-grained image classification tasks, the proposed method can successfully generate informative synthetic datasets for the same tasks. Moreover, we demonstrate that the proposed method outperforms the baselines even on benchmark datasets such as SVHN, CIFAR-10, and CIFAR-100. Finally, we demonstrate the high applicability of the proposed method by applying it to continual learning tasks., Comment: Accepted at ICML 2022

Automatic few-shot font generation aims to solve a well-defined, real-world problem because manual font designs are expensive and sensitive to the expertise of designers. Existing methods learn to disentangle style and content elements by developing a universal style representation for each font style. However, this approach limits the model in representing diverse local styles, because it is unsuitable for complicated letter systems, for example, Chinese, whose characters consist of a varying number of components (often called "radical") -- with a highly complex structure. In this paper, we propose a novel font generation method that learns localized styles, namely component-wise style representations, instead of universal styles. The proposed style representations enable the synthesis of complex local details in text designs. However, learning component-wise styles solely from a few reference glyphs is infeasible when a target script has a large number of components, for example, over 200 for Chinese. To reduce the number of required reference glyphs, we represent component-wise styles by a product of component and style factors, inspired by low-rank matrix factorization. Owing to the combination of strong representation and a compact factorization strategy, our method shows remarkably better few-shot font generation results (with only eight reference glyphs) than other state-of-the-art methods. Moreover, strong locality supervision, for example, location of each component, skeleton, or strokes, was not utilized. The source code is available at https://github.com/clovaai/lffont and https://github.com/clovaai/fewshot-font-generation., Comment: First two authors contributed equally. This is a journal extension of our AAAI 2021 paper arXiv:2009.11042; Code: https://github.com/clovaai/lffont and https://github.com/clovaai/fewshot-font-generation

Recent neural sequence-to-sequence models with a copy mechanism have achieved remarkable progress in various text generation tasks. These models addressed out-of-vocabulary problems and facilitated the generation of rare words. However, the identification of the word which needs to be copied is difficult, as observed by prior copy models, which suffer from incorrect generation and lacking abstractness. In this paper, we propose a novel supervised approach of a copy network that helps the model decide which words need to be copied and which need to be generated. Specifically, we re-define the objective function, which leverages source sequences and target vocabularies as guidance for copying. The experimental results on data-to-text generation and abstractive summarization tasks verify that our approach enhances the copying quality and improves the degree of abstractness., Comment: 8 pages, 3 figures, 8 tables and 4 pages of appendices

Recently, fairness-aware learning have become increasingly crucial, but most of those methods operate by assuming the availability of fully annotated demographic group labels. We emphasize that such assumption is unrealistic for real-world applications since group label annotations are expensive and can conflict with privacy issues. In this paper, we consider a more practical scenario, dubbed as Algorithmic Group Fairness with the Partially annotated Group labels (Fair-PG). We observe that the existing methods to achieve group fairness perform even worse than the vanilla training, which simply uses full data only with target labels, under Fair-PG. To address this problem, we propose a simple Confidence-based Group Label assignment (CGL) strategy that is readily applicable to any fairness-aware learning method. CGL utilizes an auxiliary group classifier to assign pseudo group labels, where random labels are assigned to low confident samples. We first theoretically show that our method design is better than the vanilla pseudo-labeling strategy in terms of fairness criteria. Then, we empirically show on several benchmark datasets that by combining CGL and the state-of-the-art fairness-aware in-processing methods, the target accuracies and the fairness metrics can be jointly improved compared to the baselines. Furthermore, we convincingly show that CGL enables to naturally augment the given group-labeled dataset with external target label-only datasets so that both accuracy and fairness can be improved. Code is available at https://github.com/naver-ai/cgl_fairness., Comment: Accepted to CVPR 2022; Code is available at https://github.com/naver-ai/cgl_fairness

We study $L^p$-Sobolev regularity estimate for the restricted X-ray transforms generated by nondegenerate curves. Making use of the inductive strategy in the recent work by the authors, we establish the sharp $L^p$-regularity estimates for the restricted X-ray transforms in $\mathbb R^{d+1}$, $d\ge 3$. This extends the result due to Pramanik and Seeger in $\mathbb R^3$ to every dimension., Comment: 18 pages

Observations indicate that the star formation rate (SFR) of nuclear rings varies considerably with time and is sometimes asymmetric rather than being uniform across a ring. To understand what controls temporal and spatial distributions of ring star formation, we run semi-global, hydrodynamic simulations of nuclear rings subject to time-varying and/or asymmetric mass inflow rates. These controlled variations in the inflow lead to variations in the star formation, while the ring orbital period ($18\,{\rm Myr}$) and radius ($600\,{\rm pc}$) remain approximately constant. We find that both the mass inflow rate and supernova feedback affect the ring SFR. An oscillating inflow rate with period $\Delta \tau_\text{in}$ and amplitude 20 causes large-amplitude (a factor of $\gtrsim 5$), quasi-periodic variations of the SFR, when $\Delta \tau_\text{in} \gtrsim 50\,{\rm Myr}$. We find that the time-varying ISM weight and midplane pressure track each other closely, establishing an instantaneous vertical equilibrium. The measured time-varying depletion time is consistent with the prediction from self-regulation theory provided the time delay between star formation and supernova feedback is taken into account. The supernova feedback is responsible only for small-amplitude (a factor of $\sim 2$) fluctuations of the SFR with a timescale $\lesssim 40\,{\rm Myr}$. Asymmetry in the inflow rate does not necessarily lead to asymmetric star formation in nuclear rings. Only when the inflow rate from one dust lane is suddenly increased by a large factor, the rings undergo a transient period of lopsided star formation., Comment: 15 pages, 6 figures, accepted for publication in ApJ. Replaced with minor text corrections

We prove new $L^p$- $L^q$ estimates for averages over dilates of the circle with respect to $\alpha$-dimensional fractal measure, which unify different types of maximal estimates for the circular average. Our results are consequences of $L^p$- $L^q$ smoothing estimates for the wave operator relative to fractal measures. We also discuss similar results concerning the spherical averages., Comment: 25pages

Model-agnostic meta-learning (MAML) is a well-known optimization-based meta-learning algorithm that works well in various computer vision tasks, e.g., few-shot classification. MAML is to learn an initialization so that a model can adapt to a new task in a few steps. However, since the gradient norm of a classifier (head) is much bigger than those of backbone layers, the model focuses on learning the decision boundary of the classifier with similar representations. Furthermore, gradient norms of high-level layers are small than those of the other layers. So, the backbone of MAML usually learns task-generic features, which results in deteriorated adaptation performance in the inner-loop. To resolve or mitigate this problem, we propose contextual gradient scaling (CxGrad), which scales gradient norms of the backbone to facilitate learning task-specific knowledge in the inner-loop. Since the scaling factors are generated from task-conditioned parameters, gradient norms of the backbone can be scaled in a task-wise fashion. Experimental results show that CxGrad effectively encourages the backbone to learn task-specific knowledge in the inner-loop and improves the performance of MAML up to a significant margin in both same- and cross-domain few-shot classification., Comment: Accepted to WACV2022

We investigate the spectral structure of the Neumann-Poincar\'e operator on thin ellipsoids. Two types of thin ellipsoids are considered: long prolate ellipsoids and flat oblate ellipsoids. We show that the totality of eigenvalues of the Neumann-Poincar\'e operators on a sequence of the prolate spheroids is densely distributed in the interval [0,1/2] as their eccentricities tend to 1, namely, as they become longer. We then prove that eigenvalues of the Neumann-Poincar\'e operators on the oblate ellipsoids are densely distributed in the interval [-1/2, 1/2] as the ellipsoids become flatter. In particular, this shows that even if there are at most finitely many negative eigenvalues on the oblate ellipsoids, more and more negative eigenvalues appear as the ellipsoids become flatter. We also show a similar spectral property for flat three dimensional domains., Comment: 24 pages, 1 figure

Transformers are transforming the landscape of computer vision, especially for recognition tasks. Detection transformers are the first fully end-to-end learning systems for object detection, while vision transformers are the first fully transformer-based architecture for image classification. In this paper, we integrate Vision and Detection Transformers (ViDT) to build an effective and efficient object detector. ViDT introduces a reconfigured attention module to extend the recent Swin Transformer to be a standalone object detector, followed by a computationally efficient transformer decoder that exploits multi-scale features and auxiliary techniques essential to boost the detection performance without much increase in computational load. Extensive evaluation results on the Microsoft COCO benchmark dataset demonstrate that ViDT obtains the best AP and latency trade-off among existing fully transformer-based object detectors, and achieves 49.2AP owing to its high scalability for large models. We will release the code and trained models at https://github.com/naver-ai/vidt, Comment: This is a revised version on Nov. 29