Search

We report a framework that enables the wide adoption of authentic research educational methodology at various schools by addressing common barriers. The guiding principles we present were applied to implement a program in which teams of students with complementary skills develop useful artificial intelligence (AI) solutions for researchers in natural sciences. To accomplish this, we work with research laboratories that reveal/specify their needs, and then our student teams work on the discovery, design, and development of an AI solution for unique problems using a consulting-like arrangement. To date, our group has been operating at New York University (NYU) for seven consecutive semesters, has engaged more than a hundred students, ranging from first-year college students to master's candidates, and has worked with more than twenty projects and collaborators. While creating education benefits for students, our approach also directly benefits scientists, who get an opportunity to evaluate the usefulness of machine learning for their specific needs.

Non-autoregressive generative transformers recently demonstrated impressive image generation performance, and orders of magnitude faster sampling than their autoregressive counterparts. However, optimal parallel sampling from the true joint distribution of visual tokens remains an open challenge. In this paper we introduce Token-Critic, an auxiliary model to guide the sampling of a non-autoregressive generative transformer. Given a masked-and-reconstructed real image, the Token-Critic model is trained to distinguish which visual tokens belong to the original image and which were sampled by the generative transformer. During non-autoregressive iterative sampling, Token-Critic is used to select which tokens to accept and which to reject and resample. Coupled with Token-Critic, a state-of-the-art generative transformer significantly improves its performance, and outperforms recent diffusion models and GANs in terms of the trade-off between generated image quality and diversity, in the challenging class-conditional ImageNet generation., Comment: Accepted to ECCV 2022

Gamma-ray bursts (GRBs) are divided into short gamma-ray bursts (SGRBs) and long gamma-ray bursts (LGRBs) based on the bimodal distribution of their durations. LGRBs and SGRBs are typically characterized by different statistical characteristics. Nevertheless, there are some samples that challenge such a framework, such as GRB 060614, a long-duration burst with short-burst characteristics. Furthermore, GRBs are generally considered to be an event with no periodic or repetitive behavior, since the progenitors usually undergo destructive events, such as massive explosions or binary compact star mergers. In this work, we investigated Fermi data for possible quasiperiodic oscillations and repetitive behaviors of GRBs using timing analysis methods and report a special event GRB 201104A, which is a long-duration burst with the characteristics of an SGRB, and it exhibits a "repetitive" behavior. We propose that such a situation may arise from lensed SGRBs and attempt to verify it by Bayesian inference. In addition, we extend the spectral analysis to Bayesian inference. In spite of the existence of at least two distinct time periods with a nearly identical spectrum, there is no strong evidence that they result from a lensing GRB. Taking the gravitational-lensing scenario out of consideration, a long burst would resemble a short burst in its repetitive behavior, which presents a challenge for the current classification scheme., Comment: 15 pages, 6 figures, 2 tables; Accepted for publication in ApJ

Generative transformers have experienced rapid popularity growth in the computer vision community in synthesizing high-fidelity and high-resolution images. The best generative transformer models so far, however, still treat an image naively as a sequence of tokens, and decode an image sequentially following the raster scan ordering (i.e. line-by-line). We find this strategy neither optimal nor efficient. This paper proposes a novel image synthesis paradigm using a bidirectional transformer decoder, which we term MaskGIT. During training, MaskGIT learns to predict randomly masked tokens by attending to tokens in all directions. At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation. Our experiments demonstrate that MaskGIT significantly outperforms the state-of-the-art transformer model on the ImageNet dataset, and accelerates autoregressive decoding by up to 64x. Besides, we illustrate that MaskGIT can be easily extended to various image editing tasks, such as inpainting, extrapolation, and image manipulation.

Creating visual layouts is a critical step in graphic design. Automatic generation of such layouts is essential for scalable and diverse visual designs. To advance conditional layout generation, we introduce BLT, a bidirectional layout transformer. BLT differs from previous work on transformers in adopting non-autoregressive transformers. In training, BLT learns to predict the masked attributes by attending to surrounding attributes in two directions. During inference, BLT first generates a draft layout from the input and then iteratively refines it into a high-quality layout by masking out low-confident attributes. The masks generated in both training and inference are controlled by a new hierarchical sampling policy. We verify the proposed model on six benchmarks of diverse design tasks. Experimental results demonstrate two benefits compared to the state-of-the-art layout transformer models. First, our model empowers layout transformers to fulfill controllable layout generation. Second, it achieves up to 10x speedup in generating a layout at inference time than the layout transformer baseline. Code is released at https://shawnkx.github.io/blt., Comment: ECCV 2022

Aggressive data augmentation is a key component of the strong generalization capabilities of Vision Transformer (ViT). One such data augmentation technique is adversarial training (AT); however, many prior works have shown that this often results in poor clean accuracy. In this work, we present pyramid adversarial training (PyramidAT), a simple and effective technique to improve ViT's overall performance. We pair it with a "matched" Dropout and stochastic depth regularization, which adopts the same Dropout and stochastic depth configuration for the clean and adversarial samples. Similar to the improvements on CNNs by AdvProp (not directly applicable to ViT), our pyramid adversarial training breaks the trade-off between in-distribution accuracy and out-of-distribution robustness for ViT and related architectures. It leads to 1.82% absolute improvement on ImageNet clean accuracy for the ViT-B model when trained only on ImageNet-1K data, while simultaneously boosting performance on 7 ImageNet robustness metrics, by absolute numbers ranging from 1.76% to 15.68%. We set a new state-of-the-art for ImageNet-C (41.42 mCE), ImageNet-R (53.92%), and ImageNet-Sketch (41.04%) without extra data, using only the ViT-B/16 backbone and our pyramid adversarial training. Our code is publicly available at pyramidat.github.io., Comment: Accepted to CVPR22 (oral, best paper finalist). 33 pages, including references & supplementary material

Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs trained on ImageNet are overly reliant on local textures and fail to make adequate use of shape information. ViTs thus have difficulties generalizing to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.

A tiny fraction of observed gamma-ray bursts (GRBs) may be lensed. The time delays induced by the gravitational lensing are milliseconds to seconds if the point lenses are intermediate-mass black holes. The prompt emission of the lensed GRBs, in principle, should have repeated pulses with identical light curves and spectra but different fluxes and slightly offset positions. In this work, we search for such candidates within the GRBs detected by Fermi/GBM, Swift/BAT, and HXMT/HE and report the identification of an attractive event GRB 200716C that consists of two pulses. Both the autocorrelation analysis and the Bayesian inference of the prompt emission light curve are in favor of the gravitational lensing scenario. Moreover, the spectral properties of the two pulses are rather similar and follow the so-called Amati relation of short GRBs rather than long duration bursts. The measured flux ratios between the two pulses are nearly constant in all channels, as expected from gravitational lensing. We therefore suggest that the long duration burst GRB 200716C was a short event being lensed. The redshifted mass of the lens was estimated to be $4.25^{+2.46}_{-1.36}$ $\times$ $10^5$ $M_{\odot}$ (90$\%$ credibility). If correct, this could point towards the existence of an intermediate-mass black hole along the line of sight of GRB 200716C., Comment: 11 pages, 3 figures, 2 tables

Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if such performance can be extended to image generation. To this end, we integrate the ViT architecture into generative adversarial networks (GANs). For ViT discriminators, we observe that existing regularization methods for GANs interact poorly with self-attention, causing serious instability during training. To resolve this issue, we introduce several novel regularization techniques for training GANs with ViTs. For ViT generators, we examine architectural choices for latent and pixel mapping layers to facilitate convergence. Empirically, our approach, named ViTGAN, achieves comparable performance to the leading CNN-based GAN models on three datasets: CIFAR-10, CelebA, and LSUN bedroom., Comment: Accepted to ICLR 2022 (Spotlight)

Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, $F_2$-XEnDec, to combine self-supervised and supervised learning to optimize NMT models. To exploit complementary self-supervised signals for supervised learning, NMT models are trained on examples that are interbred from monolingual and parallel sentences through a new process called crossover encoder-decoder. Experiments on two resource-rich translation benchmarks, WMT'14 English-German and WMT'14 English-French, demonstrate that our approach achieves substantial improvements over several strong baseline methods and obtains a new state of the art of 46.19 BLEU on English-French when incorporating back translation. Results also show that our approach is capable of improving model robustness to input perturbations such as code-switching noise which frequently appears on social media., Comment: Accepted by ICML 2021