Search

Far beyond learning long-range interactions of natural language, transformers are becoming the de-facto standard for many vision tasks with their power and scalability. Especially with cross-modal tasks between image and text, vector quantized variational autoencoders (VQ-VAEs) are widely used to make a raw RGB image into a sequence of feature vectors. To better leverage the correlation between image and text, we propose L-Verse, a novel architecture consisting of feature-augmented variational autoencoder (AugVAE) and bidirectional auto-regressive transformer (BiART) for image-to-text and text-to-image generation. Our AugVAE shows the state-of-the-art reconstruction performance on ImageNet1K validation set, along with the robustness to unseen images in the wild. Unlike other models, BiART can distinguish between image (or text) as a conditional reference and a generation target. L-Verse can be directly used for image-to-text or text-to-image generation without any finetuning or extra object detection framework. In quantitative and qualitative experiments, L-Verse shows impressive results against previous methods in both image-to-text and text-to-image generation on MS-COCO Captions. We furthermore assess the scalability of L-Verse architecture on Conceptual Captions and present the initial result of bidirectional vision-language representation learning on general domain., Accepted to CVPR 2022 as Oral Presentation (18 pages, 14 figures, 4 tables)

In this paper, we estimate depth information using two defocused images from dual aperture camera. Recent advances in deep learning techniques have increased the accuracy of depth estimation. Besides, methods of using a defocused image in which an object is blurred according to a distance from a camera have been widely studied. We further improve the accuracy of the depth estimation by training the network using two images with different degrees of depth-of-field. Using images taken with different apertures for the same scene, we can determine the degree of blur in an image more accurately. In this work, we propose a novel deep convolutional network that estimates depth map using dual aperture images based on boundary cue. Our proposed method achieves state-of-the-art performance on a synthetically modified NYU-v2 dataset. In addition, we built a new camera using fast variable apertures to build a test environment in the real world. In particular, we collected a new dataset which consists of real world vehicle driving scenes. Our proposed work shows excellent performance in the new dataset.

In this work, we propose an algorithm to estimate the depth map of a scene using defocused images. In particular, the depth map is estimated using two defocused images with different depth-of-field for the same scene. Similar to the approach of the general depth from defocus (DFD), the proposed algorithm obtains the depth information from the blurredness of the object. Moreover, our proposed algorithm dramatically improves the accuracy by using both the shallow and deep depth-of-field images, simultaneously. Especially, we propose a novel depth estimation network for dual defocused images using convolutional neural network (CNN). We evaluate our proposed network on the NYU-v2 dataset and show superior performance compared to the existing techniques.

In this paper, we address the problem of interactive image segmentation which segments an image based on user-supplied scribbles. For this purpose, we propose a novel framework that provides consistent performance robust to the location of input seeds. Most of the existing methods, especially random walk-based approaches, strongly depend on initial seed positions, which differ from one user to another. To overcome this drawback, the proposed algorithm incorporates the seed expansion step, in which robust seed information is secured and improved. We also address the computation issue using a coarse-to-fine random walk technique. We evaluate our algorithm using challenging datasets, such as Grabcut, PASCAL VOC and Alpha matting datasets. Our algorithm produces a more accurate segmentation results than the existing methods.