Search

We propose a novel framework for spatiotemporal action detection using only video-level class labels as weak supervision. Traditional fully-supervised approaches rely on labor-intensive manual annotation of bounding boxes for each frame. In contrast, collecting video-level class labels is significantly less tedious and more feasible compared to annotating frame-level sequences with bounding boxes. To address this challenge, we propose a discriminative action tubelet detector, called DAT-detector, designed to discern discriminative tubelets from action tubelet proposals (ATPs). Whereas the previous approaches have only focused on tubelet selection among the predefined object proposals, our DAT-detector prioritizes the generation of more precise action tubelets using regression and attention modules. Moreover, we introduce an ATP generation method that enhances the quality of tubelet proposals. Our approach achieves state-of-the-art performance on several benchmarks, and also demonstrates competitive performance even with fully-supervised approaches. • We propose a DAT-detector for spatiotemporal action detection using video-level class labels as weak supervision. • The DAT-detector generates precise actions tubes via proposed attention and regression modules. • We enhace tubelet proposal quality with our action tubelet proposal generation method. • Our method significantly outperforms the state-of-the-art action proposal methods. • We achieve remarkable performance in spatiotemporal action detection across multiple benchmarks, effectively competing with fully supervised approaches. [ABSTRACT FROM AUTHOR]

Despite significant progress in machine learning, pedestrian detection in the real-world is still regarded as one of the challenging problems, limited by occluded appearances, cluttered backgrounds, and bad visibility at night. This has caused detection approaches using multi-spectral sensors such as color and thermal which could be complementary to each other. In this paper, we propose a novel sensor fusion framework for detecting pedestrians even in challenging real-world environments. We design a convolutional neural network (CNN) architecture that consists of three-branch detection models taking different modalities as inputs. Unlike existing methods, we consider all detection probabilities from each modality in a unified CNN framework and selectively use them through a channel weighting fusion (CWF) layer to maximize the detection performance. An accumulated probability fusion (APF) layer is also introduced to combine probabilities from different modalities at the proposal-level. We formulate these sub-networks into a unified network, so that it is possible to train the whole network in an end-to-end manner. Our extensive evaluation demonstrates that the proposed method outperforms the state-of-the-art methods on the challenging KAIST, CVC-14, and DIML multi-spectral pedestrian datasets. [ABSTRACT FROM AUTHOR]

Establishing correspondences is a fundamental task in many image processing and computer vision applications. In particular, finding the correspondences between a non-linearly deformed image pair induced by different modality conditions is a challenging problem. This paper describes a simple but powerful image transform called local area transform (LAT) for modality-robust correspondence estimation. Specifically, LAT transforms an image from the intensity domain to the local area domain, which is invariant under nonlinear intensity deformations, especially radiometric, photometric, and spectral deformations. Experimental results show that LATransformed images provide a consistency for nonlinearly deformed images, even under random intensity deformations. LAT reduces the mean absolute difference by approximately 0.20 and the different pixel ratio by approximately 58% on average, as compared to conventional methods. Furthermore, the reformulation of descriptors with LAT shows superiority to conventional methods, which is a promising result for the tasks of cross-spectral and modality correspondence matching. LAT gains an approximately 23% improvement in the correct detection ratio and a 10% improvement in the recognition rate for the tasks of RGB-NIR cross-spectral template matching and cross-spectral feature matching, respectively. LAT reduces the bad pixel percentage by approximately 15% and the root mean squared errors by 13.5 in the task of cross-radiation stereo matching. LAT also improves the cross-modal dense flow estimation task in terms of warping error, providing 50% error reduction. [ABSTRACT FROM AUTHOR]

Foreground detection is one of the most important and fundamental tasks in many computer vision applications such as real-time video surveillance. Although there have been many efforts to find solutions to this problem, many obstacles such as illumination changes, noises, dynamic backgrounds, and computational complexities have prevented them from being used in real surveillance systems. In this paper, to alleviate these inherent limitations of conventional methods, we propose a fast illumination-robust foreground detection (FIFD) system that provides robustness against illumination variations and noises from various real circumstances with an efficient computational scheme. In contrast to the conventional approaches, our method focuses on efficiently formulating the foreground object detection system by leveraging a foreground candidate region detection and hierarchical distribution map. Specifically, our approach consists of three parts. First, for a query image, foreground candidates are detected by fusing multiple methods. The existence and the block size of the foreground object are determined through the use of the foreground continuity. Second, the foreground block is found from the estimated distribution map and then detected from the extracted valid blocks. Finally, with a labeling scheme, the foreground is detected. To intensively evaluate our approach compared to the conventional methods, we use the publicly available I2R and traffic datasets, and we build a novel electron multiplying charge-coupled device foreground detection benchmark taken in an environment with light lower than 10lux. Experimental results show that our approach provides satisfactory performance compared to the state-of-the-art methods even under very challenging circumstances. Furthermore, our approach is very efficient in that it takes only approximately 31 ms per frame. [ABSTRACT FROM AUTHOR]

A sensor-based vision localization system is one of the most essential technologies in computer vision applications like an autonomous navigation, surveillance, and many others. Conventionally, sensor-based vision localization systems have three inherent limitations, These include, sensitivity to illumination variations, viewpoint variations, and high computational complexity. To overcome these problems, we propose a robust image matching method to provide invariance to the illumination and viewpoint variations by focusing on how to solve these limitations and incorporate this scheme into the vision-based localization system. Based on the proposed image matching method, we design a robust localization system that provides satisfactory localization performance with low computational complexity. Specifically, in order to solve the problem of illumination and viewpoint, we extract a key point using a virtual view from a query image and the descriptor based on the local average patch difference, similar to HC-LBP. Moreover, we propose a key frame selection method and a simple tree scheme for fast image search. Experimental results show that the proposed localization system is four times faster than existing systems, and exhibits better matching performance compared to existing algorithms in challenging environments with difficult illumination and viewpoint conditions. [ABSTRACT FROM AUTHOR]

Diffusion models have recently shown significant advancement in the generative models with their impressive fidelity and diversity. The success of these models can be often attributed to their use of sampling guidance techniques, such as classifier or classifier-free guidance, which provide effective mechanisms to trade-off between fidelity and diversity. However, these methods are not capable of guiding a generated image to be aware of its geometric configuration, e.g., depth, which hinders their application to downstream tasks such as scene understanding that require a certain level of depth awareness. To overcome this limitation, we propose a novel sampling guidance method for diffusion models that uses self-predicted depth information derived from the rich intermediate representations of diffusion models. Concretely, we first present a label-efficient depth estimation framework using internal representations of diffusion models. Subsequently, we propose the incorporation of two guidance techniques during the sampling phase. These methods involve using pseudo-labeling and depth-domain diffusion prior to self-condition the generated image using the estimated depth map. Experiments and comprehensive ablation studies demonstrate the effectiveness of our method in guiding the diffusion models toward the generation of geometrically plausible images. Our project page is available at https://ku-cvlab.github.io/DAG/. • Prove that depth information is in learned U-Net diffusion models. • Train depth predictors in U-Net efficiently using pretrained models • A novel framework injects depth awareness on images from diffusion model. • New guidance methods apply consistency regularization and learned from depth map prior. [ABSTRACT FROM AUTHOR]

Existing CNN-based style transfer frameworks have suffered from inaccurate control of pixel-wise stylization. As the CNN operation is designed based on kernel-wise operation, such a design unavoidably makes pixels affect each other. To mitigate this problem, we propose a controllable style transfer framework that leverages Implicit Neural Representation to encode each pixel respectively and optimize each style and content pair via test-time training. Unlike previous CNN-based style transfer frameworks, this formulation naturally enables accurate pixel-wise stylization control. In addition, to give explicit controllability on the degree of stylization, we define two vectors that represent the content and style respectively, enabling control by interpolating these vectors. We further demonstrate that, after being test-time trained once, our framework can show a various range of applications by precisely controlling the stylized images pixel-wise and freely adjusting image resolution and degree of stylization without further optimization or training. • A novel controllable style transfer framework based on Implicit Neural Representation that pixel-wisely controls the stylized output via test-time training. • Exponential reweighting technique for mitigating undesirable entanglement of content and style. • Demonstrating the various applications, such as resolution control, pixel-wise control of stylization degree, spatial control using masks, and gradation style transfer effect. [ABSTRACT FROM AUTHOR]