Search

Video frame interpolation is a challenging problem that involves various scenarios depending on the variety of foreground and background motions, frame rate, and occlusion. Therefore, generalizing across different scenes is difficult for a single network with fixed parameters. Ideally, one could have a different network for each scenario, but this will be computationally infeasible for practical applications. In this work, we propose MetaVFI, an adaptive video frame interpolation algorithm that uses additional information readily available at test time but has not been exploited in previous works. We initially show the benefits of test-time adaptation through simple fine-tuning of a network and then greatly improve its efficiency by incorporating meta-learning. Thus, we obtain significant performance gains with only a single gradient update without introducing any additional parameters. Moreover, the proposed MetaVFI algorithm is model-agnostic which can be easily combined with any video frame interpolation network. We show that our adaptive framework greatly improves the performance of baseline video frame interpolation networks on multiple benchmark datasets.

Few-shot learning is an emerging yet challenging problem in which the goal is to achieve generalization from only few examples. Meta-learning tackles few-shot learning via the learning of prior knowledge shared across tasks and using it to learn new tasks. One of the most representative meta-learning algorithms is the model-agnostic meta-learning (MAML), which formulates prior knowledge as a common initialization, a shared starting point from where a learner can quickly adapt to unseen tasks. However, forcibly sharing an initialization can lead to conflicts among tasks and the compromised (undesired by tasks) location on optimization landscape, thereby hindering task adaptation. Furthermore, the degree of conflict is observed to vary not only among the tasks but also among the layers of a neural network. Thus, we propose task-and-layer-wise attenuation on the compromised initialization to reduce its adverse influence on task adaptation. As attenuation dynamically controls (or selectively forgets) the influence of the compromised prior knowledge for a given task and each layer, we name our method Learn to Forget (L2F). Experimental results demonstrate that the proposed method greatly improves the performance of the state-of-the-art MAML-based frameworks across diverse domains: few-shot classification, cross-domain few-shot classification, regression, reinforcement learning, and visual tracking.

In interactive segmentation, the role of seed information provided by the user is significant. A seed is a clue to ease the ambiguity of the problem by making the object segmentation task interactive. However, in most deep network-based works, seed information has been used as an additional channel for input images. In this paper, we propose a novel bi-directional attention module for more actively using seed information. The proposed bi-directional seed attention module (BSA) operates based on the feature map of the segmentation network and the input seed map. Through our attention module, the network feature map is affected by the seed map, while the feature also updates the seed information. As a result, our system concentrates on the seed information and more accurately derives the segmentation result required by the user. We have conducted validation experiments on the four standard benchmark datasets, including SBD, GrabCut, Berkeley, and DAVIS, and achieved the state-of-the-art results.

State-of-the-art video deblurring methods cannot handle blurry videos recorded in dynamic scenes since they are built under a strong assumption that the captured scenes are static. Contrary to the existing methods, we propose a new video deblurring algorithm that can deal with general blurs inherent in dynamic scenes. To handle general and locally varying blurs caused by various sources, such as moving objects, camera shake, depth variation, and defocus, we estimate pixel-wise varying non-uniform blur kernels. We infer bidirectional optical flows to handle motion blurs, and also estimate Gaussian blur maps to remove optical blur from defocus. Therefore, we propose a single energy model that jointly estimates optical flows, defocus blur maps and latent frames. We also provide a framework and efficient solvers to minimize the proposed energy model. By optimizing the energy model, we achieve significant improvements in removing general blurs, estimating optical flows, and extending depth-of-field in blurry frames. Moreover, in this work, to evaluate the performance of non-uniform deblurring methods objectively, we have constructed a new realistic dataset with ground truths. In addition, extensive experimental results on publicly available challenging videos demonstrate that the proposed method produces qualitatively superior performance than the state-of-the-art methods which often fail in either deblurring or optical flow estimation.

When a human matches two images, the viewer has a natural tendency to view the wide area around the target pixel to obtain clues of right correspondence. However, designing a matching cost function that works on a large window in the same way is difficult. The cost function is typically not intelligent enough to discard the information irrelevant to the target pixel, resulting in undesirable artifacts. In this paper, we propose a novel learn a stereo matching cost with a large-sized window. Unlike conventional pooling layers with strides, the proposed per-pixel pyramid-pooling layer can cover a large area without a loss of resolution and detail. Therefore, the learned matching cost function can successfully utilize the information from a large area without introducing the fattening effect. The proposed method is robust despite the presence of weak textures, depth discontinuity, illumination, and exposure difference. The proposed method achieves near-peak performance on the Middlebury benchmark., published in SPL

A novel tracking algorithm is proposed, which robustly tracks a target by finding the state that minimizes the likelihood uncertainty. Likelihood uncertainty is estimated by determining the gap between the lower and upper bounds of likelihood. By minimizing the gap between the two bounds, the proposed method identifies the confident and reliable state of the target. In this study, the state that provides the Minimum Uncertainty Gap (MUG) between likelihood bounds is shown to be more reliable than the state that provides the maximum likelihood only, especially when severe illumination changes, occlusions, and pose variations occur. A rigorous derivation of the lower and upper bounds of the likelihood for the visual tracking problem is provided to address this issue. Additionally, an efficient inference algorithm that uses Interacting Markov Chain Monte Carlo (IMCMC) approach is presented to find the best state that maximizes the average of the lower and upper bounds of likelihood while minimizing the gap between the two bounds. We extend our method to update the target model adaptively. To update the model, the current observation is combined with a previous target model with the adaptive weight, which is calculated according to the goodness of the current observation. The goodness of the observation is measured using the proposed uncertainty gap estimation of likelihood. Experimental results demonstrate that the proposed method robustly tracks the target in realistic videos and outperforms conventional tracking methods.

A novel approach for event summarization and rare event detection is proposed. Unlike conventional methods that deal with event summarization and rare event detection independently, our method solves them in a single framework by transforming them into a graph editing problem. In our approach, a video is represented by a graph, each node of which indicates an event obtained by segmenting the video spatially and temporally. The edges between nodes describe the relationship between events. Based on the degree of relations, edges have different weights. After learning the graph structure, our method finds subgraphs that represent event summarization and rare events in the video by editing the graph, that is, merging its subgraphs or pruning its edges. The graph is edited to minimize a predefined energy model with the Markov Chain Monte Carlo (MCMC) method. The energy model consists of several parameters that represent the causality, frequency, and significance of events. We design a specific energy model that uses these parameters to satisfy each objective of event summarization and rare event detection. The proposed method is extended to obtain event summarization and rare event detection results across multiple videos captured from multiple views. For this purpose, the proposed method independently learns and edits each graph of individual videos for event summarization or rare event detection. Then, the method matches the extracted multiple graphs to each other, and constructs a single composite graph that represents event summarization or rare events from multiple views. Experimental results show that the proposed approach accurately summarizes multiple videos in a fully unsupervised manner . Moreover, the experiments demonstrate that the approach is advantageous in detecting rare transition of events .

We propose the visual tracker sampler, a novel tracking algorithm that can work robustly in challenging scenarios, where several kinds of appearance and motion changes of an object can occur simultaneously. The proposed tracking algorithm accurately tracks a target by searching for appropriate trackers in each frame. Since the real-world tracking environment varies severely over time, the trackers should be adapted or newly constructed depending on the current situation, so that each specific tracker takes charge of a certain change in the object. To do this, our method obtains several samples of not only the states of the target but also the trackers themselves during the sampling process. The trackers are efficiently sampled using the Markov Chain Monte Carlo (MCMC) method from the predefined tracker space by proposing new appearance models, motion models, state representation types, and observation types, which are the important ingredients of visual trackers. All trackers are then integrated into one compound tracker through an Interacting MCMC (IMCMC) method, in which the trackers interactively communicate with one another while running in parallel. By exchanging information with others, each tracker further improves its performance, thus increasing overall tracking performance. Experimental results show that our method tracks the object accurately and reliably in realistic videos, where appearance and motion drastically change over time, and outperforms even state-of-the-art tracking methods.

A novel tracking algorithm is proposed for targets with drastically changing geometric appearances over time. To track such objects, we develop a local patch-based appearance model and provide an efficient online updating scheme that adaptively changes the topology between patches. In the online update process, the robustness of each patch is determined by analyzing the likelihood landscape of the patch. Based on this robustness measure, the proposed method selects the best feature for each patch and modifies the patch by moving, deleting, or newly adding it over time. Moreover, a rough object segmentation result is integrated into the proposed appearance model to further enhance it. The proposed framework easily obtains segmentation results because the local patches in the model serve as good seeds for the semi-supervised segmentation task. To solve the complexity problem attributable to the large number of patches, the Basin Hopping (BH) sampling method is introduced into the tracking framework. The BH sampling method significantly reduces computational complexity with the help of a deterministic local optimizer. Thus, the proposed appearance model could utilize a sufficient number of patches. The experimental results show that the present approach could track objects with drastically changing geometric appearance accurately and robustly.

In this paper, we propose a new efficient topology estimation algorithm to construct a multiresolution polygonal mesh from measured three-dimensional (3-D) range data. The topology estimation problem is defined under the constraints of cognition, compactness, and regularity, and the algorithm is designed to be applied to either a cloud of points or a dense mesh. The proposed algorithm initially segments the range data into a finite number of Voronoi patches using the K-means clustering algorithm. Each patch is then approximated by an appropriate polygonal and eventually a triangular mesh model. In order to improve the equiangularity of the mesh, we employ a dynamic mesh model, in which the mesh finds its equilibrium state adaptively, according to the equiangularity constraint. Experimental results demonstrate that satisfactory equiangular triangular mesh models can be constructed rapidly at various resolutions, while yielding tolerable modeling error.

The authors propose to combine a triangular element surface model with a linearized reflectance map to formulate the shape-from-shading problem. The main idea is to approximate a smooth surface by the union of triangular surface patches called triangular elements and express the approximating surface as a linear combination of a set of nodal basis functions. Since the surface normal of a triangular element is uniquely determined by the heights of its three vertices (or nodes), image brightness can be directly related to nodal heights using the linearized reflectance map. The surface height can then be determined by minimizing a quadratic cost functional corresponding to the squares of brightness errors and solved effectively with the multigrid computational technique. The proposed method does not require any integrability constraint or artificial assumptions on boundary conditions. Simulation results for synthetic and real images are presented to illustrate the performance and efficiency of the method. >