Search

Video frame interpolation is a challenging problem that involves various scenarios depending on the variety of foreground and background motions, frame rate, and occlusion. Therefore, generalizing across different scenes is difficult for a single network with fixed parameters. Ideally, one could have a different network for each scenario, but this will be computationally infeasible for practical applications. In this work, we propose MetaVFI, an adaptive video frame interpolation algorithm that uses additional information readily available at test time but has not been exploited in previous works. We initially show the benefits of test-time adaptation through simple fine-tuning of a network and then greatly improve its efficiency by incorporating meta-learning. Thus, we obtain significant performance gains with only a single gradient update without introducing any additional parameters. Moreover, the proposed MetaVFI algorithm is model-agnostic which can be easily combined with any video frame interpolation network. We show that our adaptive framework greatly improves the performance of baseline video frame interpolation networks on multiple benchmark datasets.

Few-shot learning is an emerging yet challenging problem in which the goal is to achieve generalization from only few examples. Meta-learning tackles few-shot learning via the learning of prior knowledge shared across tasks and using it to learn new tasks. One of the most representative meta-learning algorithms is the model-agnostic meta-learning (MAML), which formulates prior knowledge as a common initialization, a shared starting point from where a learner can quickly adapt to unseen tasks. However, forcibly sharing an initialization can lead to conflicts among tasks and the compromised (undesired by tasks) location on optimization landscape, thereby hindering task adaptation. Furthermore, the degree of conflict is observed to vary not only among the tasks but also among the layers of a neural network. Thus, we propose task-and-layer-wise attenuation on the compromised initialization to reduce its adverse influence on task adaptation. As attenuation dynamically controls (or selectively forgets) the influence of the compromised prior knowledge for a given task and each layer, we name our method Learn to Forget (L2F). Experimental results demonstrate that the proposed method greatly improves the performance of the state-of-the-art MAML-based frameworks across diverse domains: few-shot classification, cross-domain few-shot classification, regression, reinforcement learning, and visual tracking.

Objectives To develop a convolutional neural network system to jointly segment and classify a hepatic lesion selected by user clicks in ultrasound images. Methods In total, 4309 anonymized ultrasound images of 3873 patients with hepatic cyst (n = 1214), hemangioma (n = 1220), metastasis (n = 1001), or hepatocellular carcinoma (HCC) (n = 874) were collected and annotated. The images were divided into 3909 training and 400 test images. Our network is composed of one shared encoder and two inference branches used for segmentation and classification and takes the concatenation of an input image and two Euclidean distance maps of foreground and background clicks provided by a user as input. The performance of hepatic lesion segmentation was evaluated based on the Jaccard index (JI), and the performance of classification was based on accuracy, sensitivity, specificity, and the area under the receiver operating characteristic curve (AUROC). Results We achieved performance improvements by jointly conducting segmentation and classification. In the segmentation only system, the mean JI was 68.5%. In the classification only system, the accuracy of classifying four types of hepatic lesions was 79.8%. The mean JI and classification accuracy were 68.5% and 82.2%, respectively, for the proposed joint system. The optimal sensitivity and specificity and the AUROC of classifying benign and malignant hepatic lesions of the joint system were 95.0%, 86.0%, and 0.970, respectively. The respective sensitivity, specificity, and the AUROC for classifying four hepatic lesions of the joint system were 86.7%, 89.7%, and 0.947. Conclusions The proposed joint system exhibited fair performance compared to segmentation only and classification only systems. Key Points • The joint segmentation and classification system using deep learning accurately segmented and classified hepatic lesions selected by user clicks in US examination. • The joint segmentation and classification system for hepatic lesions in US images exhibited higher performance than segmentation only and classification only systems. • The joint segmentation and classification system could assist radiologists with minimal experience in US imaging by characterizing hepatic lesions.

Prevailing video frame interpolation techniques rely heavily on optical flow estimation and require additional model complexity and computational cost; it is also susceptible to error propagation in challenging scenarios with large motion and heavy occlusion. To alleviate the limitation, we propose a simple but effective deep neural network for video frame interpolation, which is end-to-end trainable and is free from a motion estimation network component. Our algorithm employs a special feature reshaping operation, referred to as PixelShuffle, with a channel attention, which replaces the optical flow computation module. The main idea behind the design is to distribute the information in a feature map into multiple channels and extract motion information by attending the channels for pixel-level frame synthesis. The model given by this principle turns out to be effective in the presence of challenging motion and occlusion. We construct a comprehensive evaluation benchmark and demonstrate that the proposed approach achieves outstanding performance compared to the existing models with a component for optical flow computation.

In interactive segmentation, the role of seed information provided by the user is significant. A seed is a clue to ease the ambiguity of the problem by making the object segmentation task interactive. However, in most deep network-based works, seed information has been used as an additional channel for input images. In this paper, we propose a novel bi-directional attention module for more actively using seed information. The proposed bi-directional seed attention module (BSA) operates based on the feature map of the segmentation network and the input seed map. Through our attention module, the network feature map is affected by the seed map, while the feature also updates the seed information. As a result, our system concentrates on the seed information and more accurately derives the segmentation result required by the user. We have conducted validation experiments on the four standard benchmark datasets, including SBD, GrabCut, Berkeley, and DAVIS, and achieved the state-of-the-art results.

Most image super-resolution (SR) methods are developed on synthetic low-resolution (LR) and high-resolution (HR) image pairs that are constructed by a predetermined operation, e.g., bicubic downsampling. As existing methods typically learn an inverse mapping of the specific function, they produce blurry results when applied to real-world images whose exact formulation is different and unknown. Therefore, several methods attempt to synthesize much more diverse LR samples or learn a realistic downsampling model. However, due to restrictive assumptions on the downsampling process, they are still biased and less generalizable. This study proposes a novel method to simulate an unknown downsampling process without imposing restrictive prior knowledge. We propose a generalizable low-frequency loss (LFL) in the adversarial training framework to imitate the distribution of target LR images without using any paired examples. Furthermore, we design an adaptive data loss (ADL) for the downsampler, which can be adaptively learned and updated from the data during the training loops. Extensive experiments validate that our downsampling model can facilitate existing SR methods to perform more accurate reconstructions on various synthetic and real-world examples than the conventional approaches., Accepted at TPAMI

Convolutional neural networks are data models. Their design should embrace the structural properties of the data being modeled, e.g., the natural images. We argue that this also holds for discriminators of adversarial training frameworks for photo-realistic image restoration. We develop this idea to highlight three essential structural features of natural images: translation equivariance, rotation invariance, and hierarchy of scale. The analysis leads to a new discriminator, Structure-Resonant Discriminator (SRD), which can capture image structures in need. The proposed SRD is demonstrated in the perceptual single image super-resolution task. By replacing only the discriminator, our method restores more visually pleasing high-resolution images than the previous state-of-the-art techniques, while exhibits the least distortions.

Most current action recognition methods heavily rely on appearance information by taking an RGB sequence of entire image regions as input. While being effective in exploiting contextual information around humans, e.g., human appearance and scene category, they are easily fooled by out-of-context action videos where the contexts do not exactly match with target actions. In contrast, pose-based methods, which take a sequence of human skeletons only as input, suffer from inaccurate pose estimation or ambiguity of human pose per se. Integrating these two approaches has turned out to be non-trivial; training a model with both appearance and pose ends up with a strong bias towards appearance and does not generalize well to unseen videos. To address this problem, we propose to learn pose-driven feature integration that dynamically combines appearance and pose streams by observing pose features on the fly. The main idea is to let the pose stream decide how much and which appearance information is used in integration based on whether the given pose information is reliable or not. We show that the proposed IntegralAction achieves highly robust performance across in-context and out-of-context action video datasets. The codes are available in https://github.com/mks0601/IntegralAction_RELEASE., Comment: Published at CVPRW 2021

Super-Resolution (SR) is a fundamental computer vision task that aims to obtain a high-resolution clean image from the given low-resolution counterpart. This paper reviews the NTIRE 2021 Challenge on Video Super-Resolution. We present evaluation results from two competition tracks as well as the proposed solutions. Track 1 aims to develop conventional video SR methods focusing on the restoration quality. Track 2 assumes a more challenging environment with lower frame rates, casting spatio-temporal SR problem. In each competition, 247 and 223 participants have registered, respectively. During the final testing phase, 14 teams competed in each track to achieve state-of-the-art performance on video SR tasks.

OBJECTIVES: The objective of our study was to evaluate the association between occupational exposure to trichloroethylene (TCE), a suspected lymphomagen, and serum levels of miRNAs in a cross-sectional molecular epidemiology study of TCE exposed workers and comparable unexposed controls in China. METHODS: Serum levels of 40 miRNAs were compared in 74 workers exposed to TCE (median: 12 ppm) and 90 unexposed control workers. Linear regression models were used to test for differences in serum miRNA levels between exposed and unexposed workers and to evaluate exposure-response relationships across TCE exposure categories using a three-level ordinal variable (i.e., unexposed

ObjectivesThe occupational exposure limit for trichloroethylene (TCE) in different countries varies from 1 to 100 ppm as an 8-hour time-weighted average (TWA). Many countries currently use 10 ppm as the regulatory standard for occupational exposures, but the biological effects in humans at this level of exposure remain unclear. The objective of our study was to evaluate alterations in immune and renal biomarkers among workers occupationally exposed to low levels of TCE below current regulatory standards.MethodsWe conducted a cross-sectional molecular epidemiology study of 80 healthy workers exposed to a wide range of TCE (ie, 0.4–229 ppm) and 96 comparable unexposed controls in China, and previously reported that TCE exposure was associated with multiple candidate biological markers related to immune function and kidney toxicity. Here, we conducted further analyses of all of the 31 biomarkers that we have measured to determine the magnitude and statistical significance of changes in the subgroup of workers (n=35) exposed to ResultsSix immune biomarkers (ie, CD4+ effector memory T cells, sCD27, sCD30, interleukin-10, IgG and IgM) were significantly decreased (% difference ranged from −16.0% to −72.1%) and one kidney toxicity marker (kidney injury molecule-1, KIM-1) was significantly increased (% difference: +52.5%) among workers exposed to ConclusionOur results suggest that occupational exposure to TCE below 10 ppm as an 8-hour TWA may alter levels of key markers of immune function and kidney toxicity.

Background: We aimed to report the prevalence and correlates of high-risk alcohol consumption and types of alcoholic beverages. Methods: The baseline data of the Health Examinees-Gem (HEXA-G) study participants, including 43,927 men and 85,897 women enrolled from 2005 through 2013, were used for analysis. Joinpoint regression was performed to estimate trends in the age-standardized prevalence of alcohol consumption. Associations of demographic and behavioral factors, perceived health-related effects, social relationships, and the diagnostic history of diseases with alcohol consumption were assessed using multinomial logistic regression. Results: The prevalence of alcohol consumption remained higher in men during the study period and increased in women. The amount of alcohol consumed has increased in women, especially that from beer and makgeolli, a traditional Korean fermented rice wine. Older participants were less likely to be high-risk drinkers (men and women who drink more than 40 or 20 g/day of alcohol, respectively) and drink Soju, a distilled liquor, and beer, and more likely to drink makgeolli. Educational level was negatively associated with high-risk drinking. However, it was positively associated with the consumption of strong spirits and wine. Smoking was associated with high-risk drinking and the consumption of soju and strong spirits. Engaging in regular exercise and having stress were associated with drinking all types of beverages except for soju. Conclusions: Sex-specific trends in alcohol consumption were influenced by demographic, behavioral, and perceived health-related factors. The findings will help improve the understanding of alcohol-related problems and provide evidence for establishing country-specific policies and campaigns in Korea.

Deep CNNs have achieved significant successes in image processing and its applications, including single image super-resolution (SR). However, conventional methods still resort to some predetermined integer scaling factors, e.g., x2 or x4. Thus, they are difficult to be applied when arbitrary target resolutions are required. Recent approaches extend the scope to real-valued upsampling factors, even with varying aspect ratios to handle the limitation. In this paper, we propose the SRWarp framework to further generalize the SR tasks toward an arbitrary image transformation. We interpret the traditional image warping task, specifically when the input is enlarged, as a spatially-varying SR problem. We also propose several novel formulations, including the adaptive warping layer and multiscale blending, to reconstruct visually favorable results in the transformation process. Compared with previous methods, we do not constrain the SR model on a regular grid but allow numerous possible deformations for flexible and diverse image editing. Extensive experiments and ablation studies justify the necessity and demonstrate the advantage of the proposed SRWarp method under various transformations., Accepted to CVPR 2021

OBJECTIVES: This study compared the results of meta-analysis with and without adjustment for the healthy worker effect on the association between working in the semiconductor industry and cancer mortality.METHODS: Six studies that reported standardized mortality ratios (SMRs) for cancers were selected for meta-analysis. Using a random-effects model, the SMR results from each study were combined for all cancers and leukemias to estimate the summary SMRs (95% confidence interval, CI). To adjust for the healthy worker effect, the relative standardized mortality ratio (rSMR=SMRx/SMRnot x) were calculated using observed and expected counts for the specific cause of interest (i.e., all cancers and leukemias) and the observed and expected counts for all other causes of mortality. Then, the rSMR results were combined to estimate the summary rSMRs (95% CIs).RESULTS: The SMRs for all causes of mortality among semiconductor industry workers ranged from 0.25 to 0.80, which reflects a significant healthy worker effect. A remarkable difference was found between the summary SMRs and the summary rSMRs. The summary SMR for all cancers was 0.70 (95% CI, 0.63 to 0.79) whereas the summary rSMR was 1.38 (95% CI, 1.20 to 1.59). The summary SMR for leukemia was 0.88 (95% CI, 0.72 to 1.07), and the summary rSMR was 1.88 (95% CI, 1.20 to 2.95).CONCLUSIONS: Our results suggest that adjustment for the healthy worker effect (i.e., rSMR) may be useful in meta-analyses of cohort studies reporting SMRs.

Video super-resolution has recently become one of the most important mobile-related problems due to the rise of video communication and streaming services. While many solutions have been proposed for this task, the majority of them are too computationally expensive to run on portable devices with limited hardware resources. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based video super-resolution solutions that can achieve a real-time performance on mobile GPUs. The participants were provided with the REDS dataset and trained their models to do an efficient 4X video upscaling. The runtime of all models was evaluated on the OPPO Find X2 smartphone with the Snapdragon 865 SoC capable of accelerating floating-point networks on its Adreno GPU. The proposed solutions are fully compatible with any mobile GPU and can upscale videos to HD resolution at up to 80 FPS while demonstrating high fidelity results. A detailed description of all models developed in the challenge is provided in this paper., Comment: Mobile AI 2021 Workshop and Challenges: https://ai-benchmark.com/workshops/mai/2021/. arXiv admin note: substantial text overlap with arXiv:2105.07825. substantial text overlap with arXiv:2105.08629, arXiv:2105.07809, arXiv:2105.08630

Most conventional supervised super-resolution (SR) algorithms assume that low-resolution (LR) data is obtained by downscaling high-resolution (HR) data with a fixed known kernel, but such an assumption often does not hold in real scenarios. Some recent blind SR algorithms have been proposed to estimate different downscaling kernels for each input LR image. However, they suffer from heavy computational overhead, making them infeasible for direct application to videos. In this work, we present DynaVSR, a novel meta-learning-based framework for real-world video SR that enables efficient downscaling model estimation and adaptation to the current input. Specifically, we train a multi-frame downscaling module with various types of synthetic blur kernels, which is seamlessly combined with a video SR network for input-aware adaptation. Experimental results show that DynaVSR consistently improves the performance of the state-of-the-art video SR models by a large margin, with an order of magnitude faster inference time compared to the existing blind SR approaches.

Recent advances in Siamese network-based visual tracking methods have enabled high performance on numerous tracking benchmarks. However, extensive scale variations of the target object and distractor objects with similar categories have consistently posed challenges in visual tracking. To address these persisting issues, we propose novel TridentAlign and context embedding modules for Siamese network-based visual tracking methods. The TridentAlign module facilitates adaptability to extensive scale variations and large deformations of the target, where it pools the feature representation of the target object into multiple spatial dimensions to form a feature pyramid, which is then utilized in the region proposal stage. Meanwhile, context embedding module aims to discriminate the target from distractor objects by accounting for the global context information among objects. The context embedding module extracts and embeds the global context information of a given frame into a local feature representation such that the information can be utilized in the final classification stage. Experimental results obtained on multiple benchmark datasets show that the performance of the proposed tracker is comparable to that of state-of-the-art trackers, while the proposed tracker runs at real-time speed. (Code available on https://github.com/JanghoonChoi/TACT).

Motion blur is one of the most common degradation artifacts in dynamic scene photography. This paper reviews the NTIRE 2020 Challenge on Image and Video Deblurring. In this challenge, we present the evaluation results from 3 competition tracks as well as the proposed solutions. Track 1 aims to develop single-image deblurring methods focusing on restoration quality. On Track 2, the image deblurring methods are executed on a mobile platform to find the balance of the running speed and the restoration accuracy. Track 3 targets developing video deblurring methods that exploit the temporal relation between input frames. In each competition, there were 163, 135, and 102 registered participants and in the final testing phase, 9, 4, and 7 teams competed. The winning methods demonstrate the state-of-the-art performance on image and video deblurring tasks.

In this paper, we estimate depth information using two defocused images from dual aperture camera. Recent advances in deep learning techniques have increased the accuracy of depth estimation. Besides, methods of using a defocused image in which an object is blurred according to a distance from a camera have been widely studied. We further improve the accuracy of the depth estimation by training the network using two images with different degrees of depth-of-field. Using images taken with different apertures for the same scene, we can determine the degree of blur in an image more accurately. In this work, we propose a novel deep convolutional network that estimates depth map using dual aperture images based on boundary cue. Our proposed method achieves state-of-the-art performance on a synthetically modified NYU-v2 dataset. In addition, we built a new camera using fast variable apertures to build a test environment in the real world. In particular, we collected a new dataset which consists of real world vehicle driving scenes. Our proposed work shows excellent performance in the new dataset.

Most of the recent deep learning-based 3D human pose and mesh estimation methods regress the pose and shape parameters of human mesh models, such as SMPL and MANO, from an input image. The first weakness of these methods is the overfitting to image appearance, due to the domain gap between the training data captured from controlled settings such as a lab, and in-the-wild data in inference time. The second weakness is that the estimation of the pose parameters is quite challenging due to the representation issues of 3D rotations. To overcome the above weaknesses, we propose Pose2Mesh, a novel graph convolutional neural network (GraphCNN)-based system that estimates the 3D coordinates of human mesh vertices directly from the 2D human pose. The 2D human pose as input provides essential human body articulation information without image appearance. Also, the proposed system avoids the representation issues, while fully exploiting the mesh topology using GraphCNN in a coarse-to-fine manner. We show that our Pose2Mesh significantly outperforms the previous 3D human pose and mesh estimation methods on various benchmark datasets. The codes are publicly available(https://github.com/hongsukchoi/Pose2Mesh_RELEASE).

Despite the recent success of single image-based 3D human pose and shape estimation methods, recovering temporally consistent and smooth 3D human motion from a video is still challenging. Several video-based methods have been proposed; however, they fail to resolve the single image-based methods' temporal inconsistency issue due to a strong dependency on a static feature of the current frame. In this regard, we present a temporally consistent mesh recovery system (TCMR). It effectively focuses on the past and future frames' temporal information without being dominated by the current static feature. Our TCMR significantly outperforms previous video-based methods in temporal consistency with better per-frame 3D pose and shape accuracy. We also release the codes. For the demo video, see https://youtu.be/WB3nTnSQDII. For the codes, see https://github.com/hongsukchoi/TCMR_RELEASE., Comment: Accepted to CVPR 2021, 10 pages

Videos in the real-world contain various dynamics and motions that may look unnaturally discontinuous in time when the recorded frame rate is low. This paper reports the second AIM challenge on Video Temporal Super-Resolution (VTSR), a.k.a. frame interpolation, with a focus on the proposed solutions, results, and analysis. From low-frame-rate (15 fps) videos, the challenge participants are required to submit higher-frame-rate (30 and 60 fps) sequences by estimating temporally intermediate frames. To simulate realistic and challenging dynamics in the real-world, we employ the REDS_VTSR dataset derived from diverse videos captured in a hand-held camera for training and evaluation purposes. There have been 68 registered participants in the competition, and 5 teams (one withdrawn) have competed in the final testing phase. The winning team proposes the enhanced quadratic video interpolation method and achieves state-of-the-art on the VTSR task.

Most of the previous image-based 3D human pose and mesh estimation methods estimate parameters of the human mesh model from an input image. However, directly regressing the parameters from the input image is a highly non-linear mapping because it breaks the spatial relationship between pixels in the input image. In addition, it cannot model the prediction uncertainty, which can make training harder. To resolve the above issues, we propose I2L-MeshNet, an image-to-lixel (line+pixel) prediction network. The proposed I2L-MeshNet predicts the per-lixel likelihood on 1D heatmaps for each mesh vertex coordinate instead of directly regressing the parameters. Our lixel-based 1D heatmap preserves the spatial relationship in the input image and models the prediction uncertainty. We demonstrate the benefit of the image-to-lixel prediction and show that the proposed I2L-MeshNet outperforms previous methods. The code is publicly available (https://github.com/mks0601/I2L-MeshNet_RELEASE).

Video frame interpolation is a challenging problem because there are different scenarios for each video depending on the variety of foreground and background motion, frame rate, and occlusion. It is therefore difficult for a single network with fixed parameters to generalize across different videos. Ideally, one could have a different network for each scenario, but this is computationally infeasible for practical applications. In this work, we propose to adapt the model to each video by making use of additional information that is readily available at test time and yet has not been exploited in previous works. We first show the benefits of `test-time adaptation' through simple fine-tuning of a network, then we greatly improve its efficiency by incorporating meta-learning. We obtain significant performance gains with only a single gradient update without any additional parameters. Finally, we show that our meta-learning framework can be easily employed to any video frame interpolation network and can consistently improve its performance on multiple benchmark datasets., Comment: CVPR 2020

Analysis of hand-hand interactions is a crucial step towards better understanding human behavior. However, most researches in 3D hand pose estimation have focused on the isolated single hand case. Therefore, we firstly propose (1) a large-scale dataset, InterHand2.6M, and (2) a baseline network, InterNet, for 3D interacting hand pose estimation from a single RGB image. The proposed InterHand2.6M consists of 2.6 M labeled single and interacting hand frames under various poses from multiple subjects. Our InterNet simultaneously performs 3D single and interacting hand pose estimation. In our experiments, we demonstrate big gains in 3D interacting hand pose estimation accuracy when leveraging the interacting hand data in InterHand2.6M. We also report the accuracy of InterNet on InterHand2.6M, which serves as a strong baseline for this new dataset. Finally, we show 3D interacting hand pose estimation results from general images. Our code and dataset are available (https://mks0601.github.io/InterHand2.6M/).

State-of-the-art video deblurring methods cannot handle blurry videos recorded in dynamic scenes since they are built under a strong assumption that the captured scenes are static. Contrary to the existing methods, we propose a new video deblurring algorithm that can deal with general blurs inherent in dynamic scenes. To handle general and locally varying blurs caused by various sources, such as moving objects, camera shake, depth variation, and defocus, we estimate pixel-wise varying non-uniform blur kernels. We infer bidirectional optical flows to handle motion blurs, and also estimate Gaussian blur maps to remove optical blur from defocus. Therefore, we propose a single energy model that jointly estimates optical flows, defocus blur maps and latent frames. We also provide a framework and efficient solvers to minimize the proposed energy model. By optimizing the energy model, we achieve significant improvements in removing general blurs, estimating optical flows, and extending depth-of-field in blurry frames. Moreover, in this work, to evaluate the performance of non-uniform deblurring methods objectively, we have constructed a new realistic dataset with ground truths. In addition, extensive experimental results on publicly available challenging videos demonstrate that the proposed method produces qualitatively superior performance than the state-of-the-art methods which often fail in either deblurring or optical flow estimation.

One of the major challenges of model-free visual tracking problem has been the difficulty originating from the unpredictable and drastic changes in the appearance of objects we target to track. Existing methods tackle this problem by updating the appearance model on-line in order to adapt to the changes in the appearance. Despite the success of these methods however, inaccurate and erroneous updates of the appearance model result in a tracker drift. In this paper, we introduce a novel real-time visual tracking algorithm based on a template selection strategy constructed by deep reinforcement learning methods. The tracking algorithm utilizes this strategy to choose the appropriate template for tracking a given frame. The template selection strategy is self-learned by utilizing a simple policy gradient method on numerous training episodes randomly generated from a tracking benchmark dataset. Our proposed reinforcement learning framework is generally applicable to other confidence map based tracking algorithms. The experiment shows that our tracking algorithm runs in real-time speed of 43 fps and the proposed policy network effectively decides the appropriate template for successful visual tracking.

When a human matches two images, the viewer has a natural tendency to view the wide area around the target pixel to obtain clues of right correspondence. However, designing a matching cost function that works on a large window in the same way is difficult. The cost function is typically not intelligent enough to discard the information irrelevant to the target pixel, resulting in undesirable artifacts. In this paper, we propose a novel learn a stereo matching cost with a large-sized window. Unlike conventional pooling layers with strides, the proposed per-pixel pyramid-pooling layer can cover a large area without a loss of resolution and detail. Therefore, the learned matching cost function can successfully utilize the information from a large area without introducing the fattening effect. The proposed method is robust despite the presence of weak textures, depth discontinuity, illumination, and exposure difference. The proposed method achieves near-peak performance on the Middlebury benchmark., published in SPL

A novel tracking algorithm is proposed, which robustly tracks a target by finding the state that minimizes the likelihood uncertainty. Likelihood uncertainty is estimated by determining the gap between the lower and upper bounds of likelihood. By minimizing the gap between the two bounds, the proposed method identifies the confident and reliable state of the target. In this study, the state that provides the Minimum Uncertainty Gap (MUG) between likelihood bounds is shown to be more reliable than the state that provides the maximum likelihood only, especially when severe illumination changes, occlusions, and pose variations occur. A rigorous derivation of the lower and upper bounds of the likelihood for the visual tracking problem is provided to address this issue. Additionally, an efficient inference algorithm that uses Interacting Markov Chain Monte Carlo (IMCMC) approach is presented to find the best state that maximizes the average of the lower and upper bounds of likelihood while minimizing the gap between the two bounds. We extend our method to update the target model adaptively. To update the model, the current observation is combined with a previous target model with the adaptive weight, which is calculated according to the goodness of the current observation. The goodness of the observation is measured using the proposed uncertainty gap estimation of likelihood. Experimental results demonstrate that the proposed method robustly tracks the target in realistic videos and outperforms conventional tracking methods.

Videos contain various types and strengths of motions that may look unnaturally discontinuous in time when the recorded frame rate is low. This paper reviews the first AIM challenge on video temporal super-resolution (frame interpolation) with a focus on the proposed solutions and results. From low-frame-rate (15 fps) video sequences, the challenge participants are asked to submit higher-frame-rate (60 fps) video sequences by estimating temporally intermediate frames. We employ the REDS_VTSR dataset derived from diverse videos captured in a hand-held camera for training and evaluation purposes. The competition had 62 registered participants, and a total of 8 teams competed in the final testing phase. The challenge winning methods achieve the state-of-the-art in video temporal super-resolution.

Fast and reliable adaptability to appearance variations of any target object has been the holy grail of visual tracking. Recently, Siamese-based trackers have demonstrated outstanding speed, however at the cost of adaptability and accuracy. We propose to model a temporal evolution of appearance features, allowing for adaptability without online training. Specifically, we introduce a memory-augmented convolutional recurrent neural network (RNN), named Past-to-Future (P2FNet), that takes appearance features as an input at each frame and predicts the next-frame features. RNN allows for fast adaptability to dynamically varying appearance, while the memory provides the generalization capability over longer sequences via template management. For reliability, we propose a new augmentation to train RNN to disregard corrupted features. A novel visualization method illustrates the reliable template management of the memory. The experimental results on benchmarks demonstrate the tracker shows competitive performance among real-time state-of-the-art trackers.

Catastrophic forgetting is a critical challenge in training deep neural networks. Although continual learning has been investigated as a countermeasure to the problem, it often suffers from the requirements of additional network components and the limited scalability to a large number of tasks. We propose a novel approach to continual learning by approximating a true loss function using an asymmetric quadratic function with one of its sides overestimated. Our algorithm is motivated by the empirical observation that the network parameter updates affect the target loss functions asymmetrically. In the proposed continual learning framework, we estimate an asymmetric loss function for the tasks considered in the past through a proper overestimation of its unobserved sides in training new tasks, while deriving the accurate model parameter for the observable sides. In contrast to existing approaches, our method is free from the side effects and achieves the state-of-the-art accuracy that is even close to the upper-bound performance on several challenging benchmark datasets., ICCV 2019

ObjectivesThis study evaluated whether individuals with affected family member adhered to healthy behaviours.Design and settingThis was a cross-sectional study of participants selected from health examinees who underwent the national health check-up programme of Korea in 39 centres between 2004 and 2013.ParticipantsThe baseline data of 128 520 participants enrolled in the Health Examinees-Gem study were used for analysis.Main outcomes and measuresAssociations of family history of diabetes with adherence to regular exercise, healthy diet and body composition, and clusters of healthy behaviours were evaluated while adjusting for potential confounders selected by a directed acyclic graph.ResultsParticipants with a family history of diabetes were more likely to adhere to a regular exercise regimen (OR=1.12, 95% CI 1.06 to 1.18 for men and OR=1.10, 95% CI 1.07 to 1.14 for women) and healthy diet (OR=1.06, 95% CI 1.01 to 1.12 for men and OR=1.06, 95% CI 1.01 to 1.12 for women) but were less likely to have a normal body composition (OR=0.83, 95% CI 0.78 to 0.87 for men and OR=0.83, 95% CI 0.80 to 0.86 for women). These associations were strengthened when the affected family members were siblings, the number of affected members was increased or the age at diagnosis of the affected member was younger than 50 years. In men and women, having a normal body composition is important in determining the cluster of behaviours, and those with a family history of diabetes were less likely to adhere to the normal body composition cluster.ConclusionsThe group with high risk of diabetes showed healthy behaviors, but they did not have a normal body composition. Policies and campaigns targeting integrated health behaviors will be needed to reduce the burden of diseases and improve public health.

Few-shot learning is a challenging problem where the goal is to achieve generalization from only few examples. Model-agnostic meta-learning (MAML) tackles the problem by formulating prior knowledge as a common initialization across tasks, which is then used to quickly adapt to unseen tasks. However, forcibly sharing an initialization can lead to conflicts among tasks and the compromised (undesired by tasks) location on optimization landscape, thereby hindering the task adaptation. Further, we observe that the degree of conflict differs among not only tasks but also layers of a neural network. Thus, we propose task-and-layer-wise attenuation on the compromised initialization to reduce its influence. As the attenuation dynamically controls (or selectively forgets) the influence of prior knowledge for a given task and each layer, we name our method as L2F (Learn to Forget). The experimental results demonstrate that the proposed method provides faster adaptation and greatly improves the performance. Furthermore, L2F can be easily applied and improve other state-of-the-art MAML-based frameworks, illustrating its simplicity and generalizability., CVPR 2020. Code at https://github.com/baiksung/L2F

This paper reviews the first NTIRE challenge on video deblurring (restoration of rich details and high frequency components from blurred video frames) with focus on the proposed solutions and results. A new REalistic and Diverse Scenes dataset (REDS) was employed. The challenge was divided into 2 tracks. Track 1 employed dynamic motion blurs while Track 2 had additional MPEG video compression artifacts. Each competition had 109 and 93 registered participants. Total 13 teams competed in the final testing phase. They gauge the state-of-the-art in video deblurring problem.

Performance of deep metric learning depends heavily on the capability of mining hard negative examples during training. However, many metric learning algorithms often require intractable computational cost due to frequent feature computations and nearest neighbor searches in a large-scale dataset. As a result, existing approaches often suffer from trade-off between training speed and prediction accuracy. To alleviate this limitation, we propose a stochastic hard negative mining method. Our key idea is to adopt class signatures that keep track of feature embedding online with minor additional cost during training, and identify hard negative example candidates using the signatures. Given an anchor instance, our algorithm first selects a few hard negative classes based on the class-to-sample distances and then performs a refined search in an instance-level only from the selected classes. As most of the classes are discarded at the first step, it is much more efficient than exhaustive search while effectively mining a large number of hard examples. Our experiment shows that the proposed technique improves image retrieval accuracy substantially; it achieves the state-of-the-art performance on the several standard benchmark datasets.

Recurrent neural networks (RNNs) are widely used for sequential data processing. Recent state-of-the-art video deblurring methods bank on convolutional recurrent neural network architectures to exploit the temporal relationship between neighboring frames. In this work, we aim to improve the accuracy of recurrent models by adapting the hidden states transferred from past frames to the frame being processed so that the relations between video frames could be better used. We iteratively update the hidden state via re-using RNN cell parameters before predicting an output deblurred frame. Since we use existing parameters to update the hidden state, our method improves accuracy without additional modules. As the architecture remains the same regardless of iteration number, fewer iteration models can be considered as a partial computational path of the models with more iterations. To take advantage of this property, we employ a stochastic method to optimize our iterative models better. At training time, we randomly choose the iteration number on the fly and apply a regularization loss that favors less computation unless there are considerable reconstruction gains. We show that our method exhibits state-of-the-art video deblurring performance while operating in real-time speed.

This paper introduces a novel large dataset for video deblurring, video super-resolution and studies the state-of-the-art as emerged from the NTIRE 2019 video restoration challenges. The video deblurring and video super-resolution challenges are each the first challenge of its kind, with 4 competitions, hundreds of participants and tens of proposed solutions. Our newly collected REalistic and Diverse Scenes dataset (REDS) was employed by the challenges. In our study, we compare the solutions from the challenges to a set of representative methods from the literature and evaluate them on our proposed REDS dataset. We find that the NTIRE 2019 challenges push the state-of-the-art in video deblurring and super-resolution, reaching compelling performance on our newly proposed REDS dataset.

This paper reviews the first NTIRE challenge on video super-resolution (restoration of rich details in low-resolution video frames) with focus on proposed solutions and results. A new REalistic and Diverse Scenes dataset (REDS) was employed. The challenge was divided into 2 tracks. Track 1 employed standard bicubic downscaling setup while Track 2 had realistic dynamic motion blurs. Each competition had 124 and 104 registered participants. There were total 14 teams in the final testing phase. They gauge the state-of-the-art in video super-resolution., NA

Although significant improvement has been achieved recently in 3D human pose estimation, most of the previous methods only treat a single-person case. In this work, we firstly propose a fully learning-based, camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. The pipeline of the proposed system consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. Our system achieves comparable results with the state-of-the-art 3D single-person pose estimation models without any groundtruth information and significantly outperforms previous 3D multi-person pose estimation methods on publicly available datasets. The code is available in https://github.com/mks0601/3DMPPE_ROOTNET_RELEASE , https://github.com/mks0601/3DMPPE_POSENET_RELEASE., Comment: Published at ICCV 2019

We propose an efficient Stereographic Projection Neural Network (SPNet) for learning representations of 3D objects. We first transform a 3D input volume into a 2D planar image using stereographic projection. We then present a shallow 2D convolutional neural network (CNN) to estimate the object category followed by view ensemble, which combines the responses from multiple views of the object to further enhance the predictions. Specifically, the proposed approach consists of four stages: (1) Stereographic projection of a 3D object, (2) view-specific feature learning, (3) view selection and (4) view ensemble. The proposed approach performs comparably to the state-of-the-art methods while having substantially lower GPU memory as well as network parameters. Despite its lightness, the experiments on 3D object classification and shape retrievals demonstrate the high performance of the proposed method.

Multi-person pose estimation from a 2D image is an essential technique for human behavior understanding. In this paper, we propose a human pose refinement network that estimates a refined pose from a tuple of an input image and input pose. The pose refinement was performed mainly through an end-to-end trainable multi-stage architecture in previous methods. However, they are highly dependent on pose estimation models and require careful model design. By contrast, we propose a model-agnostic pose refinement method. According to a recent study, state-of-the-art 2D human pose estimation methods have similar error distributions. We use this error statistics as prior information to generate synthetic poses and use the synthesized poses to train our model. In the testing stage, pose estimation results of any other methods can be input to the proposed method. Moreover, the proposed model does not require code or knowledge about other methods, which allows it to be easily used in the post-processing step. We show that the proposed approach achieves better performance than the conventional multi-stage refinement models and consistently improves the performance of various state-of-the-art pose estimation methods on the commonly used benchmark. The code is available in this https URL\footnote{\url{https://github.com/mks0601/PoseFix_RELEASE}}., Published at CVPR 2019

The purpose of this study is to evaluate the associations of sleep duration and with metabolic syndrome and its components. A total of 72,673 participants from the Health Examinees Study were analyzed to investigate the association of sleep duration with metabolic syndrome and its components. Using the NCEP-ATP III guideline and multiple logistic regression models, odds ratios (ORs) were estimated, stratified by sex. Long-sleep duration was significantly associated with an increased OR for metabolic syndrome in women (OR 1.53, 95 % CI 1.32–1.78), but not in men (1.19, 0.98–1.46). Both short- and long-sleep duration were significantly associated with elevated triglycerides among men (short and long sleep, respectively: 1.09, 1.00–1.19; 1.26, 1.04–1.52). Women were associated with long-sleep duration and all of metabolic syndrome components, especially triglycerides (1.29, 1.11–1.50). In conclusion, long-sleep duration is significantly associated with metabolic syndrome in women. Moreover, triglycerides revealed association across sleep duration among components of metabolic syndrome.

Objectives: In Korea, the system of management of medical waste largely relies on the incineration method. Our study aimed to identify any regional imbalances between the generation and treatment of medical waste, and provide reference data for future policy in Korea. Methods: We analyzed data on the amount of medical waste from 2,000 hospitals in 2012, 2013, and 2014 obtained from the Korea Environment Corporation. In the Arc GIS program (version 10.2.3), the addresses of hospitals and incinerators were transformed into map coordinates. The amount of waste generated by each hospital and the amount incinerated were represented by density and size of a triangle symbol, respectively. Results: As of 2014, the total amount of medical waste from the top 2,000 hospitals was 136,073 tons, out of which about half (49%) was generated in the capital area. Eleven incineration companies (three in the capital area, two in the Chungcheong Provinces area, one in the Jeolla Provinces area, and five in the Gyeongsang Provinces area) treated this waste. For the years 2012, 2013, and 2014, about 60% of the medical waste generated from the hospitals in the capital area was treated within the capital area and about 40% was transported to other areas, especially the Gyeongsang Provinces area, for treatment. On the other hand, about 90% of the medical waste incinerated in the capital area originated from the capital area. Conclusion: Our results suggest a spatial imbalance between the generation and treatment of medical waste in Korea and warrants multilateral policies, including the expansion of on-site treatment, strengthening regulation of the containment of medical wastes, promoting reductions in medical waste and more.