Search

Safeguarding personal identity in various surveillance videos, dashcams, and on-street videos is crucial. One way to do this is to detect faces and blur them, but a better solution is to replace them with non-existent ones to maintain the naturalness of the videos. While face swapping methods have already been used in the media industry with high-quality faces, it is challenging to apply them for identity protection to faces in-the-wild where faces are often occluded and of low-resolution. Therefore, we propose a new framework for face swapping specifically designed to work with face images taken in real-world scenarios, making it useful as a privacy protection method. To tackle the issue of low-resolution images, we introduce a Cross-Resolution Contrastive Loss (CRCL) technique, which allows our neural network model to be trained using triplets of varying resolutions. This enables the model to learn and use identity information across different resolutions, thereby improving its accuracy. We also propose a plug-and-play framework that can be easily applied to existing face swapping models to handle occlusions. By explicit swapping of facial features and filling of occluded regions, our framework provides a more seamless blend. To demonstrate the effectiveness of our method in handling faces in-the-wild, we create an occluded VGGFace2 dataset consisting of face images augmented with various facial masks and hand occlusions. Through quantitative and qualitative assessments on this dataset, our proposed method demonstrates robust performance under low-resolution or occluded scenarios. Significant improvements are made in the quality of swapped faces while preserving their identity and attributes, highlighting the effectiveness of our framework in advancing face swapping as a reliable privacy protection measure.

When implementing a super-resolution (SR) model on an edge device, it is common to train the model on a cloud using pre-determined training images. This is due to the lack of large-scale training data and computation power available on the edge device. However, such frameworks may encounter a domain gap issue because input images to these devices often have different characteristics than those used in training. Therefore, it is essential to continually update the model parameters through on-device learning, which takes into account the limited computation power of edge devices and makes use of on-site input images. In this paper, we present a fast and efficient on-device learning framework for an SR model that aims to overcome the challenges posed by restricted computation and domain gap issues. Specifically, we propose an architecture for training the SR model in a quantized domain, which helps to reduce the quantization errors that accumulate during training. Additionally, we propose cost-constrained gradient pruning and meta-learning-based fast training schemes to enhance restoration performance within a smaller number of iterations. Experimental results show that our approach can maintain the restoration performance for unseen inputs on a lightweight model achieved by our quantization scheme.

Constructing a high dynamic range (HDR) image from multi-exposure low dynamic range (LDR) images is challenging mainly due to two major problems. One is a large misalignment between the LDR images taken at different moments due to camera and object motions. The other is missing content in over- or under-exposed areas, which can be very large in the LDR images with long or short exposure times. These problems lead to ghosting artifacts or saturated regions in the reconstructed HDR image. In this paper, we propose a method using convolutional neural networks (CNNs) to address these problems. Specifically, each LDR image is fed to a two-branch CNN that extracts two kinds of features separately, named content features and global features. Then, another CNN aligns and merges the features to create the HDR output. The content features are used to create spatial attention masks, which are used to align the features in the time domain with exposure-invariant attributes. Further, Fourier coefficient vectors of global features are utilized to modulate intermediate features in the frequency domain during the reconstruction process. Experimental results show that our method achieves state-of-the-art HDR reconstruction performances on several benchmarks. Our code is available at https://github.com/KeuntekLee/FDM-HDR.

Multi-exposure high dynamic range (HDR) imaging aims to generate an HDR image from multiple low dynamic range (LDR) images. There are two main challenges in this task; camera and object motions causing ghost artifacts and over/under-exposure areas having no information. To tackle these challenges, we propose a deep neural network that aligns the multiple inputs through feature transformation and hallucinates the washed-out areas after fusion. Specifically, we propose a pyramidal feature adjustment network (PFAN) that adjusts the reference image features’ brightness and structure with respect to LDR inputs and regard them as the aligned features of the LDR inputs. Then, the features from the PFANs are integrated in a coarse-to-fine manner. They are first merged into an HDR image in a fusion network, and the washed-out regions are restored using valid information in a hallucination network. To deal with larger displacements in motion alignment, we also present a progressive training strategy, which begins the training with easy samples having little motion and then transfers the network with more dynamic samples. Extensive experiments demonstrate that our method achieves state-of-the-art performance on benchmark datasets. Our codes are available at https://github.com/haesoochung/PFAN.

For the reduction of JPEG compression artifacts, there have been many methods using deep neural networks. Most of them use the JPEG compression quality factor (QF) as prior knowledge in designing and training the networks. However, since the images we get from the Internet are often recompressed, the given QF is not so informative or misleading. Also, early works validated their methods on low QFs less than 50, while recent smartphones use high QFs larger than or equal to 90. In this paper, we propose a new JPEG artifacts reduction network considering the above-stated problems. Specifically, to extract quality information from the input image itself instead of the QF provided in the header of the JPEG file, we use a variational autoencoder (VAE) and regard its latent vector as quality information. In designing the artifact reduction network, we let the network change flexibly according to the input image quality by employing a deformable offset gating (DOG) network. The gating network and VAE are merged as our overall network, dubbed DOG-VAE, where the information from the VAE is used to adjust the DOG network according to the input quality. The DOG-VAE is trained end-to-end with the QFs in the range of [10], [90]. Extensive experiments validate that our method achieves comparable results to the state-of-the-art method for monochrome images and better results for color images. Our codes are available at https://github.com/yunjh410/DOGNet.

Recent studies have significantly enhanced the performance of single-image super-resolution (SR) using convolutional neural networks (CNNs). While there can be many high-resolution (HR) solutions for a given input, most existing CNN-based methods do not explore alternative solutions during the inference. A typical approach to obtaining alternative SR results is to train multiple SR models with different loss weightings and exploit the combination of these models. Instead of using multiple models, we present a more efficient method to train a single adjustable SR model on various combinations of losses by taking advantage of multi-task learning. Specifically, we optimize an SR model with a conditional objective during training, where the objective is a weighted sum of multiple perceptual losses at different feature levels. The weights vary according to given conditions, and the set of weights is defined as a style controller. Also, we present an architecture appropriate for this training scheme, which is the Residual-in-Residual Dense Block equipped with spatial feature transformation layers. At the inference phase, our trained model can generate locally different outputs conditioned on the style control map. Extensive experiments show that the proposed SR model produces various desirable reconstructions without artifacts and yields comparable quantitative performance to state-of-the-art SR methods. Code and trained models are available at https://github.com/seungho-snu/FxSR.

This paper presents a new lossless image compression method based on the learning of pixel values and contexts through multilayer perceptrons (MLPs). The prediction errors and contexts obtained by MLPs are forwarded to adaptive arithmetic encoders, like the conventional lossless compression schemes. The MLP-based prediction has long been attempted for lossless compression, and recently convolutional neural networks (CNNs) are also adopted for the lossy/lossless coding. While the existing MLP-based lossless compression schemes focused only on accurate pixel prediction, we jointly predict the pixel values and contexts. We also adopt and design channel-wise progressive learning, residual learning, and duplex network in this MLP-based framework, which leads to improved coding gain compared to the conventional methods. Experiments show that the proposed method performs better than the conventional non-learning algorithms and also recent learning-based compression methods with practical computation time.

Deep convolutional neural networks (CNNs) are widely used to improve the performance of image restoration tasks, including single-image super-resolution (SISR). Generally, researchers are manually designing more complex and deeper CNNs to further increase the given problems’ performance. Instead of this hand-crafted CNN architecture design, neural architecture search (NAS) methods have been developed to find an optimal architecture for a given task automatically. For example, NAS-based SR methods find optimized network connections and operations by reinforcement learning (RL) or evolutionary algorithms (EA). These methods enable finding an optimal system automatically, but most of them need a very long search time. In this paper, we propose a new search method for the SISR that can significantly reduce the overall design time by applying a weight-sharing scheme. We also employ a multi-branch structure to enlarge the search space for capturing multi-scale features, resulting in better reconstruction on the textured region. Experiments show that the proposed method finds an optimal SISR network about twenty times faster than the existing methods, while showing comparable performance in terms of PSNR vs. parameters. Comparison of visual quality validates that the obtained SISR network reconstructs texture areas better than the previous methods because of the enlarged search space to find multi-scale features.

The accurate and reproducible localization of cephalometric landmarks is an important procedure for treatment planning and clinical practice in orthodontics and maxillofacial surgery. In this paper, we propose a new multistage cephalometric landmark localization method that exploits local appearances and global characteristics simultaneously. To be precise, a convolutional neural network(CNN) is trained by minimizing the sum of all landmark errors. Since landmarks are considered simultaneously, global hard/soft tissue characteristics, as well as landmark relations, can be reflected in this stage. Then, we exploit local appearances by using high-resolution cropped images. In this second stage, we train CNNs for individual landmarks, respectively. Finally, we improve the localization performance of cephalometric landmarks of the mandible with linear estimators. Experiments on ISBI2015 dataset have shown that the proposed method outperforms conventional methods. Also, the proposed method allows us to evaluate confidence (e.g., standard deviational ellipses) due to its probabilistic formulation.

Most of existing compression artifacts reduction methods focused on the application for low-quality images and usually assumed a known compression quality factor. However, images compressed with high quality should also be manipulated because even small artifacts become noticeable when we enhance the compressed image. Also, the use of quality factor from the decoder is not practical because there are too many recompressed or transcoded images whose quality factor are not reliable and spatially varying. To address these issues, we propose a quality-adaptive artifacts removal network based on the gating scheme, with a quality estimator that works for a wide range of quality factor. Specifically, the estimator gives a pixel-wise quality factor, and our gating scheme generates gate-weights from the quality factor. Then, the gate-weights control the magnitudes of feature maps in our artifacts removal network. Thus, our gating scheme guarantees the proposed network to perform adaptively without changing the parameters according to the change of quality factor. Moreover, we exploit the Discrete Cosine Transform (DCT) scheme with 3D convolution for capturing both spatial and frequency dependencies of images. Experiments show that the proposed network provides better performance than the state-of-the-art methods over a wide range of quality factor. Also, the proposed method provides robust results in real-world scenarios such as the manipulation of transcoded images and videos.

Detecting the pupil center plays a key role in human-computer interaction, especially for gaze tracking. The conventional deep learning-based method for this problem is to train a convolutional neural network (CNN), which takes the eye image as the input and gives the pupil center as a regression result. In this paper, we propose an indirect use of the CNN for the task, which first segments the pupil region by a CNN as a classification problem, and then finds the center of the segmented region. This is based on the observation that CNN works more robustly for the pupil segmentation than for the pupil center-point regression when the inputs are noisy IR images. Specifically, we use the UNet model for the segmentation of pupil regions in IR images and then find the pupil center as the center of mass of the segment. In designing the loss function for the segmentation, we propose a new loss term that encodes the convex shape-prior for enhancing the robustness to noise. Precisely, we penalize not only the deviation of each predicted pixel from the ground truth label but also the non-convex shape of pupils caused by the noise and reflection. For the training, we make a new dataset of 111,581 images with hand-labeled pupil regions from 29 IR eye video sequences. We also label commonly used datasets (ExCuSe and ElSe dataset) that are considered real-world noisy ones to validate our method. Experiments show that the proposed method performs better than the conventional methods that directly find the pupil center as a regression result.

Convolutional neural networks (CNNs) generally provide higher performance gain for single image super-resolution (SISR) as the depth and number of parameters are increasing. However, just increasing the layers of straightforward deep networks has a problem that it requires an impractically large number of parameters for obtaining state-of-the-art performance. Instead, some researchers proposed lightweight networks, which is designed with more sophisticated network structures for achieving better performance than the straightforward networks at the same parameter requirement. In this paper, we propose new lightweight Multi-scale Spatial Attention Networks (MSAN) for SISR, which attempt to bring out a better performance from the relatively small number of parameters. Specifically, we adopt a dense connection with feature fusion layers to broadcast abundant features to every level of layers, and propose a double residual structure that provides an additional skip-connection. We also design a Multi-scale Spatial Attention Block (MSAB) to exploit multi-scale spatial contextual information. Furthermore, we introduce a spatial attention module which adaptively focuses on the most informative feature scale in a given region of the image. In the experiments, we validate that the proposed MSAN achieves significant accuracy compared to recent lightweight models and comparable performance to the state-of-the-art methods.

This paper presents a deep end-to-end network for high dynamic range (HDR) imaging of dynamic scenes with background and foreground motions. Generating an HDR image from a sequence of multi-exposure images is a challenging process when the images have misalignments by being taken in a dynamic situation. Hence, recent methods first align the multi-exposure images to the reference by using patch matching, optical flow, homography transformation, or attention module before the merging. In this paper, we propose a deep network that synthesizes the aligned images as a result of blending the information from multi-exposure images, because explicitly aligning photos with different exposures is inherently a difficult problem. Specifically, the proposed network generates under/over-exposure images that are structurally aligned to the reference, by blending all the information from the dynamic multi-exposure images. Our primary idea is that blending two images in the deep-feature-domain is effective for synthesizing multi-exposure images that are structurally aligned to the reference, resulting in better-aligned images than the pixel-domain blending or geometric transformation methods. Specifically, our alignment network consists of a two-way encoder for extracting features from two images separately, several convolution layers for blending deep features, and a decoder for constructing the aligned images. The proposed network is shown to generate the aligned images with a wide range of exposure differences very well and thus can be effectively used for the HDR imaging of dynamic scenes. Moreover, by adding a simple merging network after the alignment network and training the overall system end-to-end, we obtain a performance gain compared to the recent state-of-the-art methods.

In this article, we present a new multi-lingual Optical Character Recognition (OCR) system for scanned documents. In the case of Latin characters, current open source systems such as Tesseract provide very high accuracy. However, the accuracy of the multi-lingual documents, including Asian characters, is usually lower than that for Latin-only documents. For example, when the document is the mix of English, Chinese and/or Korean characters, the OCR accuracy is lowered than English-only because the character/text properties of Chinese and Korean are quite different from Latin-type characters. To tackle these problems, we propose a new framework using three neural blocks (a segmenter, a switcher, and multiple recognizers) and the reinforcement learning of the segmenter: The segmenter partitions a given word image into multiple character images, the switcher assigns a recognizer for each sub-image, and the recognizers perform the recognition of assigned sub-images. The training of recognizers and switcher can be considered traditional image classification tasks and we train them with a supervised learning method. However, the supervised learning of the segmenter has two critical drawbacks: Its objective function is sub-optimal and its training requires a large amount of annotation efforts. Thus, by adopting the REINFORCE algorithm, we train the segmenter so as to optimize the overall performance, i.e., we minimize the edit distance of final recognition results. Experimental results have shown that the proposed method significantly improves the performance for multi-lingual scripts and large character set languages without using character boundary labels.

This paper presents a new framework for jointly enhancing the resolution and the dynamic range of an image, i.e., simultaneous super-resolution (SR) and high dynamic range imaging (HDRI), based on a convolutional neural network (CNN). From the common trends of both tasks, we train a CNN for the joint HDRI and SR by focusing on the reconstruction of high-frequency details. Specifically, the high-frequency component in our work is the reflectance component according to the Retinex-based image decomposition, and only the reflectance component is manipulated by the CNN while another component (illumination) is processed in a conventional way. In training the CNN, we devise an appropriate loss function that contributes to the naturalness quality of resulting images. Experiments show that our algorithm outperforms the cascade implementation of CNN-based SR and HDRI.

The conventional image denoising methods based on the convolutional neural network (CNN) focus on the non-blind training, and hence many networks are required to cope with various noise levels at the test. Although there are blind training methods that deal with multiple noise levels with a single network, their performance gain is generally lower than the non-blind ones, especially at low noise levels. In this paper, we propose a new denoising scheme that controls the feature maps of a single denoising network according to the noise level at the test phase, without changing the network parameters. This is achieved by employing a gating scheme where the feature maps of the denoising network are multiplied with appropriate weights from a gate-weight generating network which is trained along with the denoising network. We train the overall network on a wide range of noise level such that the proposed method can be used for both blind and non-blind cases. The experiments show that the proposed system yields better denoising performance than the other CNN-based methods, especially for the untrained noise levels. Finally, it is shown that the proposed system can manage spatially variant unknown noises and real noises without changing the whole CNN parameters.

Deep neural networks are widely used and exhibit excellent performance in many areas. However, they are vulnerable to adversarial attacks that compromise networks at inference time by applying elaborately designed perturbations to input data. Although several defense methods have been proposed to address specific attacks, other types of attacks can circumvent these defense mechanisms. Therefore, we propose Purifying Variational AutoEncoder (PuVAE), a method to purify adversarial examples. The proposed method eliminates an adversarial perturbation by projecting an adversarial example on the manifold of each class and determining the closest projection as a purified sample. We experimentally illustrate the robustness of PuVAE against various attack methods without any prior knowledge about the attacks. In our experiments, the proposed method exhibits performances that are competitive with state-of-the-art defense methods, and the inference time is approximately 130 times faster than that of Defense-GAN which is a state-of-the art purifier method.

This paper presents an algorithm for high dynamic range (HDR) and super-resolution (SR) imaging from a single image. First, we propose a new single image HDR imaging (HDRI) method based on the Retinex approach and exploit a recent single image SR method based on a convolutional neural network (CNN). Among many possible configurations of HDR and SR, we find an optimal system configuration and color manipulation strategy from the extensive experiments. Specifically, the best results are obtained when we first process the luminance component (Y) of input with our single image HDRI algorithm and then feed the enhanced HDR luminance to the CNN-based SR architecture that is trained by only luminance component. The ranges of chromatic components (U and V) are just scaled in proportion to the enhanced HDR luminance, and then they are bicubic interpolated or fed to the above CNN-based SR. Subjective and objective assessments for various experiments are presented to validate the effectiveness of the proposed HDR/SR imaging scheme.

It has been shown that deep convolutional neural networks (CNNs) reduce JPEG compression artifacts better than the previous approaches. However, the latest video compression standards have more complex artifacts than the JPEG, including the flickering which is not well reduced by the CNN-based methods developed for still images. Moreover, recent video compression algorithms include in-loop filters which reduce the blocking artifacts, and thus post-processing barely improves the performance. In this paper, we propose a temporal-CNN architecture to reduce the artifacts in video compression standards as well as in JPEG. Specifically, we exploit a simple CNN structure and introduce a new training strategy that captures the temporal correlation of the consecutive frames in videos. The similar patches are aggregated from the neighboring frames by a simple motion search method, and they are fed to the CNN, which further reduces the artifacts. Experiments show that our approach shows improvements over the conventional CNN-based methods with similar complexities for image and video compression standards, such as MPEG-2, AVC, and HEVC, with average PSNR gain of 1.27, 0.47, and 0.23 dB, respectively.

Abstract This paper presents a textline detection method for degraded historical documents. Our method follows a conventional two-step procedure that the binarization is first performed and then the textlines are extracted from the binary image. In order to address the challenges in historical documents such as document degradation, structure noise, and skews, we develop new methods for the binarization and textline extraction. First, we improve the performance of binarization by detecting the non-text regions and processing only text regions. We also improve the textline detection method by extracting main textblock and compensating the skew angle and writing style. Experimental results show that the proposed method yields the state-of-the-art performance for several datasets.

Two watermarking algorithms for 3D nonuniform rational B-spline (NURBS) graphic data are proposed: one is appropriate for the steganography, and the other for watermarking. Instead of directly embedding data into the parameters of NURBS, the proposed algorithms embed data into the 2D virtual images extracted by parameter sampling of 3D model. As a result, the proposed steganography algorithm can embed information into more places of the surface than the conventional algorithm, while preserving the data size of the model. Also, any existing 2D watermarking technique can be used for the watermarking of 3D NURBS surfaces. From the experiment, it is found that the algorithm for the watermarking is robust to the attacks on weights, control points, and knots. It is also found to be robust to the remodeling of NURBS models.