Search

3D face reconstruction has witnessed considerable progress in recovering 3D face shapes and textures from in-the-wild images. However, due to a lack of texture detail information, the reconstructed shape and texture based on deep learning could not be used to re-render a photorealistic facial image since it does not work in harmony with weak supervision only from the spatial domain. In the paper, we propose a method of spatio-frequency decoupled weak-supervision for face reconstruction, which applies the losses from not only the spatial domain but also the frequency domain to learn the reconstruction process that approaches photorealistic effect based on the output shape and texture. In detail, the spatial domain losses cover image-level and perceptual-level supervision. Moreover, the frequency domain information is separated from the input and rendered images, respectively, and is then used to build the frequency-based loss. In particular, we devise a spectrum-wise weighted Wing loss to implement balanced attention on different spectrums. Through the spatio-frequency decoupled weak-supervision, the reconstruction process can be learned in harmony and generate detailed texture and high-quality shape only with labels of landmarks. The experiments on several benchmarks show that our method can generate high-quality results and outperform state-of-the-art methods in qualitative and quantitative comparisons.

The advancement of hardware and computing power has enabled deep learning to be used in a variety of fields, particularly in AI medical applications in intelligent medicine and medical metaverse. Deep learning models are aiding in many clinical medical image analysis tasks, including fusion, registration, detection, classification and segmentation. In recent years, many deep learning-based approaches have been developed for medical image recognition, including classification and segmentation. However, these models are susceptible to adversarial samples, posing a threat to their real world application and making them unsuitable for clinical use. This paper provides an overview of adversarial attack strategies that have been proposed against medical image models and the defense methods used to protect them. We assessed the advantages and disadvantages of these strategies and compared their efficiency. We then examined the existing state and restrictions of research methods involving the adversarial attack and defense of deep learning models for medical image recognition. Additionally, several suggestions were given on how to enhance the robustness of medical image deep learning models in intelligent medicine and medical metaverse.

This paper proposes a convolutional neural network (CNN) with three branches based on the three-view drawing principle and depth panorama for 3D shape recognition. The three-view drawing principle provides three key views of a 3D shape. A depth panorama contains the complete 2.5D information of each view. 3V-DepthPano CNN is a CNN system with three branches designed for depth panoramas generated from the three key views. This recognition system, i.e., 3V-DepthPano CNN, applies a three-branch convolutional neural network to aggregate the 3D shape depth panorama information into a more compact 3D shape descriptor to implement the classification of 3D shapes. Furthermore, we adopt a fine-tuning technique on 3V-DepthPano CNN and extract shape features to facilitate the retrieval of 3D shapes. The proposed method implements a good tradeoff state between higher accuracy and training time. Experiments show that the proposed 3V-DepthPano CNN with 3 views obtains approximate accuracy to MVCNN with 12/80 views. But the 3V-DepthPano CNN frame takes much shorter time to obtain depth panoramas and train the network than MVCNN. It is superior to all other existing advanced methods for both classification and shape retrieval.

Designing efficient deep learning models for 3D point cloud perception is becoming a major research direction. Point-voxel convolution (PVConv) Liu et al. (2019) is a pioneering research work in this topic. However, since with quite a few layers of simple 3D convolutions and linear point-voxel feature fusion operations, it still has considerable room for improvement in performance. In this paper, we propose a novel pyramid point-voxel convolution (PyraPVConv) block with two key structural modifications to address the above issues. First, PyraPVConv uses a voxel pyramid module to fully extract voxel features in the manner of feature pyramid, such that sufficient voxel features can be obtained efficiently. Second, a sharable attention module is utilized to capture compatible features between multi-scale voxels in pyramid and point cloud for aggregation, as well as to reduce the complexity via structure sharing. Extensive results on three point cloud perception tasks, i.e., indoor scene segmentation, object part segmentation and 3D object detection, validate that the networks constructed by stacking PyraPVConv blocks are efficient in terms of both GPU memory consumption and computational complexity, and are superior to the state-of-the-art methods.

It is a challenging task to handle the vector field visualization at local critical points. Generally, topological based methods firstly divide critical regions into different categories, and then process the different types of critical regions to improve the effect, which pipeline is complex. In the paper, a learning based multi-task super resolution (SR) method is proposed to improve the refinement of vector field, and enhance the visualization effect, especially at the critical region. In detail, the multi-task model consists of two important designs on task branches: one task is to simulate the interpolation of discrete vector fields based on an improved super-resolution network; and the other is a classification task to identify the types of critical vector fields. It is an efficient end-to-end architecture for both training and inferencing stages, which simplifies the pipeline of critical vector field visualization and improves the visualization effect. In experiment, we compare our method with both traditional interpolation and pure SR network on both simulation data and real data, and the reported results indicate our method lower the error and improve PSNR significantly.

A number of methods have been proposed for face reconstruction from single/multiple image(s). However, it is still a challenge to do reconstruction for limited number of wild images, in which there exists complex different imaging conditions, various face appearance, and limited number of high-quality images. And most current mesh model based methods cannot generate high-quality face model because of the local mapping deviation in geometric optics and distortion error brought by discrete differential operation. In this paper, accurate geometrical consistency modeling on B-spline parameter domain is proposed to reconstruct high-quality face surface from the various images. The modeling is completely consistent with the law of geometric optics, and B-spline reduces the distortion during surface deformation. In our method, 0th- and 1st-order consistency of stereo are formulated based on low-rank texture structures and local normals, respectively, to approach the pinpoint geometric modeling for face reconstruction. A practical solution combining the two consistency as well as an iterative algorithm is proposed to optimize high-detailed B-spline face effectively. Extensive empirical evaluations on synthetic data and unconstrained data are conducted, and the experimental results demonstrate the effectiveness of our method on challenging scenario, e.g., limited number of images with different head poses, illuminations, and expressions.

Two-scale image representation of base and detail in the spatial-domain is a well-known decomposition scheme for its lower computational complexity than that performed in the transform-domain in the field of image fusion. Unfortunately, for a pseudo-colour input image, the base and detail images in the spatial-domain obtained via image decomposition scheme always display in greyscale. In this paper, a two-scale image fusion method with adaptive threshold obtained by Otsu's method is proposed for pseudo-colour image in the colour space domain. For greyscale image, detail and base image are obtained using structural information extracted from the difference image between a global and a local patch size. Consequently, local edge-preserving filter for preserving luminance information and local energy with the discussed window size are adopted to combine base and detail image. Experimental results show that structural and luminance information has been better preserved in terms of subjective and objective evaluations for medical image and protein image fusion. Specially, a two-step non-parametric statistical test (Friedman test and Nemenyi post-hoc test) with p-values is adopted to analysis the statistical significant of the relative difference between the proposed and compared methods in terms of values of objective metrics including 30 co-registered pairs of imaging data.

Facial landmark placement is a key step in many biomedical and biometrics applications. This paper presents a computational method that efficiently performs automatic 3D facial landmark placement based on training images containing manually placed anthropological facial landmarks. After 3D face registration by an iterative closest point (ICP) technique, a visual analytics approach is taken to generate local geometric patterns for individual landmark points. These individualized local geometric patterns are derived interactively by a user’s initial visual pattern detection. They are used to guide the refinement process for landmark points projected from a template face to achieve accurate landmark placement. Compared to traditional methods, this technique is simple, robust, and does not require a large number of training samples (e.g. in machine learning based methods) or complex 3D image analysis procedures. This technique and the associated software tool are being used in a 3D biometrics project that aims to identify links between human facial phenotypes and their genetic association.

The existing methods for visualizing volumetric data are mostly based on piecewise linear models. And all kinds of analysis based on them have to be substituted by coarse interpolations. So both accuracy and reliability of the traditional framework for visualization and analysis of volumetric data are far from our needs of digging information implied in volumetric data fields. In this paper, we propose a novel framework based on a C 2 -continuous seven-directional box spline, under which reconstruction is of high accuracy and differential computations relative to analysis based on the reconstruction model are accurate. We introduce a polynomial differential operator to improve the reconstruction accuracy. In order to settle the difficulty of evaluating upon the seven-directional box spline, we convert it into Bezier form and propose effective theories and algorithms of extracting iso-surfaces, critical points and curvatures. Plentiful of examples are also given in this paper to illustrate that the novel framework is suitable for analysis, the improved reconstruction method has high accuracy, and our algorithms are fast and stable.