Search

Existing approaches to animatable NeRF-based head avatars are either built upon face templates or use the expression coefficients of templates as the driving signal. Despite the promising progress, their performances are heavily bound by the expression power and the tracking accuracy of the templates. In this work, we present LatentAvatar, an expressive neural head avatar driven by latent expression codes. Such latent expression codes are learned in an end-to-end and self-supervised manner without templates, enabling our method to get rid of expression and tracking issues. To achieve this, we leverage a latent head NeRF to learn the person-specific latent expression codes from a monocular portrait video, and further design a Y-shaped network to learn the shared latent expression codes of different subjects for cross-identity reenactment. By optimizing the photometric reconstruction objectives in NeRF, the latent expression codes are learned to be 3D-aware while faithfully capturing the high-frequency detailed expressions. Moreover, by learning a mapping between the latent expression code learned in shared and person-specific settings, LatentAvatar is able to perform expressive reenactment between different subjects. Experimental results show that our LatentAvatar is able to capture challenging expressions and the subtle movement of teeth and even eyeballs, which outperforms previous state-of-the-art solutions in both quantitative and qualitative comparisons. Project page: https://www.liuyebin.com/latentavatar., Comment: Accepted by SIGGRAPH 2023

Face reenactment methods attempt to restore and re-animate portrait videos as realistically as possible. Existing methods face a dilemma in quality versus controllability: 2D GAN-based methods achieve higher image quality but suffer in fine-grained control of facial attributes compared with 3D counterparts. In this work, we propose StyleAvatar, a real-time photo-realistic portrait avatar reconstruction method using StyleGAN-based networks, which can generate high-fidelity portrait avatars with faithful expression control. We expand the capabilities of StyleGAN by introducing a compositional representation and a sliding window augmentation method, which enable faster convergence and improve translation generalization. Specifically, we divide the portrait scenes into three parts for adaptive adjustments: facial region, non-facial foreground region, and the background. Besides, our network leverages the best of UNet, StyleGAN and time coding for video learning, which enables high-quality video generation. Furthermore, a sliding window augmentation method together with a pre-training strategy are proposed to improve translation generalization and training performance, respectively. The proposed network can converge within two hours while ensuring high image quality and a forward rendering time of only 20 milliseconds. Furthermore, we propose a real-time live system, which further pushes research into applications. Results and experiments demonstrate the superiority of our method in terms of image quality, full portrait video generation, and real-time re-animation compared to existing facial reenactment methods. Training and inference code for this paper are at https://github.com/LizhenWangT/StyleAvatar., Comment: 8 pages, 5 figures, SIGGRAPH 2023 Conference Proceedings

Synthesizing high-fidelity and emotion-controllable talking video portraits, with audio-lip sync, vivid expressions, realistic head poses, and eye blinks, has been an important and challenging task in recent years. Most existing methods suffer in achieving personalized and precise emotion control, smooth transitions between different emotion states, and the generation of diverse motions. To tackle these challenges, we present GMTalker, a Gaussian mixture-based emotional talking portraits generation framework. Specifically, we propose a Gaussian mixture-based expression generator that can construct a continuous and disentangled latent space, achieving more flexible emotion manipulation. Furthermore, we introduce a normalizing flow-based motion generator pretrained on a large dataset with a wide-range motion to generate diverse head poses, blinks, and eyeball movements. Finally, we propose a personalized emotion-guided head generator with an emotion mapping network that can synthesize high-fidelity and faithful emotional video portraits. Both quantitative and qualitative experiments demonstrate our method outperforms previous methods in image quality, photo-realism, emotion accuracy, and motion diversity., Comment: Project page: https://bob35buaa.github.io/GMTalker. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

The ability to animate photo-realistic head avatars reconstructed from monocular portrait video sequences represents a crucial step in bridging the gap between the virtual and real worlds. Recent advancements in head avatar techniques, including explicit 3D morphable meshes (3DMM), point clouds, and neural implicit representation have been exploited for this ongoing research. However, 3DMM-based methods are constrained by their fixed topologies, point-based approaches suffer from a heavy training burden due to the extensive quantity of points involved, and the last ones suffer from limitations in deformation flexibility and rendering efficiency. In response to these challenges, we propose MonoGaussianAvatar (Monocular Gaussian Point-based Head Avatar), a novel approach that harnesses 3D Gaussian point representation coupled with a Gaussian deformation field to learn explicit head avatars from monocular portrait videos. We define our head avatars with Gaussian points characterized by adaptable shapes, enabling flexible topology. These points exhibit movement with a Gaussian deformation field in alignment with the target pose and expression of a person, facilitating efficient deformation. Additionally, the Gaussian points have controllable shape, size, color, and opacity combined with Gaussian splatting, allowing for efficient training and rendering. Experiments demonstrate the superior performance of our method, which achieves state-of-the-art results among previous methods., Comment: The link to our projectpage is https://yufan1012.github.io/MonoGaussianAvatar

Creating high-fidelity 3D head avatars has always been a research hotspot, but there remains a great challenge under lightweight sparse view setups. In this paper, we propose Gaussian Head Avatar represented by controllable 3D Gaussians for high-fidelity head avatar modeling. We optimize the neutral 3D Gaussians and a fully learned MLP-based deformation field to capture complex expressions. The two parts benefit each other, thereby our method can model fine-grained dynamic details while ensuring expression accuracy. Furthermore, we devise a well-designed geometry-guided initialization strategy based on implicit SDF and Deep Marching Tetrahedra for the stability and convergence of the training procedure. Experiments show our approach outperforms other state-of-the-art sparse-view methods, achieving ultra high-fidelity rendering quality at 2K resolution even under exaggerated expressions., Comment: Projectpage: https://yuelangx.github.io/gaussianheadavatar, Code: https://github.com/YuelangX/Gaussian-Head-Avatar

While high fidelity and efficiency are central to the creation of digital head avatars, recent methods relying on 2D or 3D generative models often experience limitations such as shape distortion, expression inaccuracy, and identity flickering. Additionally, existing one-shot inversion techniques fail to fully leverage multiple input images for detailed feature extraction. We propose a novel framework, \textbf{Incremental 3D GAN Inversion}, that enhances avatar reconstruction performance using an algorithm designed to increase the fidelity from multiple frames, resulting in improved reconstruction quality proportional to frame count. Our method introduces a unique animatable 3D GAN prior with two crucial modifications for enhanced expression controllability alongside an innovative neural texture encoder that categorizes texture feature spaces based on UV parameterization. Differentiating from traditional techniques, our architecture emphasizes pixel-aligned image-to-image translation, mitigating the need to learn correspondences between observation and canonical spaces. Furthermore, we incorporate ConvGRU-based recurrent networks for temporal data aggregation from multiple frames, boosting geometry and texture detail reconstruction. The proposed paradigm demonstrates state-of-the-art performance on one-shot and few-shot avatar animation tasks. Code will be available at https://github.com/XChenZ/invertAvatar.

Modeling animatable human avatars from RGB videos is a long-standing and challenging problem. Recent works usually adopt MLP-based neural radiance fields (NeRF) to represent 3D humans, but it remains difficult for pure MLPs to regress pose-dependent garment details. To this end, we introduce Animatable Gaussians, a new avatar representation that leverages powerful 2D CNNs and 3D Gaussian splatting to create high-fidelity avatars. To associate 3D Gaussians with the animatable avatar, we learn a parametric template from the input videos, and then parameterize the template on two front & back canonical Gaussian maps where each pixel represents a 3D Gaussian. The learned template is adaptive to the wearing garments for modeling looser clothes like dresses. Such template-guided 2D parameterization enables us to employ a powerful StyleGAN-based CNN to learn the pose-dependent Gaussian maps for modeling detailed dynamic appearances. Furthermore, we introduce a pose projection strategy for better generalization given novel poses. To tackle the realistic relighting of animatable avatars, we introduce physically-based rendering into the avatar representation for decomposing avatar materials and environment illumination. Overall, our method can create lifelike avatars with dynamic, realistic, generalized and relightable appearances. Experiments show that our method outperforms other state-of-the-art approaches., Comment: An extended version of Animatable Gaussians, Projectpage: https://animatable-gaussians.github.io/relight

We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D., Comment: Project Page: https://mrtornado24.github.io/DreamCraft3D

The problem of modeling an animatable 3D human head avatar under light-weight setups is of significant importance but has not been well solved. Existing 3D representations either perform well in the realism of portrait images synthesis or the accuracy of expression control, but not both. To address the problem, we introduce a novel hybrid explicit-implicit 3D representation, Facial Model Conditioned Neural Radiance Field, which integrates the expressiveness of NeRF and the prior information from the parametric template. At the core of our representation, a synthetic-renderings-based condition method is proposed to fuse the prior information from the parametric model into the implicit field without constraining its topological flexibility. Besides, based on the hybrid representation, we properly overcome the inconsistent shape issue presented in existing methods and improve the animation stability. Moreover, by adopting an overall GAN-based architecture using an image-to-image translation network, we achieve high-resolution, realistic and view-consistent synthesis of dynamic head appearance. Experiments demonstrate that our method can achieve state-of-the-art performance for 3D head avatar animation compared with previous methods.

With NeRF widely used for facial reenactment, recent methods can recover photo-realistic 3D head avatar from just a monocular video. Unfortunately, the training process of the NeRF-based methods is quite time-consuming, as MLP used in the NeRF-based methods is inefficient and requires too many iterations to converge. To overcome this problem, we propose AvatarMAV, a fast 3D head avatar reconstruction method using Motion-Aware Neural Voxels. AvatarMAV is the first to model both the canonical appearance and the decoupled expression motion by neural voxels for head avatar. In particular, the motion-aware neural voxels is generated from the weighted concatenation of multiple 4D tensors. The 4D tensors semantically correspond one-to-one with 3DMM expression basis and share the same weights as 3DMM expression coefficients. Benefiting from our novel representation, the proposed AvatarMAV can recover photo-realistic head avatars in just 5 minutes (implemented with pure PyTorch), which is significantly faster than the state-of-the-art facial reenactment methods. Project page: https://www.liuyebin.com/avatarmav., Comment: Accepted by SIGGRAPH 2023

3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Towards fine-grained control over facial attributes, recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. Explicit methods provide fine-grained expression control but cannot handle topological changes caused by hair and accessories, while implicit ones can model varied topologies but have limited generalization caused by the unconstrained deformation fields. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images. To achieve both deformation accuracy and topological flexibility, we propose a 3D representation called Generative Texture-Rasterized Tri-planes. The proposed representation learns Generative Neural Textures on top of parametric mesh templates and then projects them into three orthogonal-viewed feature planes through rasterization, forming a tri-plane feature representation for volume rendering. In this way, we combine both fine-grained expression control of mesh-guided explicit deformation and the flexibility of implicit volumetric representation. We further propose specific modules for modeling mouth interior which is not taken into account by 3DMM. Our method demonstrates state-of-the-art 3D-aware synthesis quality and animation ability through extensive experiments. Furthermore, serving as 3D prior, our animatable 3D representation boosts multiple applications including one-shot facial avatars and 3D-aware stylization., Comment: Accepted by CVPR 2023. Project page: https://mrtornado24.github.io/Next3D

We present FaceVerse, a fine-grained 3D Neural Face Model, which is built from hybrid East Asian face datasets containing 60K fused RGB-D images and 2K high-fidelity 3D head scan models. A novel coarse-to-fine structure is proposed to take better advantage of our hybrid dataset. In the coarse module, we generate a base parametric model from large-scale RGB-D images, which is able to predict accurate rough 3D face models in different genders, ages, etc. Then in the fine module, a conditional StyleGAN architecture trained with high-fidelity scan models is introduced to enrich elaborate facial geometric and texture details. Note that different from previous methods, our base and detailed modules are both changeable, which enables an innovative application of adjusting both the basic attributes and the facial details of 3D face models. Furthermore, we propose a single-image fitting framework based on differentiable rendering. Rich experiments show that our method outperforms the state-of-the-art methods., Comment: https://github.com/LizhenWangT/FaceVerse

Existing 3D-aware facial generation methods face a dilemma in quality versus editability: they either generate editable results in low resolution or high-quality ones with no editing flexibility. In this work, we propose a new approach that brings the best of both worlds together. Our system consists of three major components: (1) a 3D-semantics-aware generative model that produces view-consistent, disentangled face images and semantic masks; (2) a hybrid GAN inversion approach that initialize the latent codes from the semantic and texture encoder, and further optimized them for faithful reconstruction; and (3) a canonical editor that enables efficient manipulation of semantic masks in canonical view and product high-quality editing results. Our approach is competent for many applications, e.g. free-view face drawing, editing, and style control. Both quantitative and qualitative results show that our method reaches the state-of-the-art in terms of photorealism, faithfulness, and efficiency., Comment: Project Page: https://mrtornado24.github.io/IDE-3D

Wang L, Duan Y, Zhang W, et al. Clin Epidemiol. 2020;12:387–391. Page 388, left column, last sentence of the first paragraph, the text “Here we report early clinical features of 25 patients with confirmed COVID-19 who were admitted to Liaocheng Infectious Disease Hospital in Shandong Province” should read “Here we report early clinical features of 26 patients with confirmed COVID-19 who were admitted to Liaocheng Infectious Disease Hospital inShandong Province”. Page 389, Discussion section, first sentence, the text “We report here a cohort of 25 cases of laboratory-confirmed COVID-19 at our hospital caused by patient-to-patient transmission of 2019-nCoV” should read “We report here a cohort of 26 cases of laboratory-confirmed COVID-19 at our hospital caused by patient-to-patient transmission of 2019-nCoV”. The authors apologize for these errors. Read the original article

Lizhen Wang, 1 Yueting Duan, 1 Wenming Zhang, 2 Juan Liang, 2 Jianxiang Xu, 3 Yongqing Zhang, 3 Changgang Wu, 3 Yangling Xu, 3 Hua Li 2 1Department of Tuberculosis, Liaocheng Infectious Disease Hospital, Liaocheng, Shandong, People’s Republic of China; 2Intensive Care Unit, Liaocheng Infectious Disease Hospital, Liaocheng, Shandong, People’s Republic of China; 3Department of Respiration, Liaocheng Infectious Disease Hospital, Liaocheng, Shandong, People’s Republic of ChinaCorrespondence: Hua LiDepartment of Respiration, Liaocheng Infectious Disease Hospital, 45 Jianshe Eastern Road, Dongchang, Liaocheng, Shandong 252000, People’s Republic of ChinaEmail weiyichunguang@163.comIntroduction: Over 40,000 cases of coronavirus (CoV) disease 2019 (COVID-19) have been confirmed in China. The causative agent, 2019 novel CoV (2019-nCoV), has spread rapidly to more than 25 countries worldwide. Human-to-human transmission has accounted for most of the infections outside Wuhan. Most studies to date on COVID-19 have focused on disease etiology and the genomics of 2019-nCoV, with few reports on the epidemiologic and clinical characteristics of infected patients.Methods: We report early clinical features of 26 patients with confirmed COVID-19 who were admitted to Liaocheng Infectious Disease Hospital in Shandong Province.Results: The median age of the 26 patients with COVID-19 in this study was 42 years. The most common occupation was retail staff (16 patients, 61.54%), with 11 patients or their family members working at the same supermarket. Only 2 patients had visited Wuhan since December 2019; the other cases of 2019-nCoV infection arose from patient-to-patient transmission. Twelve patients had more than one sign or symptom; however, seven patients had no sign or symptom. The average time from symptom onset to admission was 4.5 days. CT revealed signs of bronchitis in 2 patients and unilateral and bilateral pneumonia in 9 and 15 patients, r

We propose NormalGAN, a fast adversarial learning-based method to reconstruct the complete and detailed 3D human from a single RGB-D image. Given a single front-view RGB-D image, NormalGAN performs two steps: front-view RGB-D rectification and back-view RGBD inference. The final model was then generated by simply combining the front-view and back-view RGB-D information. However, inferring backview RGB-D image with high-quality geometric details and plausible texture is not trivial. Our key observation is: Normal maps generally encode much more information of 3D surface details than RGB and depth images. Therefore, learning geometric details from normal maps is superior than other representations. In NormalGAN, an adversarial learning framework conditioned by normal maps is introduced, which is used to not only improve the front-view depth denoising performance, but also infer the back-view depth image with surprisingly geometric details. Moreover, for texture recovery, we remove shading information from the front-view RGB image based on the refined normal map, which further improves the quality of the back-view color inference. Results and experiments on both testing data set and real captured data demonstrate the superior performance of our approach. Given a consumer RGB-D sensor, NormalGAN can generate the complete and detailed 3D human reconstruction results in 20 fps, which further enables convenient interactive experiences in telepresence, AR/VR and gaming scenarios., Comment: 10 pages, 11 figures, ECCV 2020

Qisheng Xiong,1,2 Xuesong Wang,3 Lizhen Wang,1,2 Yan Huang,1,2 Xiaodong Tian,3 Yubo Fan,1,2 Chia-Ying Lin1,2,4,5 1School of Biological Science and Medical Engineering, Beihang University, Beijing, China; 2Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, China; 3Spine Department, The No 2 Affiliated Hospital of Qingdao University, Qingdao, China; 4Department of Orthopaedic Surgery, University of Cincinnati Academic Health Center, Cincinnati, OH, USA; 5Department of Biomedical Engineering, University of Cincinnati, Cincinnati, OH, USA Background: Bone morphogenetic proteins (BMPs), members of the TGF-β superfamily, are known to regulate cell proliferation, differentiation, apoptosis, chemotaxis, and angiogenesis. BMPs also participate in the development of most tissues and organs in vertebrates. Recombinant human (rh) BMPs, such as rhBMP-2, rhBMP-4, and rhBMP-7, have been recently approved to augment spinal fusion and recalcitrant long-bone non-unions because of their equivalent or superior efficacy to autogenous bone graft in enhancing bony fusion. Nonetheless, the use of BMPs is contraindicated in surgery for bone tumors because of concerns that this anabolic growth factor may cause tumor proliferation. However, we have repeatedly reported that BMP-2 is effective in inducing osteogenic differentiation of a subpopulation of osteosarcoma (OSA) cells that acquire stem cell attributes and are capable of reconstituting tumor masses, which in turn suppress the malignancy of the bone tumor.Methods: 3×105/20 µL human OSA 143B cells were inoculated into 5–6 weeks old BABL/c nude mice to establish orthotopic OSA. X-ray device was used to monitor the developed tumors in animals. Necropsy was performed and the pathology of lung metastasis were tested by Haemotoxylin and Eosin. Moreover, bone formation induced by rhBMP-2 was investigated through micro-computed tomography. In addition, immunohistoch

Shanghui Zhou,1 Ling Chen,2 Mubarak Mashrah,1 Yun Zhu,1 Jiannan Liu,1 Xi Yang,1 Zhijing He,3 Lizhen Wang,4 Tingxiu Xiang,2 Zhigang Yao,5 Feng Guo,6 Wenjun Yang,1 Chenping Zhang1 1Department of Oral and Maxillofacial-Head and Neck Oncology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 2Molecular Oncology and Epigenetics Laboratory, The First Affiliated Hospital of Chongqing Medical University, Chongqing, 3Department of Oral and Maxillofacial Surgery, The Second Xiangya Hospital, Central South University, Changsha, Hunan, 4Department of Oral Pathology, Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 5Department of Oral Pathology, Xiangya Stomatological Hospital, 6Department of Oral and Maxillofacial Surgery, Xiangya Hospital, Central South University, Changsha, Hunan, People’s Republic of China Abstract: Secreted frizzled-related proteins (SFRPs), the first identified Wnt antagonists, have been well recognized as tumor suppressors in multiple human cancers through suppressing the Wnt/β-catenin pathway. To better elucidate the mechanisms of SFRPs involved in the carcinogenesis of oral submucous fibrosis (OSF), one of the precancerous lesions of oral squamous cell carcinoma (OSCC), we investigated expression and localization of SFRP1, SFRP5, and β-catenin in normal oral epithelium, OSF, and OSCC tissues. We found that SFRP1 and SFRP5 were readily expressed in normal oral mucous tissues but gradually decreased in OSF early, moderately advanced, and advanced tissues and rarely expressed in OSCC tissues. We found the changes of SFRP1 localization and SFRP5 localization from nucleus to cytoplasm in the carcinogenesis of OSF. There is a significant association among reduced SFRP1, SFRP5, and cytoplasmic/nuclear β-catenin expression, which is correlated with higher tumor grade and stage of OSCC. We further found that SFRP1 and

In animal breeding programs, deoxyribonucleic acid (DNA) markers can be used to identify sires that are less susceptible to disease. These DNA markers are typically discovered in populations that display differences in susceptibility. To find those differences, it was hypothesized that sires influence their offspring responses to infection with H. parasuis. To identify differences in susceptibility, colostrum-deprived pigs derived from 6 sires were inoculated with a virulent strain of H. parasuis serovar 5. Pigs were infected at 21-d of age and euthanized 1, 2, or 3 days post-infection. Rectal temperatures, bacterial detection, clinical signs, and lesions were measured by comparing disease susceptibility in the offspring from each sire. The effect of the sire on the severity of disease in the offspring was statistically analyzed using to a 2-way ANOVA with sire and test day as fixed effects. Significant differences among sires were found for lesions, rectal temperatures from days 0-1 and 0-2 (P < 0.05) and marginal effects for clinical signs (P = 0.08). On average, the offspring of sire H94 was the most susceptible to challenge. Responses to infection were categorized to determine the clinical responses and analyzed by Chi square. Overall, 10% of all pigs infected were fully resistant to H. parasuis infection. Boar H94 didn't produce any fully resistant offspring. Differences in susceptibility to H. parasuis were observed, and the results support the hypothesis that sires influence their offspring's response to infection. Tissues from this population could be used to identify DNA markers for genetic selection of sires that produce offspring more resistant to H. parasuis infection.