Search

Given an uncalibrated network of video cameras, we are tasked with the problem of building a 3D model of every person's face as they move within the network. The process of doing so requires overcoming many challenges including person detection, tracking, cross-camera correspondence (how to determine if a person in one camera is the same person in another), and the final 3D model reconstruction from multiple views in real-time or at near real-time speeds (efficient modeling and data fusion from multiple hardware sources). Towards this goal, a wireless camera network was designed and built from the ground up, a new tracking algorithm and a cross-camera human signature method was developed, and face modeling using multiple cameras in a real-world setting was performed.

Synthesis of realistic human faces is still one of the biggest challenges in computer graphics. Similarly, analysis of human faces in images has been a traditional topic of research in computer vision. Both research areas related to human faces have many practical applications including face modeling, face tracking, face relighting, and face recognition in diverse industry fields such as games, virtual reality, digital photography, biometrics and security. In this dissertation we explore the problem of model-based 3D face synthesis and analysis using various kind of image ensembles including multi-view image silhouettes, sequential video frames, and single photographs. We design and develop various image-model difference metrics that can be used in an optimization framework to recover the optimal model parameters of our 3D face model from the input images. In the first part of this dissertation we present a novel method for 3D face reconstruction from a set or sequence of 2D binary silhouettes. Experiments with a multi-camera rig as well as monocular video sequences demonstrate the advantages of our 3D modeling framework. In order to determine the set of optimal views required for silhouette-based shape reconstruction, we build on our modeling framework and extend it by aggressive pruning of the view-sphere with view clustering and various imaging constraints. In the second part of this dissertation we present a novel framework that acquires the 3D shape, texture, pose and illumination of a face from a single photograph. Using a custom-built face scanning system which is equipped with 16 digital cameras and 146 directional LED light sources, a large-scale dataset was collected, which consists of 3D face scans and light reflection images of a diverse group of human subjects. From this dataset, we derive a novel measurement-based illumination model that implicitly incorporates cast-shadows and specularities. We also propose a novel fitting framework that estimates the parameters of the morphable model by minimizing the distance of the input image to the dynamically changing illumination subspace. We leverage our modeling and fitting methods to solve two challenging problems in computer graphics and computer vision - face relighting and face recognition.