Facial landmarking for in-the-wild images with local inference based on global appearance

We present a novel method that tackles the problem of facial landmarking in unconstrained conditions within the part-based framework. Part-based methods alternate the evaluation of local appearance models to produce a per-point response map and a shape fitting step which finds a valid face shape that maximises the sum of the per-point responses. Our approach focuses on obtaining better appearance models for the creation of the response maps, and it can be used in combination with any shape fitting strategy. Local appearance models need to tackle very heterogeneous data when dealing with in-the-wild imagery due to factors as varying head poses, facial expressions, identity, lighting conditions, or image quality among others. Pose-wise experts are typically used in this scenario so that each expert deals with more homogeneous data. However, the computation cost at test time is significantly increased. Furthermore, choosing the right expert is not straightforward, which can lead to gross errors. We propose to dynamically select at test time the training examples used for inference. We use a global similarity measure to select the most adequate training examples for inference, and create a single test sample-specific expert using a localised inference technique. To illustrate the validity of these ideas, we capitalise on the recently proposed use of regression to generate local appearance models. In particular, we use Gaussian processes, as their non-parametric nature easily allows for localised regression. This novel way of constructing the response maps is combined with two state-of-the-art standard shape fitting algorithms, the popular Constrained Local Models framework and the Consensus of Exemplars method. We validate our method on two publicly available datasets as well as on a cross-dataset experiment, showing a considerable performance improvement of the proposed approach. Display Omitted The problem of facial landmark detection in unconstrained conditions is tackled.A new way of constructing per-landmark response maps is proposed.The method is based on the selection at test time of globally similar images.We use local Gaussian process regression for inference.