Search

Tracking the head in a video stream is a common thread seen within computer vision literature, supplying the research community with a large number of challenging and interesting problems. Head pose estimation from monocular cameras is often considered an extended application after the face tracking task has already been performed. This often involves passing the resultant 2D data through a simpler algorithm that best fits the data to a static 3D model to determine the 3D pose estimate. This work describes the 2.5D constrained local model, combining a deformable 3D shape point model with 2D texture information to provide direct estimation of the pose parameters, avoiding the need for additional optimization strategies. It achieves this through an analytical derivation of a Jacobian matrix describing how changes in the parameters of the model create changes in the shape within the image through a full-perspective camera model. In addition, the model has very low computational complexity and can run in real-time on modern mobile devices such as tablets and laptops. The point distribution model of the face is built in a unique way, so as to minimize the effect of changes in facial expressions on the estimated head pose and hence make the solution more robust. Finally, the texture information is trained via local neural fields—a deep learning approach that utilizes small discriminative patches to exploit spatial relationships between the pixels and provide strong peaks at the optimal locations. [ABSTRACT FROM AUTHOR]

We present an end-to-end pipeline for the export of 3D scenes from content creation tools to a real-time rendering engine in an embeddable web-page, including a novel system for compression/decompression of textured polygonal meshes. We show that the compression/decompression outperforms the best state-of-the-art non-progressive alternative, especially as bandwidth increases, showing that web-specific techniques should consider the whole user pipeline. Our pipeline also includes progressivity, which is paramount for a good interactive user experience, and permits full user interaction with lower resolution versions of the 3D scenes, while progressively higher resolution data is downloaded. Finally, we discuss how our method may be used in the future to facilitate the transfer of animated meshes. [ABSTRACT FROM AUTHOR]

We present an approach for real-time pen-and-ink hatching renderings on large scenes. Starting with 3D models including photorealistic textures and materials, we aim to propose a solution that produces hatched renderings. As we consider scene objects described as polygonal meshes with their own textures, we produce once hatching patterns at different tones and resolutions considering the material of each object. To achieve that, we create a flow direction map per texture pixel, using contour characteristics extracted from the original texture and then interpolated. Stroke trajectories are thus generated depending on the flow direction and using B-splines, providing tones from light-to-dark. Tones are then stored in a mutli-resolution tonal art map. Moreover, we aim to overcome the limitations of existing hatching rendering methods by introducing an illumination model, fully implemented on GPU and able to manage three shading types: regular shadow, soft/cast shadow and self-shadowing. Tones and hatching resolutions are, therefore, assigned according to local/global illumination supporting multiple light sources. Our model, both dedicated for 3D static model renderings and 3D model animation, supports model deformations and is also spatially and temporally coherent since it gives continuous hatching strokes during object animations and/or light displacements. [ABSTRACT FROM AUTHOR]

This paper outlines a digital workflow that allows artists to add visual detail to 3D models whilst adhering to the strict polygonal budget that is vital to successfully render in real-time. Two versions of a model – a high-poly model, and a low-poly model – are produced to create a 2D normal map through a process known as ‘baking’, to achieve the illusion of the high resolution detail on the low-resolution model. [ABSTRACT FROM PUBLISHER]

We present a simple and effective method for constructing a gallery that consists of weathering effect elements called time-dependent appearance manifolds (TDAMs). Since TDAMs are computed from sample video clips showing dynamic weathering phenomena, they represent very smooth changes in the appearance of weathered pixels over time. Once a gallery with a variety of weathering effects is prepared, users can interactively choose and apply the predefined effects onto the surface of 3D graphic models and then finally assign the most appropriate one. This video-based weathering method can be implemented with very simple algorithms and it supports predictable, intuitive, and natural effects. Our system allows users to produce photorealistic augmented videos that include 3D graphic models weathered by our method. Moreover, users can easily enhance the realism of the augmented videos by manipulating rendering parameters such as the degree of weathering, texturing, lighting, and shadowing through user-friendly graphical user interfaces (GUIs). [ABSTRACT FROM AUTHOR]

In economy, society and personal life map-based interactive geospatial visualization becomes a natural element of a growing number of applications and systems. The visualization of 3D geospatial information, however, raises the question how to represent the information in an effective way. Considerable research has been done in technology-driven directions in the fields of cartography and computer graphics (e.g., design principles, visualization techniques). Here, non-photorealistic rendering (NPR) represents a promising visualization category - situated between both fields - that offers a large number of degrees for the cartography-oriented visual design of complex 2D and 3D geospatial information for a given application context. Still today, however, specifications and techniques for mapping cartographic design principles to the state-of-the-art rendering pipeline of 3D computer graphics remain to be explored. This paper revisits cartographic design principles for 3D geospatial visualization and introduces an extended 3D semiotic model that complies with the general, interactive visualization pipeline. Based on this model, we propose NPR techniques to interactively synthesize cartographic renditions of basic feature types, such as terrain, water, and buildings. In particular, it includes a novel iconification concept to seamlessly interpolate between photorealistic and cartographic representations of 3D landmarks. Our work concludes with a discussion of open challenges in this field of research, including topics, such as user interaction and evaluation. [ABSTRACT FROM AUTHOR]

Street-level visualization is an important application of 3D city models. Challenges to street-level visualization include the cluttering of buildings due to fine detail and visualization performance. In this paper, a novel method is proposed for street-level visualization based on visual saliency evaluation. The basic idea of the method is to preserve these salient buildings in a scene while removing those that are non-salient. The method can be divided into pre-processing procedures and real-time visualization. The first step in pre-processing is to convert 3D building models at higher Levels of Detail (LoDs) into LoD1 models with simplified ground plans. Then, a number of index viewpoints are created along the streets; these indices refer to both the position and the direction of each street site. A visual saliency value is computed for each building, with respect to the index site, based on a visual difference between the original model and the generalized model. We calculate and evaluate three methods for visual saliency: local difference, global difference and minimum projection area. The real-time visualization process begins by mapping the observer to its closest indices. The street view is then generated based on the building information stored in those indexes. A user study shows that the local visual saliency method performs better than do the global visual saliency, area and image-based methods and that the framework proposed in this paper may improve the performance of 3D visualization. [ABSTRACT FROM AUTHOR]

In this paper, we present an on-line real-time physics-based approach to motion control with contact repositioning based on a low-dimensional dynamics model using example motion data. Our approach first generates a reference motion in run time according to an on-line user request by transforming an example motion extracted from a motion library. Guided by the reference motion, it repeatedly generates an optimal control policy for a small time window one at a time for a sequence of partially overlapping windows, each covering a couple of footsteps of the reference motion, which supports an on-line performance. On top of this, our system dynamics and problem formulation allow to derive closed-form derivative functions by exploiting the low-dimensional dynamics model together with example motion data. These derivative functions and their sparse structures facilitate a real-time performance. Our approach also allows contact foot repositioning so as to robustly respond to an external perturbation or an environmental change as well as to perform locomotion tasks such as stepping on stones effectively. [ABSTRACT FROM AUTHOR]

We introduce a novel efficient technique for automatically transforming a generic renderable 3D scene into a simple graph representation named ExploreMaps, where nodes are nicely placed point of views, called probes, and arcs are smooth paths between neighboring probes. Each probe is associated with a panoramic image enriched with preferred viewing orientations, and each path with a panoramic video. Our GPU-accelerated unattended construction pipeline distributes probes so as to guarantee coverage of the scene while accounting for perceptual criteria before finding smooth, good looking paths between neighboring probes. Images and videos are precomputed at construction time with off-line photorealistic rendering engines, providing a convincing 3D visualization beyond the limits of current real-time graphics techniques. At run-time, the graph is exploited both for creating automatic scene indexes and movie previews of complex scenes and for supporting interactive exploration through a low-DOF assisted navigation interface and the visual indexing of the scene provided by the selected viewpoints. Due to negligible CPU overhead and very limited use of GPU functionality, real-time performance is achieved on emerging web-based environments based on WebGL even on low-powered mobile devices. [ABSTRACT FROM AUTHOR]

We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character. [ABSTRACT FROM AUTHOR]

We present a new algorithm for realtime face tracking on commodity RGB-D sensing devices. Our method requires no user-specific training or calibration, or any other form of manual assistance, thus enabling a range of new applications in performance-based facial animation and virtual interaction at the consumer level. The key novelty of our approach is an optimization algorithm that jointly solves for a detailed 3D expression model of the user and the corresponding dynamic tracking parameters. Realtime performance and robust computations are facilitated by a novel subspace parameterization of the dynamic facial expression space. We provide a detailed evaluation that shows that our approach significantly simplifies the performance capture workflow, while achieving accurate facial tracking for realtime applications. [ABSTRACT FROM AUTHOR]

We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, the pose and expressions of the face are recovered by fitting a user-specific blendshape model to them. The main technical contribution of this work is the 3D regression algorithm that learns an accurate, user-specific face alignment model from an easily acquired set of training data, generated from images of the user performing a sequence of predefined facial poses and expressions. Experiments show that our system can accurately recover 3D face shapes even for fast motions, non-frontal faces, and exaggerated expressions. In addition, some capacity to handle partial occlusions and changing lighting conditions is demonstrated. [ABSTRACT FROM AUTHOR]

Whereas 3D surface models are often used for augmented reality (e.g., for occlusion handling or model-based camera tracking), the creation and the use of such dense 3D models in augmented reality applications usually are two separated processes. The 3D surface models are often created in offline preparation steps, which makes it difficult to detect changes and to adapt the 3D model to these changes. This work presents a 3D change detection and model adjustment framework that combines AR techniques with real-time depth imaging to close the loop between dense 3D modeling and augmented reality. The proposed method detects the differences between a scene and a 3D model of the scene in real time. Then, the detected geometric differences are used to update the 3D model, thus bringing AR and 3D modeling closer together. The accuracy of the geometric difference detection depends on the depth measurement accuracy as well as on the accuracy of the intrinsic and extrinsic parameters. To evaluate the influence of these parameters, several experiments were conducted with simulated ground truth data. Furthermore, the evaluation shows the applicability of AR and depth image-based 3D modeling for model-based camera tracking. [ABSTRACT FROM AUTHOR]

Non-photorealistic (illustrative) rendering augments typical rendering models to selectively emphasize or de-emphasize specific structures of rendered objects. Illustrative techniques may affect not only the rendering style of specific portions of an object but also their visibility, ensuring that less important regions do not occlude more important ones. Cutaway views completely remove occluding, unimportant structures-possibly also removing valuable context information-while existing solutions for smooth reduction of occlusion based on importance lack precise visibility control, simplicity and generality. We introduce a new front-to-back fragment composition equation that directly takes into account a measure of sample importance and allows smooth and precise importance-based visibility control. We demonstrate the generality of our composition equation with several illustrative effects, which were obtained by using a set of importance measures calculated on the fly or defined by the user. The presented composition method is suitable for direct volume rendering as well as rendering of layered 3D models. We discuss both cases and show examples, though focusing mainly on illustration of volumetric data. [ABSTRACT FROM AUTHOR]

Virtual endoscopy is a technique to explore hollow organs and anatomical cavities using 3D medical imaging and computer graphics. In this paper, boundary model and local feature structure are used to realize tissue segmentation, and a new efficient algorithm is presented to solve path planning. As to real-time processing, a frame in virtual endoscopy is divided into near viewpoint part and far viewpoint part based on volume data characteristics in our method. In the aspect of scene rendering, a ray casting algorithm based on the boundary voxel is proposed. Thus the voyage images can be rendered in real time with high quality in virtual endoscopy system by using these techniques. The experiments show that application results of our algorithm in tissue segmentation, path planning, scene rendering are better than other algorithms. Copyright © 2011 John Wiley & Sons, Ltd. Path planning and navigation images in virtual endoscopy system for virtual medicine. [ABSTRACT FROM AUTHOR]

Facial animation details like wrinkles or bulges are very useful for the analysis and the interpretation of facial emotions and expressions. However, outfitting a virtual face with expression details for real-time applications is a difficult task. In this paper, we propose a mono-camera acquisition technique of facial animation details and a technique which add a wrinkle map layer (fine-scale animation) to a skinning layer (large-scale animation) for real-time rendering of a virtual 3D face. The acquisition is based on ratio image computed from two pictures of a same face, with and without expression. The real-time dynamic wrinkles technique is based on a small set of reference poses. These two methods offer an easy and low-cost way to capture facial animation details and use it for real-time facial animation. Copyright © 2011 John Wiley & Sons, Ltd. In this paper, we propose a mono-camera acquisition technique of facial animation details and a technique which add a wrinkle map layer (fine-scale animation) to a skinning layer (large-scale animation) for real-time rendering of a virtual 3D face. The acquisition is based on ratio image computed from two pictures of a same face, with and without expression. The real-time dynamic wrinkles technique is based on a small set of reference poses. [ABSTRACT FROM AUTHOR]

Real-time rendering can benefit from global illumination methods to make the 3D environments look more convincing and lifelike. On the other hand, the conventional global illumination algorithms for the estimation of the diffuse surface interreflection make heavy usage of intra- and interobject visibility calculations, so they are time-consuming, and using them in real-time graphics applications can be prohibitive for complex scenes. Modern illumination approximations, such as ambient occlusion variants, use precalculated or frame-dependent data to reduce the problem to a local shading one. This paper presents a fast real-time method for visibility sampling using volumetric data in order to produce accurate inter- and intraobject ambient occlusion. The proposed volume sampling technique disassociates surface representation data from the visibility calculations, and therefore, makes the method suitable for both primitive-order or screen-order rendering, such as deferred rendering. The sampling mechanism can be used in any application that performs visibility queries or ray marching. [ABSTRACT FROM AUTHOR]

Glare is a consequence of light scattered within the human eye when looking at bright light sources. This effect can be exploited for tone mapping since adding glare to the depiction of high-dynamic range (HDR) imagery on a low-dynamic range (LDR) medium can dramatically increase perceived contrast. Even though most, if not all, subjects report perceiving glare as a bright pattern that fluctuates in time, up to now it has only been modeled as a static phenomenon. We argue that the temporal properties of glare are a strong means to increase perceived brightness and to produce realistic and attractive renderings of bright light sources. Based on the anatomy of the human eye, we propose a model that enables real-time simulation of dynamic glare on a GPU. This allows an improved depiction of HDR images on LDR media for interactive applications like games, feature films, or even by adding movement to initially static HDR images. By conducting psychophysical studies, we validate that our method improves perceived brightness and that dynamic glare-renderings are often perceived as more attractive depending on the chosen scene. [ABSTRACT FROM AUTHOR]

In this article we present TopoPlan, a topological planner dedicated to real-time humanoid path-planning and motion adaptation to floor and ceiling constraints inside complex static environments. This planner analyzes unstructured 3D triangular meshes in order to automatically determine their topology. The analysis is based on a prismatic spatial subdivision which is analyzed, taking into account humanoid characteristics, in order to extract navigable surfaces and precisely identify environmental constraints such as floors, ceilings, walls, steps and bottlenecks. The technique also provides a lightweight roadmap computation covering all accessible free space. We demonstrate the properties of our topological planner within the context of two reactive motion control processes: an on-the-fly trajectory optimization and foot print generation process that correctly handles climbing of complex staircases, and a reactive ceiling adaptation process that handles beam avoidance and motion adaptation to irregular floors and ceilings. We further show that the computation cost of these processes is compatible with the real time animation of several dozens of virtual humans. [ABSTRACT FROM AUTHOR]

Real-time rendering of large-scale vector maps over terrain surfaces requires displaying substantial numbers of polylines and polygons. Because a long latency in display and manipulation is fatal to maintaining presence in a virtual environment, a new method efficiently simplifies and renders such maps. This method consists of three steps. First, it simplifies the vector map while maintaining the map's topological consistency and preventing local conflicts such as intersections or self-intersections. Second, it generates view-dependent level-of-detail (LOD) models. Finally, it overlays the maps onto multiresolution terrain models through the stencil shadow volume algorithm and other techniques. Experiments demonstrated the method's efficiencies in real-time rendering of large-scale vector maps over various LOD terrain surfaces. [ABSTRACT FROM PUBLISHER]