Search

International audience; This paper proposes a novel framework to produce 3D, high-precision models of humans from multi-view capture. This method's inputs are a visual hull and several sets of multi-baseline views. For each such view set, a surface is reconstructed with a multi-baseline stereovision method, then used to carve the visual hull. Carved visual hulls from different view sets are then fused pairwise to deliver the intended 3D model. The contributions of this paper are threefold: (i) the addition of visual hull guidance to a multi-baseline stereovision method, (ii) a carving solution to a visual hull from an interpolated and smooth stereovision surface, and (iii) a fusion solution to merge differently carved volumes differing in several areas. The paper shows that the proposed approach helps recovering a high quality carved volume, a 3D representation of the human to be modelled, that is precise even for small details and in concave areas subjected to occlusion.

High dynamic range (HDR) imaging and stereoscopic 3D are active research fields, with challenges posed in various directions: acquisition, processing, and display. This chapter analyzes the latest advances in stereo HDR imaging acquisition from multiple exposures, in order to highlight the current progress toward a common target: stereoscopic 3D HDR video. A method to build multiscopic HDR images from low dynamic range (LDR) multiexposures is proposed. This method is based on a patch-match algorithm which has been adapted and improved to take advantage of epipolar geometry constraints of stereo images. Experimental results show accurate matching between stereo images that allow generating HDR for each LDR view of the sequence.

4D multi-view reconstruction of moving actors has many applications in the entertainment industry and although studios providing such services become more accessible, efforts have to be done in order to improve the underlying technology to produce high-quality 4D contents. In this paper, we present a method to derive a time-evolving surface representation from a sequence of binary volumetric data representing an arbitrary motion in order to introduce coherence in the data. The context is provided by an indoor multi-camera system which performs synchronized video captures from multiple viewpoints in a chroma-key studio. Our input is given by a volumetric silhouette-based reconstruction algorithm that generates a visual hull at each frame of the video sequence. These 3D volumetric models lack temporal coherence, in terms of structure and topology, as each frame is generated independently. This prevents an easy post-production editing with 3D animation tools. Our goal is to transform this input sequence of independent 3D volumes into a single dynamic structure, directly usable in post-production. Our approach is based on a motion estimation procedure. An unsigned distance function on the volumes is used as the main shape descriptor and a 3D surface matching algorithm minimizes the interference between unrelated surface regions. Experimental results, tested on our multi-view datasets, show that our method outperforms other approaches based on optical flow when considering robustness over several frames.

Relighting algorithms make it possible to take a model of a real-world scene and virtually modify its lighting. Unlike other methods that require controlled conditions, we introduce a new radiance capture method that allows the user to capture parts of the scene under different lighting conditions. A novel calibration method is presented that finds the positions of reflective spheres and their mathematically accurate projection onto the scene geometry. The resulting radiance distribution is used to estimate a diffuse reflectance for each object, computed coherently using the appropriate light probe image. Finally, the scene is relit using a novel illumination pattern.

The convergence between high dynamic range (HDR) images/video and stereoscopic/3D images/video is an active research field, with limitations coming from various aspects: acquisition, processing, and display. This work analyzes the latest advances in HDR video and stereo HDR video acquisition from multiple exposures in order to highlight the current progress towards a common target: 3D HDR video. The most relevant existing techniques are presented and discussed.

We propose a fully automatic time-varying surface reconstruction of an actor’s performance captured from a production stage through omnidirectional video. The resulting mesh and its texture can then directly be edited in post-production. Our method makes no assumption on the costumes or accessories present in the recording. We take as input a raw sequence of volumetric static poses reconstructed from video sequences acquired in a multi-viewpoint chroma-key studio. The first frame is chosen as the reference mesh. An iterative approach is applied throughout the sequence in order to induce a deformation of the reference mesh for all input frames. At first, a pseudo-rigid transformation adjusts the pose to match the input visual hull as closely as possible. Then, local deformation is added to reconstruct fine details. We provide examples of actors’ performance inserted into virtual scenes, including dynamic interaction with the environment.

This paper proposes a novel stereovision framework for multi-view 3D reconstruction relying on inputs of both several sets of multi-baseline views and a visual hull [3]. The pipeline is illustrated in figure 1. Our Contributions of this paper are threefold: (i) improvement of our multi-baseline stereovision method [2] by VH guidance, (ii) carving VH from stereovision surface and (iii) merging differently carved volumes. Our approach builds on a previously proposed framework [2] for multi-baseline stereo-vision which provides upon the Disparity Space (DS) introduced by [5], a materiality map expressing the probability for 3D sample points to lie on a visible surface. Our acquisition system [4] composed of the cameras which are scattered around the observed scene in order to build the VH, with several groups laid as multi-scopic units dedicated to multi-baseline stereovision. Multi-scopic units are composed of aligned and evenly distributed cameras.

This paper outlines a study into the effects of co-location (the term `co-location' is used throughout to refer to the co-location of haptic and visual sensory modes, except where otherwise specified) of haptic and visual sensory modes in VR simulations. The study hypothesis is that co-location of these sensory modes will lead to improved task performance within a VR environment. Technical challenges and technological limitations are outlined prior to a description of the implementation adopted for this study. Experiments were conducted to evaluate the effect on user performance of co-located haptics (force feedback) in a 3D virtual environment. Results show that co-location is an important factor, and when coupled with haptic feedback the performance of the user is greatly improved.

A mixed reality (MR) represents an environment composed both by real and virtual objects. MR applications are used more and more, for instance in surgery, architecture, cultural heritage, entertainment, etc. For some of these applications it is important to merge the real and virtual elements using consistent illumination. This paper proposes a classification of illumination methods for MR applications that aim at generating a merged environment in which illumination and shadows are consistent. Three different illumination methods can be identified: common illumination, relighting and methods based on inverse illumination. In this paper a classification of the illumination methods for MR is given based on their input requirements: the amount of geometry and radiance known of the real environment. This led us to define four categories of methods that vary depending on the type of geometric model used for representing the real scene, and the sdifferent radiance information available for each point of the real scene. Various methods are described within their category. The classification points out that in general the quality of the illumination interactions increases with the amount of input information available. On the other hand, the accessibility of the method decreases since its pre-processing time increases to gather the extra information. Recent developed techniques managed to compensate unknown data with clever techniques using an iterative algorithm, hardware illumination or recent progress in stereovision. Finally, a review of illumination techniques for MR is given with a discussion on important properties such as the possibility of interactivity or the amount of complexity in the simulated illumination.

This paper proposes a novel framework for multi-baseline stereovision exploiting the information redundancy to deal with known problems related to occluded regions. Inputs are multiple images shot or rectified in simplified geometry which induces a convenient sampling scheme of scene space: the disparity space. Instead of uniquely relying on image-space information like most multi-view stereovision methods, we work in this sampled scene space. We use fuzzy visibility reasoning and pixel neighborhood similarity measures in order to optimize fuzzy 3D discrete maps of materiality yielding precise reconstruction even in low texture and semi occluded regions. Our main contribution is to build on the disparity space to propose a new materiality map which locates the object surfaces within the actual scene.

International audience; This chapter analyzes camera calibration from a geometric and colorimetric perspective. The first part of the chapter introduces the mathematical model that describes a camera such as position, orientation, focal, as well as its applications such as 3D reconstructions. It analyzes different geometric processes for stereoscopic images such as corrections of radial distortion as well as image rectification. The second part of the chapter focuses on colorimetric models relating to digital acquisition systems. It demonstrates how to characterize a colorimetric camera and then analyze different elements involved in color correction for images taken from a system of cameras.

Multiview acquisition, the focus of this chapter, relates to the capture of synchronized video data representing different viewpoints of a single scene. In contrast to video surveillance systems, which deploy multiple cameras to visually cover a large scale environment to be monitored with little redundancy, the materials, devices or systems used in multiview acquisition are designed to cover several perspectives of a single, often fairly restricted, physical space and use redundancy in images for specific aims. Depending on the final application, the number, layout and settings of cameras can fluctuate greatly. This chapter will introduce the main camera configurations in a purely video multiview capture context, using notable practical examples and their use. Each time, we will also propose links to databases providing access to media produced by devices within each category.

We propose a new methodology to acquire HDR video content for autostereoscopic displays by adapting and augmenting an eight view video camera with standard sensors. To augment the intensity capacity of the sensors, we combine images taken at dierent exposures. Since the exposure has to be the same for all objectives of our camera, we x the exposure variation by applying neutral density lters on each objective. Such an approach has two advantages: several exposures are known for each video frame and we do not need to worry about synchronization. For each pixel of each view, an HDR value is computed by a weighted average function applied to the values of matching pixels from all views. The building of the pixel match list is simplied by the property of our camera which has eight aligned, equally distributed objectives. At each frame, this results in an individual HDR image for each view while only one exposition per view was taken. The nal eight HDR images are tone-mapped and interleaved for autostereoscopic display.

Computer graphics is always growing, in both the techniques it employs and the areas in which it’s applied. We continually see waves of new approaches and applications coming from other fields. At first, these new ideas meet some resistance. People question, “Is this really computer graphics?” However, these original, controversial views often become central to computer graphics. This special issue features four new areas of expansion in computer graphics. These new approaches and applications come from recent research in human–computer interaction (HCI) and computer vision. Each of these new areas is a “big thing” that computer graphics practitioners and researchers need to be aware of. Each area has emerged as the result of advanced technologies that have improved data acquisition—whether through devices that sense touch or imaging systems that produce output in multiple sharply defined wavelength bands. The first big thing is “Visualization beyond the Desktop.” Visualization long ago grew past being simply a tool for individual scientists to examine their data. It’s used in virtually every discipline, including the humanities and social sciences. It has become commonplace in public communication—from the 3D visualizations in weather forecasts to interactive analysis of housing prices in online newspapers. To create and interact with visualizations, researchers have for many years discussed post-WIMP (windows, icons, menus, pointing devices) interfaces. New devices such as smart phones with cameras and GPS and multitouch tables have made such interfaces feasible. In their article, Jonathan Roberts and his colleagues present a view of visualization that’s no longer bound to the desktop and includes interactions with all of a user’s senses. The second big thing is “Automatic Emotion Recognition Based on Body Movement Analysis.” New devices such as inexpensive high-resolution cameras have made it possible to extend user interaction to natural movements. In the past, much HCI research investigated detection of human emotion, often under the name “affective computing.” Emotion recognition, particularly from full-body measurements, has reached the point that it’s suitable for incorporation into computer graphics systems such as computer games and intelligent tutors. In their article, Haris Zacharatos and his colleagues present a short survey of emotion recognition. On the basis of the progress so far, they also outline four major lines of inquiry for additional research. The third big thing is “ElectroEncephaloGraphics: Making Waves in Computer Graphics Research.” Electroencephalography (EEG) is well known as a medical-sensing technology for sensing brain activity through electrical measurements on a person’s scalp. Recent research has made EEG devices much less expensive. Furthermore, signalprocessing research has made it possible to detect subtler signals in the often-noisy EEG data. One possible application of EEG is in HCI. However, in their article, Maryam Mustafa and Marcus Magnor explore an alternative application of EEG in graphics. EEG makes it possible to record an individual’s response to viewing images. Rather than explicitly asking subjects questions such as, “which of these images is better?” in computer graphics psychophysical tests, researchers can measure EEG response. This opens a new approach to improving computer graphics techniques by gathering passive, rather than explicit, responses. Mustafa and Magnor give examples of applying this idea to improving rendering techniques.

International audience; This article proposes using search engine results data such as the number of results containing relevant terms, to measure the evolution of Haptics, the field devoted to the science and technology of the sense of touch. Haptics is a complex discipline which is at the intersection of the knowledge of several specialized fields like robotics, computer science, psychology, and mathematics. It can also appear as a new and emergent discipline due to the fact that many promising haptic interfaces, which allow innovative multimodal applications in many fields, have become mature only recently. The study presented in this article uses data collected at different periods of time (in December 1999, January 2004, January 2005, November 2006 and April 2007) on Web search engines from requests on three different terminologies: haptique, haptik and haptics, taken respectively from French, German, and English languages. The evolution of Haptics is seemingly reflected by to the online frequency of these specific terms over time. This evolution has been measured by considering the Internet community through search engines such as Google or Yahoo!.

In this paper, we propose a hierarchical approach to path planning that scales well for large crowds. Crowds have become an important research field that presents challenges associated to the high complexity of potential behaviors. In most related work crowds are designed to follow global or local rules that infer intelligent navigation and behavior. Path planning methods can be applied in simulation where it is required to have an exact distribution of the crowd in the environment. However, the simulation is often created offline as path planning algorithms do not scale well with large numbers of agents, if each is considered individually. Our approach provides reliable paths that can be used for environment crowd distribution evaluations, and scales well with a high number of agents to run at interactive from rates for up to a few thousand agents.

A face is capable of producing about twenty thousand different facial expressions [2]. Many researchers on Virtual Characters have selected a limited set of emotional facial expressions and defined them as basic emotions, which are universally recognized facial expressions. These basic emotions have been well studied since 1969 and employed in many applications [3]. However, real life communication usually entails more complicated emotions. For instance, communicative emotions like "convinced", "persuaded" and "bored" are difficult to describe adequately with basic emotions. Our daily face-to-face interaction is already accompanied by more complex mental states, so an empathic animation system should support them. Compared to basic emotions, complex mental states are harder to model because they require knowledge of temporal changes in facial displays and head movements as opposed to a static snapshot of the facial expression. We address this by building animation models for complex emotions based on video clips of professional actors displaying these emotions.

We describe the development and evaluation of a large-scale multimodal virtual reality simulation suitable for the visualization of cultural heritage sites and architectural planning. The system is demonstrated with a reconstruction of an ancient Greek temple in Messene that was created as part of a EU funded cultural heritage project (CREATE). The system utilizes a CAVE-like theatre consisting of head-tracked user localization, a haptic interface with two arms, and 3D sound. The haptic interface was coupled with a realistic physics engine allowing users to experience and fully appreciate the effort involved in the construction of architectural components and their changes through the ages. Initial user-based studies were carried out, to evaluate the usability and performance of the system. A simple task of stacking blocks was used to compare errors and timing in a haptics-enabled system with a haptics-disabled system. In addition, a qualitative study of the final system took place while it was installed in a museum.

The incorporation of virtual populations into virtual environmentsenhances the realism of these environments. Many effective algo-rithms have been developed to govern the movement of individualavatars. One [Loscos et al. 2003] combines local navigation rulesbased on a multi-layer data system as in [Tecchia et al. 2001] whileusing goals for way finding. We designed a new system buildingon those ideas that adds new levels to this behavior in the form ofpoints of attraction, evacuations and walking in small groups.

Figure 1: (a) The 5 LDRI segments show HCM (walking people) and the resulting HDRI (large image) shows an undesirable duplication of the human shapes (inset). (b) HDRI generation using the presented method, HCM is removed using a variance image, VI. (c) HDRI of a dynamic scene (larger image) with LCM (leaves). (d) HDRI after LCM removal using an uncertainty measure, UI. (e, top) VI segmentation. (e, bottom) UI segmentation.

International audience; In the context of virtual reality, the simulation of complex environments with many animated objects is becoming more and more common. Virtual reality applications have always promoted the development of new efficient algorithms and image-based rendering techniques for real-time interaction. In this paper, we propose a technique which allows the real-time simulation in a city of the reflections of static geometry (eg. building) on specular dynamic objects (vehicles). For this, we introduce the idea of multiple environment maps. We pre-compute a set of reference environment maps at strategic positions in the scene, that are used at run time and for each visible dynamic object, to compute local environment maps by resampling images. To efficiently manage a small number of reference environment maps, compared to the scene dimension, for each vertex of the reconstructed environment we perform a ray tracing in a heightfield representation of the scene. We control the frame rate by adaptative reconstruction of environment maps. We have implemented this approach, and the results show that it is efficient and scalable to many dynamic objects while maintaining interactive frame rates.

International audience; In games, entertainment, medical and architectural applications , the creation of populated virtual city environments has recently become widespread. In this paper we want to provide a technique that allows the simulation of up to 10,000 pedestrians walking in real-time. Simulation for such environments is difficult as a trade off needs to be found between realism and real-time simulation. This paper presents a pedestrian crowd simulation method aiming at improving the local and global reactions of the pedestrians. The method uses a subdivision of space into a 2D grid for pedestrian-to-pedestrian collision avoidance, while assigning goals to pedestrians to make their trajectories smoother and coherent. Goals are computed automatically and connected into a graph that reflects the structure of the city and triggers a spatial repartition of the density of pedestrians. In order to create realistic reactions when areas become crowded, local directions are stored and updated in real-time, allowing the apparition of pedestrians streams. Combining the different methods contributes to a more realistic model, while keeping a real-time frame rate for up to 10,000 simulated pedestrians.

In this paper, we address the problem of shadow computation for large environments including thousands of dynamic objects. The method we propose is based on the assumption that the environment is 2.5D, which is often the case for virtual cities, thus avoiding complex visibility computation. We apply our method for virtual cities populated by thousands of walking humans, which we render with impostors, allowing real time simulation.In this paper, we treat the cases of shadows cast by buildings on humans, and by humans on the ground. To avoid 3D computation, we represent the shadows cast by buildings onto the environment with a 2.5D shadow map. When humans move, we quickly access the shadow information at the current location with a 2D grid. For each new position of a human, we compute its coverage by the shadow, and we render the shadow on top of the impostor with low cost using multi-texturing hardware. We also use the property of an impostor to display the shadow of humans on the ground plane, by projecting the impostor relatively to the light source.The method is currently limited to sharp shadows and a single light source. However approximations could be made to allow non-accurate soft-shadows. We show in the results that the computation of the shadows, as well as the display is done in real time, and that the method could be easily extended to real time moving light sources.

International audience; We present new method allowing interactive modification of real light sources for augmented reality. Real light shadows are removed from captured textures, and a modified interactive radiosity system is used.

International audience; Lighting design is often tedious due to the required physical manipulation of real light sources and objects. As an alternative, we present an interactive system to virtually modify the lighting and geometry of scenes with both real and synthetic objects, including mixed real/virtual lighting and shadows. In our method, real scene geometry is first approximately reconstructed from photographs. Additional images are taken from a single viewpoint with a real light in different positions to estimate reflectance. A filtering process is used to compensate for inaccuracies, and per image reflectances are averaged to generate an approximate reflectance image for the given viewpoint, removing shadows in the process. This estimate is used to initialise a global illumination hierarchical radiosity system, representing real-world secondary illumination; the system is optimized for interactive updates. Direct illumination from lights is calculated separately using ray-casting and a table for efficient reuse of data where appro-priate. Our system allows interactive modification of light emission and object positions, all with mixed real/virtual illumination effects. Real objects can also be virtually removed using texture-filling algorithms for reflectance estimation.

International audience; Interactive rendering of soft shadows (or penumbra) in scenes with moving objects is a challenging problem. High quality walkthrough rendering of static scenes with penumbra can be achieved using pre-calculated discontinuity meshes, which provide a triangulation well adapted to penumbral boundaries, and backprojections which provide exact illumination computation at vertices very efficiently. However, recomputation of the complete mesh and backprojection structures at each frame is prohibitively expensive in environments with changing geometry. This recomputation would in any case be wasteful: only a limited part of these structures actually needs to be recalculated. We present a novel algorithm which uses spatial coherence of movement as well as the rich visibility information existing in the discontinuity mesh to avoid unnecessary recomputation after object motion. In particular we isolate all modifications required for the update of the discontinuity mesh by using an augmented spatial subdivision structure and we restrict intersections of discontinuity surfaces with the scene. In addition, we develop an algorithm which identifies visibility changes by exploiting information contained in the planar discontinuity mesh of each scene polygon, obviating the need for many expensive searches in 3D space. A full implementation of the algorithm is presented, which allows interactive updates of high-quality soft shadows for scenes of moderate complexity. The algorithm can also be directly applied to global illumination.

International audience; Computer augmented reality (CAR) is a rapidly emerging field which enables users to mix real and virtual worlds. Our goal is to provide interactive tools to perform common illumination, i.e., light interactions between real and virtual objects, including shadows and relighting (real and virtual light source modification). In particular, we concentrate on virtually modifying real light source intensities and inserting virtual lights and objects into a real scene; such changes can be very useful for virtual lighting design and prototyping. To achieve this, we present a three-step method. We first reconstruct a simplified representation of real scene geometry using semi-automatic vision-based techniques. With the simplified geometry, and by adapting recent hierarchical radiosity algorithms, we construct an approximation of real scene light exchanges. We next perform a preprocessing step, based on the radiosity system, to create unoccluded illumination textures. These replace the original scene textures which contained real light effects such as shadows from real lights. This texture is then modulated by a ratio of the radiosity (which can be changed) over a display factor which corresponds to the radiosity for which occlusion has been ignored. Since our goal is to achieve a convincing relighting effect, rather than an accurate solution, we present a heuristic correction process which results in visually plausible renderings. Finally, we perform an interactive process to compute new illumination with modified real and virtual light intensities. Our results show that we are able to virtually relight real scenes interactively, including modifications and additions of virtual light sources and objects.

With the increasing usage of 3D models, the need for efficient shape retrieval techniques became mandatory. In this chapter, several approaches for 3D retrieval and shape recognition are presented. Invariant descriptor issues for 3D static mesh are exposed, before addressing action and face recognition in 3D videos. Then classical 3D-search engine performance evaluation tools are thoroughly explained. Finally, examples of significant applications using 3D retrieval techniques are given.

In this thesis, we present a method to automatically register an environment map with a local scene.In the literature, many methods need to orient the environmental map to be coherent with a local scene. This orientation is mostly done manually by a user. A characteristic of our approach is that we do not need to create a complete 3D model of the local scene nor interact with the user. We propose a pipeline to create a virtual representation of the scene using our input data. This representation includes the global scene represented by the environment map and the local scene represented by a reference object and its shadow. By knowing the position of the main light source on the environment map, we can simulate the lighting and project a computed shadow on the ground. It is possible to compare the computed shadow shape with the input one to recover the correct position of the main light source. The nal orientation of the environment map is directly related to the position of this main light source. We provide an evaluation of the proposed approach by calculating two metrics that compare our angle estimate with actual ground truth directions. Our orientation estimation shows that our method recovers a correct environment map orientation. In this thesis, we are interested in real input data. The environment map and the local scene are extracted from photographs or videos, which already contain a lighting rendering. It is therefore important to orient the environment map in a way that is consistent with the existing lighting in the local scene. We propose an automatic method, to orient an environment map to a local scene. This method is driven by the behavior of light, drawing rays of light towards an object and attempting to match two shadows, one given as input and one calculated. We also use 3D data from the object we are considering. The originality is that we base our method on how light behaves in order to calculate and match shadows. By matching the shadows, we can estimate the correct position of the environment map., Dans cette thèse, nous présentons une méthode pour recaler automatiquement une carte d'environne-ment avec une scène locale. Dans la littérature, de nombreuses méthodes ont besoin d'orienter la carte d'environnement pour obtenir une cohérence avec une scène locale. Cette orientation est dans la majorité des cas e ectuée de manière manuelle par un utilisateur. Une caractéristique de notre approche est que nous n'avons pas besoin de créer un modèle 3D complet de la scène locale, ni d'interagir avec l'utilisateur.Nous proposons un pipeline pour créer une représentation virtuelle de la scène en utilisant nos données d'entrée. Cette représentation comprend la scène globale représentée par la carte d'environnement et la scène locale représentée par un objet de référence et son ombre. En utilisant la connaissance de la position de la source lumineuse principale sur la carte d'environnement, nous pouvons simuler l'éclairage et projeter une ombre sur le sol. Il est alors possible de comparer la forme de l'ombre calculée avec celle de l'ombre d'entrée pour récupérer la position correcte de la source lumineuse principale. L'orientation nale de la carte d'environnement est directement liée à la position de cette source lumineuse principale. Nous fournissons une évaluation de l'approche proposée en calculant deux métriques qui comparent notre estimation d'angle avec les directions réelles de référence. Notre estimation d'orientation montre que notre méthode récupère une orientation de carte d'environnement correcte.