Search

We present Light4AR, a light source estimation solution based on the detection of real cast shadows in an image captured by a mobile device. Given the camera pose of the image and a local 3D model of the scene, the approach consists in analyzing shadows cast by a reference object onto its supporting plane to determine the 3D position and the intensity of multiple light sources. By creating virtual point lights based on the resulting parameters and adding them to the AR scene, all the virtual objects can be illuminated and cast virtual shadows consistent with the real environment lighting, thereby enhancing user experience and object presence. In addition to offering the ability to share results over multiple users while preserving device resources, the server-based GPU implementation provides results in interactive time and makes photorealistic AR experiences accessible to most mobile devices, not limiting them to recent ones or specific OS. The proposed approach requires small manual input, limited to placing in the scene a reference object of known geometry, then selecting a region including the shadows cast by this object. We show the potential of this approach on several challenging scenes with various lighting configurations and background textures.

Precise knowledge of the real environment is a prerequisite for the integration of the real and virtual worlds in mixed-reality applications. However, real-time updating of a real environment model is a costly and difficult process, therefore, hybrid approaches have been developed: An updated world model can be inferred from an offline acquisition of the 3D world, which is then updated online using live image sequences under the condition of developing fast and robust change detection algorithms. Current algorithms are biased toward object insertion and often fail in object removal detection, in an environment where there is uniformity in the background—in color and intensity—the disappearances of foreground objects between the 3D scan of a scene and the capture of several new pictures of said scene are difficult to detect. The novelty of our approach is that we circumvent this issue by focusing on areas of least change in parts of the scene that should be occluded by the foreground. Through experimentation on realistic datasets, we show that this approach results in better detection and localization of removed objects. This technique can be paired with an insertion detection algorithm to provide a complete change detection framework.

We introduce a system to assign navigation tasks to a self-moving robot using an Augmented Reality (AR) application running on a smartphone. The system relies on a robot controller and a central server hosted on a PC. The user points at a target location in the phone camera view and the robot moves accordingly. The robot and the phone are independently located in the 3D space thanks to registration methods running on the server, hence they do not need to be spatially registered to each other nor in direct line of sight.

International audience; In this work, we consider the challenge of achieving a coherent blending between real and virtual worlds in the context of a Mixed Reality (MR) scenario. Specifically, we have designed and implemented an interactive demonstrator that shows a realistic MR application without using any light probe. The proposed system takes as input the RGB stream of the real scene, and uses these data to recover both the position and intensity of light sources. The lighting can be static and/or dynamic and the geometry of the scene can be partially altered. Our system is robust in presence of specular effects and handles both uniform and/or textured surfaces.

Mesh texturing is an essential part of 3D scene reconstruction that enables a more realistic perception than the geometry alone and even compensates for inaccurate geometry. In this work we present a flexible formulation for color correction of textured scenes based on color augmentation per face. It can be employed as a post-processing step after selecting the best keyframe per face to compensate for color differences between pairs of neighboring faces. We present a Markov Random Field (MRF) formulation to find the best keyframes as well as the optimal color augmentations. We use a simple model to avoid reflection and camera vignetting during the view selection. Our model for color correction finds the piecewise-linear augmentation to be added to the texture patches of faces. It encourages smoothness inside every fragment while compensating color differences along view transitions. Moreover, we speed up the optimization by breaking down the formulation into multiple binary MRFs that estimate the best augmentations from coarse to fine resolutions. The experimental results prove our method outperforming the state of the art methods.

International audience; This chapter offers a view of future technical and scientific prospects, related to major evolutions in use. It explains the impact of technological advances on applications in the entertainment field and, more generally, on the use of VR‐AR by the general public. The chapter then discusses the potential of brain‐computer interactions (BCI). It explains the working principle of BCIs and analyzes the possibilities opened up by alternative perception mechanisms for interactions in virtual reality (VR). The chapter looks at how user perceptions can be altered in VR using pseudo‐sensory feedback. Finally, it shows that it is possible to generate illusions of movement for an immobile user by altering their movement in VR, and thus overcoming the current limitations of VR technology related to movement in the VE.

A multi-user experience extending a standard TV content with AR elements is presented. It runs with both a standard tablet and a premium MR headset, the Microsoft HoloLens. A virtual TV mosaic is displayed around the TV screen and used as a GUI to control both TV and MR content. This paper focuses on the collaborative and personalized dimension offered by the experience. Unlike most AR applications, it can be simultaneously run by several users using different devices. The users can share content with others while keeping a personalized display. The added-value of such an extended TV experience has been demonstrated through complementary types of content, and user feedback confirms a real interest in this new kind of home entertainment, at the same time immersive, interactive, collaborative and personalized.

Technicolor has been investigating how Mixed Reality technology could impact the future of home entertainment. We have designed and implemented a system to extend a standard TV experience with AR content, using a consumer tablet or a headset. A virtual TV mosaic is displayed around the TV screen and used as a GUI to control both TV and MR content. Using this interface, the user is able to switch TV content, display meta-data in AR (subtitles, text information or program guide), enhance TV content with interactive 3D objects blended in the environment, or play a game in interaction with the real world. The interactions between the real and the virtual worlds are handled thanks to a scene analysis pre-processing stage, which provides information about both the geometry and the lighting of the real environment. The real-virtual interactions strongly contribute to reinforcement of the immersion feeling. User feedback shows that the concept is very promising.

The Extended TV application allows to enhance an audiovisual content displayed on a TV, using Mixed Reality technology. During preprocessing, the close environment of the TV is scanned using a consumer depth camera. The captured RGB-D data are analyzed, providing models for both the 3D geometry and the lighting of the real scene. During runtime, the TV is watched through a tablet, and virtual objects can apparently come out of the screen and start populating the user's environment. Virtual objects can be occluded by real objects, and virtual shadows are consistent with the real ones.

In this paper an approach is proposed for creating realistic models of existing roads adapted to driving simulation. Unlike most previous work, based on generic construction rules, urbanism patterns or sociological behaviour, our approach aims at reproducing existing road networks. First a data model based on a multi-layered graph is presented. This model can handle the three representation levels required by traffic simulation: the road network, the graphical level and the traffic level. In the second part of the paper, a method for modelling existing roads is proposed. The novelty of the approach is the use of existing road data bases to automatically create a virtual environment (3D model and traffic organisation) close to ground truth. An existing 3D GIS data base provides accurate information about road axes, ground and building 3D geometry, whereas a navigation road database helps refining the model by providing clues on network topology and traffic rules. The resulting virtual road system reproduces the real world and has been successfully interfaced with a driving simulator.

A new method is described for automatically reconstructing 3D planar faces from multiple images of a scene. The novelty of the approach lies in the use of inter-image homographies to validate and best estimate the plane, and in the minimal initialization requirements-only a single 3D line with a textured neighbourhood is required to generate a plane hypothesis. The planar facets enable line grouping and also the construction of parts of the wireframe which were missed due to the inevitable shortcomings of feature detection and matching. The method allows a piecewise planar model of a scene to be built completely automatically, with no user intervention at any stage, given only the images and camera projection matrices as input. The robustness and reliability of the method are illustrated on several examples, from both aerial and interior views.

We describe a method to completely automatically recover 3D scene structure together with a camera for each frame from a sequence of images acquired by an unknown camera undergoing unknown movement. Previous approaches have used calibration objects or landmarks to recover this information, and are therefore often limited to a particular scale. The approach of this paper is far more general, since the “landmarks” are derived directly from the imaged scene texture. The method can be applied to a large class of scenes and motions, and is demonstrated here for sequences of interior and exterior scenes using both controlled-motion and hand-held cameras.

In this paper, we address the problem of handling intricate urban scenes with stereo pairs of images, for the reconstruction of the 3D objects, when one has no idea of the types of 3D objects and their arrangement to be found. Firstly we argue for a two stage focusing analysis: first, an attention focusing stage based only on image criteria provides areas of interest, then a model-driven characterization of these areas leads to the model-based reconstruction of the objects.

A new approach to the detection of above-ground from a pair of stereoscopic images in a general urban context is proposed. It includes a stereoscopic matching stage well- adapted to our task in order to provide a digital surface model (DSM). Then a segmentation of the DSM is performed, and regions are classified as ground or above-ground. The interest of the method is its ability to manage extended above-ground with several heights and any shape, as well as the case of a sloping ground. An application to digital terrain models (DTM) generation in urban areas is discussed. Assessment of both above-ground extraction and DTM generation on difficult scenes shows the feasibility of the approach.