Search

One of the main goals of many augmented reality applications is to provide a seamless integration of a real scene with additional virtual data. To fully achieve that goal, such applications must typically provide high-quality real-world tracking, support real-time performance and handle the mutual occlusion problem, estimating the position of the virtual data into the real scene and rendering the virtual content accordingly. In this survey, we focus on the occlusion handling problem in augmented reality applications and provide a detailed review of 161 papers published in this field between January 1992 and August 2020. To do so, we present a historical overview of the most common strategies employed to determine the depth order between real and virtual objects, to visualize hidden objects in a real scene, and to build occlusion-capable visual displays. Moreover, we look at the state-of-the-art techniques, highlight the recent research trends, discuss the current open problems of occlusion handling in augmented reality, and suggest future directions for research.

In 2015, Guth proved that for any set of $k$-dimensional bounded complexity varieties in $\mathbb{R}^d$ and for any positive integer $D$, there exists a polynomial of degree at most $D$ whose zero set divides $\mathbb{R}^d$ into open connected sets, so that only a small fraction of the given varieties intersect each of these sets. Guth's result generalized an earlier result of Guth and Katz for points. Guth's proof relies on a variant of the Borsuk-Ulam theorem, and for $k>0$, it is unknown how to obtain an explicit representation of such a partitioning polynomial and how to construct it efficiently. In particular, it is unknown how to effectively construct such a polynomial for bounded-degree algebraic curves (or even lines) in $\mathbb{R}^3$. We present an efficient algorithmic construction for this setting. Given a set of $n$ input algebraic curves and a positive integer $D$, we efficiently construct a decomposition of space into $O(D^3\log^3{D})$ open "cells," each of which meets $O(n/D^2)$ curves from the input. The construction time is $O(n^2)$. For the case of lines in $3$-space we present an improved implementation, whose running time is $O(n^{4/3} \log^{O(1)} n)$. The constant of proportionality in both time bounds depends on $D$ and the maximum degree of the polynomials defining the input curves. As an application, we revisit the problem of eliminating depth cycles among non-vertical lines in $3$-space, recently studied by Aronov and Sharir (2018), and show an algorithm that cuts $n$ such lines into $O(n^{3/2+\epsilon})$ pieces that are depth-cycle free, for any $\epsilon > 0$. The algorithm runs in $O(n^{3/2+\epsilon})$ time, which is a considerable improvement over the previously known algorithms., Comment: 20 pages, 0 figures. v2: final version, to appear in SIAM J. Comput. A preliminary version of this work was presented in Proc. 30th Annual ACM-SIAM Sympos. Discrete Algorithms, 2019

International audience; This paper studies the effect of the order of depth of mention on nested named entity recognition (NER) models. NER is an essential task in the extraction of biomedical information, and nested entities are common since medical concepts can assemble to form larger entities. Conventional NER systems only predict disjointed entities. Thus, iterative models for nested NER use multiple predictions to enumerate all entities, imposing a predefined order from largest to smallest or smallest to largest. We design an order-agnostic iterative model and a procedure to choose a custom order during training and prediction. To accommodate for this task, we propose a modification of the Transformer architecture to take into account the entities predicted in the previous steps. We provide a set of experiments to study the model's capabilities and the effects of the order on performance. Finally, we show that the smallest to largest order gives the best results.

Deep images store multiple fragments perpixel, each of which includes colour and depth, unlike traditional 2D flat images which store only a single colour value and possibly a depth value. Recently, deep images have found use in an increasing number of applications, including ones using transparency and compositing. A step in compositing deep images requires merging per-pixel fragment lists in depth order; little work has so far been presented on fast approaches. This paper explores GPU based merging of deep images using different memory layouts for fragment lists: linked lists, linearised arrays, and interleaved arrays. We also report performance improvements using techniques which leverage GPU memory hierarchy by processing blocks of fragment data using fast registers, following similar techniques used to improve performance of transparency rendering. We report results for compositing from two deep images or saving the resulting deep image before compositing, as well as for an iterated pairwise merge of multiple deep images. Our results show a 2 to 6 fold improvement by combining efficient memory layout with fast register based merging.

In three-dimensional (3-D) cluttered scenes such as foliage, deeper surfaces often are more shadowed and hence darker, and so depth and luminance often have negative covariance. We examined whether the sign of depth-luminance covariance plays a role in depth perception in 3-D clutter. We compared scenes rendered with negative and positive depth-luminance covariance where positive covariance means that deeper surfaces are brighter and negative covariance means deeper surfaces are darker. For each scene, the sign of the depth-luminance covariance was given by occlusion cues. We tested whether subjects could use this sign information to judge the depth order of two target surfaces embedded in 3-D clutter. The clutter consisted of distractor surfaces that were randomly distributed in a 3-D volume. We tested three independent variables: the sign of the depth-luminance covariance, the colors of the targets and distractors, and the background luminance. An analysis of variance showed two main effects: Subjects performed better when the deeper surfaces were darker and when the color of the target surfaces was the same as the color of the distractors. There was also a strong interaction: Subjects performed better under a negative depth-luminance covariance condition when targets and distractors had different colors than when they had the same color. Our results are consistent with a "dark means deep" rule, but the use of this rule depends on the similarity between the color of the targets and color of the 3-D clutter.

Photos compress 3D visual data to 2D. However, it is still possible to infer depth information even without sophisticated object learning. We propose a solution based on small-scale defocus blur inherent in optical lens and tackle the estimation problem by proposing a non-parametric matching scheme for natural images. It incorporates a matching prior with our newly constructed edgelet dataset using a non-local scheme, and includes semantic depth order cues for physically based inference. Several applications are enabled on natural images, including geometry based rendering and editing.

2D Visual saliency has been widely explored for decades. Several comprehensive and well performing models have been proposed, but they are not totally adapted to stereoscopic 3D content. To date only few tentatives of 3D saliency prediction can be found in the literature and most of them rely on binocular depth/disparity. The latter information cannot be correctly obtained in the case of asymmetric processing of the stereo-pair, exploiting the phenomenon of binocular suppression. Based on this aspect, we propose in this paper a new saliency model for stereoscopic 3D images. The proposed model considers two features: (1) spatial feature based on the characteristics of interest points and (2) depth feature based on monocular cues. The latter feature is adapted to asymmetric content and uses occlusions for predicting depth order of the image objects. A tunable fusion strategy is proposed in order to take advantage of different modalities of combining conspicuity maps. For the needs of performance evaluation, an eye-tracking database is created using stereo-pairs with different content. The proposed model gives very good performance in comparison to the literature. The results show that the use of monocular cues outperforms the use of disparity. HighlightsA saliency model for stereoscopic 3D images (asymmetric or symmetric) is proposed.Interest point is exploited for the construction of the spatial conspicuity map.Monocular cues (occlusion) are used for the construction of depth conspicuity map.Different fusion strategies are applied to combine spatial and depth features.An eye-tracking experiment is conducted for the validation of the proposed model.

This study introduces a novel depth estimation method that can automatically generate plausible depth map from a single image with unstructured environment. Our goal is to extrapolate depth map with more correct, rich, and distinct depth order, which is both quantitatively accurate as well as visually pleasing. Based on the preexisting DepthTransfer algorithm, our approach primarily transfers depth information at the level of superpixels from the most photometrically similar retrieval images under the framework of non-parametric learning. Posteriorly, we propose to concurrently warp the corresponding superpixels in multi-scale levels, where we employ an improved SLIC technique to segment the RGBD images from coarse to fine. Then, modified Cross Bilateral Filter is leveraged to refine the final depth field. With respect to training and evaluation, we perform our experiment on the popular Make3D dataset and demonstrate that our method outperforms the state-of-the-art in both efficacy and computational efficiency. Especially, the final results show that in qualitatively evaluation, our results are visually superior in realism and simultaneously more immersive.

This study proposes a system to estimate the depth order of regions belonging to a monocular image sequence. For each frame, the regions are ordered according to their relative depth using information from the previous and following frames. The algorithm estimates occlusions relying on a hierarchical region-based representation of the image by means of a binary tree. This representation is used to define the final depth order partition which is obtained through an energy minimisation process. Finally, to achieve a global and consistent depth ordering, a depth order graph is constructed and used to eliminate contradictory local cues. The system is evaluated and compared with the state-of-the-art figure/ground labelling systems showing very good results.

In the present study, the effects of depth order on forward and backward vection were examined using optical flows simulating motion in depth (i.e., approaching or receding). In an experiment, space extending 10 or 20 m in depth was simulated, and the space was divided into foreground and background spaces. In each space, a random-dot pattern was presented and the binocular disparity, size, and velocity of each dot were continuously manipulated in a way consistent with the depth being simulated. Participants reported whether they perceived vection. Latency, total duration (i.e., the amount of time that participants reported perceiving vection during a 60-s presentation), and strong-vection duration (i.e., the amount of time that participants reported perceiving strong vection) were measured. The results indicated that, even though the dots making up the optical flow were much smaller and slower moving in the background space than in the foreground space, vection was strongly dependent on flow motion in the background space. This supports the idea that the perceptual system uses background stimulus motion as a reliable cue for self-motion perception.

The overall objectives of the TeleFOT project are to evaluate the impact of different functions provided by nomadic and after market devices (ND) on four impact areas: efficiency, environment, mobility and safety. The fifth area to be investigated is user uptake. The paper presents preliminary results from one of 13 field operational tests (FOTs), the Swedish L-FOT2 focusing on user uptake. One device and three functions were tested: a Green Driving Support System (GD), a Navigation Support System (NAV) and a Traffic Information Service (TI). Preliminary results indicate the importance of addressing the relative rather than the absolute benefits of a function, i.e. the benefits relative other functions offered by other channels/devices and, further, that the specific characteristics and contents of a function need to be analysed in depth order to understand user uptake.

The hollow-face illusion involves a misperception of depth order: our perception follows our top–down knowledge that faces are convex, even though bottom–up depth information reflects the actual concave surface structure. While pictorial cues can be ambiguous, stereopsis should unambiguously indicate the actual depth order. We used computer-generated stereo images to investigate how, if at all, the sign and magnitude of binocular disparities affect the perceived depth of the illusory convex face. In experiment 1 participants adjusted the disparity of a convex comparison face until it matched a reference face. The reference face was either convex or hollow and had binocular disparities consistent with an average face or had disparities exaggerated, consistent with a face stretched in depth. We observed that apparent depth increased with disparity magnitude, even when the hollow faces were seen as convex (ie when perceived depth order was inconsistent with disparity sign). As expected, concave faces appeared flatter than convex faces, suggesting that disparity sign also affects perceived depth. In experiment 2, participants were presented with pairs of real and illusory convex faces. In each case, their task was to judge which of the two stimuli appeared to have the greater depth. Hollow faces with exaggerated disparities were again perceived as deeper.

We give an output-sensitive algorithm for computing the visibility map of a set of n constant-complexity convex fat polyhedra or curved objects in 3-space. Our algorithm runs in O((n+k) polylog n) time, where k is the combinatorial complexity of the visibility map. This is the first algorithm for computing the visibility map of fat objects that does not require a depth order on the objects and is faster than the best known algorithm for general objects. It is also the first output-sensitive algorithm for curved objects that does not require a depth order.

Under the condition of colour transparency, the colour of a region beyond a coloured transparent filter looks closer to its original colour when compared with the same colour presented without any impression of transparency. It is thought that this colour perception is caused by colour constancy based on the recognition of colour transparency. However, colour induction may also be involved in this phenomenon. In order to clarify how this colour constancy works, the property of colour induction was tested under the condition of colour transparency. By changing the depth order of the stimuli and radius of colour-inducing stimulus, perceived colour changed due to the effect of change in colour transparency and induction. Our results suggest that colour induction is affected by a higher cognitive function that recognizes the order of coloured surfaces, which are represented separately in the visual system.

The rotation direction and depth order of a rotating sphere consisting of random dots often reverses while it is viewed under orthographic projection. However, if a short viewing distance is simulated under perspective projection, the correct rotation direction can be perceived. There are two motion cues for the rotation direction and depth order. One is the speed cue; points with higher velocities are closer to the observer. The other is the vertical motion cue; vertical motion is induced when the dots recede from or approach the observer. It was examined whether circular motion, which does not have any depth information but induces vertical velocities, masks the vertical motion cue. In Experiment 1, the effects of circular motion on the judgment of the rotation direction of a rotating sphere were examined. The magnitude of the two cues (the speed cue and the vertical velocity cue) as well as the angular speed of circular motion was varied. It was found that the performance improved as the vertical velocity increased and that the speed cue had slight effects on the judgment of the rotation direction. It was also found that the performance worsened as the angular speed of the circular motion was increased. In Experiment 2, the effects of circular motion on depth judgment of a rotating half sphere were investigated. The performance worsened as the angular speed of the circular motion increased, as in Experiment 1. These results suggest that the visual system cannot compensate perfectly for circular motion for the judgment of the rotation direction and depth order.

The sign of an accommodative response is provided by differences in chromatic aberration between under- and over-accommodated images. We asked whether these differences enable people to judge the depth order of two stimuli in the absence of other depth cues. Two vertical edges separated by an illuminated gap were presented at random relative distances. Exposure was brief, or prolonged with fixed or changing accommodation. The gap was illuminated with tungsten light or monochromatic light. Subjects could detect image blur with brief exposure for both types of light. But they could detect depth order only in tungsten light with long exposure, with or without changes in accommodation.

Three neighbouring opaque surfaces may appear split into two layers, one transparent and one opaque beneath, if an outline contour is drawn that encompasses two of them. The phenomenon was originally observed by Kanizsa [1955 Rivista di Psicologia69 3–19; 1979 Organization in Vision: Essays on Gestalt Psychology (New York: Praeger)], for the case where an outline contour is drawn to encompass one of the two parts of a bicoloured figure and a portion of a background of lightest (or darkest) luminance. Preliminary observations revealed that the outline contour yields different effects: in addition to the stratification into layers described by Kanizsa, a second split, opposite in depth order, may occur when the outline contour is close in luminance to one of the three surfaces. An initial experiment was designed to investigate what conditions give rise to the two phenomenal transparencies: this led to the conclusion that an outline contour superimposed on an opaque surface causes this surface to emerge as a transparent layer when the luminances of the contour and the surface differ, in absolute value, by no more than 13.2 cd m−2. We have named this phenomenon ‘transparency of the intercepted surface’, to distinguish it from the phenomenal transparency arising when the contour and surface are very different in luminance. When such a difference exists, the contour acts as a factor of surface definition and grouping: the portion of the homogeneous surface it bounds emerges as a fourth surface and groups with a nearby surface if there is one close in luminance. The transparency phenomena (‘transparency of the contoured surface’) perceived in this context conform to the constraints of Metelli's model, as demonstrated by a second experiment, designed to gather ‘opacity’ ratings of stimuli. The observer judgments conformed to the values predicted by Metelli's formula for perceived degree of transparency, a. The role of the outline contour in conveying figural and intensity information is discussed.

Two experiments examined the effects of relative movements of the head with respect to a stimulus display on the magnitude and direction of the perceived depth from motion parallax. In Experiment 1, a velocity gradient of random-dots moving in opposite directions was presented in three viewing conditions: (a) stationary head-and-display, (b) display-movement, and (c) head-movement. The observer's task was to report the direction and magnitude of the perceived depth. The results with 10 observers showed that the magnitude of the perceived depth in condition (b) was lower than those in conditions (a) and (c), while the direction was unambiguous in (b) and (c), but ambiguous in (a). In Experiment 2, the direction of the perceived depth was examined in six conditions including the three of Experiment 1; in the other three conditions both head and display were moving in the same direction but with different ratios of the display velocity to the head one: (d) 0.5, (e) 1.0, and (f) 2.0. The results with 11 observers showed that unambiguous depth perception was obtained in conditions (b), (c), and (f) but not in conditions (a), (d), and (e).Furthermore, the depth order perceptions in (b) and (a) were identical to those in (f), and (e), respectively. The experimental results suggest that relative movement of the head with respect to the display irrespective of either movement is processed to determine the perceived depth order and that the velocity gradient within the display is processed to determine the magnitude of the perceived depth.

Objective : Vection, a feeling of self-motion while being physically stationary, and postural sway can be modulated by various visual factors. Moreover, vection and postural sway are often found to be closely related when modulated by such visual factors, suggesting a common neural mechanism. One well-known visual factor is the depth order of the stimulus. The density, i.e. number of objects per unit area, is proposed to interact with the depth order in the modulation of vection and postural sway, which has only been studied to a limited degree. Methods : We therefore exposed 17 participants to 18 different stimuli containing a stationary pattern and a pattern rotating around the naso-occipital axis. The density of both patterns was varied between 10 and 90%; the densities combined always added up to 100%. The rotating pattern occluded or was occluded by the stationary pattern, suggesting foreground or background motion, respectively. During pattern rotation participants reported vection by pressing a button, and postural sway was recorded using a force plate. Results : Participants always reported more vection and swayed significantly more when rotation was perceived in the background and when the rotating pattern increased in density. As hypothesized, we found that the perceived depth order interacted with pattern density. A pattern rotating in the background with a density between 60 and 80% caused significantly more vection and postural sway than when it was perceived to rotate in the foreground. Conclusions : The findings suggest that the ratio between fore- and background pattern densities is an important factor in the interaction with the depth order, and it is not the density of rotating pattern per se. Moreover, the observation that vection and postural sway were modulated in a similar way points towards a common neural origin regulating both variables.