Search

In this paper, we propose a lightweight asymmetric refining fusion network (LARFNet) for real-time semantic segmentation to solve the problem that some existing models cannot achieve good segmentation accuracy with real-time inference speed in mobile devices due to the huge computational overhead. Specifically, LARFNet adopts an asymmetric encoder–decoder structure. The depth-wise separable asymmetric interaction module (DSAI module) is designed in the encoding process, which effectively extracted local and surrounding information under different receptive fields with optimized convolution in the condition of ensuring communication between channels. In the decoder, we design the bilateral pyramid pooling attention module (BPPA module) and the multi-stage refinement fusion module (MRF Module). The BPPA module is used to integrate the high-level output multi-scale context information. Based on spatial and channel attention mechanisms, the MRF module is proposed to refine the feature maps of different resolutions and guide the feature fusion. Experimental results show that LARFNet achieves 69.2% mIoU and 65.6% mIoU on Cityscapes and Camvid datasets at 127 FPS and 222 FPS respectively, only using a single NVIDIA GeForce GTX2080Ti GPU and 0.72M parameters without any pre-training or pre-processing. Compared with most of the existing state-of-the-art models, the proposed method realizes the efficient use of network parameters at a faster speed, reduces the number of network parameters, and still achieves the accuracy of good segmentation. [Display omitted] • In this paper, we propose a lightweight real-time semantic segmentation network which considers inference speed, number of model parameters and segmentation accuracy as a whole, named lightweight asymmetric refining fusion network (LARFNet). The network mainly consists of three modules. The DSAI module is used to extract different features. The BPPA module is used to provide pixel-level attention for features, and the MRF module is used to guide feature fusion after optimizing features. [ABSTRACT FROM AUTHOR]

Fatigue driving is an important cause of traffic accidents, so it is particularly important to achieve effective fatigue driving detection. The current fatigue driving detection algorithm has the problems of poor comfort, vulnerable to external factors (light intensity, wearing masks and sunglasses), and poor real-time performance. To solve these problems, we design a system for detecting driver's facial features (FFD-System) and an algorithm for judging driver's fatigue state (MF-Algorithm). In FFD-System, we designed three networks M1-FDNet, M2-PENet and M3-SJNet for driver face detection, head pose estimation and eye-mouth state judgment respectively. M1-FDNet is a lightweight face detection network. We design a new loss function to allow the network to output five key face points in addition to the output face. M2-PENet converts the regression problem into a classification problem, and the head pose can be estimated only by a single face image. M3-SJNet can use the global information of eyes and mouth to determine its state. Wearing sunglasses or masks while driving can cause the driver's eyes or mouth to be obscured, which makes it impossible to accurately detect all of the driver's facial features. MF-Algorithm can solve this problem well. MF-Algorithm uses the output of FFD-System to calculate single eye closure time, blinking frequency, eye closure rate, mouth opening frequency and head misalignment time, and combines these parameters to comprehensively judge the driver's fatigue state. Thus, even if a single facial feature cannot be accurately identified, our algorithm can accurately determine the driver's fatigue status based on other features. The experimental results show that the algorithm meets the requirements of real-time detection in terms of detection speed and accuracy. [Display omitted] • Two lightweight neural networks are designed to detect the driver' s facial features. • A fatigue state judgment algorithm based on multiple facial factors of the driver is proposed. • This paper uses multiple modules to interact with each other to reduce the interference of external factors. [ABSTRACT FROM AUTHOR]

• The current 3D human body reconstruction technology requires a large number of 3D scanning equipment and a complex dynamic capture process, which significantly limits its rapid adoption in these application domains. The purpose of this article is to simplify the process of 3D reconstruction. • By combining deep learning methods, an end-to-end 3D human body reconstruction framework is proposed to achieve the goal of reconstructing a high-precision 3D human body model based on RGB images. [Display omitted] In recent years, with the improvement of artificial intelligence technology, it has become possible to reconstruct high-precision 3D human body models based on ordinary RGB images. The current 3D human body reconstruction technology requires complex external equipment to scan all angles of the human body, which is complicated to be implemented and cannot be popularized. In order to solve this problem, this paper applies deep learning models on reconstructing 3D human body based on monocular images. First of all, this paper uses Stacked Hourglass network to perform convolution operations on monocular images collected from different views. Then Multi-Layer Perceptrons (MLPs) are used to decode the encoded high-level images. The feature codes in the two views(main and side) are fused, and the interior and exterior points are classified by the fusion features, so as to obtain the corresponding 3D occupancy field. At last, the Marching Cube algorithm is used for 3D reconstruction with a specific threshold and then we use Laplace smoothing algorithm to remove artifacts. This paper proposes a dense sampling strategy based on the important joint points of the human body, which has a certain optimization effect on the realization of high-precision 3D reconstruction. The performance of the proposed scheme has been validated on the open source datasets, MGN dataset and the THuman dataset, provided by Tsinghua University. The proposed scheme can reconstruct features such as clothing folds, color textures, and facial details,and has great potential to be applied in different applications. [ABSTRACT FROM AUTHOR]

The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene's materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of 7.85 × 10−4 when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes. [Display omitted] • Automatic end-to-end method to estimate the environment lighting in XR applications. • A CNN architecture that learns a latent-space of the environment lighting. • A methodology to generate egocentric mixed-reality-views from HDR panoramas. • Real-time lighting estimation that does not make assumptions about the XR scene. [ABSTRACT FROM AUTHOR]

• We solve the problem of representation and mapping of non-manifold geometry, appropriately loosen the theoretical restrictions by defining the key parts of the graphics in Sections 3.1 and 3.3. • The approximation method is robust to small planes and damaged data because of our method in Section 3.2. • Stepwise approximation reduce 12 parameters from to 3 parameters at most in each approximation in Section 3.2. Pairwise orthogonal planes are important semantic structures in indoor scenes. The method of Manhattan world orientation calculation and constrained planes selection can solve most segmentation problems of point cloud, but it is inefficient when it comes to small areas on beams, columns or other small architectural primitives. In this paper, we propose a novel method to extract pairwise orthogonal planes (abbr. POP) from point cloud building model. We firstly define the orthogonal structure, and then utilize it as a primitive shape to extract the pairwise orthogonal planes using RANSAC. Our method can segment small regions of beams and columns. Moreover, it can also extract non-manifold structures with broken point cloud data. Experimental results show that our proposed method can extract the POP model efficiently and accurately. [ABSTRACT FROM AUTHOR]

Simulating multi-body dynamics with both rigid and flexible parts and with frictional contacts is a hard problem. We solve this by expressing the couplings between the bodies as position level constraints. The implicit treatment of the constraint directions gives us improved stability over velocity based methods. Then by employing regularization of nonlinear constraints and a convex minimization formulation, we bridge constraint-based methods to traditional force-based methods. In fact, the former are just a dual variables formulation of the latter. We solve this dual problem using position based dynamics (PBD). We show how PBD is a completely valid modeling technique and we extend it with an accurate contact and Coulomb friction model. We further show for the first time how the same solver can be used to simulate both rigid and deformable solids with two way coupling. For the soft bodies we introduce a novel form of linear finite elements expressed as constraints, that is more accurate than PBD mass-spring systems. More of our results include the energy conserving Newmark integrator and the accelerated Jacobi solver suitable for parallel architectures. Note that this paper is an extended and revised version of the conference paper published in [1]. [ABSTRACT FROM AUTHOR]

• Hierarchical path finding can highly boost performance once the connecting S and G problem is solved. • Multi-agent simulation can also benefit from speedups larger than a basic parallel A* method. • There is a trade-off in the memory cost of methods that require pre-computation compared to full parallel solutions. • Pre-computed methods offer the highest speedups. One of the main challenges in video games is to compute paths as efficiently as possible for groups of agents. As both the size of the environments and the number of autonomous agents increase, it becomes harder to obtain results in real time under the constraints of memory and computing resources. Hierarchical approaches, such as HNA* (Hierarchical A* for Navigation Meshes) can compute paths more efficiently, although only for certain configurations of the hierarchy. For other configurations, the method suffers from a bottleneck in the step that connects the Start and Goal positions with the hierarchy. This bottleneck can drop performance drastically. In this paper we present two approaches to solve the HNA* bottleneck and thus obtain a performance boost for all hierarchical configurations. The first method relies on further memory storage, and the second one uses parallelism on the GPU. Our comparative evaluation shows that both approaches offer speed-ups as high as 9x faster than A*, and show no limitations based on hierarchical configuration. Finally we show how our CUDA based parallel implementation of HNA* for multi-agent path finding can now compute paths for over 500K agents simultaneously in real-time, with speed-ups above 15x faster than a parallel multi-agent implementation using A*. [ABSTRACT FROM AUTHOR]

Highlights • We propose a specific optimization algorithm that allows to find a global minimum for a desired reflector in a fast way. • A tree data structure is constructed on the fly, where each node is the evaluation of a reflector shape. • The new tree branches are constructed from the stochastically selected nodes following heuristic rules. • Our algorithm is able to refine the reflector shape where it better approaches the global minimum, avoiding local minima. • When we are close to the solution, a classic local search optimization algorithm is used to reach the minimum faster. Graphical abstract Abstract This paper presents a global optimization algorithm specifically tailored for inverse reflector design problems. In such problems, the goal is to obtain a reflector shape that produces a light distribution as close as possible to a user-provided one. The optimization is an iterative process where each step evaluates the difference between the current reflector illumination and the desired one. We propose a tree-based stochastic method that drives the optimization process, using some heuristic rules, to reach a minimum below a user-provided threshold that satisfies the requirements. When we are close to the solution, we resort to the Hooke and Jeeves method, to reach the minimum faster. Extending our previous work Mas et al. (2010), we show that our method reaches a solution in fewer steps than most other classic optimization methods, and also avoids many local minima. The method has been tested on a real case study based on European road lighting safety regulations. [ABSTRACT FROM AUTHOR]

The radiosity problem can be expressed as a linear system, where light interactions between patches of the scene are considered. Its resolution has been one of the main subjects in Computer Graphics, which has led to the development of methods focused on different goals. For instance, in inverse lighting problems, it is convenient to solve the radiosity equation thousands of times for static geometries. Also, this calculation needs to consider many (or infinite) light bounces to achieve accurate global illumination results. Several methods have been developed to solve the linear system by finding approximations or other representations of the radiosity matrix, because the full storage of this matrix is memory demanding. Some examples are hierarchical radiosity, progressive refinement approaches, or wavelet radiosity, which may become slow for many bounces. Recently, new direct methods have been developed based on matrix factorization. This paper introduces a novel and efficient error-bounded factorization method based on the use of multiple singular value decompositions and the Z-order curve to sort the patches of the model. This technique accelerates the factorization of in-core matrices, and allows to work with out-of-core matrices passing only one time over them. Using this method, the inverse of the radiosity matrix can be efficiently approximated, reducing the memory and time resources needed to compute radiosity with infinite bounces. In the experimental analysis, the presented method is applied to scenes up to 163 K patches. After a precomputation stage, it is used to solve the radiosity problem for fixed geometries at interactive times. [ABSTRACT FROM AUTHOR]

Recent surface acquisition technologies based on microsensors produce three-space tangential curve data which can be transformed into a network of space curves with surface normals. This paper addresses the problem of surfacing an arbitrary closed 3D curve network with given surface normals. Thanks to the normal vector input, the patch finding problem can be solved unambiguously and an initial piecewise smooth triangle mesh is computed. The input normals are propagated throughout the mesh. Together with the initial mesh, the propagated normals are used to compute mean curvature vectors. We then compute the final mesh as the solution of a new variational optimization method based on the mean curvature vectors. The intuition behind this original approach is to guide the standard Laplacian-based variational methods by the curvature information extracted from the input normals. The normal input increases shape fidelity and allows to achieve globally smooth and visually pleasing shapes. [ABSTRACT FROM AUTHOR]

In this paper, we present symmetry-aware embedding for shape correspondence, which is robust against symmetric (left–right) flips and rotational (front–back) flips. Unlike previous embedding approaches that embed surfaces into a high-dimensional space, our technique is based on a low dimensional (3D) embedding. Our method can solve left–right flips by finding a 3D rigid transformation between two embedding surfaces without reflections. Using the global reflectional symmetry plane to align two surfaces, we can further reduce the problem to that of finding the rotation that corrects the signs of the front–back and up–down directions (there are four possible solutions). We exploit this simple problem formulation and alleviate the front–back flips, by explicitly comparing the front and back of embedding surfaces based on the global and local extrinsic shape characteristics. Consequently, reasonably accurate point-to-point correspondences can be established simply by performing the nearest neighbor search in our embedding space. Experimental results based on a shape correspondence benchmark showed that our method produces stable matching results. [ABSTRACT FROM AUTHOR]

Re-combining parts of existing 3D object models is an interesting and efficient technique to create novel shape collections. However, due to the lack of direct parts’ correspondence across different shape families, such data-driven modeling approaches in literature are mostly limited to the synthesis of in-class shapes only. To address the problem, this paper proposes a novel approach to create 3D shapes via re-combination of cross-category object parts from an existing database of different model families. In our approach, a reference shape containing multi-functional constituent parts is pre-specified by users, and its design style is then reused to guide the creation process. To this end, the functional substructures are first extracted for the reference shape. After that, we explore a series of category pairs which are potential replacements for the functional substructures of the reference shape to make interesting variations. We demonstrate our ideas using various examples, and present a user study to evaluate the usability and effectiveness of our technique. [ABSTRACT FROM AUTHOR]

Many applications require manipulation and visualization of complex and highly detailed models at realtime. In this paper, we present a new mesh process and rendering method for realtime high quality rendering. The basic idea is to send a simplified mesh to hardware pipeline, while use the online tessellation on the GPU to facilitate the rendering of complex geometric details. We formulate it into an inverse tessellation problem that first computes the simplified mesh, and then optimizes the tessellated mesh with geometric details to approximate the original mesh. To solve this problem, we propose a two-stage algorithm. In the first stage, we employ an iterative surface simplification technique, where we take the requirement of hardware tessellation into consideration to obtain an optimal simplified mesh. In the second stage, to better utilize the hardware tessellation, we propose a moving vertex strategy to approximate the tessellated mesh to the original mesh. Results show that our method achieves 2–4 times faster at rendering but still retains high quality geometrical details. [ABSTRACT FROM AUTHOR]

Video segmentation is an important preprocessing step for many computer vision and graphics tasks. Its main goal is to group the voxels in the video volume with similar appearance and motion into spatio-temporally consistent supervoxels. In this paper, we formulate video segmentation as an L 0 gradient minimization problem, so that the spatio-temporal coherence can be effectively enforced through a gradient sparsity pursuit way. In our method, the appearance and motion descriptor space is first built for over-segmented image patches of each video frame. Then the L 0 gradient minimization is performed in the descriptor space, for both spatial and temporal dimensions. To solve the non-convex L 0 norm minimization problem, we extend the fused coordinate descent algorithm from 2D image grids to 3D video volume. We conduct quantitative evaluation of our method in a public video segmentation benchmark LIBSVX. The experimental results demonstrate our superior performance to state-of-the-arts in segmentation accuracy and undersegmentation error, and comparable performance in boundary recall and explained variation. [ABSTRACT FROM AUTHOR]

Despite the benefits of presence, immersive virtual reality is still too limiting and ineffective to be widely adopted in people׳s everyday lives. One important reason for this is its inability to handle highly diverse 3D interaction tasks, such as object manipulation from different scales, perspectives, reference frames, and dimensions. Inspired by preliminary development of an immersive world builder tool, this paper offers an in-depth presentation of our solution to this problem, using a coordinated, tablet- and HMD-based, hybrid virtual environment (HVE) system. Wearing a non-occlusive HMD, the user is able to view and interact with a tablet mounted on the non-dominant forearm, which provides a multi-touch interaction surface, as well as an exocentric God view of the virtual world. To reduce transition gaps across 3D interaction tasks and interfaces, four coordination mechanisms are proposed, two of which were implemented, and one was evaluated in a user study featuring complex level-editing tasks. Based on subjective ratings, task performance, interview feedback, and video analysis, we found that having multiple seamless Interaction Contexts (ICs) with complementary benefits can lead to good performance and user experience, despite the complexity of learning and using the hybrid system. The results also suggest keeping 3DUI tasks synchronized across the ICs, as this can help users understand their relationships, smoothen within- and between-task IC transitions, and inspire more creative use of different interfaces. Finally, based on these findings, a four-step design process is proposed to aid the process of designing and implementing HVE systems, as well as applying them to appropriate task scenarios. [ABSTRACT FROM AUTHOR]

3-D object modeling from single images has many applications in computer graphics and multimedia. Most previous 3-D modeling methods which directly recover 3-D geometry from single images require user interactions during the whole modeling process. In this paper, we propose a semi-automatic 3-D modeling approach to recover accurate 3-D geometry from a single image of a piecewise planar object with less user interaction. Our approach concentrates on these three aspects: (1) requiring rough sketch input only, (2) accurate modeling for a large class of objects, and (3) automatically recovering the invisible part of an object and providing a complete 3-D model. Experimental results on various objects show that the proposed approach provides a good solution to these three problems. [ABSTRACT FROM AUTHOR]

Using 2D contour sketches as input is an attractive solution for easing the creation of 3D models. This paper tackles the problem of creating 3D models of animals from a single, side-view sketch. We use the a priori assumptions of smoothness and structural symmetry of the animal about the sagittal plane to inform the 3D reconstruction. Our contributions include methods for identifying and inferring the contours of shape parts from the input sketch, a method for identifying the hierarchy of these structural parts including the detection of approximate symmetric pairs, and a hierarchical algorithm for positioning and blending these parts into a consistent 3D implicit-surface-based model. We validate this pipeline by showing that a number of plausible animal shapes can be automatically constructed from a single sketch. [ABSTRACT FROM AUTHOR]

Acquiring natural-looking in-betweens is a fundamental shape interpolation problem in computer graphics. Several previous studies showed that as-isometric-as-possible interpolation is the key to natural and intuitive results. With this presumption, this paper describes a novel method for acquiring an isometric shape interpolation for given key-frame models. The technological and theoretical contributions of our method lie in the introduction of a new coordinate system, isometry-invariant intrinsic coordinates. This simplifies the nonlinear isometric interpolation into a simple linear algebraic problem. The method is shown to yield an effective isometric interpolation of the given key-frame models and is easy to implement. Experimental results confirm that the proposed method produces minimal metric distortion and exhibits reasonable efficiency. [ABSTRACT FROM AUTHOR]

While object deformation has received a lot of attention in Computer Graphics in recent years, with several good surveys that summarize the state-of-the-art in the field, a comparable comprehensive literature review is still needed for the related problem of crack and fracture modeling. In this paper we present such a review, with a special focus on the latest advances in this area, and a careful analysis of the open issues along with the avenues for further research. With this survey, we hope to provide the community not only a fresh view of the topic, but also an incentive to delve into and explore these unsolved problems further. [ABSTRACT FROM AUTHOR]