Search

In the last decade, many academic and industrial research groups around the globe are focusing on XR, leading to rapid progress of the field. Next to incremental hardware advances, new software techniques also play an important role in the recent success. The software improvements range from new tracking algorithms, which allow a more accurate and robust localization of the XR headset, to new rendering engines, which are able to render photo realistic environments in real time. In the first half of this thesis, I present my work on camera tracking of low-power mobile devices such as XR headsets. The proposed tracking pipeline uses several algorithmic tricks to reduce the computational complexity. For example, I present a novel decoupled formulation for visual inertial bundle adjustment, which makes the optimization more efficient and can be run in parallel. Furthermore, I show how recursive matrix algebra can be used to speed up the nonlinear optimization problems of a typical tracking pipeline. Overall, the proposed pipeline achieves a similar or better accuracy than the state-of-the-art while being substantially faster. On integrated or low-power computers, my method can process over 60 frames per second, which exceeds the frame rate of commodity cameras by a factor of two. Later in this thesis, I show how the output of my tracking system can be used to generate high-quality RGB and depth images from arbitrary locations in the scene. The proposed method renders triangulated depth images of keyframes to a target view and fuses them in the fragment shader. This approach is very efficient and allows large scene updates of the tracking system since no global volumetric model is built. AR applications can use the resulting images to visualize the scene or display virtual objects with correct occlusion. Finally, I present Approximate Differentiable One-Pixel Point Rendering (ADOP), a novel point-based neural rendering approach for real-time novel view synthesis. The input is an initial reconstruction of a scene using standard photogrammetry software. During a short training stage, neural point descriptors are learned as well as the parameters of a rendering network and a tone-mapper. After that we are able to synthesize photo-realistic views of these scenes at arbitrary camera locations. Due to a novel differentiable point rasterizer, we are also able to optimize the initial camera parameters and point cloud provided by the photogrammetry software. In several experiments, I show that this input optimization can significantly improve the image quality and make ADOP to one of the best performing neural rendering approaches. Im letzten Jahrzehnt haben sich viele akademische und industrielle Forschungsgruppen auf virtuelle und erweiterte Realität (XR) konzentriert, was zu einem raschen Fortschritt in diesem Bereich geführt hat. Neben inkrementellen Hardware-Fortschritten spielen auch neue Softwaretechniken eine wichtige Rolle für den jüngsten Erfolg. Die Software-Verbesserungen reichen von neuen Tracking-Algorithmen, die eine genauere und robustere Lokalisierung des XR-Headsets ermöglichen, bis hin zu neuen Rendering-Engines, die in der Lage sind, fotorealistische Umgebungen in Echtzeit darzustellen. In der ersten Hälfte dieser Arbeit stellen ich meine Forschungsergebnisse zur Kamera Lokalisierung von effizienten mobilen Geräten wie XR-Headsets vor. Die vorgeschlagene Tracking-Pipeline verwendet mehrere algorithmische Tricks, um die Rechenkomplexität zu reduzieren. So stellen ich beispielsweise eine neuartige entkoppelte Formulierung für das inertial-visuelle Bundleadjustment vor, welche die Berechnung effizienter macht und parallel ausgeführt werden kann. Darüber hinaus zeige ich, wie rekursive Matrix Algebra verwendet werden kann, um die nichtlinearen Optimierungsprobleme dieser Pipeline zu beschleunigen. Insgesamt erreicht unsere Methode eine ähnliche oder bessere Genauigkeit als der Stand der Technik und ist dabei wesentlich schneller. Auf integrierten Computern mit geringem Stromverbrauch kann unsere Methode mehr als 60 Bilder pro Sekunde verarbeiten, was die Bildrate handelsüblicher Kameras um den Faktor zwei übersteigt. Im weiteren Verlauf dieser Arbeit zeige ich, wie die Ergebnisse dieser Trackingpipeline verwendet werden können, um qualitativ hochwertige RGB- und Tiefenbilder von beliebigen Orten in der Szene zu erzeugen. Die vorgeschlagene Methode rendert triangulierte Tiefenbilder von Keyframes zu einer Zielansicht und verschmilzt sie im Fragmentshader. Dieser Ansatz ist sehr effizient und ermöglicht umfangreiche Szenenaktualisierungen des Trackingsystems, da kein globales volumetrisches Modell erstellt wird. AR-Anwendungen können die resultierenden Bilder verwenden, um die Szene zu visualisieren oder virtuelle Objekte mit korrekter Verdeckung anzuzeigen. Schließlich stelle ich Approximate Differentiable One-Pixel Point Rendering (ADOP) vor - einen neuartigen punktbasierten neuronalen Rendering-Ansatz. Die Eingabe ist eine initiale Rekonstruktion einer Szene, die zuvor mit Hilfe von Photogrammetrie erstellt wurde. Daraufhin werden während einer kurzen Trainingsphase die neuronalen Punktdeskriptoren, die Parameter eines Rendering-Netzwerks und ein Tone-Mapping-Modell gelernt. Danach sind wir in der Lage, fotorealistische Ansichten dieser Szenen an beliebigen Kamerapositionen zu synthetisieren. Dank eines neuartigen differenzierbaren Punktrasterisieres ist es zusätzlich möglich, die anfänglichen Kameraparameter und die Punktwolke zu optimieren. In mehreren Experimenten zeigen wir, dass diese Eingabeoptimierung die Bildqualität deutlich verbessert und ADOP eines der aktuell besten neuronalen Rendering-Ansätze ist.

Neural rendering with implicit neural networks has recently emerged as an attractive proposition for scene reconstruction, achieving excellent quality albeit at high computational cost. While the most recent generation of such methods has made progress on the rendering (inference) times, very little progress has been made on improving the reconstruction (training) times. In this work we present Neural Adaptive Scene Tracing (NAScenT ), that directly trains a hybrid explicit-implicit neural representation. NAScenT uses a hierarchical octree representation with one neural network per leaf node and combines this representation with a two-stage sampling process that concentrates ray samples where they matter most - near object surfaces. As a result, NAScenT is capable of reconstructing challenging scenes including both large, sparsely populated volumes like UAV captured outdoor environments, as well as small scenes with high geometric complexity. NAScenT outperforms existing neural rendering approaches in terms of both quality and training time., Joint Session, Rui Li, Darius Rückert, Yuanhao Wang, Ramzi Idoughi, and Wolfgang Heidrich

In this paper we present ADOP, a novel point-based, differentiable neural rendering pipeline. Like other neural renderers, our system takes as input calibrated camera images and a proxy geometry of the scene, in our case a point cloud. To generate a novel view, the point cloud is rasterized with learned feature vectors as colors and a deep neural network fills the remaining holes and shades each output pixel. The rasterizer renders points as one-pixel splats, which makes it very fast and allows us to compute gradients with respect to all relevant input parameters efficiently. Furthermore, our pipeline contains a fully differentiable physically-based photometric camera model, including exposure, white balance, and a camera response function. Following the idea of inverse rendering, we use our renderer to refine its input in order to reduce inconsistencies and optimize the quality of its output. In particular, we can optimize structural parameters like the camera pose, lens distortions, point positions and features, and a neural environment map, but also photometric parameters like camera response function, vignetting, and per-image exposure and white balance. Because our pipeline includes photometric parameters, e.g.~exposure and camera response function, our system can smoothly handle input images with varying exposure and white balance, and generates high-dynamic range output. We show that due to the improved input, we can achieve high render quality, also for difficult input, e.g. with imperfect camera calibrations, inaccurate proxy geometry, or varying exposure. As a result, a simpler and thus faster deep neural network is sufficient for reconstruction. In combination with the fast point rasterization, ADOP achieves real-time rendering rates even for models with well over 100M points., Source Code: https://github.com/darglein/ADOP

Training and inference of convolutional neural networks (CNNs) on truncated signed distance fields (TSDFs) is a challenging task. Large parts of the scene are usually empty, which makes dense implementations inefficient in terms of memory consumption and compute throughput. However, due to the truncation distance, non-zero values are grouped around the surface creating small dense blocks inside the large empty space. We show that this structure can be exploited by storing the TSDF in a block sparse tensor and then decomposing it into rectilinear super blocks. A super block is a dense 3d cuboid of variable size and can be processed by conventional CNNs. We analyze the rectilinear decomposition and present a formulation for computing the bandwidth-optimal solution given a specific network architecture. However, this solution is NP-complete, therefore we also a present a heuristic approach for fast training and inference tasks. We verify the effectiveness of SuBloNet and report a speedup of 4x towards dense implementations and 1.7x towards state-of-the-art sparse implementations. Using the super block architecture, we show that recurrent volumetric fusion is now possible on large scale scenes. Such a systems is able to reconstruct high-quality surfaces from few noisy depth images., Vision, Modeling, and Visualization, Capturing and Rendering, 91, 98, Darius Rückert and Marc Stamminger, CCS Concepts: Computing methodologies --> Reconstruction; Mixed / augmented reality; 3D imaging