Search

Neural Radiance Fields (NeRFs) are a novel approach that is being intensively investigated in 3D scene reconstruction and similar fields to overcome challenges of conventional methods. In this paper, we address the problem of estimating missing camera poses in a six degrees of freedom setting, pushing the capabilities of NeRFs to address scenarios where only the primary camera’s pose is known. Specifically, we focus on dual-camera setups with this constraint. Our core contribution is a novel pose correction model that operates alongside an unmodified NeRF model, for which we have chosen Nerfacto in the Nerfstudio framework. The pose correction model learns the necessary relative translation and rotation adjustments for the secondary camera solely through the NeRF loss. This allows us to integrate our correction model directly into the Nerfacto training pipeline without altering the core functionality. Through extensive experiments on different camera configurations in a synthetic scene, we rigorously evaluate our model’s performance across diverse scenarios, pushing it to its limits. Our findings reveal that our model can effectively learn pose correction parameters within a constrained range, with increased sensitivity to larger translations and particular challenges in rotation corrections. This research highlights the potential of NeRFs for machine learning-driven 3D reconstruction on dual- and multi-camera platforms, expanding the applicability of NeRFs to more complex, real-world setups despite the inherent challenges.

Benefiting from advancements in algorithms and computing capabilities, supervised deep learning models offer significant advantages in accurately mapping individual tree canopy cover, which is a fundamental component of forestry management. In contrast to traditional field measurement methods, deep learning models leveraging remote sensing data circumvent access limitations and are more cost-effective. However, the efficiency of models depends on the accuracy of the tree crown annotations, which are often obtained through manual labeling. The intricate features of the tree crown, characterized by irregular contours, overlapping foliage, and frequent shadowing, pose a challenge for annotators. Therefore, this study explores a novel approach that integrates the annotations of multiple annotators for the same region of interest. It further refines the labels by leveraging information extracted from multi-spectral aerial images. This approach aims to reduce annotation inaccuracies caused by personal preference and bias and obtain a more balanced integrated annotation.

3D point clouds are a relevant source of information for multiple applications, including digital twins, building modeling, disaster and risk management, forestry, autonomous driving, and many others. Assigning points to the semantic classes is one of the essential data interpretation steps to effectively use them for further analysis. Deep learning models for semantic segmentation, such as RandLA-Net, are state-of-the-art methods for this task. Although the overall accuracy of classification is usually satisfactory,there are still several shortcomings not allowing assigning correct labels across all the classes. For instance, the receptive field of these networks is often too small to correctly classify point clouds in all cases. These networks suffer also from class imbalance, typical in real-world data sets, and tend to oversmooth small classes. Post-processing approaches help to overcome these problems and achieve better classification accuracy. In this work, we investigate the feasibility of improving the deep-learning outputs by introducing prior knowledge. To do this, the output probabilities of point classes obtained using RandLA-Net are post-processed with a workflow based on Markov Random Fields, in which the unary potentials are adjusted to preserve smaller classes while the pairwise potentials take into account. a hand-tailored inter-class reliability matrix. To validate our method, we apply it to the Hessigheim benchmark. Our MRF-based approach further optimizes these prediction results, effectively and efficiently improving the overall accuracy by approximately 1 to 2 percentage points.

A 3D model can be useful for inventory management and monitoring of forests. For this task, we present our prototype mobile mapping backpack system for collecting 3D point clouds of forest vegetation. However, data collection in forests is challenging due to unreliable GNSS positioning, moving objects caused by winds, unclear object edges, and uneven ground to walk on. We include LiDAR and IMU with Simultaneous Localisation and Mapping (SLAM). We describe in detail our backpack system, its development for the forest environment, and evaluate it. Furthermore, we compare two open code SLAM algorithms for ROS, as well as data collection and laser scan quality between TLS and MLS for forest environments. Finally, we installed and tested the MicaSense multi-spectral camera on our backpack and discuss the advantages and drawbacks of it. We conclude, that the backpack is convenient to use in forest environments and produces a good point cloud. It can be carried easily off trail and on rough terrain. The system needs less storage space, computation and less collection time than TLS.

The aim of this study is to propose evaluation methodology for the quality assessment of depth cameras for indoor mapping applications. Specifically, we evaluate the RGBD sensor Intel Realsense D455 w.r.t. measurement accuracy and noise while investigating their dependence on two central parameters characterizing the measurement scenario simultaneously: measurement distance and inclination of observed surfaces. The evaluation results are presented two-dimensionally as a function of both parameters. To this aim, two evaluation studies are conducted. First, a checkerboard pattern is used as reference object in a controlled setting. Secondly, actual indoor mapping data resulting from the deployment of the evaluated sensor in the context of an RGBD-SLAM application is used to conduct a similar evaluation in representative conditions for the intended usage scenario of indoor mapping. In this case, a TLS scan of the respective indoor environment is used as reference for evaluation.

Open science is an important attribute for developing new approaches. Especially, the data component plays a significant role. The FAIR principle provides a good orientation towards open data. One part of FAIR is findability. Thus, domain specific dataset search platforms were developed: the Earth Observation Database and our Benchmark Metadata Database (BeMeDa). In addition to the search itself, the datasets found by this platforms can be compared with each other with regard to their interoperability. We compare these two platforms and present an update of our platform BeMeDa. This update includes additional location information about the datasets and a new frontend design with improved usability. We rely on user feedback for further improvements and enhancements.

The FAIR principle (find, access, interoperability, reuse) forms a sustainable resource for scientific exchange between researchers. Currently, the implementation of this principle is an important process for future research projects. To support this process in the ISPRS community, the usage of data repositories for dataset publication has the potential to bring closer the achievement of the FAIR principle. Therefore, we (1) analysed available data repositories, (2) identified common keywords in ISPRS publications and (3) developed a tool for searching appropriate repositories. Thus, infrastructures from the field of geosciences, that can already be used, become more accessible.

Plants signal their health in a broader spectrum than we can see with our eyes. We compared sunlight reflectance on plants at spectral wavelengths ranging from 430 nm to 870 nm in our study. These are based on multispectral images captured at a distance of 2 m. Indoor plants were observed over a period of 18 days and stressed due to a lack of sunlight or water. Wild sedge photographed on the forest floor at close range and with a difficult capture setup produced results comparable to published multispectral signatures derived from aerial imagery. Changes of leaf reflectance were noticed in spectral signatures and in vegetation indices. When calculating vegetation indices, our results show that comparing red and red edge reflectance values is superior to comparing red and NIR reflectance values.

Scan2BIM approaches, i.e. the automated reconstruction of building models from point cloud data, is typically evaluated against the same point clouds which are used as input for the reconstruction process. In doing so, the point clouds are often used as ground truth without considering their own inaccuracies. Thus, in this research, we investigate the manual creation of an accurate ground truth, with a process which takes into account the measurement accuracy as well as the modeling accuracy. Therefore we created a ground truth to an existing laser scan data with a total station, based on the assumption that a total station generally measures points more reliably. In addition, a manual selection and classification of points on the wall surfaces during the measurement, serves a reliable detection of the walls via plane fitting. This allows for the creation of a more reliable ground truth, which is determined by the intersection of the planes from corners and edges. The ground truth is aligned parallel to the axes of a local coordinate system. From MLS and TLS point clouds of the same building area, walls are manually classified and corners and edges are determined in a similar way to the total station. These TLS and MLS corners are registered to this ground truth using least squares optimisation at the vertices. The transformation thus determined is used to transform the laser scanning point clouds as well. The resulting errors in the corners and the whole point cloud are evaluated. We conclude that the standard deviation of wall surfaces alone isn’t sufficient to determine the quality of the reconstructed building model. Despite low measurement noise in single wall surfaces, deviations in the reconstructed room model may arise.

3D indoor mapping is becoming increasingly critical for a variety of applications such as path planning and navigation for robots. In recent years, there is a growing interest in how low-cost sensors, such as monocular or depth cameras, can be used for 3D mapping. In our paper, we present an octree-based approach for real-time 3D indoor mapping using a handheld RGB depth camera. One benefit of the generated octree map is that it requires less storage and computational resources than point cloud models. Moreover, it explicitly represents free space and unmapped areas, which are essential for the robot's navigation tasks. In this work, on the basis of the ORB-SLAM3 system (Campos et al., 2021), we developed an octree mapping system, which directly calls the keyframes and estimated poses provided by ORB-SLAM3 algorithms. Furthermore, we used point cloud library (PCL) for the dense point cloud mapping and then OctoMap for the point cloud to octree map conversion. Finally, we implemented an efficient probabilistic 3D mapping in the robot operating system (ROS) environment. We used the TUM RGB-D dataset to evaluate the estimated trajectories of the camera. The evaluation shows an average translational RMSE of 5.9 cm on the TUM RGB-D dataset. Besides, we also compared the ground truth point clouds and our generated point clouds. The result shows the mean cloud-to-cloud distance in the corridor scene is about 6 cm. All the evaluation results show our proposed approach is a promising solution for advanced indoor voxel mapping and robotic navigation systems.

Forests are irreplaceable and are being studied extensively. Better forest inventory and understanding necessitate effective mapping, modeling, and automatic analysis. As a result, considerable research effort is being devoted to digitizing forest environments. Recently, digital twins have come to the attention of the geospatial community as a virtual representation of the Earth’s surface linked to its corresponding physical asset. This concept is applicable to forests and has been studied in the literature. This requires initial input data obtained through reality capture. Among mapping techniques, laser scanning has emerged as a state-of-the-art technology for vegetation modeling. In this paper, we look into the potential of mobile laser scanning for forest digital twinning. While most studies concentrate on single tree detection, modeling, and estimation of dendrometric parameters, we also include lower vegetation in our investigations. To accomplish this, we first detect single trees and then investigate different vegetation densities and levels using geometric metrics. We also demonstrate how to model the underlying layers of vegetation in a digital twin. We perform the tests on data from mobile laser scanning (MLS) and compare the results to those from airborne laser scanning (ALS).We show that single tree detection based on crown separation using MLS data works similarly to or slightly better than ALS data. Furthermore, we demonstrate that MLS data allows for more detailed analysis of understory vegetation taking into account different height levels and a multi-level representation, whereas ALS data only allows for rough analysis of the lower parts of forest vegetation.

Mobile mapping systems are commonly used for surveying buildings. The acquisition of the buildings’ indoor spaces with laser scanning or photogrammetry generates data in the form of point clouds. These point clouds are often used to create a model of those buildings, but so far with a low degree of automation. To automate this process, it is important to extract geometric information about corners, edges, and planes from unorganized indoor point clouds. In an indoor scenario consisting of several rooms including furniture and other objects, a point cloud is expected to show occlusions. Therefore, the detection of hidden corners and edges is of importance. In this work one approach based on contour point clouds and one approach based on planes are examined for the detection of corners and edges. Both approaches use RANSAC to extract either straight lines or planes. Through their intersection, edges and corners are determined. To examine the influence of the data quality on the results, the approaches are applied to and evaluated on different datasets of the same area of a building, which are captured by various measurement methods, including mobile mapping systems and terrestrial laser scanning. Therefore, we are creating a ground truth for parts of the building to evaluate the completeness and correctness of the corner detection. The approach based on planes presents itself to be more reliable in noisy and incomplete point clouds. The approach based on contour point clouds indicates advantages in terms of the complexity of a building´s indoor geometry.

Data are a key component for many applications and methods in the domain of photogrammetry and remote sensing. Especially data-driven approaches such as deep learning rely heavily on available annotated data. The amount of data is increasing significantly every day. However, reference data is not increasing at the same rate and finding relevant data for a specific domain is still difficult. Thus, it is necessary to make existing reference data more accessible to the scientific community as far as possible in order to make optimal use of it. In this paper we provide an overview of the development of our photogrammetry and remote sensing specific Benchmark Metadata Database (BeMeDa). BeMeDa is based on MongoDB, a NoSQL database system. In addition, the development of a user-oriented metadata schema serves for data structuring. BeMeDa enables easy searching of benchmark datasets in the field of photogrammetry and remote sensing.

Having a good estimate of the position and orientation of a mobile agent is essential for many application domains such as robotics, autonomous driving, and virtual and augmented reality. In particular, when using LiDAR and IMU sensors as the inputs, most existing methods still use classical filter-based fusion methods to achieve this task. In this work, we propose DeepLIO, a modular, end-to-end learning-based fusion framework for odometry estimation using LiDAR and IMU sensors. For this task, our network learns an appropriate fusion function by considering different modalities of its input latent feature vectors. We also formulate a loss function, where we combine both global and local pose information over an input sequence to improve the accuracy of the network predictions. Furthermore, we design three sub-networks with different modules and architectures derived from DeepLIO to analyze the effect of each sensory input on the task of odometry estimation. Experiments on the benchmark dataset demonstrate that DeepLIO outperforms existing learning-based and model-based methods regarding orientation estimation and shows a marginal position accuracy difference.

Classification, and in particular semantic segmentation, plays a major role in remote sensing. In remote sensing, the classes usually correspond to landcover or landuse types while the data elements are image pixels. The results are so-called semantically segmented pixels describing the content of the data for each pixel. The identification of misclassified pixels is essential to perceive the overall performance of the classification algorithm. In the case of semantic segmentation, it is typically done with ground truth labels. However, such ground truth labels are rare and mostly reserved for training only. Especially deep learning approaches are data-hungry algorithms requesting a lot of labeled examples. In this work, we explore the possibility of using Monte-Carlo dropout for the identification of model-induced misclassifications. In particular, we obtain uncertainty measures from several inferences induced by the Monte-Carlo dropout. Furthermore, we examine how Markov Random Field optimization can reduce the number of misclassifications and facilitate their identification. The extent to which uncertainties provide information about misclassifications is assessed. Our results allow detecting 51 % of the misclassifications using uncertainties. Application of Markov Random Field optimization leads to a reduction of the percentage of misclassifications while detecting 0.4 % more misclassifications as without.

Indoor mapping has been gaining importance recently. One of the main applications of indoor maps is personal navigation. For this application, the connection to the outdoor map is very important, as users typically enter the building from outside and navigate to their destination inside. Obtaining this connection, however, is challenging, as the georeferencing of indoor maps is difficult due to the weak or total lack of GPS signal which makes positioning impossible in general. One solution for this problem could be matching indoor and outdoor datasets. Unfortunately, this is difficult due to the very low or non-existing overlap between the indoor and outdoor datasets as well as the differences in different. To overcome this problem, we propose a mobile mapping system, which can seamlessly capture the outdoor and indoor scene. Our prototype system contains three laser scanners, six RGB cameras, two GPS receivers and one IMU. In this paper, we propose an approach to seamlessly map a building and define the requirements for the mapping system. We primarily describe the construction phase of this system. Finally, we evaluate the performance of our mapping system with regard to the defined requirements.

Indoor maps are required for multiple applications, such as, navigation, building maintenance and robotics. One of common methods for map generation is laser scanning. In such maps, not only geometry of the map is of interest, but also its quality. This study aims at developing methods for real-time generation of indoor maps using features extracted from pointclouds obtained by a robot with their simultaneous quality assessment. We investigate, how this quality can be quantified for feature-based maps. First, we introduce a method for modeling 2D maps into 3D models that enable their usage for localization. Second, we review and evaluate a number of algorithms that can enable us to address features in a map. Hence, we enable the generation of objects from a pointcloud that has been sensed. Finally, we study several aspects of the map quality and we formalize them into metrics that can be applied to quantify their quality.

In the last decade, we have observed an increasing demand for indoor scene modeling in various applications, such as mobility inside buildings, emergency and rescue operations, and maintenance. Automatically distinguishing between structural elements of buildings, such as walls, ceilings, floors, windows, doors etc., and typical objects in buildings, such as chairs, tables and shelves, is particularly important for many reasons, such as 3D building modeling or navigation. This information can be generally retrieved through semantic labeling. In the past few years, convolutional neural networks (CNN) have become the preferred method for semantic labeling. Furthermore, there is ongoing research on fusing RGB and depth images in CNN frameworks. For pixel-level labeling, encoder-decoder CNN frameworks have been shown to be the most effective. In this study, we adopt an encoder-decoder CNN architecture to label structural elements in buildings and investigate the influence of using depth information on the detection of typical objects in buildings. For this purpose, we have introduced an approach to combine depth map with RGB images by changing the color space of the original image to HSV and then substitute the V channel with the depth information (D) and use it utilize it in the CNN architecture. As further variation of this approach, we also transform back the HSD images to RGB color space and use them within the CNN. This approach allows for using a CNN, designed for three-channel image input, and directly comparing our results with RGB-based labeling within the same network. We perform our tests using the Stanford 2D-3D-Semantics Dataset (2D-3D-S), a widely used indoor dataset. Furthermore, we compare our approach with results when using four-channel input created by stacking RGB and depth (RGBD). Our investigation shows that fusing RGB and depth improves results on semantic labeling; particularly, on structural elements of buildings. On the 2D- 3D-S dataset, we achieve up to 92.1 % global accuracy, compared to 90.9 % using RGB only and 93.6 % using RGBD. Moreover, the scores of Intersection over Union metric have improved using depth, which shows that it gives better labeling results at the boundaries.

Thermal properties of the building hull became an important topic of the last decade. Combining the thermal data with building models makes it possible to analyze thermal data in a 3D scene. In this paper we combine thermal images with 3D building models by texture mapping. We present a method for texture extraction from oblique airborne thermal infrared images. We put emphasis on quality assessment of these textures and evaluation of their usability for thermal inspections. The quality measures used for assessment are divided to resolution, occlusion and matching quality.

Thermal infrared imagery of urban areas became interesting for urban climate investigations and thermal building inspections. Using a flying platform such as UAV or a helicopter for the acquisition and combining the thermal data with the 3D building models via texturing delivers a valuable groundwork for large-area building inspections. However, such thermal textures are useful for further analysis if they are geometrically correctly extracted. This can be achieved with a good coregistrations between the 3D building models and thermal images, which cannot be achieved by direct georeferencing. Hence, this paper presents methodology for alignment of 3D building models and oblique TIR image sequences taken from a flying platform. In a single image line correspondences between model edges and image line segments are found using accumulator approach and based on these correspondences an optimal camera pose is calculated to ensure the best match between the projected model and the image structures. Among the sequence the linear features are tracked based on visibility prediction. The results of the proposed methodology are presented using a TIR image sequence taken from helicopter in a densely built-up urban area. The novelty of this work is given by employing the uncertainty of the 3D building models and by innovative tracking strategy based on a priori knowledge from the 3D building model and the visibility checking.

This paper presents a method for identification of errors in 3D building models which are results of inaccurate creation process. Error detection is carried out within the camera pose estimation. As observations, parameters of the building corners and of the line segments detected in the image are used and conditions for the coplanarity of corresponding edges are defined. For the estimation, the uncertainty of the 3D building models and image features are taken into account.

ESPACE is an interdisciplinary Master's study programme of Technische Universitaet Muenchen (TUM) positioned at the interface between space technology and the engineering and natural science-based use of satellite data. It combines the technical aspects of the satellite and observation systems with scientific and commercial applications. A core topic beside space engineering and satellite applications engineering is satellite and remote sensing data analysis. This paper explains the background and goals of the Masters's study programme, addresses the target groups and qualification profile, and gives an overview about the specialization fields. The special role of Photogrammetry and Remote Sensing is further explained in this context. The structure and embedding of international alliances by a double degree program with Wuhan University is shown.

3D city models are used in many fields. Photorealistic building textures find applications such as façade reconstruction, thermal building inspections and heat leakage detection using thermal infrared (TIR) images, quantitative evaluation or study of the materials lying on the object’s surface using multispectral images. Often texturing cannot be done using the same data which was used for 3D reconstruction or textures have to be updated. In such cases co-registration between 3D building models and images has to be carried out. In this paper we present a method for model-to-image matching and texture extraction with best texture selection procedure. We present results for two data sets, first for TIR image sequences taken from a helicopter and second for VIS images taken from an Unmanned Aerial Vehicle (UAV).

Data fusion techniques require a good registration of all the used datasets. In remote sensing, images are usually geo-referenced using the GPS and IMU data. However, if more precise registration is required, image processing techniques can be employed. We propose a method for multi-modal image coregistration between hyperspectral images (HSI) and digital surface models (DSM). The method is divided in three parts: object and line detection of the same object in HSI and DSM, line matching and determination of transformation parameters. Homogeneous coordinates are used to implement matching and adjustment of transformation parameters. The common object in HSI and DSM are building boundaries. They have apparent change in height and material, that can be detected in DSM and HSI, respectively. Thus, before the matching and transformation parameter computation, building outlines are detected and adjusted in HSI and DSM. We test the method on a HSI and two DSM, using extracted building outbounds and for comparison also extracted lines with a line detector. The results show that estimated building boundaries provide more line assignments, than using line detector.

Generation and texturing of building models is a fast developing field of research. Several techniques have been developed to extract building geometry and textures from multiple images and image sequences. In this paper, these techniques are discussed and extended to automatically add new textures from infrared (IR) image sequences to existing building models. In contrast to existing work, geometry and textures are not generated together from the same dataset but the textures are extracted from the image sequence and matched to an existing geo-referenced 3D building model. The texture generation is divided in two main parts. The first part deals with the estimation and refinement of the exterior camera orientation. Feature points are extracted in the images and used as tie points in the sequence. A recorded exterior orientation of the camera s added to these homologous points and a bundle adjustment is performed starting on image pairs and combining the hole sequence. A given 3d model of the observed building is additionally added to introduce further constraint as ground control points in the bundle adjustment. The second part includes the extraction of textures from the images and the combination of textures from different images of the sequence. Using the reconstructed exterior camera orientation for every image of the sequence, the visible facades are projected into the image and texture is extracted. These textures normally contain only parts of the facade. The partial textures extracted from all images are combined to one facade texture. This texture is stored with a 3D reference to the corresponding facade. This allows searching for features in textures and localising those features in 3D space. It will be shown, that the proposed strategy allows texture extraction and mapping even for big building complexes with restricted viewing possibilities and for images with low optical resolution.

The façades of buildings are almost always organized according to Gestalt principles such as good continuation, repetition in similarity, or symmetry etc. Coding such principles in production systems yields a very flexible frame to explore the usefulness of such principles in automatic façade understanding. Capturing images and image sequences of façades in the thermal domain and understanding such data is of importance e.g. for energy saving. In this contribution two different production systems are compared using the same data and interpreter.

Thermal building textures can be used for detection of damaged and weak spots in the building structure. These textures can be extracted from airborne infrared (IR) image sequences by projecting the 3D building model into the images. However, the direct georeferencing is often not sufficiently accurate and the projected 3D model does not match the structures in the image. Thus we present a technique with the main goal to find the best fit between the existing 3D building model and the IR image sequence. For this purpose we developed a hierarchical approach consisting of two working stages. In the first stage we correct exterior orientation via line based matching. In the adjustment we consider both uncertainties: the ones of the model and the ones of the image. In the second stage we match each edge separately in its closest surrounding. Thanks to this approach a better fit between the 3D building model and the IR image was found. The originally unmodeled roof overlap was reconstructed.

Due to their low cost and portability, using entertainment devices for indoor mapping applications has become a hot research topic. However, the impact of user behavior on indoor mapping evaluation with entertainment devices is often overlooked in previous studies. This article aims to assess the indoor mapping performance of entertainment devices under different mapping strategies. We chose two entertainment devices, the HoloLens 2 and iPhone 14 Pro, for our evaluation work. Based on our previous mapping experience and user habits, we defined four simplified indoor mapping strategies: straight-forward mapping (SFM), left-right alternating mapping (LRAM), round-trip straight-forward mapping (RT-SFM), and round-trip left-right alternating mapping (RT-LRAM). First, we acquired triangle mesh data under each strategy with the HoloLens 2 and iPhone 14 Pro. Then, we compared the changes in data completeness and accuracy between the different devices and indoor mapping applications. Our findings show that compared to the iPhone 14 Pro, the triangle mesh accuracy acquired by the HoloLens 2 has more stable performance under different strategies. Notably, the triangle mesh data acquired by the HoloLens 2 under the RT-LRAM strategy can effectively compensate for missing wall and floor surfaces, mainly caused by furniture occlusion and the low frame rate of the depth-sensing camera. However, the iPhone 14 Pro is more efficient in terms of mapping completeness and can acquire a complete triangle mesh more quickly than the HoloLens 2. In summary, choosing an entertainment device for indoor mapping requires a combination of specific needs and scenes. If accuracy and stability are important, the HoloLens 2 is more suitable; if efficiency and completeness are important, the iPhone 14 Pro is better.