Search

Classification, and in particular semantic segmentation, plays a major role in remote sensing. In remote sensing, the classes usually correspond to landcover or landuse types while the data elements are image pixels. The results are so-called semantically segmented pixels describing the content of the data for each pixel. The identification of misclassified pixels is essential to perceive the overall performance of the classification algorithm. In the case of semantic segmentation, it is typically done with ground truth labels. However, such ground truth labels are rare and mostly reserved for training only. Especially deep learning approaches are data-hungry algorithms requesting a lot of labeled examples. In this work, we explore the possibility of using Monte-Carlo dropout for the identification of model-induced misclassifications. In particular, we obtain uncertainty measures from several inferences induced by the Monte-Carlo dropout. Furthermore, we examine how Markov Random Field optimization can reduce the number of misclassifications and facilitate their identification. The extent to which uncertainties provide information about misclassifications is assessed. Our results allow detecting 51 % of the misclassifications using uncertainties. Application of Markov Random Field optimization leads to a reduction of the percentage of misclassifications while detecting 0.4 % more misclassifications as without.

Having a good estimate of the position and orientation of a mobile agent is essential for many application domains such as robotics, autonomous driving, and virtual and augmented reality. In particular, when using LiDAR and IMU sensors as the inputs, most existing methods still use classical filter-based fusion methods to achieve this task. In this work, we propose DeepLIO, a modular, end-to-end learning-based fusion framework for odometry estimation using LiDAR and IMU sensors. For this task, our network learns an appropriate fusion function by considering different modalities of its input latent feature vectors. We also formulate a loss function, where we combine both global and local pose information over an input sequence to improve the accuracy of the network predictions. Furthermore, we design three sub-networks with different modules and architectures derived from DeepLIO to analyze the effect of each sensory input on the task of odometry estimation. Experiments on the benchmark dataset demonstrate that DeepLIO outperforms existing learning-based and model-based methods regarding orientation estimation and shows a marginal position accuracy difference.

Indoor maps are required for multiple applications, such as, navigation, building maintenance and robotics. One of common methods for map generation is laser scanning. In such maps, not only geometry of the map is of interest, but also its quality. This study aims at developing methods for real-time generation of indoor maps using features extracted from pointclouds obtained by a robot with their simultaneous quality assessment. We investigate, how this quality can be quantified for feature-based maps. First, we introduce a method for modeling 2D maps into 3D models that enable their usage for localization. Second, we review and evaluate a number of algorithms that can enable us to address features in a map. Hence, we enable the generation of objects from a pointcloud that has been sensed. Finally, we study several aspects of the map quality and we formalize them into metrics that can be applied to quantify their quality.

Semantic segmentation has been an active field in computer vision and photogrammetry communities for over a decade. Pixel-level semantic labeling of images is generally achieved by assigning labels to pixels using machine learning techniques. Among others, the encoder–decoder convolutional neural networks have become the baseline approach for this problem recently. The majority of papers on this topic use only RGB images as input, despite the availability of other data sources, such as depth, which can improve segmentation and labeling. In this chapter, we investigate a number of encoder–decoder CNN architectures for semantic labeling, where the depth data is fused with the RGB data using three different approaches: (1) fusion with RGB image through color space transformation, (2) stacking depth images and RGB images, and (3) using Siamese network structures, such as FuseNet or VNet. The chapter also presents our approach to using surface normals in place of depth data. The advantage of using surface normal representation is to introduce viewpoint independency, meaning that the direction of a surface normal vector remains the same when the camera pose changes. This is a clear advantage over raw depth data where the depth value for a single scene point changes when the camera moves. The chapter provides a comprehensive analysis of three different fusion approaches using SegNet, FuseNet and VNet deep learning architectures. The analysis is conducted on both the Stanford 2D-3D-Semantics indoor dataset and aerial images from the ISPRS's Vaihingen dataset. Depth images of the Stanford dataset are acquired directly by flash LiDAR, and the ISPRS dataset depth images are generated by dense 3D reconstruction. We show that the surface normal representation better generalizes to different scenes. In our experiments, using surface normals with FuseNet achieved 5% improvement compared to using depth, and this resulted in 81.5% global accuracy on the Stanford dataset.

This paper presents a method for identification of errors in 3D building models which are results of inaccurate creation process. Error detection is carried out within the camera pose estimation. As observations, parameters of the building corners and of the line segments detected in the image are used and conditions for the coplanarity of corresponding edges are defined. For the estimation, the uncertainty of the 3D building models and image features are taken into account.

Thermal properties of the building hull became an important topic of the last decade. Combining the thermal data with building models makes it possible to analyze thermal data in a 3D scene. In this paper we combine thermal images with 3D building models by texture mapping. We present a method for texture extraction from oblique airborne thermal infrared images. We put emphasis on quality assessment of these textures and evaluation of their usability for thermal inspections. The quality measures used for assessment are divided to resolution, occlusion and matching quality.

Thermal infrared imagery of urban areas became interesting for urban climate investigations and thermal building inspections. Using a flying platform such as UAV or a helicopter for the acquisition and combining the thermal data with the 3D building models via texturing delivers a valuable groundwork for large-area building inspections. However, such thermal textures are useful for further analysis if they are geometrically correctly extracted. This can be achieved with a good coregistrations between the 3D building models and thermal images, which cannot be achieved by direct georeferencing. Hence, this paper presents methodology for alignment of 3D building models and oblique TIR image sequences taken from a flying platform. In a single image line correspondences between model edges and image line segments are found using accumulator approach and based on these correspondences an optimal camera pose is calculated to ensure the best match between the projected model and the image structures. Among the sequence the linear features are tracked based on visibility prediction. The results of the proposed methodology are presented using a TIR image sequence taken from helicopter in a densely built-up urban area. The novelty of this work is given by employing the uncertainty of the 3D building models and by innovative tracking strategy based on a priori knowledge from the 3D building model and the visibility checking.

Generation and texturing of building models is a fast developing field of research. Several techniques have been developed to extract building geometry and textures from multiple images and image sequences. In this paper, these techniques are discussed and extended to automatically add new textures from infrared (IR) image sequences to existing building models. In contrast to existing work, geometry and textures are not generated together from the same dataset but the textures are extracted from the image sequence and matched to an existing geo-referenced 3D building model. The texture generation is divided in two main parts. The first part deals with the estimation and refinement of the exterior camera orientation. Feature points are extracted in the images and used as tie points in the sequence. A recorded exterior orientation of the camera s added to these homologous points and a bundle adjustment is performed starting on image pairs and combining the hole sequence. A given 3d model of the observed building is additionally added to introduce further constraint as ground control points in the bundle adjustment. The second part includes the extraction of textures from the images and the combination of textures from different images of the sequence. Using the reconstructed exterior camera orientation for every image of the sequence, the visible facades are projected into the image and texture is extracted. These textures normally contain only parts of the facade. The partial textures extracted from all images are combined to one facade texture. This texture is stored with a 3D reference to the corresponding facade. This allows searching for features in textures and localising those features in 3D space. It will be shown, that the proposed strategy allows texture extraction and mapping even for big building complexes with restricted viewing possibilities and for images with low optical resolution.

The aim of this article is to investigate methods for the automatic extraction of the infrared (IR) textures for the roofs and facades of existing building models. We focus on the correction of the measured exterior orientation parameters of the IR camera mounted on a mobile platform. The developed method is based on point-to-point matching of the features extracted from IR images with a wire-frame building model. Firstly, the extraction of different feature types is studied on a sample IR image; Forstner and intersection points are chosen for a representation of the image features. Secondly, the three-dimensional (3D) building model is projected into each frame of the IR video sequence using orientation parameters; only coarse exterior orientation parameters are known. Then the automatic co-registration of a 3D building model projection into the image sequence with image features is carried out. The matching of a model and extracted features is applied iteratively, and exterior orientation parameters are adjusted with least square adjustment. The method is tested on a dataset of a dense urban area. Finally, an evaluation of the developed method is presented with five quality parameters, i.e. efficiency of the method, completeness and correctness of matching and extraction.

This contribution is focused on the selection of building textures extracted from thermal infrared (TIR) image sequences acquired both from terrestrial and aerial platforms by introducing a quality assessment. Extracted quality features are completeness of the texture, projection accuracy, viewing angle, and geometric resolution. The calculation of the quality features is discussed and results are presented. The proposed method allows to compare the quality of different recordings from different positions and even sensors to obtain the most complete and accurate buildings textures of high geometric resolution.

For image fusion in remote sensing applications the georeferencing accuracy using position, attitude, and camera calibration measurements can be insufficient. Thus, image processing techniques should be employed for precise coregistration of images. In this article a method for multimodal object-based image coregistration refinement between hyperspectral images (HSI) and digital surface models (DSM) is presented. The method is divided in three parts: object outline detection in HSI and DSM, matching, and determination of transformation parameters. The novelty of our proposed coregistration refinement method is the use of material properties and height information of urban objects from HSI and DSM, respectively. We refer to urban objects as objects which are typical in urban environments and focus on buildings by describing them with 2D outlines. Furthermore, the geometric accuracy of these detected building outlines is taken into account in the matching step and for the determination of transformation parameters. Hence, a stochastic model is introduced to compute optimal transformation parameters. The feasibility of the method is shown by testing it on two aerial HSI of different spatial and spectral resolution, and two DSM of different spatial resolution. The evaluation is carried out by comparing the accuracies of the transformations parameters to the reference parameters, determined by considering object outlines at much higher resolution, and also by computing the correctness and the quality rate of the extracted outlines before and after coregistration refinement. Results indicate that using outlines of objects instead of only line segments is advantageous for coregistration of HSI and DSM. The extraction of building outlines in comparison to the line cue extraction provides a larger amount of assigned lines between the images and is more robust to outliers, i.e. false matches.

Thermal inspections of buildings contribute to detection of damaged and weak spots in the building hull. 3D spatial reference for this purpose can be achieved combining infrared images with 3D building models via texture mapping. Using terrestrial image sequences from a camera mounted in a mobile platform frontal faces can be captured, while airborne image sequences can be taken for roofs and inner yards. However, according to different geometry of acquisition, two different methods for matching of terrestrial and airborne images are required. In this paper we present a concept for texturing of an existing 3D building model with complementary terrestrial and airborne IR image sequences. First we present a method for matching of terrestrial image sequences based on relative orientation of the point cloud. Second, we introduce a method for matching of airborne images including system calibration and best texture selection. Further, the combination of the terrestrial and airborne data in one 3D building model is introduced.

The façades of buildings are almost always organized according to Gestalt principles such as good continuation, repetition in similarity, or symmetry etc. Coding such principles in production systems yields a very flexible frame to explore the usefulness of such principles in automatic façade understanding. Capturing images and image sequences of façades in the thermal domain and understanding such data is of importance e.g. for energy saving. In this contribution two different production systems are compared using the same data and interpreter.

Data fusion techniques require a good registration of all the used datasets. In remote sensing, images are usually geo-referenced using the GPS and IMU data. However, if more precise registration is required, image processing techniques can be employed. We propose a method for multi-modal image coregistration between hyperspectral images (HSI) and digital surface models (DSM). The method is divided in three parts: object and line detection of the same object in HSI and DSM, line matching and determination of transformation parameters. Homogeneous coordinates are used to implement matching and adjustment of transformation parameters. The common object in HSI and DSM are building boundaries. They have apparent change in height and material, that can be detected in DSM and HSI, respectively. Thus, before the matching and transformation parameter computation, building outlines are detected and adjusted in HSI and DSM. We test the method on a HSI and two DSM, using extracted building outbounds and for comparison also extracted lines with a line detector. The results show that estimated building boundaries provide more line assignments, than using line detector.

Thermal building textures can be used for detection of damaged and weak spots in the building structure. These textures can be extracted from airborne infrared (IR) image sequences by projecting the 3D building model into the images. However, the direct georeferencing is often not sufficiently accurate and the projected 3D model does not match the structures in the image. Thus we present a technique with the main goal to find the best fit between the existing 3D building model and the IR image sequence. For this purpose we developed a hierarchical approach consisting of two working stages. In the first stage we correct exterior orientation via line based matching. In the adjustment we consider both uncertainties: the ones of the model and the ones of the image. In the second stage we match each edge separately in its closest surrounding. Thanks to this approach a better fit between the 3D building model and the IR image was found. The originally unmodeled roof overlap was reconstructed.