Search

Super resolution for satellite imagery possesses specific challenges due to its unique feature geometry compared to classic computer vision. Specifically, satellite images contain a high amount of relatively small, distributed high-frequency features that are hard to preserve when sampling up to a higher resolution. General adversarial networks (GANs) are suitable for super resolution tasks but need special attention concerning the geometric and radiometric accuracy of the results. We propose a network for versatile satellite imagery super resolution (VSISR) that focuses on high-frequency detail preservation and radiometric consistency. As a novelty, it is able to utilize a reference high resolution image during inference to lower hallucinations without altering the source image’s radiometric characteristics.A GAN that is already handling well high frequencies in standard computer vision cases is adapted for satellite imagery and supplemented with a mixed pixel approach for data augmentation. Training on a diverse RGB dataset from four satellite missions results in a versatile super resolution model that is optimized to preserve radiometric features and minimize hallucinations. Compared to other neural networks for satellite image super resolution in their respective datasets, VSISR performs in the middle regarding mean PSNR (25.30 dB) and SSIM (0.8098). Nevertheless, it is the first time that, using mixed pixel training on a comparably small dataset, this versatility concerning the satellite data source is achieved while maintaining their unique radiometric traits.

Numerous navigation applications rely on data from global navigation satellite systems (GNSS), even though their accuracy is compromised in urban areas, posing a significant challenge, particularly for precise autonomous car localization. Extensive research has focused on enhancing localization accuracy by integrating various sensor types to address this issue. This paper introduces a novel approach for car localization, leveraging image features that correspond with highly detailed semantic 3D building models. The core concept involves augmenting positioning accuracy by incorporating prior geometric and semantic knowledge into calculations. The work assesses outcomes using Level of Detail 2 (LoD2) and Level of Detail 3 (LoD3) models, analyzing whether facade-enriched models yield superior accuracy. This comprehensive analysis encompasses diverse methods, including off-the-shelf feature matching and deep learning, facilitating thorough discussion. Our experiments corroborate that LoD3 enables detecting up to 69% more features than using LoD2 models. We believe that this study will contribute to the research of enhancing positioning accuracy in GNSS-denied urban canyons. It also shows a practical application of under-explored LoD3 building models on map-based car positioning.

Autonomous vehicles must navigate independently in an outdoor environment using features or objects. However, some objects may be more or less suitable for localization due to their attributes. Therefore, this work investigates the suitability of landmarks for camera- and object-based outdoor localization methods. First, object attributes are methodically derived from the requirements of object-based localization. The physical representation on the camera image plane, probability of occurrence, and persistence were identified as influencing the object localization suitability. The influence of the object’s camera image plane representation regarding object recognition algorithms is not considered or discussed, but advice on the minimum object pixel size is provided. The first milestone was the creation of an equation for object localization suitability calculation by normalizing and multiplying the identified attributes. Simultaneously, potential objects from the outdoor environment were identified, resulting in a structured object catalog. The results of the equation and catalog are a ranked according to the object localization suitability in a comparison table. Our comparison demonstrates that objects such as buildings or trees are more suitable than street lane markings for self-localization. However, most current datasets do not include the proposed instantiated objects. The paper addresses this issue, assists in the object selection for outdoor localization methods and provides input for the creation of future-oriented datasets and autonomous driving maps.

Semantic 3D building models are widely available and used in numerous applications. Such 3D building models display rich semantics but no façade openings, chiefly owing to their aerial acquisition techniques. Hence, refining models’ façades using dense, street-level, terrestrial point clouds seems a promising strategy. In this paper, we propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. Voxels are also used to identify conflicts between laser observations and 3D models. The semantic voxels and conflicts are combined in a Bayesian network to classify and delineate façade openings, which are reconstructed using a 3D model library. Unaffected building semantics is preserved while the updated one is added, thereby upgrading the building model to LoD3. Moreover, Bayesian network results are back-projected onto point clouds to improve points’ classification accuracy. We tested our method on a municipal CityGML LoD2 repository and the open point cloud datasets: TUM-MLS-2016 and TUM-FAÇADE. Validation results revealed that the method improves the accuracy of point cloud semantic segmentation and upgrades buildings with façade elements. The method can be applied to enhance the accuracy of urban simulations and facilitate the development of semantic segmentation algorithms.

This contribution presents a method for extracting a 3D model of facades and windows from a point cloud. The point cloud is segmented based on a voxel octree, in which the facades are sought as planes. These can be used to filter out potential window points within the building, which are then analysed on their visibility by checking the occupancy grid of the voxel space. Here, methods of digital image processing are used for analysing both point clusters behind the facade and holes in the estimated facade planes as window candidates. Facades and windows are both simplified as rectangles. The test data set was gathered in a Mobile Laser Scanning campaign. While the segmentation fails in some cases, the extraction of facades and windows shows good results: 25 facades with 702 detected windows yield a detection rate of 86% with a false alarm rate of 13%. The reconstructed sizes of the windows differ from reference measurements in the range of centimetres to a few decimetres. These refined geometries can be used to enrich existing building models or for vehicle navigation without GNSS.

Gravitational mass movements represent a significant hazard potential in Alpine regions. Due to climate change and an associated increases in extreme weather events, this risk is growing. For a better predictability of such events and the monitoring of affected areas, a precise determination of the ongoing movements is necessary. In this paper, a method for monitoring of Alpine slope movements based on image sequences is presented. By means of SIFT features, corresponding key points in the images of different epochs are found. Then, using forward section, the object coordinates of the points are computed. By using these coordinates and the detected correspondences, three-dimensional motion vectors can be determined. The calculated vectors are checked for significance based on their accuracy. The vectors found have a high spatial density compared to manually marked points and are detected automatically. In order to detect even small-scale movements, they are determined with an accuracy of a few millimeters. The data basis is a sequence of images of an active landslide on the Hochvogel mountain (Alps, Germany) which were taken in 2018 and 2021. On average, the calculated motion vectors show a movement of 75 mm.

Point clouds are widely regarded as one of the best dataset types for urban mapping purposes. Hence, point cloud datasets are commonly investigated as benchmark types for various urban interpretation methods. Yet, few researchers have addressed the use of point cloud benchmarks for façade segmentation. Robust façade segmentation is becoming a key factor in various applications ranging from simulating autonomous driving functions to preserving cultural heritage. In this work, we present a method of enriching existing point cloud datasets with façade-related classes that have been designed to facilitate façade segmentation testing. We propose how to efficiently extend existing datasets and comprehensively assess their potential for façade segmentation. We use the method to create the TUM-FAÇADE dataset, which extends the capabilities of TUM-MLS-2016. Not only can TUM-FAÇADE facilitate the development of point-cloud-based façade segmentation tasks, but our procedure can also be applied to enrich further datasets.

Nowadays, the number of connected devices providing unstructured data is rapidly rising. These devices acquire data with a temporal and spatial resolution at an unprecedented level creating an influx of geoinformation which, however, lacks semantic information. Simultaneously, structured datasets like semantic 3D city models are widely available and assure rich semantics and high global accuracy but are represented by rather coarse geometries. While the mentioned downsides curb the usability of these data types for nowadays’ applications, the fusion of both shall maximize their potential. Since testing and developing automated driving functions stands at the forefront of the challenges, we propose a pipeline fusing structured (CityGML and HD Map datasets) and unstructured datasets (MLS point clouds) to maximize their advantages in the automatic 3D road space models reconstruction domain. The pipeline is a parameterized end-to-end solution that integrates segmentation, reconstruction, and modeling tasks while ensuring geometric and semantic validity of models. Firstly, the segmentation of point clouds is supported by the transfer of semantics from a structured to an unstructured dataset. The distinction between horizontal- and vertical-like point cloud subsets enforces a further segmentation or an immediate refinement while only adequately depicted models by point clouds are allowed. Then, based on the classified and filtered point clouds the input 3D model geometries are refined. Building upon the refinement, the semantic enrichment of the 3D models is presented. The deployment of a simulation engine for automated driving research and a city model database tool underlines the versatility of possible application areas.

In our daily lives, trees can be seen as the tallest and most noticeable representatives of the plant kingdom. Especially in urban areas, the individual tree is of high significance and responsible for a manifold of positive effects on the environment and residents. In the context of urban tree registers and thus monitoring of urban vegetation, we propose a general concept for the segmentation of trees from 3D point clouds. Mobile Laser Scanning (MLS) is introduced as the preferred sensor. Based on an analysis of earlier work in this field, we gather arguments and methods in order to involve segmentation in the bigger frame of a tree register workflow, including detailed modeling and change detection. Our concept for segmentation is based on a voxel-structure. In a first step, region growing approaches are used for ground removal and rough segmentation. Later, graph-based optimization will separate neighboring trees. For now, only the general concept can be introduced—quantitative analysis and optimization of the steps will follow in future work.

When purchasing a premium car for a substantial sum, first impressions count. Key to that first impression is a flawless exterior appearance, something self-explanatory for the customer, but a far greater challenge for production than one might initially assume. Fortunately, photogrammetric technologies and evaluation methods are enabling an ever greater degree of oversight in the form of comprehensive quality data at different automotive production stages, namely stamping, welding, painting and finishing. A drawback lies in the challenging production environment, which complicates inline integratability of certain technologies. In recent years, machine vision and deep learning have been applied to photogrammetric surface inspection with ever increasing success. Given comprehensive surface quality information throughout the entire production chain, production parameters can be dialed in ever tighter in a data-driven fashion, leading to a sustainable increase in quality. This paper provides a review of current and potential contributions of photogrammetry to this end, discussing several recent advances in research along the way. Particular emphasis will be placed on early production stages, as well as the application of machine vision and deep learning to this challenging task. An outline for further research conducted by the authors will conclude this paper.

Registration of multiple point clouds acquired via terrestrial laser scanning (TLS) is usually compulsory to obtain the scanned data covering a whole urban scene. However, the automated processing of aligning multiple scans is still a concern because of the complex urban environment. To this end, we propose a fast and sturdy estimation of 3D shifts between point clouds by an automated markerfree process using global features, converting translation measurement between two point clouds in the space domain to the frequency domain and estimating the phase difference. By using the low-frequency components from the normalized cross-power spectrum, accurate 3D shifts are calculated by solving parameters in the linear equation representing phase difference angles, with the help of a robust estimator. The results of experiments using TLS datasets of different scenes show that the proposed approach is both practical and efficient. In particular, the proposed approach can achieve results with a translation error of less than about 1.0 m on test datasets.

Image registration is a fundamental issue in photogrammetry and remote sensing, which targets to find the alignment between different images. Recently, registration of images from difference sensors become the hot topic. The registered images from different sensors are able to offer additional information, which help with different tasks like segmentation, classification, and even emergency analysis. In this paper, we proposed a registration strategy to calculate the dominant orientation difference and then achieve the dense alignment of Thermal Infrared (TIR) image and RGB image with MINDflow. Firstly, the orientation difference of TIR images and RGB images is calculated by finding the dominant image orientations based on phase congruency. Then, the modality independent neighborhood descriptor (MIND) together with global optical flow algorithm are adopted as MINDflow for dense matching. Our method is tested in the image sets containing TIR images and RGB images captured separately but in the same construction site areas. The results show that it is able to achieve the optimal results with features of significance even for dramatically radiometric differences between TIR images and RGB images. By comparing the results with other descriptor, our method is more robust and keep the features of objects in the images.

This contribution discusses the accuracy and the applicability of Photogrammetric point clouds based on dense image matching for the monitoring of gravitational mass movements caused by crevices. Four terrestrial image sequences for three different time epochs have been recorded and oriented using ground control point in a local reference frame. For the first epoch, two sequences are recorded, one in the morning and one in the afternoon to evaluate the noise level within the point clouds for a static geometry and changing light conditions. The second epoch is recorded a few months after the first epoch where also no significant change has occurred in between. The third epoch is recorded after one year with changes detected. As all point clouds are given in the same local coordinate frame and thus are co-registered via the ground control points, change detection is based on calculating the Multiscale-Model-to-Model-Cloud distances (M3C2) of the point clouds. Results show no movements for the first year, but identify significant movements comparing the third epoch taken in the second year. Besides the noise level estimation, the quality checks include the accuracy of the camera orientations based on ground control points, the covariances of the bundle adjustment, and a comparison the Geodetic measurements of additional control points and a laser scanning point cloud of a part of the crevice. Additionally, geological measurements of the movements have been performed using extensometers.

In this work, we discussed how to directly combine thermal infrared image (TIR) and the point cloud without additional assistance from GCPs or 3D models. Specifically, we propose a point-based co-registration process for combining the TIR image and the point cloud for the buildings. The keypoints are extracted from images and point clouds via primitive segmentation and corner detection, then pairs of corresponding points are identified manually. After that, the estimated camera pose can be computed with EPnP algorithm. Finally, the point cloud with thermal information provided by IR images can be generated as a result, which is helpful in the tasks such as energy inspection, leakage detection, and abnormal condition monitoring. This paper provides us more insight about the probability and ideas about the combining TIR image and point cloud.

This paper proposes a method to get semantic information of changes in bathymetric point clouds. This method aims for assigning labels to river ground points which determine if either the point can be compared with a reference DEM, if there are no data in the reference or if there are no water points inside the new Data of wet areas of the reference data. This labels can be further used to specify areas where differences of DEMS can be calculated, the comparable areas. The Areas where no reference data is found specify areas where the reference DEM will have a higher variance due to interpolation which should be considered in the comparison. The areas where no water in the new data was found specify areas there no refraction correction in the new data can be done and which should be considered with a higher variance of the ground points or there the water surface should be tried to reconstruct. The proposed approach uses semantic reference data to specify water areas in the new scan. An occupancy analysis is used to specify if voxels of the new data exist in the reference or not. In case of occupancy, the labels of the reference are assigned to the new data and in case of no occupancy, the label of changed data is assigned. A histogram based method is used to separate ground and water points in wet areas and a second occupancy analysis is used to specify the semantic changes in wet areas. The proposed method is evaluated on a proposed data set of the Mangfall area where the ground truth is manually labelled.

This work proposes an approach for semantic classification of an outdoor-scene point cloud acquired with a high precision Mobile Mapping System (MMS), with major goal to contribute to the automatic creation of High Definition (HD) Maps. The automatic point labeling is achieved by utilizing the combination of a feature-based approach for semantic classification of point clouds and a deep learning approach for semantic segmentation of images. Both, point cloud data, as well as the data from a multi-camera system are used for gaining spatial information in an urban scene. Two types of classification applied for this task are: 1) Feature-based approach, in which the point cloud is organized into a supervoxel structure for capturing geometric characteristics of points. Several geometric features are then extracted for appropriate representation of the local geometry, followed by removing the effect of local tendency for each supervoxel to enhance the distinction between similar structures. And lastly, the Random Forests (RF) algorithm is applied in the classification phase, for assigning labels to supervoxels and therefore to points within them. 2) The deep learning approach is employed for semantic segmentation of MMS images of the same scene. To achieve this, an implementation of Pyramid Scene Parsing Network is used. Resulting segmented images with each pixel containing a class label are then projected onto the point cloud, enabling label assignment for each point. At the end, experiment results are presented from a complex urban scene and the performance of this method is evaluated on a manually labeled dataset, for the deep learning and feature-based classification individually, as well as for the result of the labels fusion. The achieved overall accuracy with fusioned output is 0.87 on the final test set, which significantly outperforms the results of individual methods on the same point cloud. The labeled data is published on the TUM-PF Semantic-Labeling-Benchmark.

Reconstruction of dense photogrammetric point clouds is often based on depth estimation of rectified image pairs by means of pixel-wise matching. The main drawback lies in the high computational complexity compared to that of the relatively straightforward task of laser triangulation. Dense image matching needs oriented and rectified images and looks for point correspondences between them. The search for these correspondences is based on two assumptions: pixels and their local neighborhood show a similar radiometry and image scenes are mostly homogeneous, meaning that neighboring points in one image are most likely also neighbors in the second. These rules are violated, however, at depth changes in the scene. Optimization strategies tend to find the best depth estimation based on the resulting disparities in the two images. One new field in neural networks is the estimation of a depth image from a single input image through learning geometric relations in images. These networks are able to find homogeneous areas as well as depth changes, but result in a much lower geometric accuracy of the estimated depth compared to dense matching strategies. In this paper, a method is proposed extending the Semi-Global-Matching algorithm by utilizing a-priori knowledge from a monocular depth estimating neural network to improve the point correspondence search by predicting the disparity range from the single-image depth estimation (SIDE). The method also saves resources through path optimization and parallelization. The algorithm is benchmarked on Middlebury data and results are presented both quantitatively and qualitatively.

With the rapid development of subpixel matching algorithms, the estimation of image shifts with an accuracy of higher than 0.05 pixels is achieved, which makes the narrow baseline stereovision possible. Based on the subpixel matching algorithm using the robust phase correlation (PC), in this work, we present a novel hierarchical and adaptive disparity estimation scheme for narrow baseline stereo, which consists of three main steps: image coregistration, pixel-level disparity estimation, and subpixel refinement. The Fourier-Mellin transform with subpixel PC is used to co-register two input images. Then, the pixel-level disparities are estimated in an iterative manner, which is achieved through multiscale superpixels. The pixel-level PC is performed with the window sizes and locations adaptively determined according to superpixels, with the disparity values calcualted. Fast weighted median filtering based on edge-aware filter is adopted to refine the disparity results. At last, the accurate disparities are calculated via a robust subpixel PC method. The combination of multiscale superpixel hierarchy, adaptive determination of the window size and location of correlation, fast weighted median filtering and subpixel PC make the proposed scheme be able to overcome the issues of either low-texture areas or fattening effect. Experimental results on a pair of UAV images and the comparison with the fixed-window PC methods, the iterative scheme with fixed variation strategy, and a sophisticated implementation using global optimization demonstrate the superiority and reliability of the proposed scheme.

This paper shows a method to register point clouds from images of UAV-mounted airborne cameras as well as airborne laser scanner data. The focus is a general technique which does rely neither on linear or planar structures nor on the point cloud density. Therefore, the proposed approach is also suitable for rural areas and water bodies captured via different sensor configurations. This approach is based on a regular 2.5D grid generated from the segmented ground points of the 3D point cloud. It is assumed that initial values for the registration are already estimated, e.g. by measured exterior orientation parameters with the UAV mounted GNSS and IMU. These initial parameters are finely tuned by minimizing the distances between the 3D points of a target point cloud to the generated grid of the source point cloud in an iteration process. To eliminate outliers (e.g., vegetation points) a threshold for the distances is defined dynamically at each iteration step, which filters ground points during the registration. The achieved accuracy of the registration is up to 0.4 m in translation and up to 0.3 degrees in rotation, by using a raster size of the DEM of 2 m. Considering the ground sampling distance of the airborne data which is up to 0.4 m between the scan lines, this result is comparable to the result achieved by an ICP algorithm, but the proposed approach does not rely on point densities and is therefore able to solve registrations where the ICP have difficulties.

Obtaining accurate 3D descriptions in the thermal infrared (TIR) is a quite challenging task due to the low geometric resolutions of TIR cameras and the low number of strong features in TIR images. Combining the radiometric information of the thermal infrared with 3D data from another sensor is able to overcome most of the limitations in the 3D geometric accuracy. In case of dynamic scenes with moving objects or a moving sensor system, a combination with RGB cameras and profile laserscanners is suitable. As a laserscanner is an active sensor in the visible red or near infrared (NIR) and the thermal infrared camera captures the radiation emitted by the objects in the observed scene, the combination of these two sensors for close range applications are independent from external illumination or textures in the scene. This contribution focusses on the fusion of point clouds from terrestrial laserscanners and RGB cameras with images from thermal infrared mounted together on a robot for indoor 3D reconstruction. The system is geometrical calibrated including the lever arm between the different sensors. As the field of view is different for the sensors, the different sensors record the same scene points not exactly at the same time. Thus, the 3D scene points of the laserscanner and the photogrammetric point cloud from the RGB camera have to be synchronized before point cloud fusion and adding the thermal channel to the 3D points.

This paper discusses the automatic generation of thermal infrared ortho image mosaics and the extraction of solar cells from these ortho image mosaics. Image sequences are recorded by a thermal infrared (TIR) camera mounted on a remotely piloted aerial system (RPAS). The image block is relatively oriented doing a bundle block adjustment and transferred to a local coordinate system using ground control points. The resulting ortho image mosaic is searched for solar cells. A library of templates of solar cells from thermal images is used to learn an implicit shape model. The extraction of the single solar cells is done by estimating corners and centre points of cells using these shape models in a Markov-Chain-Monte-Carlo algorithm by combining four corners and a centre point. As for the limited geometric resolution and radiometric contrast, most of the cells are not directly detected. An iterative process based on the knowledge of the regular grid structure of a solar cell installation is used to predict further cells and verify their existence by repeating the corner extraction and grammar combination. Results show that this work flow is able to detect most of the solar cells under the condition that the cells have a more or less common radiometric behaviour and no reflections i.e. from the sun occur. The cells need a rectangular shape and have the same orientation so that the model of the grammar is applicable to the solar cells.

This paper presents a method to register photogrammetric point clouds generated from optical images acquired by UAV and aerial LIDAR point clouds. Normally, the registration of two airborne scans of the same scene is solved by the use of control points and the direct registration using GNSS and INS information. However, the registration of multi-sensor point clouds without control points is more complicated and challenging. For the scene of non urban areas, the registration task gets even more complicated, because it is hard to extract sufficient geometric primitives from the building structures. For our proposed method, an outdoor scene is tested providing almost no man-made objects. Therefore, it is nearly impossible to search for planar objects and use them for registration. With no geometric primitives extracted, the proposed method utilizes the structure of the 2.5D DEM created from the ground points of the point cloud. Besides, instead of using control points or key points, the method automatic detect key planes from the 2.5D DEM as correspondences. These key planes are detected on a regular grid by the use of a predefined mask. To mark a DEM grid cell as key plane the histogram of sums of the angles between the center cell is used. Afterwards, similarity values between two key planes are calculated based on the histogram differences and a RANSAC based strategy is adopted to find corresponding key planes and estimate the transformation parameters. Experiments conducted in this paper indicate that it is feasible to register multi sensor point clouds with a big difference in their ground sampling distances with respect to the used cell size of the 2.5D DEM.

In this work, we propose a classification method designed for the labeling of MLS point clouds, with detrended geometric features extracted from the points of the supervoxel-based local context. To achieve the analysis of complex 3D urban scenes, acquired points of the scene should be tagged with individual labels of different classes. Thus, assigning a unique label to the points of an object that belong to the same category plays an essential role in the entire 3D scene analysis workflow. Although plenty of studies in this field have been reported, this work is still a challenging task. Specifically, in this work: 1) A novel geometric feature extraction method, detrending the redundant and in-salient information in the local context, is proposed, which is proved to be effective for extracting local geometric features from the 3D scene. 2) Instead of using individual point as basic element, the supervoxel-based local context is designed to encapsulate geometric characteristics of points, providing a flexible and robust solution for feature extraction. 3) Experiments using complex urban scene with manually labeled ground truth are conducted, and the performance of proposed method with respect to different methods is analyzed. With the testing dataset, we have obtained a result of 0.92 for overall accuracy for assigning eight semantic classes.

Visual SLAM algorithms allow localizing the camera by mapping its environment by a point cloud based on visual cues. To obtain the camera locations in a metric coordinate system, the metric scale of the point cloud has to be known. This contribution describes a method to calculate the metric scale for a point cloud of an indoor environment, like a parking garage, by fusing multiple individual scale values. The individual scale values are calculated from structures and objects with a-priori known metric extension, which can be identified in the unscaled point cloud. Extensions of building structures, like the driving lane or the room height, are derived from density peaks in the point distribution. The extension of objects, like traffic signs with a known metric size, are derived using projections of their detections in images onto the point cloud. The method is tested with synthetic image sequences of a drive with a front-looking mono camera through a virtual 3D model of a parking garage. It has been shown, that each individual scale value improves either the robustness of the fused scale value or reduces its error. The error of the fused scale is comparable to other recent works.

In this work, we report a novel way of generating ground truth dataset for analyzing point cloud from different sensors and the validation of algorithms. Instead of directly labeling large amount of 3D points requiring time consuming manual work, a multi-resolution 3D voxel grid for the testing site is generated. Then, with the help of a set of basic labeled points from the reference dataset, we can generate a 3D labeled space of the entire testing site with different resolutions. Specifically, an octree-based voxel structure is applied to voxelize the annotated reference point cloud, by which all the points are organized by 3D grids of multi-resolutions. When automatically annotating the new testing point clouds, a voting based approach is adopted to the labeled points within multiple resolution voxels, in order to assign a semantic label to the 3D space represented by the voxel. Lastly, robust line- and plane-based fast registration methods are developed for aligning point clouds obtained via various sensors. Benefiting from the labeled 3D spatial information, we can easily create new annotated 3D point clouds of different sensors of the same scene directly by considering the corresponding labels of 3D space the points located, which would be convenient for the validation and evaluation of algorithms related to point cloud interpretation and semantic segmentation.

For obtaining a full coverage of 3D scans in a large-scale urban area, the registration between point clouds acquired via terrestrial laser scanning (TLS) is normally mandatory. However, due to the complex urban environment, the automatic registration of different scans is still a challenging problem. In this work, we propose an automatic marker free method for fast and coarse registration between point clouds using the geometric constrains of planar patches under a voxel structure. Our proposed method consists of four major steps: the voxelization of the point cloud, the approximation of planar patches, the matching of corresponding patches, and the estimation of transformation parameters. In the voxelization step, the point cloud of each scan is organized with a 3D voxel structure, by which the entire point cloud is partitioned into small individual patches. In the following step, we represent points of each voxel with the approximated plane function, and select those patches resembling planar surfaces. Afterwards, for matching the corresponding patches, a RANSAC-based strategy is applied. Among all the planar patches of a scan, we randomly select a planar patches set of three planar surfaces, in order to build a coordinate frame via their normal vectors and their intersection points. The transformation parameters between scans are calculated from these two coordinate frames. The planar patches set with its transformation parameters owning the largest number of coplanar patches are identified as the optimal candidate set for estimating the correct transformation parameters. The experimental results using TLS datasets of different scenes reveal that our proposed method can be both effective and efficient for the coarse registration task. Especially, for the fast orientation between scans, our proposed method can achieve a registration error of less than around 2 degrees using the testing datasets, and much more efficient than the classical baseline methods.

Segmentation is the fundamental step for recognizing and extracting objects from point clouds of 3D scene. In this paper, we present a strategy for point cloud segmentation using voxel structure and graph-based clustering with perceptual grouping laws, which allows a learning-free and completely automatic but parametric solution for segmenting 3D point cloud. To speak precisely, two segmentation methods utilizing voxel and supervoxel structures are reported and tested. The voxel-based data structure can increase efficiency and robustness of the segmentation process, suppressing the negative effect of noise, outliers, and uneven points densities. The clustering of voxels and supervoxel is carried out using graph theory on the basis of the local contextual information, which commonly conducted utilizing merely pairwise information in conventional clustering algorithms. By the use of perceptual laws, our method conducts the segmentation in a pure geometric way avoiding the use of RGB color and intensity information, so that it can be applied to more general applications. Experiments using different datasets have demonstrated that our proposed methods can achieve good results, especially for complex scenes and nonplanar surfaces of objects. Quantitative comparisons between our methods and other representative segmentation methods also confirms the effectiveness and efficiency of our proposals.

Point cloud segmentation and classification is currently a research highlight. Methods in this field create labelled data, where each point has additional class information. Current approaches are to generate a graph on the basis of all points in the point cloud, calculate or learn descriptors and train a matcher for the descriptor to the corresponding classes. Since these approaches need to look on each point in the point cloud iteratively, they result in long calculation times for large point clouds. Therefore, large point clouds need a generalization, to save computation time. One kind of generalization is to cluster the raw points into a 3D grid structure, which is represented by small volume units ( i.e. voxels) used for further processing. This paper introduces a method to use such a voxel structure to cluster a large point cloud into ground and non-ground points. The proposed method for ground detection first marks ground voxels with a region growing approach. In a second step non ground voxels are searched and filtered in the ground segment to reduce effects of over-segmentations. This filter uses the probability that a voxel mostly consist of last pulses and a discrete gradient in a local neighbourhood . The result is the ground label as a first classification result and connected segments of non-ground points. The test area of the river Mangfall in Bavaria, Germany, is used for the first processing.

This paper discusses the automatic coregistration and fusion of 3d point clouds generated from aerial image sequences and corresponding thermal infrared (TIR) images. Both RGB and TIR images have been taken from a RPAS platform with a predefined flight path where every RGB image has a corresponding TIR image taken from the same position and with the same orientation with respect to the accuracy of the RPAS system and the inertial measurement unit. To remove remaining differences in the exterior orientation, different strategies for coregistering RGB and TIR images are discussed: (i) coregistration based on 2D line segments for every single TIR image and the corresponding RGB image. This method implies a mainly planar scene to avoid mismatches; (ii) coregistration of both the dense 3D point clouds from RGB images and from TIR images by coregistering 2D image projections of both point clouds; (iii) coregistration based on 2D line segments in every single TIR image and 3D line segments extracted from intersections of planes fitted in the segmented dense 3D point cloud; (iv) coregistration of both the dense 3D point clouds from RGB images and from TIR images using both ICP and an adapted version based on corresponding segmented planes; (v) coregistration of both image sets based on point features. The quality is measured by comparing the differences of the back projection of homologous points in both corrected RGB and TIR images.

Stereo camera systems in cars are often used to estimate the distance of other road users from the car. This information is important to improve road safety. Such camera systems are typically mounted behind the windshield of the car. In this contribution, the influence of the windshield on the estimated distance values is analyzed. An offline stereo camera calibration is performed with a moving planar calibration target. In a standard procedure bundle adjustment, the relative orientation of the cameras is estimated. The calibration is performed for the identical stereo camera system with and without a windshield in between. The base lengths are derived from the relative orientation in both cases and are compared. Distance values are calculated and analyzed. It can be shown, that the difference of the base length values in the two cases is highly significant. Resulting effects on the distance calculation up to a half meter occur.

Palm trees play an important role as they are widely used in a variety of products including oil and bio-fuel. Increasing demand and growing cultivation have created a necessity in planned farming and the monitoring different aspects like inventory keeping, health, size etc. The large cultivation regions of palm trees motivate the use of remote sensing to produce such data. This study proposes an object detection methodology on the aerial images, using shape feature for detecting and counting palm trees, which can support an inventory. The study uses circular autocorrelation of the polar shape matrix representation of an image, as the shape feature, and the linear support vector machine to standardize and reduce dimensions of the feature. Finally, the study uses local maximum detection algorithm on the spatial distribution of standardized feature to detect palm trees. The method was applied to 8 images chosen from different tough scenarios and it performed on average with an accuracy of 84% and 76.1%, despite being subjected to different challenging conditions in the chosen test images.

Today, cameras mounted in vehicles are used to observe the driver as well as the objects around a vehicle. In this article, an outline of a concept for image based recognition of dynamic traffic situations is shown. A dynamic traffic situation will be described by road users and their intentions. Images will be taken by a vehicle fleet and aggregated on a server. On these images, new strategies for machine learning will be applied iteratively when new data has arrived on the server. The results of the learning process will be models describing the traffic situation and will be transmitted back to the recording vehicles. The recognition will be performed as a standalone function in the vehicles and will use the received models. It can be expected, that this method can make the detection and classification of objects around the vehicles more reliable. In addition, the prediction of their actions for the next seconds should be possible. As one example how this concept is used, a method to recognize the illumination situation of a traffic scene is described. This allows to handle different appearances of objects depending on the illumination of the scene. Different illumination classes will be defined to distinguish different illumination situations. Intensity based features are extracted from the images and used by a classifier to assign an image to an illumination class. This method is being tested for a real data set of daytime and nighttime images. It can be shown, that the illumination class can be classified correctly for more than 80% of the images.

Texture mapping techniques are used to achieve a high degree of realism for computer generated large-scale and detailed 3D surface models by extracting the texture information from photographic images and applying it to the object surfaces. Due to the fact that a single image cannot capture all parts of the scene, a number of images should be taken. However, texturing the object surfaces from several images can lead to lighting variations between the neighboring texture fragments. In this paper we describe the creation of a textured 3D scene from overlapping aerial images using a Markov Random Field energy minimization framework. We aim to maximize the quality of the generated texture mosaic, preserving the resolution from the original images, and at the same time to minimize the seam visibilities between adjacent fragments. As input data we use a triangulated mesh of the city center of Munich and multiple camera views of the scene from different directions.

For many research applications like water resources evaluation, determination of glacier specific changes, and for calculation of the past and future contribution of glaciers to sea-level change, parameters about the size and spatial distribution of glaciers is crucial. In this paper, an automatic method for determination of glacier surface area using single track high resolution TerraSAR-X imagery by benefits of low resolution optical and thermal data is presented. Based on the normalized difference snow index (NDSI) and land surface temperature (LST) map generated from optical and thermal data combined with a surface slope data, a low resolution binary mask was derived used for the supervised classification of glacier using SAR imagery. Then, a set of suitable features is derived from the SAR intensity image, such as the texture information generated based on the gray level co-occurrence matrix (GLCM), and the intensity values. With these features, the glacier surface is discriminated from the background by Random Forests (RF) method.

This paper discusses the automatic texturing of building facades from thermal infrared image sequences. A fully automatic method is presented to refine GPS based positions estimating relative orientations of the image sequences including a given building model in a bundle adjustment process. The resulting refined orientation parameters are used to extract partial facade textures from all images and all sequences. The resulting partial textures of every sequence are combined to get complete facade textures in the thermal infrared domain. Textures from different image sequences are combined for object detection and extraction. These sequences are acquired either at different times for different radiometric thermal behavior of facade objects or with different viewing directions for objects located before or behind the facade plane.

The automatic analysis of 3D point clouds has become a crucial task in photogrammetry, remote sensing and computer vision. Whereas modern range cameras simultaneously provide both range and intensity images with high frame rates, other devices can be used to obtain further information which could be quite valuable for tasks such as object detection or scene interpretation. In particular thermal information offers many advantages, since people can easily be detected as heat sources in typical indoor or outdoor environments and, furthermore, a variety of concealed objects such as heating pipes as well as structural properties such as defects in isolation may be observed. In this paper, we focus on thermal 3D mapping which allows to observe the evolution of a dynamic 3D scene over time. We present a fully automatic methodology consisting of four successive steps: (i) a radiometric correction, (ii) a geometric calibration, (iii) a robust approach for detecting reliable feature correspondences and (iv) a co-registration of 3D point cloud data and thermal information via a RANSAC-based EPnP scheme. For an indoor scene, we demonstrate that our methodology outperforms other recent approaches in terms of both accuracy and applicability. We additionally show that efficient straightforward techniques allow a categorization according to background, people, passive scene manipulation and active scene manipulation.

Obtaining accurate 3d descriptions in the thermal infrared (TIR) is a quite challenging task due to the low geometric resolutions of TIR cameras and the low number of strong features in TIR images. Combining the radiometric information of the thermal infrared with 3d data from another sensor is able to overcome most of the limitations in the 3d geometric accuracy. In case of dynamic scenes with moving objects or a moving sensor system, a combination with RGB cameras of Time-of-Flight (TOF) cameras is suitable. As a TOF camera is an active sensor in the near infrared (NIR) and the thermal infrared camera captures the radiation emitted by the objects in the observed scene, the combination of these two sensors for close range applications is independent from external illumination or textures in the scene. This article is focused on the fusion of data acquired both with a time-of-flight (TOF) camera and a thermal infrared (TIR) camera. As the radiometric behaviour of many objects differs between the near infrared used by the TOF camera and the thermal infrared spectrum, a direct co-registration with feature points in both intensity images leads to a high number of outliers. A fully automatic workflow of the geometric calibration of both cameras and the relative orientation of the camera system with one calibration pattern usable for both spectral bands is presented. Based on the relative orientation, a fusion of the TOF depth image and the TIR image is used for scene segmentation and people detection. An adaptive histogram based depth level segmentation of the 3d point cloud is combined with a thermal intensity based segmentation. The feasibility of the proposed method is demonstrated in an experimental setup with different geometric and radiometric influences that show the benefit of the combination of TOF intensity and depth images and thermal infrared images.

Pose estimation is used for different applications like indoor positioning, simultaneous localization and mapping (SLAM), industrial measurement and robot calibration. For industrial applications several approaches dealing with the subject of pose estimation employ photogrammetric methods. Cameras which observe an object from a given point of view are utilized as well as cameras which are firmly mounted on the object that is to be oriented. Since it is not always possible to create an environment that the camera can observe the object, we concentrate on the latter option. A camera system shall be developed which is flexibly applicable in an indoor environment, and can cope with different occlusion situations, varying distances and density of reference marks. For this purpose in a first step a conception has been designed and a test scenario was created to evaluate different camera configurations and reference mark distributions. Both issues, the theoretical concept as well as the experimental setup are subject of this document.

Obtaining a 3D description of man-made and natural environments is a basic task in Computer Vision, Photogrammetry and Remote Sensing. New active sensors provide the possibility of capturing range information by images with a single measurement. With this new technique, image-based active ranging is possible which allows for capturing dynamic scenes, e.g. with moving pedestrians or moving vehicles. The currently available range imaging devices usually operate within the close-infrared domain to capture range and furthermore active and passive intensity images. Depending on the application, a 3D description with additional spectral information such as thermal-infrared data can be helpful and offers new opportunities for the detection and interpretation of human subjects and interactions. Therefore, thermal-infrared data combined with range information is promising. In this paper, an approach for mapping thermal-infrared data on range data is proposed. First, a camera calibration is carried out for the range imaging system (PMD[vision] CamCube 2.0) and the thermal-infrared system (InfraTec VarioCAM hr). Subsequently, a registration of close-infrared and thermal infrared intensity images derived from different sensor devices is performed. In this context, wavelength independent properties are selected in order to derive point correspondences between the different spectral domains. Finally, the thermal infrared images are enhanced with information derived from data acquired with the range imaging device and the enhanced IR texture is projected onto the respective 3D point cloud data for gaining appropriate infrared-textured 3D models. The feasibility of the proposed methodology is demonstrated for an experimental setup which is well-suited for investigating these proposed possibilities. Hence, the presented work is a first step towards the development of methods for combined thermal-infrared and range representation.

Generation and texturing of building models is a fast developing field of research. Several techniques have been developed to extract building geometry and textures from multiple images and image sequences. In this paper, these techniques are discussed and extended to automatically add new textures from infrared (IR) image sequences to existing building models. In contrast to existing work, geometry and textures are not generated together from the same dataset but the textures are extracted from the image sequence and matched to an existing geo-referenced 3D building model. The texture generation is divided in two main parts. The first part deals with the estimation and refinement of the exterior camera orientation. Feature points are extracted in the images and used as tie points in the sequence. A recorded exterior orientation of the camera s added to these homologous points and a bundle adjustment is performed starting on image pairs and combining the hole sequence. A given 3d model of the observed building is additionally added to introduce further constraint as ground control points in the bundle adjustment. The second part includes the extraction of textures from the images and the combination of textures from different images of the sequence. Using the reconstructed exterior camera orientation for every image of the sequence, the visible facades are projected into the image and texture is extracted. These textures normally contain only parts of the facade. The partial textures extracted from all images are combined to one facade texture. This texture is stored with a 3D reference to the corresponding facade. This allows searching for features in textures and localising those features in 3D space. It will be shown, that the proposed strategy allows texture extraction and mapping even for big building complexes with restricted viewing possibilities and for images with low optical resolution.

The façades of buildings are almost always organized according to Gestalt principles such as good continuation, repetition in similarity, or symmetry etc. Coding such principles in production systems yields a very flexible frame to explore the usefulness of such principles in automatic façade understanding. Capturing images and image sequences of façades in the thermal domain and understanding such data is of importance e.g. for energy saving. In this contribution two different production systems are compared using the same data and interpreter.

Thermal building textures can be used for detection of damaged and weak spots in the building structure. These textures can be extracted from airborne infrared (IR) image sequences by projecting the 3D building model into the images. However, the direct georeferencing is often not sufficiently accurate and the projected 3D model does not match the structures in the image. Thus we present a technique with the main goal to find the best fit between the existing 3D building model and the IR image sequence. For this purpose we developed a hierarchical approach consisting of two working stages. In the first stage we correct exterior orientation via line based matching. In the adjustment we consider both uncertainties: the ones of the model and the ones of the image. In the second stage we match each edge separately in its closest surrounding. Thanks to this approach a better fit between the 3D building model and the IR image was found. The originally unmodeled roof overlap was reconstructed.