Search

The implementation of multi-modal remote sensing image registration is still a challenge in various applications. In this paper, a novel framework called Heterogeneous SuperPoint Network is proposed for multi-modal remote sensing image registration. Heterogeneous SuperPoint Network operates on full-sized images and produces multi-modal interest points accompanied with fixed length descriptors in a single forward pass. Moreover, we develop a strategy for training a base detector that used in conjunction with heterogeneous homographic adaptation to generate pseudo-ground truth interest point labels for unlabeled images in a self-supervised fashion. Then, the interest points and point correspondences obtained by Heterogeneous SuperPoint Network are used to do the final matching with the nearest neighbor (NN) matching and Random sample consensus (RANSAC). Various image sets have been considered in this paper for the evaluation of the proposed approach. Experimental results demonstrate the superior performance of the proposed method over the existing design for the whole data set. [ABSTRACT FROM AUTHOR]

Automatic registration of multi-modal remote sensing data is a challenging task. Due to the significant nonlinear radiation difference and noise between multi-mode images, the register points extracted by traditional gradient algorithms (such as SIFT) have low repetition rate and poor feature similarity, which makes it difficult to register multi-mode images. To solve these problems, this paper proposes a method which is called Multi-modal Image Matching Based on Information Distribution Composite Feature (IDCF). Firstly, an adaptive information entropy map (AIEM) is proposed. AIEM not only describes the information distribution characteristics of the image, but also the distribution of the contour features of the image. Compared with traditional contour feature extraction operators, AIEM extraction results are clearer, more comprehensive and more detailed. In addition, AIEM is more robust to NRD compared to gradient, which is greatly affected by NRD. Then, IDCF extracts corner points on AIEM as feature points because corner points have better repeatability. After that, a composite feature description model based on maximum information index map (MIIM) and information trend graph (ITM) is defined. MIIM describes the main change direction of image information and ITM describes the overall change trend of image information. Finally, the similarity criterion SAD is used to match the feature points. The proposed IDCF aims to capture the structural similarity between images and has been tested with a variety of optical, Lidar, SAR, and map data. The results show that IDCF is robust against complex NRD outperforms the advanced algorithms (i.e. RIFT, PSO-SIFT and OS-SIFT) in matching performance. [ABSTRACT FROM AUTHOR]

The reliability of feature matching can decide the accuracy and robustness of the feature-based registration result. Aiming at the problem that the number of final feature matches preserved by many popular outlier removal methods is small, and the position accuracy of final feature matches is not high enough, we propose an accurate and robust image registration framework based on outlier removal and feature point adjustment in this paper. This framework increases the number and improves the position accuracy of inliers while eliminating most outliers. The increased number of inliers improves the robustness of image registration, and high accurate inliers improves the accuracy of image registration. Firstly, the initial feature matches are extracted by a commonly used feature-based registration method, such as the scale-invariant feature transform (SIFT)-based method. Then, outliers of the initial feature matches are eliminated by a frequency domain similarity measure, called PHase-based Structural SIMilarity (PH-SSIM) proposed in this paper. Considering the inherent error of the feature matches that still exist after the outlier elimination, a PH-SSIM-based feature point adjustment strategy is designed to fine-adjust the position of the preserved feature points in the reference image. Finally, the registration parameters are calculated by the fine-adjusted feature matches. The proposed framework has been evaluated by several remote sensing images with different resolution, grey-scale, texture, and scene, and compared with four state-of-the-art image registration methods. Experimental result demonstrates the high accuracy and robustness of the proposed method. [ABSTRACT FROM AUTHOR]

Applications based on the fusion of aerial imagery and Light Detection and Ranging (LiDAR) data have attracted more and more interest in recent years. Registration is the crucial prerequisite of data integration because of the spatial shift between two data modalities. However, there are many challenges for registration due to the different mechanisms of LiDAR and camera, especially in complex urban scenes. To address the problem, this paper proposes an automatic and robust registration method by using the centres and corner points of building boundaries as tie points. Primary registration is achieved by using centers of building boundaries as control points. To increase the robustness and accuracy of the registration, corner points of building boundaries are utilized to estimate the transformation model repeatedly. Two datasets of optical imagery and LiDAR data with different scene coverage are used to verify the performance of the proposed method. Experimental results show that the proposed method can achieve the registration accuracy of less than 2 pixels for both cases based on an assessment of check lines. This result can basically meet the requirements of data fusion. [ABSTRACT FROM AUTHOR]

Remote sensing image change detection (RSICD) is an essential measure for monitoring the earth's surface changes. In recent years, the explosive growth of very high-resolution (VHR) satellite sensors and the booming innovations in deep learning technology have significantly boosted RSICD development. However, most of the current RSICD models focus on locating accurate change areas while ignoring the efficiency of their method, which limits the practical application of RSICD models, especially for large-scale and emergency RSICD tasks. In this paper, we propose an Efficient Multi-scale-fusion Change Detection Network (EMS-CDNet) for bi-temporal RSICD tasks. Our EMS-CDNet pays more attention to the model's inference speed and the accuracy-efficiency trade-off rather than only pursuing detection accuracy. We designed a multi-scale fusion module for EMS-CDNet, which adopts multi-scale and multi-branch operations to extract multi-scale features simultaneously and aggregate features at different feature levels. In addition to EMS-CDNet's ability to achieve sufficient feature extraction, the multi-scale image input within the designed module alleviates the influence of image registration errors in practical applications, thereby strengthening EMS-CDNet's value for practical RSICD tasks. We also integrated a novel partition unit in EMS-CDNet to lighten the model while maintaining the detection ability of small targets, thus shortening its processing time without a severe accuracy decrease. We conducted experiments on two state-of-the-art (SOTA) public RSICD datasets and our own collected dataset. The public datasets were utilized to comparatively measure the overall accuracy and efficiency measurement of EMS-CDNet, and the dataset of images we collected was used to observe EMS-CDNet's performance under the influence of image registration errors. Our experimental results show that EMS-CDNet achieved a better accuracy-efficiency trade-off than the SOTA public datasets methods. For example, EMS-CDNet reduced the inference time by about 33% while maintaining identical detection accuracy to CLNet (the optimal method among the comparison methods). Furthermore, EMS-CDNet achieved higher accuracy on our collected dataset, with an F1 of 74% and mIoU of 0.806, demonstrating its robustness to image registration errors and showing its value for practical RSICD applications. [ABSTRACT FROM AUTHOR]

To obtain the complete coverage when scanning a large-scale scene using a laser scanner, registration of pairwise point clouds by the corresponding geometric features is usually necessary under the condition of insufficient direct geographic reference information of GNSS sensors. In this paper, we propose a target-less coarse registration method for pairwise point clouds using the corresponding voxel-based planes, which are identified by a pair of conjugate 2-plane bases. The voxel-based planes are firstly extracted from the entire point cloud with 3D cubic grids decomposed by the octree-based voxelization; Then, the 2-plane bases in each point cloud are constructed, and by comparing the dihedral angles of two 2-plane bases that come from the source and target point clouds, respectively, the conjugate 2-plane base pairs are generated one by one; Next, a set of plane correspondences is identified by a pair of conjugate 2-plane bases, and its corresponding largest consistency planes (LCP) set is calculated; Finally, a series of plane correspondence sets are obtained using the generated pairs of conjugate 2-plane bases, and the one with the highest LCP is used to compute the transformation matrix between the pairwise point clouds. Experimental results revealed that our proposed pairwise coarse registration method can be effective for aligning point clouds acquired from indoor and outdoor scenes, with rotation errors less than 0.4 degrees, translation errors less than 0.4 m, root mean square distance (RMSD) less than 0.42 m, and successful registration rate (SRR) about 98%. Furthermore, our proposed method was more efficient than the point- and line-based methods under the same hardware and software configuration conditions. [ABSTRACT FROM AUTHOR]

Earth observation using remote sensing data is a trending research field across the globe. In this perspective, image registration is a mandatory data processing step for any kind of time series data analytics. Feature-based image registration is one of the prominent categories for superimposing multi-temporal and multi-modal images over each other. The paper provides a comprehensive and comparative survey of feature detection/description techniques for remote sensing images. In addition, outlier removal algorithms are explored in detail to generate putative keypoint correspondences for accurate transformation parameter estimation. The experiments are conducted on multiple remote sensing image pairs comprising visible, Synthetic Aperture Radar (SAR) and infrared images that provide diverse characteristics of the feature target. The co-registration accuracy is quantified for all possible combinations of feature detection/description with outlier removal, and best amalgamation is visually verified to check sub-pixel spatial alignment by swiping multi-temporal or multi-modal images over each other. It has been found that SIFT performs better in optical image registration, whereas A-KAZE feature detection has an upper edge at SAR image registration task. Marginalizing Sample Consensus (MAGSAC) and Mode Guided (MG) based outlier removal techniques achieves better pruning performance for multi-temporal optical and SAR images respectively. The comparative evaluation indicates that Motion Smoothness Constraint (MSC) optimization shows optimal performance for multi-modal remote sensing images. The best possible Root Mean Square Error (RMSE) achieved for multi-temporal optical images is 0.44 pixel whereas for multi-temporal SAR images, it is 0.46 pixels. The RMSE achieved for multi-modal images is 0.48 pixel using combination of SIFT with MSC. [ABSTRACT FROM AUTHOR]

With increasing spatial resolution, the differences in nonlinear radiation and geometric properties between optical and synthetic aperture radar (SAR) images will be magnified, which will lead to lower registration accuracy. This paper addresses this problem and proposes an automatic registration method for HR optical and SAR images using the uniform optimized feature and the extended phase congruency (EPC) model. First, we combine two different gradient operators and a quadtree algorithm to improve the detection performance of the multi-scale Harris detector, especially for the repeatability and uniformity of feature points in optical and SAR images with large nonlinear radiation and severe speckle noise. Next, we use the EPC model with illumination and contrast invariance to generate the multi-scale orientation index map (MS-OIM) and the multi-orientation phase congruency amplitude map (MO-PCAM) and then build a novel GLOH-like local descriptor to obtain corresponding features based on the two feature maps above. Finally, the fast sample consensus (FSC) method is utilized to remove false matches and calculate the transformation parameters of sensed image based on the piecewise linear model. The detection results on synthetic data show that the proposed detector has better repeatability and uniformity than the other three classical detectors. The registration results obtained on real data indicate that our method is effective for optical and SAR images with speckle noise interference and nonlinear radiometric differences and achieves high registration accuracy. [ABSTRACT FROM AUTHOR]

Image matching is critical for photorealistic 3D reconstruction based on oblique images. An efficient match pair selection methodology for matching large-scale oblique UAV images is proposed in this paper. The proposed methodology effectively uses existing geospatial data and prior knowledge about data acquisition to generate precise match pairs. First, the principal point of each image is directly georeferenced. The direct georeferencing is based on a novel new DEM-aided direct georeferencing algorithm which employs an iterative binary search in the elevation domain. Second, a terrain-adaptive search radius is calculated for each image based on the calculated ground points, POS data, and prior knowledge. Finally, initial match pairs are generated based on a camera-oriented approach. The camera-oriented approach generates match pairs using k nearest neighbours (knn) search and prior knowledge. False connections in the initial match pairs are filtered using geometrical constraints. A limited number of match pairs are kept to reduce the redundancy of the generated match graph. The proposed methodology is tested on a dataset containing 28,560 images. The experimental results show that the precision of the generated match pairs is 99.99%. An incremental 3D reconstruction of the scene is conducted based on the robustly matched images. More than 99.8% of all the images are successfully oriented. The profiling analysis shows that the proposed methodology generates all the match pairs in 7.719 seconds. The precision, efficiency, and robustness of the proposed methodology are comprehensively analysed. The experimental and simulation results show that the proposed methodology is efficient and precise for large-scale UAV photogrammetry. [ABSTRACT FROM AUTHOR]

Imaging Satellites acquire multispectral images, MIs, in low resolution, LR, and panchromatic images, PANs, in high resolution, HR, due to some advantages provided in satellite design. The pansharpening, PS, is a super resolution image synthesis method that is used to generate the pansharpened image, PI, by fusion of PAN and MI. The use of PS process is unavoidable in applications such as efficient use of communication-bandwidth of imaging satellites and fusion of images derived from various image sensors. The PS process is a multi-step process consisting of various complex image processing stages, such as registration, resampling, synthesis, and fusion. The Weighted Differential Evolution Algorithm-based PS method, WDEPS, has been proposed in this paper. The WDEPS uses WDE to synthesize the intensity image, which is the blended-image of MI -bands. WDE has been used to compute the relevant image-blending weights, efficiently. In the experiments, several satellite images (QuickBird-2, Ikonos-2, and GeoEye-1) with different spatial resolutions were used. The WDEPS's experimental results have been compared with 17 well-known PS methods by using 9 full-reference and 3 blind image quality assessment metrics. The experimental results exposed that WDEPS generates high-quality PIs than traditional PS methods used in the experimental aspect of qualitative and quantitative assessment. [ABSTRACT FROM AUTHOR]

Feature matching in optical remote sensing images usually suffers from significant intensity differences and background variations. In this paper, an effective image matching approach is proposed to match salient structures in optical remote sensing images. The proposed method consists of three main steps. In the first step, EDLines algorithm is implemented to extract straight lines from two corresponding images, and these lines are further processed and divided into line segments for feature matching. In the second step, a robust descriptor namely line segment context (LSC) descriptor is computed for each line segment. An iterative procedure that combines line segment description and feature matching is presented for accurate matching. Finally, the consistency check is implemented to further refine the matching results. We conduct the experiments on pairs of optical remote sensing images, and experimental results show that the proposed method can automatically and robustly generate line segment features and achieve favourable performance compared to existing image matching approaches. [ABSTRACT FROM AUTHOR]

Over the last two decades, there has been a significant improvement in the quality and volume of the remote-sensing images. With the extensive availability of the images, these are used in many applications such as change detection, image mosaicing, and image fusion where image registration is very necessary. However, image registration is a challenging task in remote sensing due to the following reasons: geometric differences between images, intensity differences, and noise affect. Over the years, wide variety of algorithms have come into existence to handle these issues. In this paper, a comprehensive review is presented on these remote sensing image registration methods. At first, the general information of remote sensing image registration, its classifications, and application areas are provided. Then, an exhaustive review is given on the existing methods and their merits as well as shortcomings are mentioned. Finally, a comparative analysis is provided for the state-of-the-art methods and current challenges of this research are presented. [ABSTRACT FROM AUTHOR]

Features determined from keypoints or edges are widely used in multi-source image matching. However, for cases that exhibit non-linear intensity changes and significant noise, identifying sufficient identical features for multi-source image pairs can be complicated. Motivated by problems encountered in existing multi-source image matching algorithms, a robust and effective algorithm for multi-source image matching was proposed in this paper. First, keypoints uniformly and sufficiently distributed on the significant edge were extracted by a location-based boosting detector. Second, in order to effectively describe the corresponding region and reduce the influence of noise and non-linear intensity changes, a novel descriptor, denoted as the histogram of point-edge orientation (HPEO) was proposed for multi-source image matching. A bilateral matching process was then used to remove the incorrect matches. Experiments were performed with standard infrared-visible datasets and the results demonstrate that the proposed algorithm achieves a more accurate matching performance. [ABSTRACT FROM AUTHOR]

Automatic registration of multimodal remote sensing images, which is a critical prerequisite in a range of applications (e.g. image fusion, image mosaic, and image analysis), continues to be a fundamental and challenging problem. In this paper, we propose a novel extended phase correlation algorithm based on Log-Gabor filtering (LGEPC) for the registration of images with nonlinear radiometric differences and geometric differences (e.g. rotation, scale, and translation). Our algorithm focuses on two problems that the traditional extended phase correlation algorithms cannot well handle: 1) significant nonlinear radiometric differences and 2) large-scale differences between image pairs. After an over-complete multi-scale atlas space of the original image is built based on the filtered magnitudes obtained by using Log-Gabor filters with different central frequencies, the phase correlation of the single scale images is extended by LGEPC to atlases phase correlation, which is conducive to solving the problem of large scale and rotation differences between the image pairs. Subsequently, LGEPC eliminates the interface of the significant nonlinear radiometric differences by superimposing multi-scale geometric structural spectra and carrying out the phase correlation module, so that the translation can be well determined. Our experiments on synthetic images demonstrated the rationality and effectiveness of LGEPC, and the experiments on a variety of multimodal images confirmed that LGEPC can ideally achieve pixel-wise registration accuracy for multimodal image pairs that conform to the similarity transformation model. [ABSTRACT FROM AUTHOR]

To meet the high frame rate requirements of correct point correspondences with a sub-pixel precision, this paper first proposes a Field Programmable Gate Array (FPGA) architecture that consists of corner detection, corner matching, outlier rejection, and sub-pixel precision localisation. In the architecture, a combined Features from Accelerated Segment Test (FAST)+ Binary Robust Independent Elementary Features (BRIEF) algorithm is adopted for detection and matching with pixel precision, a combined algorithm of Slope-based Rejection (SR) and Correlation-Coefficient-based Rejection (CCR) is used to reject the outliers, and a gradient centroid-based algorithm is used for sub-pixel precision localisation. The whole FPGA architecture is implemented on a single FPGA platform (Xilinx XC72K325T). Five image datasets with different spatial resolutions, textures, lights, rotate angles, and viewpoints are used to evaluate the performance of the FPGA-based implementation. The experimental results show that (1) the SR and CCR algorithms are effective for outlier rejection; (2) a higher correct matching rate is achieved for the image pairs that cover artificial textures than for those that cover natural textures; (3) the proposed architecture is also suitable for image pairs with small change of lights, rotate angles, and viewpoints; (4) the speed of the FPGA-based implementation can reach 280 Frames Per Second (FPS), which is 35 times faster than the Personal Computer (PC)-based implementation; and (5) the usage of FPGA resources is acceptable for the selected FPGA platform. The speed and usage of the FPGA resources can be improved when the whole FPGA-based implementation is further optimised. [ABSTRACT FROM AUTHOR]

Many spectral matching algorithms, ranging from the traditional clustering techniques to the recent automated matching models, have evolved. This paper provides a review and up-to-date information on the past and current role of the spectral matching approaches adopted in hyperspectral satellite image processing. The need for spectral matching has been deliberated and a list of spectral matching algorithms has been compared and described. A review of the conventional spectral angle measures and the advanced automated spectral matching tools indicates that, for better performance of target detection, there is a need for combining two or more spectral matching techniques. From the studies of several authors, it is inferred that continuous improvement in the matching techniques over the past few years is due to the need to handle and analyse hyperspectral image data for various applications. The need to develop a well-built and specialized spectral library to accommodate the resources from enormous spectral data is suggested. This may improve accuracy in mineral and soil mapping, vegetation species identification and health monitoring, and target detection. The future role of cloud computing in accessing globally distributed spectral libraries and performing spectral matching is highlighted. Rather than inferring that a particular matching algorithm is the best, this paper points out the requirements of an ideal algorithm. With increasing usage of hyperspectral data for resources mapping, the review presented in this paper will certainly benefit the large and emerging community of hyperspectral image users. [ABSTRACT FROM AUTHOR]

Owing to significant geometric distortions and illumination differences, high precision and robust matching of multisource remote sensing images is a difficult task. To solve this, mutual information (MI)-based methods have been a preferred choice, as MI represents a measure of statistical dependence between the two images. However, MI only considers original grey information and neglects spatial information in the calculation of the probability distribution. In this paper, a novel similarity metric based on rotationally invariant regional mutual information (RIRMI) is proposed. The RIRMI metric is constructed by combining MI with a regional information based on the statistical relationship between rotationally invariant centre-symmetric local binary patterns of the images. The similarity metric based on RIRMI considers not only the spatial information, but the effect of the local grey variations and rotation changes on computing probability density function as well. The proposed method is tested on various simulated remote sensing images (5-30 m GSD) and real remote sensing images (2-30 m GSD) which are taken at different sensors, spectral bands, and times. Results verify that RIRMI is more robust and accurate than the common MI-based registration method. [ABSTRACT FROM AUTHOR]

A generalized search scheme for automatic registration of level-1 data of multiresolution and multi-sensor remote-sensing data was proposed. The robustness and time efficiency of the automatic registration process is critically dependant on the search scheme for identification of control points. The proposed method consists of three levels of search ranging from coarser to finer. This process was found to be capable of registering images to sub-pixel level which are independent of rotation and scale variations, and also translation that vary by few metres to kilometres. In order to reduce the low pass effect due to multiple transformations involved in the multi-level registration process, a direct correspondence between the reference image and target image was established so that a single resampling needs to be applied. This correspondence also helps to generate products at any desired pixel size and to keep the original resolution intact. In this scheme mutual information (MI) is used as a similarity measure and a non-rigid transformation, thin plate spline (TPS), is used for achieving sub-pixel registration accuracies. MI is found to be better for identification of match points even for images that are radiometrically nonlinear. Unlike global transformation methods, use of non-rigid transformations such as TPS achieves sub-pixel accuracy in the moderate hilly regions as well as high hilly regions where relief displacements are high, provided sufficient number of control points are generated. However TPS transformation demands accurate control points. A robust method for detection of inaccurate control points was adopted and explained in the paper. The scheme was tested on a number of combinations of remote sensing data of the same resolution and different resolution datasets, namely Cartosat-1 with Liss-4 of Resourcesat-1, Landsat Thematic Mapper (TM) with IRS-1C/1D and Cartosat-1 with Enhanced Thematic Mapper (ETM). The results are presented along with accuracies achieved. [ABSTRACT FROM PUBLISHER]

This paper presents a novel contour-based approach for multisource image registration. The contours are parameterized with Non-Uniform Rational B-Splines (NURBS). The control points of parametric contours are used as contour descriptor for image registration due to their invariance under affine and perspective transformations. The distance of control points, and the curvature and orientation similarity of the corresponding segments induced by the control points are considered as the matching criteria, and mismatching of control points can be avoided effectively because of the local controllability of NURBS. Therefore, the method is able to deal with the case in which the corresponding contours are locally distorted. Additionally, the NURBS description of contours has the strong global property; the method is therefore robust to image noise. In order to improve robustness, we perform the extraction and labelling of contours interactively. The experiments on both single-sensor and multisource data registration demonstrate the effectiveness and robustness of the presented method. [ABSTRACT FROM AUTHOR]

In remote sensing, because of the wide diversity of image characteristics (size, spatial and radiometric resolution, terrain relief, observation poses, etc.), image registration methods that work well on certain satellite images may not produce acceptable results for others, requiring more powerful techniques. A variety of registration techniques that account for images with non-rigid geometric deformations have been proposed, including piecewise (linear or cubic) functions, weighted mean functions, radial basis functions, B-spline functions, etc. This paper compares three of them: polynomial, piecewise-linear and thin-plate-spline functions, and analyses their performance under a variety of factors: off-nadir viewing, terrain relief, density of control points, and 3D geometric correction. Our comparison applies on panchromatic QuickBird imagery, both ortho-ready (as provided by DigitalGlobe) and orthorectified, acquired on different dates, from different observation attitudes, and sensing different land covers: urban area, high-relief terrain, and a combination of both. [ABSTRACT FROM AUTHOR]