Search

Color correction across multiple images for consistency is a challenging problem in image mosaicking. To facilitate the global color optimization, existing methods mainly use less flexible correction models, e.g., linear or gamma function, which often struggle to cover the practically existing color discrepancy. In this paper, we present a novel color consistency correction method that models the color remapping function with a parameterized spline curve for each image. Thanks to the flexible model representation, on the one hand, our method has the capability to handle very challenging cases, i.e. hundreds of images with drastic color discrepancy; On the other, some important image properties, e.g., image gradient, contrast, and dynamic range, can be effective formulated with model parameters. Thus, the visual quality of individual images, along with the global color consistency, are comprehensively considered in our specifically designed cost function, which is solved in a closed form via convex quadratic programming. In addition, considering the possibly existing alteration objects within inter-image overlaps, we also propose an functional change detection algorithm with gradient and color features utilized, which guarantees the accuracy of the extracted color correspondences. We have tested the proposed approach on several challenging dataset of diverse genres, which shows that our method substantially outperforms state-of-the-art methods in both visual quality and quantitative metrics. [ABSTRACT FROM AUTHOR]

This paper introduces a new system for reconstructing 3D scenes from Line Segments (LS) on images. A new LS matching algorithm and a novel 3D LS reconstruction algorithm are incorporated into the system. Two coplanar cues that indicates image LSs are coplanar in physical (3D) space are extensively exploited in both algorithms: (1) adjacent image LSs are coplanar in space in a high possibility; (2) the projections of coplanar 3D LSs in two images are related by the same planar homography. Based on these two cues, we efficiently match LSs from two images firstly in pairs through matching the V-junctions formed by adjacent LSs, and secondly in individuals by exploiting local homographies. We extract for each V-junction a scale and affine invariant local region to match V-junctions from two images. The local homographies estimated from V-junction matches are used to match LSs in individuals. To get 3D LSs from the obtained LS matches, we propose to first estimate space planes from clustered LS matches and then back-project image LSs onto the space planes. Markov Random Field (MRF) is introduced to help more reliable LS match clustering. Experiments shows our LS matching algorithm significantly improves the efficiency of state-of-the-art methods while achieves comparable matching performance, and our 3D LS reconstruction algorithm generates more complete and detailed 3D scene models using much fewer images. [ABSTRACT FROM AUTHOR]

While mosaicking images, especially captured from the scenes of large depth differences with respective to cameras at varying locations, the detection of seamlines within overlap regions is a key issue for creating seamless and pleasant image mosaics. In this paper, we propose a novel algorithm to efficiently detect optimal seamlines for mosaicking aerial images captured from different viewpoints and for mosaicking street-view panoramic images without a precisely common center in a graph cuts energy minimization framework. To effectively ensure that the seamlines are optimally detected in the laterally continuous regions with high image similarity and low object dislocation to magnificently conceal the parallax between images, we fuse the information of image color, gradient magnitude, and texture complexity into the data and smooth energy terms in graph cuts. Different from the traditional frame-to-frame optimization for sequentially detecting seamlines for mosaicking multiple images, our method applies a novel multi-frame joint optimization strategy to find seamlines within multi-overlapped images at one time. In addition, we propose simple but effective strategies to semi-automatically guide the seamlines by exploiting simple human–computer interaction strongly constraining the image regions that the seamlines will or won’t pass through, which is often ignored by many existing seamline detection methods. Experimental results on a large set of aerial, oblique and street-view panoramic images show that the proposed method is capable of creating high-quality seamlines for multiple image mosaicking, while not crossing majority of visually obvious foreground objects and most of overlap regions with low image similarity to effectively conceal the image parallax at different extents. [ABSTRACT FROM AUTHOR]

The rapid development of augmented reality (AR), 3D reconstruction, simultaneous localization and mapping (SLAM), and autonomous driving requires off-the-shelf camera calibration solutions that are adaptable to cameras of different configurations in different complex scenarios. To this end, we propose a generic, robust, and accurate camera calibration framework, called Meta-Calib, by using single or multiple novel designed ArUco-encoded meta-board(s), which is dedicated to estimate accurate camera intrinsic parameters and extrinsic transformations of different multi-camera configurations. The ArUco calibration board has been redesigned to facilitate learning-based robust detection and obtain higher precision control point coordinates, which is termed the meta-board. This completely replaces the widely-used chessboard based on the corner extraction scheme to greatly alleviate the impact of image distortion on control points, especially when it is located at the boundary area of the fish-eye camera. A robust two-stage deep learning detection strategy is applied to reliably localize the ArUco-encoded inner coding region of the meta-board followed by identifying two categories of circular shapes representing "0" and "1" encoded in the ArUco pattern for decoding and orientation determination. The center points of circular shapes on the meta-board in the distorted image taken under the perspective view can be approximated through elliptical fitting with contour edges. The deviation between the fitting center points and ground-truth can be greatly suppressed when the refined sub-pixel contour edges extracted on the original image are projected to the orthographic projection view based on the camera intrinsic parameters, distortion coefficients and the prior information of the meta-board. Based on this observation, we propose a systematic iterative refinement approach to achieve the high-precision intrinsic calibration of a camera. This process involves improving the estimation of camera intrinsic parameters and fitting the center control points of circular shapes on the meta-boards in an iterative manner. The progressive nature of our approach permits reliably calibrate large distortion camera models under the presence of noisy measurements, which ensures good convergence. In addition, we also propose a graph-based multi-camera extrinsic calibration method via the corrected control points to reliably estimate both the relative poses of the meta-boards and cameras in the multi-camera system. The proposed method is not constrained by the number of cameras and meta-boards used, which makes our strategy accessible even with inflexible computer vision experts. Furthermore, we have derived the mathematical form for computing the covariance of the extrinsic transformation, which makes it possible to evaluate the uncertainty of the calibration results. Extensive experiments on a large number of both real and synthetic datasets, including perspective, fish-eye, and multiple overlapping cameras, are performed to prove the effectiveness and robustness of the developed Meta-Calib calibration framework. [ABSTRACT FROM AUTHOR]

The task of color consistency correction for multiple images mainly arises from applications like orthoimage producing, panoramic image stitching and 3D reconstruction. In these applications, images usually have been geometrically aligned. So correspondences can be easily extracted and used to solve color correction models. Almost all previous methods assume that the color residuals of correspondences follow Gaussian distribution and solve color models based on least squares. However, correspondences often contain unreliable ones due to altered areas and misalignments, which results in unusual large color residuals, namely, outliers. Imposing color consistency constaints on unreliable correspondences significantly affects the performance of color correction since Gaussian is highly sensitive to outliers. In this paper, to solve this problem theoretically, we first propose a unified probabilistic framework that formulates global color correction as a maximum posteriori probability (MAP) estimation. It is flexible enough to allow for any assumptions of residual distribution. And most color correction methods can be explained in this unified framework. Then, to robust against outliers, we use t -distribution with heavier tails than Gaussian to fit the color residuals. It is more robust because higher probabilities can be assigned to outliers. We show that the MAP formulation based on t -distribution actually leads to weighted least squares, which downweights outliers adaptively. Besides, our framework requires no user-defined robustness parameter. Because all parameters of color models and t -distribution are optimized jointly. In addition, to decrease the huge computational cost of large scale dataset, we extend the proposed framework to a parallel vesion which can achieve efficiency and global optimal at the same time. In the experiments, we compare our approach with the state-of-the-art approaches of Shen et al., Xia et al., etc. on several challenging datasets with outliers. The results demonstrate that our approach achieves the best robustness (average color consistency scores CD= 5.4 , DeltaE2000= 5.7 and PSNR= 24.0) and the best efficiency (given 100 images, non-parallel/parallel runs more than 5/50 times faster than others). The implementation is available at https://github.com/yinxuanLi/ColorConsistencyCorrectionForMultipleImages. [ABSTRACT FROM AUTHOR]

Structure from motion (SfM) recovers scene structures and camera poses based on feature matching, and faces challenges from ambiguous scenes. There are a large number of ambiguous scenes in real environment, which contain many duplicate structures and textures. The ambiguity leads to incorrect feature matches between images with similar appearance, and makes geometric misalignment in SfM. To address this problem, recent methods have focused on investigating the inconsistencies in feature topology among multi-view images. However, the feature topology is directly derived from 2D images. Thus, it is susceptible to feature occlusion caused by changes in perspective. Therefore, we propose a new method that disambiguates scenes using pose consistency rather than feature consistency. The pose consistency is conducted in 3D geometric space which is less sensitive to feature occlusion. Thus, the pose consistency is more robust than feature consistency. Our core motivation lies that the incorrect matches between ambiguous images will cause pose deviation from the global poses generated by correct matches. To detect this pose deviation, we first combine local and global information of the scene to generate the global reliable camera poses. The local information of each image is obtained by image clustering, and it strengthens the global information that is represented as the verified maximum spanning tree of clusters. Then, the global poses serve as the reference for further pose consistency verification. The global poses also enable us to perform both rotation and translation consistency verification for uncertain matches. During the pose consistency verification, the pose deviation calculated on image-level may be too small to be noticed. Thus, we propose to perform pose consistency verification at cluster-level instead of image-level to amplify the pose deviation. In the experiments, we compared our approach with several state-of-the-art methods, including COLMAP, Geodesic-SfM and TC-SfM, on both ambiguous and regular datasets. The results demonstrate that our approach achieves the best robustness, only our approach succeeds on all ambiguous image sequences (14/14). The quantitative evaluation results on image sequences with ground truth also show that our approach achieves the best accuracy (average RMSE of translation = 0.109, average RMSE of rotation = 0.827) among all methods. The source code of our approach is publicly available at https://github.com/gongyeted/MA-SfM. [ABSTRACT FROM AUTHOR]

This paper studies the problem of color consistency correction for sequential images with diverse color characteristics. Existing algorithms try to adjust all images to minimize color differences among images under a unified energy framework, however, the results are prone to presenting a consistent but unnatural appearance when the color difference between images is large and diverse. In our approach, this problem is addressed effectively by providing a guided initial solution for the global consistency optimization, which avoids converging to a meaningless integrated solution. First of all, to obtain the reliable intensity correspondences in overlapping regions between image pairs, we creatively propose the histogram extreme point matching algorithm which is robust to image geometrical misalignment to some extents. In the absence of the extra reference information, the guided initial solution is learned from the major tone of the original images by searching some image subset as the reference, whose color characteristics will be transferred to the others via the paths of graph analysis. Thus, the final results via global adjustment will take on a consistent color similar to the appearance of the reference image subset. Several groups of convincing experiments on both the synthetic dataset and the challenging real ones sufficiently demonstrate that the proposed approach can achieve as good or even better results compared with the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Three-dimensional (3D) building roof reconstruction from airborne LiDAR point clouds is an important task in photogrammetry and computer vision. To automatically reconstruct the 3D building models at Level of Detail 2 (LoD-2) from airborne LiDAR point clouds, the data-driven approaches usually need to be performed in two steps: geometric primitive extraction and roof structure inference. Obviously, the traditional approaches are not end-to-end, the accumulated errors in different stages cannot be avoided and the final 3D roof models may not be optimal. In addition, the results of 3D roof models largely depend on the accuracy of geometric primitives (planes, lines, etc.). To solve these problems, we present a deep learning-based approach to directly reconstruct building roofs from airborne LiDAR point clouds, named Point2Roof. In our method, we start by extracting the deep features for each input point using PointNet++. Then, we identify a set of candidate corner points from the input point clouds using the extracted deep features. In addition, we also regress the offset for each candidate corner point to refine their locations. After that, these candidates are clustered into a set of initial vertices, and we further refine their locations to obtain the final accurate vertices. Finally, we propose a Paired Point Attention (PPA) module to predict the true model edges from an exhaustive set of candidate edges between the vertices. Unlike traditional roof modeling approaches, the proposed Point2Roof is end-to-end. However, due to the lack of a building reconstruction dataset, we construct a large-scale synthetic dataset to verify the effectiveness and robustness of the proposed Point2Roof. The experimental results conducted on the synthetic benchmark demonstrate that the proposed Point2Roof significantly outperforms the traditional roof modeling approaches. The experiments also show that the network trained on the synthetic dataset can be applied to the real point clouds after fine-tuning the trained model on a small real dataset. The large-scale synthetic dataset, the small real dataset and the source code of our approach are publicly available in https://github.com/Li-Li-Whu/Point2Roof. [ABSTRACT FROM AUTHOR]

In multiple image mosaicking, color inconsistency is a common and severe problem that degrades the quality of composite images. To avoid the appearance of visible seams, we need to optimize the color consistency before mosaicking multiple images. To facilitate the global optimization framework, existing approaches mainly use less flexible global models, e.g., the linear or gamma model, to eliminate the color differences between multiple images. These models can effectively ensure that the global tones in multiple images are consistent. However, their ability to correct local color discrepancies is usually poor. In this paper, we propose a novel image color correction approach that can correct global and local color discrepancies simultaneously and preserve image gradient as much as possible. First, we design an effective and flexible color correction model to represent the color mapping function for each image. Instead of using the same global model to correct color discrepancies for all pixels in one image, we apply a series of local linear models for color correction. For different superpixel regions, different linear models are applied to model the mapping functions. Second, based on this model, a specific global cost function that considers both gradient preservation and color consistency is designed and solved. In addition, a global color constraint is fused into this cost function to ensure that the corrected images have the similar global tone. Thus, we can jointly optimize the global and local color consistency by minimizing this cost function. The experimental results on several challenging datasets captured by different sensors demonstrate that the proposed approach outperforms the state-of-the-art color correction approaches in both visual quality and quantitative metrics. [ABSTRACT FROM AUTHOR]

Abstract For multiple orthoimages mosaicking, the detection of an optimal seamline in an overlapped region and the generation of a seamline network are two key issues for creating a seamless and pleasant large-scale digital orthophoto map. In this paper, a novel system is proposed to generate the large-scale orthophoto by mosaicking multiple orthoimages via Graph cuts. The proposed system is comprised of two parts. In the first part, to ensure that the detected seamline avoids crossing the obvious objects, a novel foreground segmentation-based approach is proposed to detect the optimal seamline for two adjacent images. The foreground objects are segmented from the overlapped region at the superpixel level followed by the pixel-level seamline optimization. In the second part, we propose a novel seamline network generation approach to produce the large-scale orthophoto by mosaicking multiple orthoimages. The pairwise and junction regions extracted from the initial network are refined using two-label and multi-label Graph cuts, respectively. The key advantage of our proposed seamline network is that junction points can be automatically and optimally found using the multi-label Graph cuts. The experimental results on two groups of orthoimages show that our proposed system can generate high-quality seamline networks with less artifacts, and that it outperforms the state-of-the-art algorithm and the commercial software based on visual comparison and statistical evaluation. [ABSTRACT FROM AUTHOR]