Search

The high sensitivity and picosecond time resolution of single-photon avalanche diodes (SPADs) can improve the operational range and imaging accuracy of underwater detection systems. When an underwater SPAD imaging system is used to detect targets, backward-scattering caused by particles in water often results in the poor quality of the reconstructed underwater image. Although methods such as simple pixel accumulation have been proven to be effective for time–photon histogram reconstruction, they perform unsatisfactorily in a highly scattering environment. Therefore, new reconstruction methods are necessary for underwater SPAD detection to obtain high-resolution images. In this paper, we propose an algorithm that reconstructs high-resolution depth profiles of underwater targets from a time–photon histogram by employing the K-nearest neighbor (KNN) to classify multiple targets and the background. The results contribute to the performance of pixel accumulation and depth estimation algorithms such as pixel cross-correlation and ManiPoP. We use public experimental data sets and underwater simulation data to verify the effectiveness of the proposed algorithm. The results of our algorithm show that the root mean square errors (RMSEs) of land targets and simulated underwater targets are reduced by 57.12% and 23.45%, respectively, achieving high-resolution single-photon depth profile reconstruction.

The timely detection of falls and alerting medical aid is critical for health monitoring in elderly individuals living alone. This paper mainly focuses on issues such as poor adaptability, privacy infringement, and low recognition accuracy associated with traditional visual sensor-based fall detection. We propose an infrared video-based fall detection method utilizing spatial-temporal graph convolutional networks (ST-GCNs) to address these challenges. Our method used fine-tuned AlphaPose to extract 2D human skeleton sequences from infrared videos. Subsequently, the skeleton data was represented in Cartesian and polar coordinates and processed through a two-stream ST-GCN to recognize fall behaviors promptly. To enhance the network’s recognition capability for fall actions, we improved the adjacency matrix of graph convolutional units and introduced multi-scale temporal graph convolution units. To facilitate practical deployment, we optimized time window and network depth of the ST-GCN, striking a balance between model accuracy and speed. The experimental results on a proprietary infrared human action recognition dataset demonstrated that our proposed algorithm accurately identifies fall behaviors with the highest accuracy of 96%. Moreover, our algorithm performed robustly, identifying falls in both near-infrared and thermal-infrared videos.

Building height is a crucial indicator when studying urban environments and human activities, necessitating accurate, large-scale, and fine-resolution calculations. However, mainstream machine learning-based methods for inferring building heights face numerous challenges, including limited sample data and slow update frequencies. Alternatively, satellite laser altimetry technology offers a reliable means of calculating building heights with high precision. Here, we initially calculated building heights along satellite orbits based on building-rooftop contour vector datasets and ICESat-2 ATL03 photon data from 2019 to 2022. By integrating multi-source passive remote sensing observation data, we used the inferred building height results as reference data to train a random forest model, regressing building heights at a 10 m scale. Compared with ground-measured heights, building height samples constructed from ICESat-2 photon data outperformed methods that indirectly infer building heights using total building floor number. Moreover, the simulated building heights strongly correlated with actual observations at a single-city scale. Finally, using several years of inferred results, we analyzed building height changes in Tianjin from 2019 to 2022. Combined with the random forest model, the proposed model enables large-scale, high-precision inference of building heights with frequent updates, which has significant implications for global dynamic observation of urban three-dimensional features.

In recent years, underwater imaging and vision technologies have received widespread attention, and the removal of the backward-scattering interference caused by impurities in the water has become a long-term research focus for scholars. With the advent of new single-photon imaging devices, single-photon avalanche diode (SPAD) devices, with high sensitivity and a high depth resolution, have become cutting-edge research tools in the field of underwater imaging. However, the high production costs and small array areas of SPAD devices make it very difficult to conduct underwater SPAD imaging experiments. To address this issue, we propose a fast and effective underwater SPAD data simulation method and develop a denoising network for the removal of backward-scattering interference in underwater SPAD images based on deep learning and simulated data. The experimental results show that the distribution difference between the simulated and real underwater SPAD data is very small. Moreover, the algorithm based on deep learning and simulated data for the removal of backward-scattering interference in underwater SPAD images demonstrates effectiveness in terms of both metrics and human observation. The model yields improvements in metrics such as the PSNR, SSIM, and entropy of 5.59 dB, 9.03%, and 0.84, respectively, demonstrating its superior performance.

With the continuous evolution of autonomous driving and unmanned driving systems, traditional limitations such as a limited field-of-view, poor ranging accuracy, and real-time display are becoming inadequate to satisfy the requirements of binocular stereo-perception systems. Firstly, we designed a binocular stereo-imaging-perception system with a wide-field-of-view and infrared- and visible light-dual-band fusion. Secondly we proposed a binocular stereo-perception optical imaging system with a wide field-of-view of 120.3°, which solves the small field-of-view of current binocular stereo-perception systems. Thirdly, For image aberration caused by the wide-field-of-view system design, we propose an ellipsoidal-image-aberration algorithm with a low consumption of memory resources and no loss of field-of-view. This algorithm simultaneously solves visible light and infrared images with an aberration rate of 45% and 47%, respectively. Fourthly, a multi-scale infrared- and visible light-image-fusion algorithm is used, which improves the situational-awareness capabilities of a binocular stereo-sensing system in a scene and enhances image details to improve ranging accuracy. Furthermore, this paper is based on the Taylor model-calibration binocular stereo-sensing system of internal and external parameters for limit correction; the implemented algorithms are integrated into an NVIDIA Jetson TX2 + FPGA hardware framework, enabling near-distance ranging experiments. The fusion-ranging accuracy within 20 m achieved an error of 0.02 m, outperforming both visible light- and infrared-ranging methods. It generates the fusion-ranging-image output with a minimal delay of only 22.31 ms at a frame rate of 50 Hz.

In recent years, the range of applications that utilize multiband imaging has significantly expanded. However, it is difficult to utilize multichannel heterogeneous images to achieve a spectral complementarity advantage and obtain accurate depth prediction based on traditional systems. In this study, we investigate CFNet, an iterative prediction network, for disparity prediction with infrared and visible light images based on common features. CFNet consists of several components, including a common feature extraction subnetwork, context subnetwork, multimodal information acquisition subnetwork, and a cascaded convolutional gated recurrent subnetwork. It leverages the advantages of dual-band (infrared and visible light) imaging, considering semantic information, geometric structure, and local matching details within images to predict the disparity between heterogeneous image pairs accurately. CFNet demonstrates superior performance in recognized evaluation metrics and visual image observations when compared with other publicly available networks, offering an effective technical approach for practical heterogeneous image disparity prediction.

Real-time compression of images with a high dynamic range into those with a low dynamic range while preserving the maximum amount of detail is still a critical technology in infrared image processing. We propose a dynamic range compression and enhancement algorithm for infrared images with local optimal contrast (DRCE-LOC). The algorithm has four steps. The first involves blocking the original image to determine the optimal stretching coefficient by using the information of the local block. In the second, the algorithm combines the original image with a low-pass filter to create the background and detailed layers, compressing the background layer with a dynamic range of adaptive gain, and enhancing the detailed layer for the visual characteristics of the human eye. Third, the original image was used as input, the compressed background layer was used as a brightness-guided image, and the local optimal stretching coefficient was used for dynamic range compression. Fourth, an 8-bit image was created (from typical 14-bit input) by merging the enhanced details and the compressed background. Implemented on FPGA, it used 2.2554 Mb of Block RAM, five dividers, and a root calculator with a total image delay of 0.018 s. The study analyzed mainstream algorithms in various scenarios (rich scenes, small targets, and indoor scenes), confirming the proposed algorithm’s superiority in real-time processing, resource utilization, preservation of the image’s details, and visual effects.

The identification of color-coated steel sheet (CCSS) roof buildings in the external environment is of great significance for the operational security of high-speed rail systems. While high-resolution remote sensing images offer an efficient approach to identify CCSS roof buildings, achieving accurate extraction is challenging due to the complex background in remote sensing images and the extensive scale range of CCSS roof buildings. This research introduces the deformation-aware feature enhancement and alignment network (DFEANet) to address these challenges. DFEANet adaptively adjusts the receptive field to effectively separate the foreground and background facilitated by the deformation-aware feature enhancement module (DFEM). Additionally, feature alignment and gated fusion module (FAGM) is proposed to refine boundaries and preserve structural details, which can ameliorate the misalignment between adjacent features and suppress redundant information during the fusion process. Experimental results on remote sensing images along the Beijing–Zhangjiakou high-speed railway demonstrate the effectiveness of DFEANet. Ablation studies further underscore the enhancement in extraction accuracy due to the proposed modules. Overall, the DFEANet was verified as capable of assisting in the external environment security of high-speed rails.

To address the problem of water surface detection imaging equipment being susceptible to water surface glints, this study demonstrates a method called De-Glints for suppressing glints and obtaining clear underwater images using a division of focal plane (DoFP) polarimeter. Based on the principle of polarization imaging, the best polarization angle and the image corresponding to the minimal average gray level of each pixel are calculated. To evaluate the improvement in image quality, the index E was designed. The results of indoor and outdoor experiments show that the error of the angle calculation of this method is within 10%, and the minimum error is only 3%. The E index is positively improved and can be relatively improved by 8.00 under the interference of strong outdoor glints, and the method proposed in this paper shows a good adaptive ability to the dynamic scene.

In order to meet the fast and accurate automatic detection requirements of equipment maintenance in railway tunnels in the era of high-speed railways, as well as adapting to the high dynamic, low-illumination imaging environment formed by strong light at the tunnel exit, we propose an automatic inspection solution based on panoramic imaging and object recognition with deep learning. We installed a hyperboloid catadioptric panoramic imaging system on an inspection vehicle to obtain a large field of view as well as to shield the high dynamic phenomena at the tunnel exit, and proposed a YOLOv5-CCFE object detection model based on railway equipment recognition. The experimental results show that the mAP@0.5 value of the YOLOv5-CCFE model reaches 98.6%, and mAP@0.5:0.95 reaches 68.9%. The FPS value is 158, which can meet the automatic inspection requirements of railway tunnel equipment along the line and has high practical application value.

Currently, automatic optical zoom setups are being extensively explored for their applications in search, detection, recognition, and tracking. In visible and infrared fusion imaging systems with continuous zoom, dual-channel multi-sensor field-of-view matching control in the process of synchronous continuous zoom can be achieved by pre-calibration. However, mechanical and transmission errors of the zoom mechanism produce a small mismatch in the field of view after co-zooming, degrading the sharpness of the fusion image. Therefore, a dynamic small-mismatch detection method is necessary. This paper presents the use of edge-gradient normalized mutual information as an evaluation function of multi-sensor field-of-view matching similarity to guide the small zoom of the visible lens after continuous co-zoom and ultimately reduce the field-of-view mismatch. In addition, we demonstrate the use of the improved hill-climbing search algorithm for autozoom to obtain the maximum value of the evaluation function. Consequently, the results validate the correctness and effectiveness of the proposed method under small changes in the field of view. Therefore, this study is expected to contribute to the improvement of visible and infrared fusion imaging systems with continuous zoom, thereby enhancing the overall working of helicopter electro-optical pods, and early warning equipment.

With the development of multi-spectral imaging techniques, many new multi-spectral imaging devices have been developed in recent years. Red-green-blue and near-infrared (RGBN) cameras are widely used because they capture visible light and near-infrared light simultaneously, but they inevitably introduce color desaturation. Because there is clear multicollinearity among the RGBN channels, the ordinary least squares regression (OLSR) color correction method performs poorly. To solve color bias and multicollinearity, an RGBN camera color correction pipeline is proposed. A large number of nonlinear regression color correction methods that consist of combinations of four regression methods and nine nonlinear transforms are evaluated in this study. The results show that the proposed OLSR-based compound transform color correction method and partial least-squares regression (PLSR) based Gaussian-core transform color correction method yield better color correction results and are more robust. These approaches reduce the multicollinearity of the RGBN camera channels and will be a valuable reference in the development of RGBN imaging applications.

The interaction between a moving submerged body and a homogeneous/stratified fluid generates a Bernoulli hump, a Kelvin wake, and an internal wake on the water surface. The height distribution of wakes is directly related to the location and motion state of the submerged body. Thus, it is possible to retrieve kinematics information of a submarine from the images of wakes obtained by photoelectric detection equipment. In order to extract the submarine's location and velocity information effectively from the wake photoelectric images, we investigate the relationship between the motion state of moving submerged bodies and wakes in this paper. Bernoulli hump, Kelvin wake, and internal wake models are established based on potential flow theory and thin-ship approximation. We analyzed the wave crestline pattern and the energy characteristics of Kelvin wake components (divergent and transverse waves) and internal wakes at different velocities and diving depths. Finally, we propose a method for estimating the velocity of a submarine based on the wake wavelength and the diving depth inversion method based on the Fourier power spectrum of the Kelvin wake. The results obtained prove the feasibility of using photoelectric equipment to obtain wake images for use in analyzing the kinematics state of submarines, which are of guiding significance for detection and information processing in real-scale submarines.

RGBN cameras that can capture visible light and near-infrared (NIR) light simultaneously produce better color image quality in low-light-level conditions. However, these RGBN cameras introduce additional color bias caused by the mixing of visible information and NIR information. The color correction matrix model widely used in current commercial color digital cameras cannot handle the complicated mapping function between biased color and ground truth color. Convolutional neural networks (CNNs) are good at fitting such complicated relationships, but they require a large quantity of training image pairs of different scenes. In order to achieve satisfactory training results, large amounts of data must be captured manually, even when data augmentation techniques are applied, requiring significant time and effort. Hence, a data generation method for training pairs that are consistent with target RGBN camera parameters, based on an open access RGB-NIR dataset, is proposed. The proposed method is verified by training an RGBN camera color restoration CNN model with generated data. The results show that the CNN model trained with the generated data can achieve satisfactory RGBN color restoration performance with different RGBN sensors.

Typical energy-efficient dynamic grooming solutions may find a detour and longer path where delay from both the IP layer and optical layer is increased on extra nodes and links. In this paper, analysis of both delay and energy is made on typical grooming operations. An integer linear programming (ILP) model for minimization of both delay and energy is formulized and an alternative model is given with more delay consideration on the distribution over expected IP links. By considering the expected resource distribution of service requests, the available network resource can be re-priced by amortized cost over the expected distribution and a delay proactive control algorithm is designed over this distribution. A proactive delay heuristic is then proposed as an independent policy by a sorting function which is distinguished from other policies by cost functions on a virtual graph. By mining a frequent subset of IP links between node pairs, the problem is simplified without loss of statistical advantage over the resource distribution. The proposed grooming heuristic is performed on a simulation platform in terms of path delay, optical energy consumption, and electrical energy consumption. The simulation results show that the proposed grooming heuristic decreased average path delay without impacting energy performance. Different factors such as sliceability of a network resource, frequent degrees of expected resource, and grooming control periods, which affect the performance of the proposed heuristic, are compared to give thorough evaluations.

Generally, the dynamic range of night vision scenes is large. Owing to the limited dynamic range of traditional low light imaging technology, the captured images are always partially overexposed or underexposed. Multi-exposure fusion is the most effective method of overcoming the dynamic range limitation of sensor, and multi-frame low dynamic range (LDR) image fusion is a key consideration. However, existing fusion methods have problems such as image detail blurring and image aliasing. This paper proposes an image multi-scale decomposition method based on gradient domain guided filter (GDGF), which can better extract image details. The fusion algorithm adopts different fusion strategies for different scales. The low-frequency layer of the image uses a new weighted sparse representation (wSR) method, which can eliminate the image boundary problems and more adequately retain the image edges.

The dynamic range of night vision scenes is typically very large. Owing to the limited dynamic range of the traditional low-light-level imaging technology, the captured images are always partially overexposed or underexposed. Multi-exposure fusion is the most effective method for overcoming the dynamic range limitations of sensors. Recently, deep learning has achieved tremendous progress in many fields. However, only a few breakthroughs have been reported on high-dynamic image fusion with the deep learning method. Additionally, many problems have been reported in conjunctions with commonly used deep-learning methods. In this study, a high-dynamic image fusion algorithm is proposed based on the decomposition convolution neural network and weighted sparse representation. Based on image decomposition, the problem of the acquisition in training samples in network training can be solved. Therefore, the classification accuracy of the network can be improved. Additionally, the decomposition structure reduces the workload of each layer and improves the efficiency and quality of the image fusion outcome.

With the development of superframe high-dynamic-range infrared imaging technology that extends the dynamic range of thermal imaging systems, a key issue that has arisen is how to choose different integration times to obtain an HDR fusion image that contains more information. This paper proposes a multi-integration time adaptive method, in order to address the lack of objective evaluation methods for the selection of superframe infrared images, consisting of the following steps: image evaluation indicators are used to obtain the best global exposure image (the optimal integration time); images are segmented by region-growing point to obtain the ambient/high-temperature regions, selecting the local optimum images with grayscale closest to the medium grayscale of the IR imaging system for the two respective regions (lowest and highest integration time); finally, the three images above are fused and enhanced to achieve HDR infrared imaging. By comparing this method with some existing integration time selection methods and applying the proposed method to some typical fusion methods, via subjective and objective evaluation, the proposed method is shown to have obvious advantages over existing algorithms, and it can optimally select the images from different integration time series images to form the best combination that contains full image information, expanding the dynamic range of the IR imaging system.

Residual interpolations are effective methods to reduce the instantaneous field-of-view error of division of focal plane (DoFP) polarimeters. However, their guide-image selection strategies are improper, and do not consider the DoFP polarimeters’ spatial sampling modes. Thus, we propose a residual interpolation method with a new guide-image selection strategy based on the spatial layout of the pixeled polarizer array to improve the sampling rate of the guide image. The interpolation performance is also improved by the proposed pixel-by-pixel, adaptive iterative process and the weighted average fusion of the results of the minimized residual and minimized Laplacian energy guide filters. Visual and objective evaluations demonstrate the proposed method’s superiority to the existing state-of-the-art methods. The proposed method proves that considering the spatial layout of the pixeled polarizer array on the physical level is vital to improving the performance of interpolation methods for DoFP polarimeters.

Infrared focal plane arrays imaging system tend to be affected by the interior thermal radiation that drifts slowly over the time, resulting in the continuous spatial low-frequency nonuniformity noise in the image. In this paper, a scene-based nonuniformity correction method for interior thermal radiation is proposed. The gradient domain least mean square (LMS) algorithm is used to estimate the gradient caused by the interior radiation noise according to the fixed spatial low-frequency information in multiframe images. After that, the calibration parameters are obtained by fitting the derivatives of the polynomial-based correction model with the calculated gradient image. The performance of the proposed method is thoroughly studied with infrared image sequences with simulated nonuniformity and real nonuniformity. The results show that the proposed algorithm can remove interior radiation noise more completely without ghosting artifacts than the state-of-art algorithms and is more robust to different scenarios.

Contrast enhancement for infrared images is important in various night vision applications. However, existing local contrast enhancement algorithms often over-enhance smooth regions in outdoor infrared images. To address this limitation, this paper presents a contrast enhancement algorithm based on local gradient-grayscale statistical feature. The proposed algorithm first extracts such features from image sub-blocks, then classifies the sub-blocks as either simple or non-simple based on textural complexity using a model trained by a support vector machine, and subsequently adopts different grayscale mapping strategies to process the two types separately. Experimental results show that the proposed algorithm avoids overenhancing simple regions while effectively improving the contrast in regions with more details.

Optical imaging systems mounted on a Sun-synchronous satellite are probably disturbed by stray light when working in the space environment. Existing research has shown that the Visible Infrared Radiometer (VIRR) onboard the FY-3C satellite is affected by external solar stray light radiation when imaging the ground. In this paper, based on analyzing the solar stray light generation mechanism, we propose a simulation and analysis method combined with the given satellite orbit attitude model to investigate the influence of external solar stray light on VIRR’s imaging quality. We use the FY-3C orbit parameters to obtain the variation pattern of the angles between the solar vector and the payload. Based on the VIRR mechanical structure and optical scattering model, light tracing is performed to investigate the spatial distribution of irradiation on the primary mirror. The results of the occurrence time and intensity of stray light obtained by the simulation are consistent with the actual data when imaging the ground, which verifies that the proposed method is a correct and effective way to investigate the regularity of the external stray light of on-orbit payload.

Image intensifiers are used internationally as advanced military night-vision devices. They have better imaging performance in low-light-level conditions than CMOS/CCD. The intensified CMOS (ICMOS) was developed to satisfy the digital demand of image intensifiers. In order to make the ICMOS capable of color imaging in low-light-level conditions, a liquid-crystal tunable filter based color imaging ICMOS was developed. Due to the time-division color imaging scheme, motion artifacts may be introduced when a moving target is in the scene. To solve this problem, a deformable kernel prediction neural network (DKPNN) is proposed for joint denoising and motion artifact removal, and a data generation method which generates images with color-channel motion artifacts is also proposed to train the DKPNN. The results show that, compared with other denoising methods, the proposed DKPNN performed better both on generated noisy data and on real noisy data. Therefore, the proposed DKPNN is more suitable for color ICMOS denoising and motion artifact removal. A new exploration was made for low-light-level color imaging schemes.

The scanning infrared focal plane array (IRFPA) suffers from stripe-like non-uniformity due to the usage of many detectors, especially when working with a large time scale. Typical calibration systems tend to block the sensor aperture and expose the detectors to an on-board blackbody calibration source. They may also point at deep space. Full aperture calibration sources of this type tend to be large and expensive. To address these problems, a dynamic non-uniformity correction (NUC) method is proposed based on a modulated internal calibration device. By employing the on-board calibration device to generate a dynamic scene and fully integrating the system characteristics of the scanning IRFPA into the scene-based non-uniformity correction (SBNUC) algorithm, on-orbit high dynamic range NUC is achieved without blocking the field of view. Here we simulate an internal calibration system alternative, where a dynamic calibration signal is superimposed on the normal imagery, thus requiring no mechanisms and a smaller size. This method using this type of calibrator shows that when the sensor is pointing at deep space for calibration, it provides an effective non-uniformity correction of the imagery. After performing the proposed method, the NU of the two evaluation images was reduced from the initial 12.99% and 8.72% to less than 2%. Compared to other on-board NUC methods that require an extended reference blackbody source, this proposed approach has the advantages of miniaturization, a short calibration time, and strong adaptability.

The problems of the neural network-based nonuniformity correction algorithm for infrared focal plane arrays mainly concern slow convergence speed and ghosting artifacts. In general, the more stringent the inhibition of ghosting, the slower the convergence speed. The factors that affect these two problems are the estimated desired image and the learning rate. In this paper, we propose a learning rate rule that combines adaptive threshold edge detection and a temporal gate. Through the noise estimation algorithm, the adaptive spatial threshold is related to the residual nonuniformity noise in the corrected image. The proposed learning rate is used to effectively and stably suppress ghosting artifacts without slowing down the convergence speed. The performance of the proposed technique was thoroughly studied with infrared image sequences with both simulated nonuniformity and real nonuniformity. The results show that the deghosting performance of the proposed method is superior to that of other neural network-based nonuniformity correction algorithms and that the convergence speed is equivalent to the tested deghosting methods.

Cooled infrared detector arrays always suffer from undesired ripple residual nonuniformity (RNU) in sky scene observations. The ripple residual nonuniformity seriously affects the imaging quality, especially for small target detection. It is difficult to eliminate it using the calibration-based techniques and the current scene-based nonuniformity algorithms. In this paper, we present a modified temporal high-pass nonuniformity correction algorithm using fuzzy scene classification. The fuzzy scene classification is designed to control the correction threshold so that the algorithm can remove ripple RNU without degrading the scene details. We test the algorithm on a real infrared sequence by comparing it to several well-established methods. The result shows that the algorithm has obvious advantages compared with the tested methods in terms of detail conservation and convergence speed for ripple RNU correction. Furthermore, we display our architecture with a prototype built on a Xilinx Virtex-5 XC5VLX50T field-programmable gate array (FPGA), which has two advantages: (1) low resources consumption; and (2) small hardware delay (less than 10 image rows). It has been successfully applied in an actual system.