Search

Over recent decades, Landsat satellite data has evolved into a highly valuable resource across diverse fields. Long-term satellite data records with integrity and consistency, such as the Landsat series, provide indispensable data for many applications. However, the malfunction of the Scan Line Corrector (SLC) on the Landsat 7 satellite in 2003 resulted in stripping in subsequent images, compromising the temporal consistency and data quality of Landsat time-series data. While various methods have been proposed to improve the quality of Landsat 7 SLC-off data, existing gap-filling methods fail to enhance the temporal resolution of reconstructed images, and spatiotemporal fusion methods encounter challenges in managing large-scale datasets. Therefore, we propose a method for reconstructing dense time series from SLC-off data. This method utilizes the Neighborhood Similar Pixel Interpolator to fill in missing values and leverages the time-series information to reconstruct high-resolution images. Taking the blue band as an example, the surface reflectance verification results show that the Mean Absolute Error (MAE) and BIAS reach minimum values of 0.0069 and 0.0014, respectively, with the Correlation Coefficient (CC) and Structural Similarity Index Metric (SSIM) reaching 0.93 and 0.94. The proposed method exhibits advantages in repairing SLC-off data and reconstructing dense time-series data, enabling enhanced remote sensing applications and reliable Earth's surface reflectance data reconstruction. [ABSTRACT FROM AUTHOR]

Monitoring ecosystems at regional or continental scales is paramount for biodiversity conservation, climate change mitigation, and sustainable land management. Effective monitoring requires satellite imagery with both high spatial resolution and high temporal resolution. However, there is currently no single, freely available data source that fulfills these needs. A seamless fusion of data from the Sentinel-3 and Sentinel-2 optical sensors could meet these monitoring requirements as Sentinel-2 observes at the required spatial resolution (10 m) while Sentinel-3 observes at the required temporal resolution (daily). We introduce the Efficient Fusion Algorithm across Spatio-Temporal scales (EFAST), which interpolates Sentinel-2 data into smooth time series (both spatially and temporally). This interpolation is informed by Sentinel-3's temporal profile such that the phenological changes occurring between two Sentinel-2 acquisitions at a 10 m resolution are assumed to mirror those observed at Sentinel-3's resolution. The EFAST consists of a weighted sum of Sentinel-2 images (weighted by a distance-to-clouds score) coupled with a phenological correction derived from Sentinel-3. We validate the capacity of our method to reconstruct the phenological profile at a 10 m resolution over one rangeland area and one irrigated cropland area. The EFAST outperforms classical interpolation techniques over both rangeland (−72% in the mean absolute error, MAE) and agricultural areas (−43% MAE); it presents a performance comparable to the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) (+5% MAE in both test areas) while being 140 times faster. The computational efficiency of our approach and its temporal smoothing enable the creation of seamless and high-resolution phenology products on a regional to continental scale. [ABSTRACT FROM AUTHOR]

Remote sensing data are becoming increasingly important for quantifying long-term changes in land surfaces. Optical sensors onboard satellite platforms face a tradeoff between temporal and spatial resolutions. Spatiotemporal fusion models can produce high spatiotemporal data, while existing models are not designed to produce moderate-spatial-resolution data, like Moderate-Resolution Imaging Spectroradiometer (MODIS), which has moderate spatial detail and frequent temporal coverage. This limitation arises from the challenge of combining coarse- and fine-spatial-resolution data, due to their large spatial resolution gap. This study presents a novel model, named multi-scale convolutional neural network for spatiotemporal fusion (MSCSTF), to generate MODIS-like data by addressing the large spatial-scale gap in blending the Advanced Very-High-Resolution Radiometer (AVHRR) and Landsat images. To mitigate the considerable biases between AVHRR and Landsat with MODIS images, an image correction module is included into the model using deep supervision. The outcomes show that the modeled MODIS-like images are consistent with the observed ones in five tested areas, as evidenced by the root mean square errors (RMSE) of 0.030, 0.022, 0.075, 0.036, and 0.045, respectively. The model makes reasonable predictions on reconstructing retrospective MODIS-like data when evaluating against Landsat data. The proposed MSCSTF model outperforms six other comparative models in accuracy, with regional average RMSE values being lower by 0.005, 0.007, 0.073, 0.062, 0.070, and 0.060, respectively, compared to the counterparts in the other models. The developed method does not rely on MODIS images as input, and it has the potential to reconstruct MODIS-like data prior to 2000 for retrospective studies and applications. [ABSTRACT FROM AUTHOR]

Medium- to high-resolution imagery is indispensable for various applications. Combining images from Landsat 8 and Sentinel-2 can improve the accuracy of observing dynamic changes on the Earth's surface. Many researchers use Sentinel-2 10 m resolution data in conjunction with Landsat 8 30 m resolution data to generate 10 m resolution data series. However, current fusion techniques have some algorithmic weaknesses, such as simple processing of coarse or fine images, which fail to extract image features to the fullest extent, especially in rapidly changing land cover areas. Facing the aforementioned limitations, we proposed a multiscale and attention mechanism-based residual spatiotemporal fusion network (MARSTFN) that utilizes Sentinel-2 10 m resolution data and Landsat 8 15 m resolution data as auxiliary data to upgrade Landsat 8 30 m resolution data to 10 m resolution. In this network, we utilized multiscale and attention mechanisms to extract features from coarse and fine images separately. Subsequently, the features outputted from all input branches are combined and further feature information is extracted through residual networks and skip connections. Finally, the features obtained from the residual network are merged with the feature information of the coarsely processed images from the multiscale mechanism to generate accurate prediction images. To assess the efficacy of our model, we compared it with existing models on two datasets. Results demonstrated that our fusion model outperformed baseline methods across various evaluation indicators, highlighting its ability to integrate Sentinel-2 and Landsat 8 data to produce 10 m resolution data. [ABSTRACT FROM AUTHOR]

The soil freeze/thaw (FT) state has emerged as a critical role in the ecosystem, hydrological, and biogeochemical processes, but obtaining representative soil FT state datasets with a long time sequence, fine spatial resolution, and high accuracy remains challenging. Therefore, we propose a decision-level spatiotemporal data fusion algorithm based on Convolutional Long Short-Term Memory networks (ConvLSTM) to expand the SMAP-enhanced L3 landscape freeze/thaw product (SMAP_E_FT) temporally. In the algorithm, the Freeze/Thaw Earth System Data Record product (ESDR_FT) is sucked in the ConvLSTM and fused with SMAP_E_FT at the decision level. Eight predictor datasets, i.e., soil temperature, snow depth, soil moisture, precipitation, terrain complexity index, area of open water data, latitude and longitude, are used to train the ConvLSTM. Direct validation using six dense observation networks located in the Genhe, Maqu, Naqu, Pali, Saihanba, and Shandian river shows that the fusion product (ConvLSTM_FT) effectively absorbs the high accuracy characteristics of ESDR_FT and expands SMAP_E_FT with an overall average improvement of 2.44% relative to SMAP_E_FT, especially in frozen seasons (averagely improved by 7.03%). The result from indirect validation based on categorical triple collocation also shows that ConvLSTM_FT performs stable regardless of land cover types, climate types, and terrain complexity. The findings, drawn from preliminary analyses on ConvLSTM_FT from 1980 to 2020 over China, suggest that with global warming, most parts of China suffer from different degrees of shortening of the frozen period. Moreover, in the Qinghai–Tibet region, the higher the permafrost thermal stability, the faster the degradation rate. [ABSTRACT FROM AUTHOR]

The coastal environment is characterized by high, multi-scale dynamics and the corresponding observations from a single remote sensing sensor are still facing challenges in achieving both high temporal and spatial resolution. This study proposed a spatiotemporal fusion model for coastal environments, which could fully enhance the efficiency of remote sensing data use and overcome the shortcomings of traditional spatiotemporal models that are insensitive to small-scale disturbances. The Enhanced Deep Super-Resolution Network (EDSR) was used to reconstruct spatial features in the lower spatial resolution GOCI-II data. The spatial features obtained instead of GOCI-II data were fed into the spatiotemporal fusion model, which enabled the fusion data to achieve an hour-by-hour observation of the water color and morphology information changes at 30 m resolution, including the changes in the spatial and temporal distributions of suspended particulate matter (SPM), the characterization of the vortex street caused by the bridge piers, the inundation process of the tidal flats, and coastline changes. In addition, this study analyzed the various factors affecting fusion accuracy, including spectral difference, errors in both temporal difference and location distance, and the structure of the EDSR model on the fusion accuracy. It is demonstrated that the location distance error and the spectral difference have the most significant impact on the fusion data, which may lead to the introduction of some ambiguous or erroneous spatial features. [ABSTRACT FROM AUTHOR]

Crop mapping is vital in ensuring food production security and informing governmental decision-making. The satellite-normalized difference vegetation index (NDVI) obtained during periods of vigorous crop growth is important for crop species identification. Sentinel-2 images with spatial resolutions of 10, 20, and 60 m are widely used in crop mapping. However, the images obtained during periods of vigorous crop growth are often covered by clouds. In contrast, time-series moderate-resolution imaging spectrometer (MODIS) images can usually capture crop phenology but with coarse resolution. Therefore, a time-series-based spatiotemporal fusion network (TSSTFN) was designed to generate TSSTFN-NDVI during critical phenological periods for finer-scale crop mapping. This network leverages multi-temporal MODIS-Sentinel-2 NDVI pairs from previous years as a reference to enhance the precision of crop mapping. The long short-term memory module was used to acquire data about the time-series change pattern to achieve this. The UNet structure was employed to manage the spatial mapping relationship between MODIS and Sentinel-2 images. The time distribution of the image sequences in different years was inconsistent, and time alignment strategies were used to process the reference data. The results demonstrate that incorporating the predicted critical phenological period NDVI consistently yields better crop classification performance. Moreover, the predicted NDVI trained with time-consistent data achieved a higher classification accuracy than the predicted NDVI trained with the original NDVI. [ABSTRACT FROM AUTHOR]

Over recent decades, Landsat satellite data has evolved into a highly valuable resource across diverse fields. Long-term satellite data records with integrity and consistency, such as the Landsat series, provide indispensable data for many applications. However, the malfunction of the Scan Line Corrector (SLC) on the Landsat 7 satellite in 2003 resulted in stripping in subsequent images, compromising the temporal consistency and data quality of Landsat time-series data. While various methods have been proposed to improve the quality of Landsat 7 SLC-off data, existing gap-filling methods fail to enhance the temporal resolution of reconstructed images, and spatiotemporal fusion methods encounter challenges in managing large-scale datasets. Therefore, we propose a method for reconstructing dense time series from SLC-off data. This method utilizes the Neighborhood Similar Pixel Interpolator to fill in missing values and leverages the time-series information to reconstruct high-resolution images. Taking the blue band as an example, the surface reflectance verification results show that the Mean Absolute Error (MAE) and BIAS reach minimum values of 0.0069 and 0.0014, respectively, with the Correlation Coefficient (CC) and Structural Similarity Index Metric (SSIM) reaching 0.93 and 0.94. The proposed method exhibits advantages in repairing SLC-off data and reconstructing dense time-series data, enabling enhanced remote sensing applications and reliable Earth’s surface reflectance data reconstruction.

As an important part of remote sensing data, weather radar plays an important role in convective weather forecasts to reduce extreme precipitation disasters. The existing radar echo extrapolation methods do not utilize the local natural characteristics of the radar echo effectively but only roughly extract the whole characteristics of the radar echo. To address these challenges, we design a spatiotemporal difference and generative adversarial fusion model (STDGAN). Specifically, a spatiotemporal difference module (STD) is designed to extract local weather patterns and model them in detail. In our model, spatiotemporal difference information and spatiotemporal features captured by the model itself are fused together. In addition, our model is trained in a generative adversarial network (GAN) framework; it helps to generate a clearer map of future radar echoes at the image level. The discriminator consists of multi-scale feature extractors, which can simulate weather models of various scales more completely. Finally, extrapolation experiments were conducted using actual radar echo data from Shijiazhuang and Nanjing. The experiments have shown that our model has a more accurate prediction performance for predicting local weather patterns and overall echo change trajectories compared with previous research models. Our model achieved MSE, PSNE, and SSIM values of 132.22, 37.87, and 0.796, respectively, on the Shijiazhuang radar echo dataset. In addition, our model also showed better performance results on the Nanjing radar echo dataset. The results show that the MSE was 49.570, the PSNR was 0.714, and the SSIM was 30.633. The CC value was 0.855. [ABSTRACT FROM AUTHOR]

The monitoring of rapidly changing land surface processes requires remote sensing images with high spatiotemporal resolution. As remote sensing satellites have different satellite orbits, satellite orbital velocities, and sensors, it is challenging to acquire remote sensing images with high resolution and dense time series within a reasonable temporal interval. Remote sensing spatiotemporal fusion is one of the effective ways to acquire high-resolution images with long time series. Most of the existing STF methods use artificially specified fusion strategies, resulting in blurry images and poor generalization ability. Additionally, some methods lack continuous time change information, leading to poor performance in capturing sharp changes in land covers. In this paper, we propose an adaptive multiscale network for spatiotemporal fusion (AMS-STF) based on a generative adversarial network (GAN). AMS-STF reconstructs high-resolution images by leveraging the temporal and spatial features of the input data through multiple adaptive modules and multiscale features. In AMS-STF, for the first time, deformable convolution is used for the STF task to solve the shape adaptation problem, allowing for adaptive adjustment of the convolution kernel based on the different shapes and types of land use. Additionally, an adaptive attention module is introduced in the networks to enhance the ability to perceive temporal changes. We conducted experiments comparing AMS-STF to the most widely used and innovative models currently available on three Landsat-MODIS datasets, as well as ablation experiments to evaluate some innovative modules. The results demonstrate that the adaptive modules significantly improve the fusion effect of land covers and enhance the clarity of their boundaries, which proves the effectiveness of AMS-STF. [ABSTRACT FROM AUTHOR]

Satellite-derived aerosol optical depth (AOD) has been extensively utilized for retrieving ground-level PM2.5 distributions. However, the presence of non-random missing data gaps in AOD poses a challenge to directly obtaining the gap-free AOD-derived PM2.5, thereby impeding accurate exposure risk assessment. Here, this study presents a novel and flexible framework that couples stacking and flexible spatiotemporal data fusion (FSDAF) approaches. By integrating multiple models and data sources, this framework aims to generate hourly (24-h) gap-free PM2.5 estimates for the Beijing–Tianjin–Hebei (BTH) region in 2018. This study effectively reconstructed data at least three times more effectively than the original AOD-derived PM2.5, achieving the Pearson coefficient (r), the coefficient determination (R2), root mean squared error (RMSE), and mean absolute error (MAE) values of 0.91, 0.84, 19.38 µg/m3, and 12.17 µg/m3, respectively, based on entire samples. Such strong predictive performance was also exhibited in spatial-based (r: 0.92–0.93, R2: 0.85–0.87, RMSE: 18.13 µg/m3–20.18 µg/m3, and MAE: 11.21 µg/m3–12.52 µg/m3) and temporal-based (r: 0.91–0.98, R2: 0.82–0.96, RMSE: 3.8 µg/m3–21.89 µg/m3, and MAE: 2.71 µg/m3–14.00 µg/m3) validations, indicating the robustness of this framework. Additionally, this framework enables the assessment of annual and seasonal PM2.5 concentrations and distributions, revealing that higher levels are experienced in the southern region, while lower levels prevail in the northern part. Winter exhibits the most severe levels, followed by spring and autumn, with comparatively lower levels in summer. Notably, the proposed framework effectively mitigates bias in calculating population-weighted exposure risk by filling data gaps with calculated values of 51.04 µg/m3, 54.17 µg/m3, 56.24 µg/m3, and 55.00 µg/m3 in Beijing, Tianjin, Hebei, and the BTH region, respectively. [ABSTRACT FROM AUTHOR]

In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels' convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies. [ABSTRACT FROM AUTHOR]

Monitoring ecosystems at regional or continental scales is paramount for biodiversity conservation, climate change mitigation, and sustainable land management. Effective monitoring requires satellite imagery with both high spatial resolution and high temporal resolution. However, there is currently no single, freely available data source that fulfills these needs. A seamless fusion of data from the Sentinel-3 and Sentinel-2 optical sensors could meet these monitoring requirements as Sentinel-2 observes at the required spatial resolution (10 m) while Sentinel-3 observes at the required temporal resolution (daily). We introduce the Efficient Fusion Algorithm across Spatio-Temporal scales (EFAST), which interpolates Sentinel-2 data into smooth time series (both spatially and temporally). This interpolation is informed by Sentinel-3’s temporal profile such that the phenological changes occurring between two Sentinel-2 acquisitions at a 10 m resolution are assumed to mirror those observed at Sentinel-3’s resolution. The EFAST consists of a weighted sum of Sentinel-2 images (weighted by a distance-to-clouds score) coupled with a phenological correction derived from Sentinel-3. We validate the capacity of our method to reconstruct the phenological profile at a 10 m resolution over one rangeland area and one irrigated cropland area. The EFAST outperforms classical interpolation techniques over both rangeland (−72% in the mean absolute error, MAE) and agricultural areas (−43% MAE); it presents a performance comparable to the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) (+5% MAE in both test areas) while being 140 times faster. The computational efficiency of our approach and its temporal smoothing enable the creation of seamless and high-resolution phenology products on a regional to continental scale.

The soil freeze/thaw (FT) state has emerged as a critical role in the ecosystem, hydrological, and biogeochemical processes, but obtaining representative soil FT state datasets with a long time sequence, fine spatial resolution, and high accuracy remains challenging. Therefore, we propose a decision-level spatiotemporal data fusion algorithm based on Convolutional Long Short-Term Memory networks (ConvLSTM) to expand the SMAP-enhanced L3 landscape freeze/thaw product (SMAP_E_FT) temporally. In the algorithm, the Freeze/Thaw Earth System Data Record product (ESDR_FT) is sucked in the ConvLSTM and fused with SMAP_E_FT at the decision level. Eight predictor datasets, i.e., soil temperature, snow depth, soil moisture, precipitation, terrain complexity index, area of open water data, latitude and longitude, are used to train the ConvLSTM. Direct validation using six dense observation networks located in the Genhe, Maqu, Naqu, Pali, Saihanba, and Shandian river shows that the fusion product (ConvLSTM_FT) effectively absorbs the high accuracy characteristics of ESDR_FT and expands SMAP_E_FT with an overall average improvement of 2.44% relative to SMAP_E_FT, especially in frozen seasons (averagely improved by 7.03%). The result from indirect validation based on categorical triple collocation also shows that ConvLSTM_FT performs stable regardless of land cover types, climate types, and terrain complexity. The findings, drawn from preliminary analyses on ConvLSTM_FT from 1980 to 2020 over China, suggest that with global warming, most parts of China suffer from different degrees of shortening of the frozen period. Moreover, in the Qinghai–Tibet region, the higher the permafrost thermal stability, the faster the degradation rate.

Medium- to high-resolution imagery is indispensable for various applications. Combining images from Landsat 8 and Sentinel-2 can improve the accuracy of observing dynamic changes on the Earth’s surface. Many researchers use Sentinel-2 10 m resolution data in conjunction with Landsat 8 30 m resolution data to generate 10 m resolution data series. However, current fusion techniques have some algorithmic weaknesses, such as simple processing of coarse or fine images, which fail to extract image features to the fullest extent, especially in rapidly changing land cover areas. Facing the aforementioned limitations, we proposed a multiscale and attention mechanism-based residual spatiotemporal fusion network (MARSTFN) that utilizes Sentinel-2 10 m resolution data and Landsat 8 15 m resolution data as auxiliary data to upgrade Landsat 8 30 m resolution data to 10 m resolution. In this network, we utilized multiscale and attention mechanisms to extract features from coarse and fine images separately. Subsequently, the features outputted from all input branches are combined and further feature information is extracted through residual networks and skip connections. Finally, the features obtained from the residual network are merged with the feature information of the coarsely processed images from the multiscale mechanism to generate accurate prediction images. To assess the efficacy of our model, we compared it with existing models on two datasets. Results demonstrated that our fusion model outperformed baseline methods across various evaluation indicators, highlighting its ability to integrate Sentinel-2 and Landsat 8 data to produce 10 m resolution data.

Remote sensing data are becoming increasingly important for quantifying long-term changes in land surfaces. Optical sensors onboard satellite platforms face a tradeoff between temporal and spatial resolutions. Spatiotemporal fusion models can produce high spatiotemporal data, while existing models are not designed to produce moderate-spatial-resolution data, like Moderate-Resolution Imaging Spectroradiometer (MODIS), which has moderate spatial detail and frequent temporal coverage. This limitation arises from the challenge of combining coarse- and fine-spatial-resolution data, due to their large spatial resolution gap. This study presents a novel model, named multi-scale convolutional neural network for spatiotemporal fusion (MSCSTF), to generate MODIS-like data by addressing the large spatial-scale gap in blending the Advanced Very-High-Resolution Radiometer (AVHRR) and Landsat images. To mitigate the considerable biases between AVHRR and Landsat with MODIS images, an image correction module is included into the model using deep supervision. The outcomes show that the modeled MODIS-like images are consistent with the observed ones in five tested areas, as evidenced by the root mean square errors (RMSE) of 0.030, 0.022, 0.075, 0.036, and 0.045, respectively. The model makes reasonable predictions on reconstructing retrospective MODIS-like data when evaluating against Landsat data. The proposed MSCSTF model outperforms six other comparative models in accuracy, with regional average RMSE values being lower by 0.005, 0.007, 0.073, 0.062, 0.070, and 0.060, respectively, compared to the counterparts in the other models. The developed method does not rely on MODIS images as input, and it has the potential to reconstruct MODIS-like data prior to 2000 for retrospective studies and applications.

Over one hundred spatiotemporal fusion algorithms have been proposed, but convolutional neural networks trained with large amounts of data for spatiotemporal fusion have not shown significant advantages. In addition, no attention has been paid to whether fused images can be used for change detection. These two issues are addressed in this work. A new dataset consisting of nine pairs of images is designed to benchmark the accuracy of neural networks using one-pair spatiotemporal fusion with neural-network-based models. Notably, the size of each image is significantly larger compared to other datasets used to train neural networks. A comprehensive comparison of the radiometric, spectral, and structural losses is made using fourteen fusion algorithms and five datasets to illustrate the differences in the performance of spatiotemporal fusion algorithms with regard to various sensors and image sizes. A change detection experiment is conducted to test if it is feasible to detect changes in specific land covers using the fusion results. The experiment shows that convolutional neural networks can be used for one-pair spatiotemporal fusion if the sizes of individual images are adequately large. It also confirms that the spatiotemporally fused images can be used for change detection in certain scenes. [ABSTRACT FROM AUTHOR]

To improve the accuracy and reliability of precipitation estimation, numerous models based on machine learning technology have been developed for integrating data from multiple sources. However, little attention has been paid to extracting the spatiotemporal correlation patterns between satellite products and rain gauge observations during the merging process. This paper focuses on this issue by proposing an integrated framework to generate an accurate and reliable spatiotemporal estimation of precipitation. The proposed framework integrates Funk-Singular Value Decomposition (F-SVD) in the recommender system to achieve the accurate spatial distribution of precipitation based on the spatiotemporal interpolation of rain gauge observations and Convolutional Long Short-Term Memory (ConvLSTM) to merge precipitation data from interpolation results and satellite observation through exploiting the spatiotemporal correlation pattern between them. The framework (FS-ConvLSTM) is utilized to obtain hourly precipitation merging data with a resolution of 0.1° in Jianxi Basin, southeast of China, from both rain gauge data and Global Precipitation Measurement (GPM) from 2006 to 2018. The LSTM and Inverse Distance Weighting (IDW) are constructed for comparison purposes. The results demonstrate that the framework could not only provide more accurate precipitation distribution but also achieve better stability and reliability. Compared with other models, it performs better in variation process description and rainfall capture capability, and the root mean square error (RSME) and probability of detection (POD) are improved by 63.6% and 22.9% from the original GPM, respectively. In addition, the merged precipitation combines the strength of different data while mitigating their weaknesses and has good agreement with observed precipitation in terms of magnitude and spatial distribution. Consequently, the proposed framework provides a valuable tool to improve the accuracy of precipitation estimation, which can have important implications for water resource management and natural disaster preparedness. [ABSTRACT FROM AUTHOR]

Due to the limitations of current technology and budget, as well as the influence of various factors, obtaining remote sensing images with high-temporal and high-spatial (HTHS) resolution simultaneously is a major challenge. In this paper, we propose the GAN spatiotemporal fusion model Based on multiscale and convolutional block attention module (CBAM) for remote sensing images (MCBAM-GAN) to produce high-quality HTHS fusion images. The model is divided into three stages: multi-level feature extraction, multi-feature fusion, and multi-scale reconstruction. First of all, we use the U-NET structure in the generator to deal with the significant differences in image resolution while avoiding the reduction in resolution due to the limitation of GPU memory. Second, a flexible CBAM module is added to adaptively re-scale the spatial and channel features without increasing the computational cost, to enhance the salient areas and extract more detailed features. Considering that features of different scales play an essential role in the fusion, the idea of multiscale is added to extract features of different scales in different scenes and finally use them in the multi loss reconstruction stage. Finally, to check the validity of MCBAM-GAN model, we test it on LGC and CIA datasets and compare it with the classical algorithm for spatiotemporal fusion. The results show that the model performs well in this paper. [ABSTRACT FROM AUTHOR]

The coastal environment is characterized by high, multi-scale dynamics and the corresponding observations from a single remote sensing sensor are still facing challenges in achieving both high temporal and spatial resolution. This study proposed a spatiotemporal fusion model for coastal environments, which could fully enhance the efficiency of remote sensing data use and overcome the shortcomings of traditional spatiotemporal models that are insensitive to small-scale disturbances. The Enhanced Deep Super-Resolution Network (EDSR) was used to reconstruct spatial features in the lower spatial resolution GOCI-II data. The spatial features obtained instead of GOCI-II data were fed into the spatiotemporal fusion model, which enabled the fusion data to achieve an hour-by-hour observation of the water color and morphology information changes at 30 m resolution, including the changes in the spatial and temporal distributions of suspended particulate matter (SPM), the characterization of the vortex street caused by the bridge piers, the inundation process of the tidal flats, and coastline changes. In addition, this study analyzed the various factors affecting fusion accuracy, including spectral difference, errors in both temporal difference and location distance, and the structure of the EDSR model on the fusion accuracy. It is demonstrated that the location distance error and the spectral difference have the most significant impact on the fusion data, which may lead to the introduction of some ambiguous or erroneous spatial features.

Crop mapping is vital in ensuring food production security and informing governmental decision-making. The satellite-normalized difference vegetation index (NDVI) obtained during periods of vigorous crop growth is important for crop species identification. Sentinel-2 images with spatial resolutions of 10, 20, and 60 m are widely used in crop mapping. However, the images obtained during periods of vigorous crop growth are often covered by clouds. In contrast, time-series moderate-resolution imaging spectrometer (MODIS) images can usually capture crop phenology but with coarse resolution. Therefore, a time-series-based spatiotemporal fusion network (TSSTFN) was designed to generate TSSTFN-NDVI during critical phenological periods for finer-scale crop mapping. This network leverages multi-temporal MODIS-Sentinel-2 NDVI pairs from previous years as a reference to enhance the precision of crop mapping. The long short-term memory module was used to acquire data about the time-series change pattern to achieve this. The UNet structure was employed to manage the spatial mapping relationship between MODIS and Sentinel-2 images. The time distribution of the image sequences in different years was inconsistent, and time alignment strategies were used to process the reference data. The results demonstrate that incorporating the predicted critical phenological period NDVI consistently yields better crop classification performance. Moreover, the predicted NDVI trained with time-consistent data achieved a higher classification accuracy than the predicted NDVI trained with the original NDVI.

The unmixing-based spatiotemporal fusion model is one of the effective ways to solve limitations in temporal and spatial resolution tradeoffs in a single satellite sensor. By using fusion data from different satellite platforms, high resolution in both temporal and spatial domains can be produced. However, due to the ill-posed characteristic of the unmixing function, the model performance may vary due to the different model setups. The key factors affecting the model stability most and how to set up the unmixing strategy for data downscaling remain unknown. In this study, we use the multisource land surface temperature as the case and focus on the three major factors to analyze the stability of the unmixing-based fusion model: (1) the definition of the homogeneous change regions (HCRs), (2) the unmixing levels, and (3) the number of HCRs. The spatiotemporal data fusion model U-STFM was used as the baseline model. The results show: (1) The clustering-based algorithm is more suitable for detecting HCRs for unmixing. Compared with the multi-resolution segmentation algorithm and k-means algorithm, the ISODATA clustering algorithm can more accurately describe LST's temporal and spatial changes on HCRs. (2) For the U-STFM model, applying the unmixing processing at the change ratio level can significantly reduce the additive and multiplicative noise of the prediction. (3) There is a tradeoff effect between the number of HCRs and the solvability of the linear unmixing function. The larger the number of HCRs (less than the available MODIS pixels), the more stable the model is. (4) For the fusion of the daily 30 m scale LST product, compared with STARFM and ESTARFM, the modified U-STFM (iso_USTFM) achieved higher prediction accuracy and a lower error ( R   2 : 0.87 and RMSE:1.09 k). With the findings of this study, daily fine-scale LST products can be predicted based on the unmixing-based spatial–temporal model with lower uncertainty and stable prediction. [ABSTRACT FROM AUTHOR]

Spatiotemporal fusion is an effective and cost-effective method to obtain both high temporal resolution and high spatial resolution images. However, existing methods do not sufficiently extract the deeper features of the image, resulting in fused images which do not recover good topographic detail and poor fusion quality. In order to obtain higher quality spatiotemporal fusion images, a novel spatiotemporal fusion method based on deep learning is proposed in this paper. The method combines an attention mechanism and a multiscale feature fusion network to design a network that more scientifically explores deeper features of the image for different input image characteristics. Specifically, a multiscale feature fusion module is introduced into the spatiotemporal fusion task and combined with an efficient spatial-channel attention module to improve the capture of spatial and channel information while obtaining more effective information. In addition, we design a new edge loss function and incorporate it into the compound loss function, which helps to generate fused images with richer edge information. In terms of both index performance and image details, our proposed model has excellent results on both datasets compared with the current mainstream spatiotemporal fusion methods. [ABSTRACT FROM AUTHOR]

It is still difficult to obtain high-resolution and fast-updated NDVI data, and spatiotemporal fusion is an effective means to solve this problem. The purpose of this study is to carry out the comparative analysis and comprehensive trade-off of spatiotemporal fusion models for NDVI generation and to provide references for scholars in this field. In this study, four spatiotemporal fusion models (STARFM, ESTARFM, FSDAF, and GF-SG) were selected to carry out NDVI image fusion in grassland, forest, and farmland test areas, and three indicators of root mean square error (RMSE), average difference (AD), and edge feature richness difference (EFRD) were used. A detailed evaluation and analysis of the fusion results and comprehensive trade-off were carried out. The results show that: (1) all four models can predict fine-resolution NDVI images well, but the phenomenon of over-smoothing generally exists, which is more serious in high-heterogeneity areas; (2) GF-SG performed well in the evaluation of the three indicators, with the highest comprehensive trade-off score (CTS) of 0.9658. Followed by ESTARFM (0.9050), FSDAF (0.8901), and STARFM (0.8789); (3) considering the comparative analysis and comprehensive trade-off results of the three test areas and the three indicators, among the four models, GF-SG has the best accuracy in generating NDVI images. GF-SG is capable of constructing NDVI time series data with high spatial and temporal resolution. [ABSTRACT FROM AUTHOR]

As an important part of remote sensing data, weather radar plays an important role in convective weather forecasts to reduce extreme precipitation disasters. The existing radar echo extrapolation methods do not utilize the local natural characteristics of the radar echo effectively but only roughly extract the whole characteristics of the radar echo. To address these challenges, we design a spatiotemporal difference and generative adversarial fusion model (STDGAN). Specifically, a spatiotemporal difference module (STD) is designed to extract local weather patterns and model them in detail. In our model, spatiotemporal difference information and spatiotemporal features captured by the model itself are fused together. In addition, our model is trained in a generative adversarial network (GAN) framework; it helps to generate a clearer map of future radar echoes at the image level. The discriminator consists of multi-scale feature extractors, which can simulate weather models of various scales more completely. Finally, extrapolation experiments were conducted using actual radar echo data from Shijiazhuang and Nanjing. The experiments have shown that our model has a more accurate prediction performance for predicting local weather patterns and overall echo change trajectories compared with previous research models. Our model achieved MSE, PSNE, and SSIM values of 132.22, 37.87, and 0.796, respectively, on the Shijiazhuang radar echo dataset. In addition, our model also showed better performance results on the Nanjing radar echo dataset. The results show that the MSE was 49.570, the PSNR was 0.714, and the SSIM was 30.633. The CC value was 0.855.

The monitoring of rapidly changing land surface processes requires remote sensing images with high spatiotemporal resolution. As remote sensing satellites have different satellite orbits, satellite orbital velocities, and sensors, it is challenging to acquire remote sensing images with high resolution and dense time series within a reasonable temporal interval. Remote sensing spatiotemporal fusion is one of the effective ways to acquire high-resolution images with long time series. Most of the existing STF methods use artificially specified fusion strategies, resulting in blurry images and poor generalization ability. Additionally, some methods lack continuous time change information, leading to poor performance in capturing sharp changes in land covers. In this paper, we propose an adaptive multiscale network for spatiotemporal fusion (AMS-STF) based on a generative adversarial network (GAN). AMS-STF reconstructs high-resolution images by leveraging the temporal and spatial features of the input data through multiple adaptive modules and multiscale features. In AMS-STF, for the first time, deformable convolution is used for the STF task to solve the shape adaptation problem, allowing for adaptive adjustment of the convolution kernel based on the different shapes and types of land use. Additionally, an adaptive attention module is introduced in the networks to enhance the ability to perceive temporal changes. We conducted experiments comparing AMS-STF to the most widely used and innovative models currently available on three Landsat-MODIS datasets, as well as ablation experiments to evaluate some innovative modules. The results demonstrate that the adaptive modules significantly improve the fusion effect of land covers and enhance the clarity of their boundaries, which proves the effectiveness of AMS-STF.

In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels’ convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies.

Satellite-derived aerosol optical depth (AOD) has been extensively utilized for retrieving ground-level PM2.5 distributions. However, the presence of non-random missing data gaps in AOD poses a challenge to directly obtaining the gap-free AOD-derived PM2.5, thereby impeding accurate exposure risk assessment. Here, this study presents a novel and flexible framework that couples stacking and flexible spatiotemporal data fusion (FSDAF) approaches. By integrating multiple models and data sources, this framework aims to generate hourly (24-h) gap-free PM2.5 estimates for the Beijing–Tianjin–Hebei (BTH) region in 2018. This study effectively reconstructed data at least three times more effectively than the original AOD-derived PM2.5, achieving the Pearson coefficient (r), the coefficient determination (R2), root mean squared error (RMSE), and mean absolute error (MAE) values of 0.91, 0.84, 19.38 µg/m3, and 12.17 µg/m3, respectively, based on entire samples. Such strong predictive performance was also exhibited in spatial-based (r: 0.92–0.93, R2: 0.85–0.87, RMSE: 18.13 µg/m3–20.18 µg/m3, and MAE: 11.21 µg/m3–12.52 µg/m3) and temporal-based (r: 0.91–0.98, R2: 0.82–0.96, RMSE: 3.8 µg/m3–21.89 µg/m3, and MAE: 2.71 µg/m3–14.00 µg/m3) validations, indicating the robustness of this framework. Additionally, this framework enables the assessment of annual and seasonal PM2.5 concentrations and distributions, revealing that higher levels are experienced in the southern region, while lower levels prevail in the northern part. Winter exhibits the most severe levels, followed by spring and autumn, with comparatively lower levels in summer. Notably, the proposed framework effectively mitigates bias in calculating population-weighted exposure risk by filling data gaps with calculated values of 51.04 µg/m3, 54.17 µg/m3, 56.24 µg/m3, and 55.00 µg/m3 in Beijing, Tianjin, Hebei, and the BTH region, respectively.

Spatiotemporal fusion (STF) is a solution to generate satellite images with both high-spatial and high-temporal resolutions. The deep learning-based STF algorithms focus on spatial dimensions to build a super-resolution (SR) model or the temporal dimensions to build a change prediction (CP) model, or the task itself to build a data-driven end-to-end model. The multi-source images used for STF usually have large spatial scale gaps and temporal spans. The large spatial scale gaps lead to poor spatial details based on a SR model; the large temporal spans make it difficult to accurately reconstruct changing areas based on a CP model. We propose a weighted dual-branch spatiotemporal fusion network based on complementarity between super-resolution and change prediction (WDBSTF), which includes the SR branch and CP branch, and a weight module representing the complementarity of the two branches. The SR branch makes full use of edge information and high-resolution reference images to obtain high-quality spatial features for image reconstruction. The CP branch decomposes complex problems via a two-layer cascaded network, changes features from the difference image, and selects high-quality spatial features through the attention mechanism. The fusion result of the CP branch has rich image details, but the fusion accuracy in the changing area is low due to the lack of detail. The SR branch has consistent and excellent fusion performances in the changing and no-changing areas, but the image details are not rich enough compared with the CP branch due to the large amplification factor. Next, a weighted network was designed to combine the advantages of the two branches to produce improved fusion results. We evaluated the performance of the WDBSTF in three representative scenarios, and both visual and quantitative evaluations demonstrate the state-of-the-art performance of our algorithm. (On the LGC dataset, our method outperforms the suboptimal method by 2.577% on SSIM. On the AHB dataset, our method outperforms the suboptimal method by 1.684% on SSIM. On the CIA dataset, our method outperforms the suboptimal method by 5.55% on SAM). [ABSTRACT FROM AUTHOR]

Remote sensing images with high temporal and spatial resolutions play a crucial role in land surface-change monitoring, vegetation monitoring, and natural disaster mapping. However, existing technical conditions and cost constraints make it very difficult to directly obtain remote sensing images with high temporal and spatial resolution. Consequently, spatiotemporal fusion technology for remote sensing images has attracted considerable attention. In recent years, deep learning-based fusion methods have been developed. In this study, to improve the accuracy and robustness of deep learning models and better extract the spatiotemporal information of remote sensing images, the existing multi-stream remote sensing spatiotemporal fusion network MSNet is improved using dilated convolution and an improved transformer encoder to develop an enhanced version called EMSNet. Dilated convolution is used to extract time information and reduce parameters. The improved transformer encoder is improved to further adapt to image-fusion technology and effectively extract spatiotemporal information. A new weight strategy is used for fusion that substantially improves the prediction accuracy of the model, image quality, and fusion effect. The superiority of the proposed approach is confirmed by comparing it with six representative spatiotemporal fusion algorithms on three disparate datasets. Compared with MSNet, EMSNet improved SSIM by 15.3% on the CIA dataset, ERGAS by 92.1% on the LGC dataset, and RMSE by 92.9% on the AHB dataset. [ABSTRACT FROM AUTHOR]

Popular deep-learning-based spatiotemporal fusion methods for creating high-temporal–high-spatial-resolution images have certain limitations. The reconstructed images suffer from insufficient retention of high-frequency information and the model suffers from poor robustness, owing to the lack of training datasets. We propose a dual-branch remote sensing spatiotemporal fusion network based on a selection kernel mechanism. The network model comprises a super-resolution network module, a high-frequency feature extraction module, and a difference reconstruction module. Convolution kernel adaptive mechanisms are added to the high-frequency feature extraction module and difference reconstruction module to improve robustness. The super-resolution module upgrades the coarse image to a transition image matching the fine image; the high-frequency feature extraction module extracts the high-frequency features of the fine image to supplement the high-frequency features for the difference reconstruction module; the difference reconstruction module uses the structural similarity for fine-difference image reconstruction. The fusion result is obtained by combining the reconstructed fine-difference image with the known fine image. The compound loss function is used to help network training. Experiments are carried out on three datasets and five representative spatiotemporal fusion algorithms are used for comparison. Subjective and objective evaluations validate the superiority of our proposed method. [ABSTRACT FROM AUTHOR]

Over one hundred spatiotemporal fusion algorithms have been proposed, but convolutional neural networks trained with large amounts of data for spatiotemporal fusion have not shown significant advantages. In addition, no attention has been paid to whether fused images can be used for change detection. These two issues are addressed in this work. A new dataset consisting of nine pairs of images is designed to benchmark the accuracy of neural networks using one-pair spatiotemporal fusion with neural-network-based models. Notably, the size of each image is significantly larger compared to other datasets used to train neural networks. A comprehensive comparison of the radiometric, spectral, and structural losses is made using fourteen fusion algorithms and five datasets to illustrate the differences in the performance of spatiotemporal fusion algorithms with regard to various sensors and image sizes. A change detection experiment is conducted to test if it is feasible to detect changes in specific land covers using the fusion results. The experiment shows that convolutional neural networks can be used for one-pair spatiotemporal fusion if the sizes of individual images are adequately large. It also confirms that the spatiotemporally fused images can be used for change detection in certain scenes.

With the development of multisource satellite platforms and the deepening of remote sensing applications, the growing demand for high-spatial resolution and high-temporal resolution remote sensing images has aroused extensive interest in spatiotemporal fusion research. However, reducing the uncertainty of fusion results caused by sensor inconsistencies and input data preprocessing is one of the challenges in spatiotemporal fusion algorithms. Here, we propose a novel sensor bias correction method to correct the input data of the spatiotemporal fusion model through a machine learning technique learning the bias between different sensors. Taking the normalized difference vegetation index (NDVI) images with low-spatial resolution (MODIS) and high-spatial resolution (Landsat) as the basic data, we generated the neighborhood gray matrices from the MODIS image and established the image bias pairs of MODIS and Landsat. The light gradient boosting machine (LGBM) regression model was used for the nonlinear fitting of the bias pairs to correct MODIS NDVI images. For three different landscape areas with various spatial heterogeneities, the fusion of the bias-corrected MODIS NDVI and Landsat NDVI was conducted by using the spatiotemporal adaptive reflection fusion model (STARFM) and the flexible spatiotemporal data fusion method (FSDAF), respectively. The results show that the sensor bias correction method can enhance the spatially detailed information in the input data, significantly improve the accuracy and robustness of the spatiotemporal fusion technology, and extend the applicability of the spatiotemporal fusion models. [ABSTRACT FROM AUTHOR]

Due to the limitations of current technology and budget, as well as the influence of various factors, obtaining remote sensing images with high-temporal and high-spatial (HTHS) resolution simultaneously is a major challenge. In this paper, we propose the GAN spatiotemporal fusion model Based on multiscale and convolutional block attention module (CBAM) for remote sensing images (MCBAM-GAN) to produce high-quality HTHS fusion images. The model is divided into three stages: multi-level feature extraction, multi-feature fusion, and multi-scale reconstruction. First of all, we use the U-NET structure in the generator to deal with the significant differences in image resolution while avoiding the reduction in resolution due to the limitation of GPU memory. Second, a flexible CBAM module is added to adaptively re-scale the spatial and channel features without increasing the computational cost, to enhance the salient areas and extract more detailed features. Considering that features of different scales play an essential role in the fusion, the idea of multiscale is added to extract features of different scales in different scenes and finally use them in the multi loss reconstruction stage. Finally, to check the validity of MCBAM-GAN model, we test it on LGC and CIA datasets and compare it with the classical algorithm for spatiotemporal fusion. The results show that the model performs well in this paper.

Landsat has provided the longest fine resolution data archive of Earth's environment since 1972; however, one of the challenges in using Landsat data for various applications is its frequent large data gaps and heavy cloud contaminations. One pressing research topic is to generate the regular time series by integrating coarse-resolution satellite data through data fusion techniques. This study presents a novel spatiotemporal fusion (STF) method based on a depthwise separable convolutional neural network (DSC), namely, STFDSC, to generate Landsat-surface reflectance time series at 8-day intervals by fusing Landsat 30 m with high-quality Moderate Resolution Imaging Spectroradiometer (MODIS) 500 m surface reflectance data. The STFDSC method consists of three main stages: feature extraction, feature fusion and prediction. Features were first extracted from Landsat and MODIS surface reflectance changes, and the extracted multilevel features were then stacked and fused. Both low-level and middle-level features that were generally ignored in convolutional neural network (CNN)-based fusion models were included in STFDSC to avoid key information loss and thus ensure high prediction accuracy. The prediction stage generated a Landsat residual image and is combined with original Landsat data to obtain predictions of Landsat imagery at the target date. The performance of STFDSC was evaluated in the Greater Khingan Mountains (GKM) in Northeast China and the Ziwuling (ZWL) forest region in Northwest China. A comparison of STFDSC with four published fusion methods, including two classic fusion methods (FSDAF, ESTARFM) and two machine learning methods (EDCSTFN and STFNET), was also carried out. The results showed that STFDSC made stable and more accurate predictions of Landsat surface reflectance than other methods in both the GKM and ZWL regions. The root-mean-square-errors (RMSEs) of TM bands 2, 3, 4, and 7 were 0.0046, 0.0038, 0.0143, and 0.0055 in GKM, respectively, and 0.0246, 0.0176, 0.0280, and 0.0141 in ZWL, respectively; it can be potentially used for generating the global surface reflectance and other high-level land products. [ABSTRACT FROM AUTHOR]

The unmixing-based spatiotemporal fusion model is one of the effective ways to solve limitations in temporal and spatial resolution tradeoffs in a single satellite sensor. By using fusion data from different satellite platforms, high resolution in both temporal and spatial domains can be produced. However, due to the ill-posed characteristic of the unmixing function, the model performance may vary due to the different model setups. The key factors affecting the model stability most and how to set up the unmixing strategy for data downscaling remain unknown. In this study, we use the multisource land surface temperature as the case and focus on the three major factors to analyze the stability of the unmixing-based fusion model: (1) the definition of the homogeneous change regions (HCRs), (2) the unmixing levels, and (3) the number of HCRs. The spatiotemporal data fusion model U-STFM was used as the baseline model. The results show: (1) The clustering-based algorithm is more suitable for detecting HCRs for unmixing. Compared with the multi-resolution segmentation algorithm and k-means algorithm, the ISODATA clustering algorithm can more accurately describe LST’s temporal and spatial changes on HCRs. (2) For the U-STFM model, applying the unmixing processing at the change ratio level can significantly reduce the additive and multiplicative noise of the prediction. (3) There is a tradeoff effect between the number of HCRs and the solvability of the linear unmixing function. The larger the number of HCRs (less than the available MODIS pixels), the more stable the model is. (4) For the fusion of the daily 30 m scale LST product, compared with STARFM and ESTARFM, the modified U-STFM (iso_USTFM) achieved higher prediction accuracy and a lower error (R 2: 0.87 and RMSE:1.09 k). With the findings of this study, daily fine-scale LST products can be predicted based on the unmixing-based spatial–temporal model with lower uncertainty and stable prediction.

Spatiotemporal fusion is an effective and cost-effective method to obtain both high temporal resolution and high spatial resolution images. However, existing methods do not sufficiently extract the deeper features of the image, resulting in fused images which do not recover good topographic detail and poor fusion quality. In order to obtain higher quality spatiotemporal fusion images, a novel spatiotemporal fusion method based on deep learning is proposed in this paper. The method combines an attention mechanism and a multiscale feature fusion network to design a network that more scientifically explores deeper features of the image for different input image characteristics. Specifically, a multiscale feature fusion module is introduced into the spatiotemporal fusion task and combined with an efficient spatial-channel attention module to improve the capture of spatial and channel information while obtaining more effective information. In addition, we design a new edge loss function and incorporate it into the compound loss function, which helps to generate fused images with richer edge information. In terms of both index performance and image details, our proposed model has excellent results on both datasets compared with the current mainstream spatiotemporal fusion methods.

It is still difficult to obtain high-resolution and fast-updated NDVI data, and spatiotemporal fusion is an effective means to solve this problem. The purpose of this study is to carry out the comparative analysis and comprehensive trade-off of spatiotemporal fusion models for NDVI generation and to provide references for scholars in this field. In this study, four spatiotemporal fusion models (STARFM, ESTARFM, FSDAF, and GF-SG) were selected to carry out NDVI image fusion in grassland, forest, and farmland test areas, and three indicators of root mean square error (RMSE), average difference (AD), and edge feature richness difference (EFRD) were used. A detailed evaluation and analysis of the fusion results and comprehensive trade-off were carried out. The results show that: (1) all four models can predict fine-resolution NDVI images well, but the phenomenon of over-smoothing generally exists, which is more serious in high-heterogeneity areas; (2) GF-SG performed well in the evaluation of the three indicators, with the highest comprehensive trade-off score (CTS) of 0.9658. Followed by ESTARFM (0.9050), FSDAF (0.8901), and STARFM (0.8789); (3) considering the comparative analysis and comprehensive trade-off results of the three test areas and the three indicators, among the four models, GF-SG has the best accuracy in generating NDVI images. GF-SG is capable of constructing NDVI time series data with high spatial and temporal resolution.

Spatiotemporal fusion (STF) is a solution to generate satellite images with both high-spatial and high-temporal resolutions. The deep learning-based STF algorithms focus on spatial dimensions to build a super-resolution (SR) model or the temporal dimensions to build a change prediction (CP) model, or the task itself to build a data-driven end-to-end model. The multi-source images used for STF usually have large spatial scale gaps and temporal spans. The large spatial scale gaps lead to poor spatial details based on a SR model; the large temporal spans make it difficult to accurately reconstruct changing areas based on a CP model. We propose a weighted dual-branch spatiotemporal fusion network based on complementarity between super-resolution and change prediction (WDBSTF), which includes the SR branch and CP branch, and a weight module representing the complementarity of the two branches. The SR branch makes full use of edge information and high-resolution reference images to obtain high-quality spatial features for image reconstruction. The CP branch decomposes complex problems via a two-layer cascaded network, changes features from the difference image, and selects high-quality spatial features through the attention mechanism. The fusion result of the CP branch has rich image details, but the fusion accuracy in the changing area is low due to the lack of detail. The SR branch has consistent and excellent fusion performances in the changing and no-changing areas, but the image details are not rich enough compared with the CP branch due to the large amplification factor. Next, a weighted network was designed to combine the advantages of the two branches to produce improved fusion results. We evaluated the performance of the WDBSTF in three representative scenarios, and both visual and quantitative evaluations demonstrate the state-of-the-art performance of our algorithm. (On the LGC dataset, our method outperforms the suboptimal method by 2.577% on SSIM. On the AHB dataset, our method outperforms the suboptimal method by 1.684% on SSIM. On the CIA dataset, our method outperforms the suboptimal method by 5.55% on SAM).

Remote sensing images with high temporal and spatial resolutions play a crucial role in land surface-change monitoring, vegetation monitoring, and natural disaster mapping. However, existing technical conditions and cost constraints make it very difficult to directly obtain remote sensing images with high temporal and spatial resolution. Consequently, spatiotemporal fusion technology for remote sensing images has attracted considerable attention. In recent years, deep learning-based fusion methods have been developed. In this study, to improve the accuracy and robustness of deep learning models and better extract the spatiotemporal information of remote sensing images, the existing multi-stream remote sensing spatiotemporal fusion network MSNet is improved using dilated convolution and an improved transformer encoder to develop an enhanced version called EMSNet. Dilated convolution is used to extract time information and reduce parameters. The improved transformer encoder is improved to further adapt to image-fusion technology and effectively extract spatiotemporal information. A new weight strategy is used for fusion that substantially improves the prediction accuracy of the model, image quality, and fusion effect. The superiority of the proposed approach is confirmed by comparing it with six representative spatiotemporal fusion algorithms on three disparate datasets. Compared with MSNet, EMSNet improved SSIM by 15.3% on the CIA dataset, ERGAS by 92.1% on the LGC dataset, and RMSE by 92.9% on the AHB dataset.

Popular deep-learning-based spatiotemporal fusion methods for creating high-temporal–high-spatial-resolution images have certain limitations. The reconstructed images suffer from insufficient retention of high-frequency information and the model suffers from poor robustness, owing to the lack of training datasets. We propose a dual-branch remote sensing spatiotemporal fusion network based on a selection kernel mechanism. The network model comprises a super-resolution network module, a high-frequency feature extraction module, and a difference reconstruction module. Convolution kernel adaptive mechanisms are added to the high-frequency feature extraction module and difference reconstruction module to improve robustness. The super-resolution module upgrades the coarse image to a transition image matching the fine image; the high-frequency feature extraction module extracts the high-frequency features of the fine image to supplement the high-frequency features for the difference reconstruction module; the difference reconstruction module uses the structural similarity for fine-difference image reconstruction. The fusion result is obtained by combining the reconstructed fine-difference image with the known fine image. The compound loss function is used to help network training. Experiments are carried out on three datasets and five representative spatiotemporal fusion algorithms are used for comparison. Subjective and objective evaluations validate the superiority of our proposed method.

With the development of multisource satellite platforms and the deepening of remote sensing applications, the growing demand for high-spatial resolution and high-temporal resolution remote sensing images has aroused extensive interest in spatiotemporal fusion research. However, reducing the uncertainty of fusion results caused by sensor inconsistencies and input data preprocessing is one of the challenges in spatiotemporal fusion algorithms. Here, we propose a novel sensor bias correction method to correct the input data of the spatiotemporal fusion model through a machine learning technique learning the bias between different sensors. Taking the normalized difference vegetation index (NDVI) images with low-spatial resolution (MODIS) and high-spatial resolution (Landsat) as the basic data, we generated the neighborhood gray matrices from the MODIS image and established the image bias pairs of MODIS and Landsat. The light gradient boosting machine (LGBM) regression model was used for the nonlinear fitting of the bias pairs to correct MODIS NDVI images. For three different landscape areas with various spatial heterogeneities, the fusion of the bias-corrected MODIS NDVI and Landsat NDVI was conducted by using the spatiotemporal adaptive reflection fusion model (STARFM) and the flexible spatiotemporal data fusion method (FSDAF), respectively. The results show that the sensor bias correction method can enhance the spatially detailed information in the input data, significantly improve the accuracy and robustness of the spatiotemporal fusion technology, and extend the applicability of the spatiotemporal fusion models.

Landsat has provided the longest fine resolution data archive of Earth’s environment since 1972; however, one of the challenges in using Landsat data for various applications is its frequent large data gaps and heavy cloud contaminations. One pressing research topic is to generate the regular time series by integrating coarse-resolution satellite data through data fusion techniques. This study presents a novel spatiotemporal fusion (STF) method based on a depthwise separable convolutional neural network (DSC), namely, STFDSC, to generate Landsat-surface reflectance time series at 8-day intervals by fusing Landsat 30 m with high-quality Moderate Resolution Imaging Spectroradiometer (MODIS) 500 m surface reflectance data. The STFDSC method consists of three main stages: feature extraction, feature fusion and prediction. Features were first extracted from Landsat and MODIS surface reflectance changes, and the extracted multilevel features were then stacked and fused. Both low-level and middle-level features that were generally ignored in convolutional neural network (CNN)-based fusion models were included in STFDSC to avoid key information loss and thus ensure high prediction accuracy. The prediction stage generated a Landsat residual image and is combined with original Landsat data to obtain predictions of Landsat imagery at the target date. The performance of STFDSC was evaluated in the Greater Khingan Mountains (GKM) in Northeast China and the Ziwuling (ZWL) forest region in Northwest China. A comparison of STFDSC with four published fusion methods, including two classic fusion methods (FSDAF, ESTARFM) and two machine learning methods (EDCSTFN and STFNET), was also carried out. The results showed that STFDSC made stable and more accurate predictions of Landsat surface reflectance than other methods in both the GKM and ZWL regions. The root-mean-square-errors (RMSEs) of TM bands 2, 3, 4, and 7 were 0.0046, 0.0038, 0.0143, and 0.0055 in GKM, respectively, and 0.0246, 0.0176, 0.0280, and 0.0141 in ZWL, respectively; it can be potentially used for generating the global surface reflectance and other high-level land products.

Remote sensing satellite images with a high spatial and temporal resolution play a crucial role in Earth science applications. However, due to technology and cost constraints, it is difficult for a single satellite to achieve both a high spatial resolution and high temporal resolution. The spatiotemporal fusion method is a cost-effective solution for generating a dense temporal data resolution with a high spatial resolution. In recent years, spatiotemporal image fusion based on deep learning has received wide attention. In this article, a spatiotemporal fusion method based on multiscale feature extraction and a spatial channel attention mechanism is proposed. Firstly, the method uses a multiscale mechanism to fully utilize the structural features in the images. Then a novel attention mechanism is used to capture both spatial and channel information; finally, the rich features and spatial and channel information are used to fuse the images. Experimental results obtained from two datasets show that the proposed method outperforms existing fusion methods in both subjective and objective evaluations.

Dense time-series remote sensing data with detailed spatial information are highly desired for the monitoring of dynamic earth systems. Due to the sensor tradeoff, most remote sensing systems cannot provide images with both high spatial and temporal resolutions. Spatiotemporal image fusion models provide a feasible solution to generate such a type of satellite imagery, yet existing fusion methods are limited in predicting rapid and/or transient phenological changes. Additionally, a systematic approach to assessing and understanding how varying levels of temporal phenological changes affect fusion results is lacking in spatiotemporal fusion research. The objective of this study is to develop an innovative hybrid deep learning model that can effectively and robustly fuse the satellite imagery of various spatial and temporal resolutions. The proposed model integrates two types of network models: super-resolution convolutional neural network (SRCNN) and long short-term memory (LSTM). SRCNN can enhance the coarse images by restoring degraded spatial details, while LSTM can learn and extract the temporal changing patterns from the time-series images. To systematically assess the effects of varying levels of phenological changes, we identify image phenological transition dates and design three temporal phenological change scenarios representing rapid, moderate, and minimal phenological changes. The hybrid deep learning model, alongside three benchmark fusion models, is assessed in different scenarios of phenological changes. Results indicate the hybrid deep learning model yields significantly better results when rapid or moderate phenological changes are present. It holds great potential in generating high-quality time-series datasets of both high spatial and temporal resolutions, which can further benefit terrestrial system dynamic studies. The innovative approach to understanding phenological changes’ effect will help us better comprehend the strengths and weaknesses of current and future fusion models.

Remote sensing products with high temporal and spatial resolution can be hardly obtained under the constrains of existing technology and cost. Therefore, the spatiotemporal fusion of remote sensing images has attracted considerable attention. Spatiotemporal fusion algorithms based on deep learning have gradually developed, but they also face some problems. For example, the amount of data affects the model’s ability to learn, and the robustness of the model is not high. The features extracted through the convolution operation alone are insufficient, and the complex fusion method also introduces noise. To solve these problems, we propose a multi-stream fusion network for remote sensing spatiotemporal fusion based on Transformer and convolution, called MSNet. We introduce the structure of the Transformer, which aims to learn the global temporal correlation of the image. At the same time, we also use a convolutional neural network to establish the relationship between input and output and to extract features. Finally, we adopt the fusion method of average weighting to avoid using complicated methods to introduce noise. To test the robustness of MSNet, we conducted experiments on three datasets and compared them with four representative spatiotemporal fusion algorithms to prove the superiority of MSNet (Spectral Angle Mapper (SAM) < 0.193 on the CIA dataset, erreur relative global adimensionnelle de synthese (ERGAS) < 1.687 on the LGC dataset, and root mean square error (RMSE) < 0.001 on the AHB dataset).

Spatiotemporal fusion (STF) is considered a feasible and cost-effective way to deal with the trade-off between the spatial and temporal resolution of satellite sensors, and to generate satellite images with high spatial and high temporal resolutions. This is achieved by fusing two types of satellite images, i.e., images with fine temporal but rough spatial resolution, and images with fine spatial but rough temporal resolution. Numerous STF methods have been proposed, however, it is still a challenge to predict both abrupt landcover change, and phenological change, accurately. Meanwhile, robustness to radiation differences between multi-source satellite images is crucial for the effective application of STF methods. Aiming to solve the abovementioned problems, in this paper we propose a hybrid deep learning-based STF method (HDLSFM). The method formulates a hybrid framework for robust fusion with phenological and landcover change information with minimal input requirements, and in which a nonlinear deep learning-based relative radiometric normalization, a deep learning-based superresolution, and a linear-based fusion are combined to address radiation differences between different types of satellite images, landcover, and phenological change prediction. Four comparative experiments using three popular STF methods, i.e., spatial and temporal adaptive reflectance fusion model (STARFM), flexible spatiotemporal data fusion (FSDAF), and Fit-FC, as benchmarks demonstrated the effectiveness of the HDLSFM in predicting phenological and landcover change. Meanwhile, HDLSFM is robust for radiation differences between different types of satellite images and the time interval between the prediction and base dates, which ensures its effectiveness in the generation of fused time-series data.

With the growing development of remote sensors, huge volumes of remote sensing data are being utilized in related applications, bringing new challenges to the efficiency and capability of processing huge datasets. Spatiotemporal remote sensing data fusion can restore high spatial and high temporal resolution remote sensing data from multiple remote sensing datasets. However, the current methods require long computing times and are of low efficiency, especially the newly proposed deep learning-based methods. Here, we propose a fast three-dimensional convolutional neural network-based spatiotemporal fusion method (STF3DCNN) using a spatial-temporal-spectral dataset. This method is able to fuse low-spatial high-temporal resolution data (HTLS) and high-spatial low-temporal resolution data (HSLT) in a four-dimensional spatial-temporal-spectral dataset with increasing efficiency, while simultaneously ensuring accuracy. The method was tested using three datasets, and discussions of the network parameters were conducted. In addition, this method was compared with commonly used spatiotemporal fusion methods to verify our conclusion.

The planting year of apple orchard not only determines the fruit output but also provides information for the governmental management of the fruit industry. However, considering that different orchards use different management and cultivation methods, this may result in some trees having similar outlines but different planting years, and it is, therefore, difficult to effectively determine the actual planting year based on textural or structural characteristics. Therefore, the monitoring method provided in this paper is not to monitor the growing year positively from the planting of orchard seedlings but to use time series remote sensing data to reverse determine the continuous growth age of each existing orchard. The city of Qixia, Shandong Province, China, was used as a case study. Firstly, the spatial distribution of apple orchards was accurately extracted using the Sentinel-2 normalized difference vegetation index (NDVI) spatiotemporally fused images and phenological vegetation information. Secondly, using region of interest (ROI) data for different vegetation types obtained from a field survey, NDVI time series were extracted from the Sentinel-2 NDVI spatiotemporally fused image. Among them, three characteristic phenological periods were selected, and the NDVI time series for apple orchards was used as a template to extract the apple orchard distribution area from 2000 to 2017. Then, the distribution area of apple orchards was defined as the area of interest in the planting year, combined with the Landsat NDVI time series image composed of three characteristic phenological periods each year from 2000 to 2017, and the apple orchard phenological curve. Subsequently, a Euclidean distance (ED) method was used to calculate the distribution area of apple orchards for each year between 2000 and 2017. Finally, a pixel-by-pixel inverse time series calculation method was used to obtain the planting year of apple orchards in the study area. This study provides a new way to accurately identify the planting year of apple orchards using satellite remote sensing images.

Satellite-based remote sensing technologies are utilized extensively to investigate urban thermal environment under rapid urban expansion. Current Moderate Resolution Imaging Spectroradiometer (MODIS) data are, however, unable to adequately represent the spatially detailed information because of its relatively coarser spatial resolution, while Landsat data cannot explore the temporally continued analysis due to the lower temporal resolution. Combining MODIS and Landsat data, “Landsat-like” data were generated by using the Flexible Spatiotemporal Data Fusion method (FSDAF) to measure land surface temperature (LST) variations, and Landsat-like data including Normalized Difference Vegetation Index (NDVI) and Normalized Difference Built Index (NDBI) were generated to analyze LST dynamic driving forces. Results show that (1) the estimated “Landsat-like” data are capable of measuring the LST variations; (2) with the urban expansion from 2013 to 2016, LST increases ranging from 1.80 °C to 3.92 °C were detected in areas where the impervious surface area (ISA) increased, while LST decreases ranging from −3.52 °C to −0.70 °C were detected in areas where ISA decreased; (3) LST has a significant negative correlation with the NDVI and a strong positive correlation with NDBI in summer. Our findings can provide information useful for mitigating undesirable thermal conditions and for long-term urban thermal environmental management.