Search

Due to the limitations inherent in satellite sensor capabilities, it remains a challenge to balance the temporal and spatial resolutions of current remote sensing observations. Spatiotemporal fusion has emerged as a viable approach to address this issue. Many spatiotemporal fusion models have been proposed to fuse surface reflectance initially; however, their adaption for land surface temperature (LST), particularly with the 'fusion-then-retrieval (FTR)' or 'retrieval-then-fusion (RTF)' strategy, has not yet been comprehensively explored. This paper describes a comparative study of the two strategies within the context of LST fusion using three classic fusion methods including the spatial and temporal adaptive reflectance fusion model (STARFM), the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) and the flexible spatiotemporal data fusion model (FSDAF). The findings illuminate nuanced applicability across different models: (1) the FTR strategy is more suitable for STARFM; (2) the strategic applicability for ESTARFM and FSDAF is strongly related to the reference images; (3) the STARFM and FSDAF models are easily affected by the time interval between the predicted date and the reference date. This work is expected to provide practical guidance for synthesizing high spatiotemporal LST data with fusion models. [ABSTRACT FROM AUTHOR]

ObjectiveThe detection of lameness in dairy cows is an important issue that needs to be solved urgently in the process of large-scale dairy farming. Timely detection and effective intervention can reduce the culling rate of young dairy cows, which has important practical significance for increasing the milk production of dairy cows and improving the economic benefits of pastures. Due to the low efficiency and low degree of automation of traditional manual detection and contact sensor detection, the mainstream cow lameness detection method is mainly based on computer vision. The detection perspective of existing computer vision-based cow lameness detection methods is mainly side view, but the side view perspective has limitations that are difficult to eliminate. In the actual detection process, there are problems such as cows blocking each other and difficulty in deployment. The cow lameness detection method from the top view will not be difficult to use on the farm due to occlusion problems. The aim is to solve the occlusion problem under the side view.MethodsIn order to fully explore the movement undulations of the trunk of the cow and the movement information in the time dimension during the walking process of the cow, a cow lameness detection method was proposed from a top view based on fused spatiotemporal flow features. By analyzing the height changes of the lame cow in the depth video stream during movement, a spatial stream feature image sequence was constructed. By analyzing the instantaneous speed of the lame cow's body moving forward and swaying left and right when walking, optical flow was used to capture the instantaneous speed of the cow's movement, and a time flow characteristic image sequence was constructed. The spatial flow and time flow features were combined to construct a fused spatiotemporal flow feature image sequence. Different from traditional image classification tasks, the image sequence of cows walking includes features in both time and space dimensions. There would be a certain distinction between lame cows and non-lame cows due to their related postures and walking speeds when walking, so using video information analysis was feasible to characterize lameness as a behavior. The video action classification network could effectively model the spatiotemporal information in the input image sequence and output the corresponding category in the predicted result. The attention module Convolutional Block Attention Module (CBAM) was used to improve the PP-TSMv2 video action classification network and build the Cow-TSM cow lameness detection model. The CBAM module could perform channel weighting on different modes of cows, while paying attention to the weights between pixels to improve the model's feature extraction capabilities. Finally, cow lameness experiments were conducted on different modalities, different attention mechanisms, different video action classification networks and comparison of existing methods. The data was used for cow lameness included a total of 180 video streams of cows walking. Each video was decomposed into 100‒400 frames. The ratio of the number of video segments of lame cows and normal cows was 1:1. For the feature extraction of cow lameness from the top view, RGB images had less extractable information, so this work mainly used depth video streams.Results and DiscussionsIn this study, a total of 180 segments of cow image sequence data were acquired and processed, including 90 lame cows and 90 non-lame cows with a 1:1 ratio of video segments, and the prediction accuracy of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features reaches 88.7%, the model size was 22 M, and the offline inference time was 0.046 s. The prediction accuracy of the common mainstream video action classification models TSM, PP-TSM, SlowFast and TimesFormer models on the data set of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features reached 66.7%, 84.8%, 87.1% and 85.7%, respectively. The comprehensive performance of the improved Cow-TSM model in this paper was the most. At the same time, the recognition accuracy of the fused spatiotemporal flow feature image was improved by 12% and 4.1%, respectively, compared with the temporal mode and spatial mode, which proved the effectiveness of spatiotemporal flow fusion in this method. By conducting ablation experiments on different attention mechanisms of SE, SK, CA and CBAM, it was proved that the CBAM attention mechanism used has the best effect on the data of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features. The channel attention in CBAM had a better effect on fused spatiotemporal flow data, and the spatial attention could also focus on the key spatial information in cow images. Finally, comparisons were made with existing lameness detection methods, including different methods from side view and top view. Compared with existing methods in the side-view perspective, the prediction accuracy of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features was slightly lower, because the side-view perspective had more effective cow lameness characteristics. Compared with the method from the top view, a novel fused spatiotemporal flow feature detection method with better performance and practicability was proposed.ConclusionsThis method can avoid the occlusion problem of detecting lame cows from the side view, and at the same time improves the prediction accuracy of the detection method from the top view. It is of great significance for reducing the incidence of lameness in cows and improving the economic benefits of the pasture, and meets the needs of large-scale construction of the pasture.

[Objective] The detection of lameness in dairy cows is an important issue that needs to be solved urgently in the process of large-scale dairy farming. Timely detection and effective intervention can reduce the culling rate of young dairy cows, which has important practical significance for increasing the milk production of dairy cows and improving the economic benefits of pastures. Due to the low efficiency and low degree of automation of traditional manual detection and contact sensor detection, the mainstream cow lameness detection method is mainly based on computer vision. The detection perspective of existing computer vision-based cow lameness detection methods is mainly side view, but the side view perspective has limitations that are difficult to eliminate. In the actual detection process, there are problems such as cows blocking each other and difficulty in deployment. The cow lameness detection method from the top view will not be difficult to use on the farm due to occlusion problems. The aim is to solve the occlusion problem under the side view. [Methods] In order to fully explore the movement undulations of the trunk of the cow and the movement information in the time dimension during the walking process of the cow, a cow lameness detection method was proposed from a top view based on fused spatiotemporal flow features. By analyzing the height changes of the lame cow in the depth video stream during movement, a spatial stream feature image sequence was constructed. By analyzing the instantaneous speed of the lame cow's body moving forward and swaying left and right when walking, optical flow was used to capture the instantaneous speed of the cow's movement, and a time flow characteristic image sequence was constructed. The spatial flow and time flow features were combined to construct a fused spatiotemporal flow feature image sequence. Different from traditional image classification tasks, the image sequence of cows walking includes features in both time and space dimensions. There would be a certain distinction between lame cows and non-lame cows due to their related postures and walking speeds when walking, so using video information analysis was feasible to characterize lameness as a behavior. The video action classification network could effectively model the spatiotemporal information in the input image sequence and output the corresponding category in the predicted result. The attention module Convolutional Block Attention Module (CBAM) was used to improve the PP-TSMv2 video action classification network and build the Cow-TSM cow lameness detection model. The CBAM module could perform channel weighting on different modes of cows, while paying attention to the weights between pixels to improve the model's feature extraction capabilities. Finally, cow lameness experiments were conducted on different modalities, different attention mechanisms, different video action classification networks and comparison of existing methods. The data was used for cow lameness included a total of 180 video streams of cows walking. Each video was decomposed into 100 ‒ 400 frames. The ratio of the number of video segments of lame cows and normal cows was 1:1. For the feature extraction of cow lameness from the top view, RGB images had less extractable information, so this work mainly used depth video streams. [Results and Discussions] In this study, a total of 180 segments of cow image sequence data were acquired and processed, including 90 lame cows and 90 non-lame cows with a 1:1 ratio of video segments, and the prediction accuracy of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features reaches 88.7%, the model size was 22 M, and the offline inference time was 0.046 s. The prediction accuracy of the common mainstream video action classification models TSM, PP-TSM, SlowFast and TimesFormer models on the data set of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features reached 66.7%, 84.8%, 87.1% and 85.7%, respectively. The comprehensive performance of the improved Cow-TSM model in this paper was the most. At the same time, the recognition accuracy of the fused spatiotemporal flow feature image was improved by 12% and 4.1%, respectively, compared with the temporal mode and spatial mode, which proved the effectiveness of spatiotemporal flow fusion in this method. By conducting ablation experiments on different attention mechanisms of SE, SK, CA and CBAM, it was proved that the CBAM attention mechanism used has the best effect on the data of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features. The channel attention in CBAM had a better effect on fused spatiotemporal flow data, and the spatial attention could also focus on the key spatial information in cow images. Finally, comparisons were made with existing lameness detection methods, including different methods from side view and top view. Compared with existing methods in the side-view perspective, the prediction accuracy of automatic detection method for dairy cow lameness based on fusion of spatiotemporal stream features was slightly lower, because the side-view perspective had more effective cow lameness characteristics. Compared with the method from the top view, a novel fused spatiotemporal flow feature detection method with better performance and practicability was proposed. [Conclusions] This method can avoid the occlusion problem of detecting lame cows from the side view, and at the same time improves the prediction accuracy of the detection method from the top view. It is of great significance for reducing the incidence of lameness in cows and improving the economic benefits of the pasture, and meets the needs of large-scale construction of the pasture. [ABSTRACT FROM AUTHOR]

Over recent decades, Landsat satellite data has evolved into a highly valuable resource across diverse fields. Long-term satellite data records with integrity and consistency, such as the Landsat series, provide indispensable data for many applications. However, the malfunction of the Scan Line Corrector (SLC) on the Landsat 7 satellite in 2003 resulted in stripping in subsequent images, compromising the temporal consistency and data quality of Landsat time-series data. While various methods have been proposed to improve the quality of Landsat 7 SLC-off data, existing gap-filling methods fail to enhance the temporal resolution of reconstructed images, and spatiotemporal fusion methods encounter challenges in managing large-scale datasets. Therefore, we propose a method for reconstructing dense time series from SLC-off data. This method utilizes the Neighborhood Similar Pixel Interpolator to fill in missing values and leverages the time-series information to reconstruct high-resolution images. Taking the blue band as an example, the surface reflectance verification results show that the Mean Absolute Error (MAE) and BIAS reach minimum values of 0.0069 and 0.0014, respectively, with the Correlation Coefficient (CC) and Structural Similarity Index Metric (SSIM) reaching 0.93 and 0.94. The proposed method exhibits advantages in repairing SLC-off data and reconstructing dense time-series data, enabling enhanced remote sensing applications and reliable Earth's surface reflectance data reconstruction. [ABSTRACT FROM AUTHOR]

Sea surface temperature (SST) prediction plays an important role in scientific research, environmental protection, and other marine-related fields. However, most of the current prediction methods are not effective enough to utilize the spatial correlation of SSTs, which limits the improvement of SST prediction accuracy. Therefore, this paper first explores spatial correlation mining methods, including regular boundary division, convolutional sliding translation, and clustering neural networks. Then, spatial correlation mining through a graph convolutional neural network (GCN) is proposed, which solves the problem of the dependency on regular Euclidian space and the lack of spatial correlation around the boundary of groups for the above three methods. Based on that, this paper combines the spatial advantages of the GCN and the temporal advantages of the long short-term memory network (LSTM) and proposes a spatiotemporal fusion model (GCN-LSTM) for SST prediction. The proposed model can capture SST features in both the spatial and temporal dimensions more effectively and complete the SST prediction by spatiotemporal fusion. The experiments prove that the proposed model greatly improves the prediction accuracy and is an effective model for SST prediction. [ABSTRACT FROM AUTHOR]

Monitoring ecosystems at regional or continental scales is paramount for biodiversity conservation, climate change mitigation, and sustainable land management. Effective monitoring requires satellite imagery with both high spatial resolution and high temporal resolution. However, there is currently no single, freely available data source that fulfills these needs. A seamless fusion of data from the Sentinel-3 and Sentinel-2 optical sensors could meet these monitoring requirements as Sentinel-2 observes at the required spatial resolution (10 m) while Sentinel-3 observes at the required temporal resolution (daily). We introduce the Efficient Fusion Algorithm across Spatio-Temporal scales (EFAST), which interpolates Sentinel-2 data into smooth time series (both spatially and temporally). This interpolation is informed by Sentinel-3's temporal profile such that the phenological changes occurring between two Sentinel-2 acquisitions at a 10 m resolution are assumed to mirror those observed at Sentinel-3's resolution. The EFAST consists of a weighted sum of Sentinel-2 images (weighted by a distance-to-clouds score) coupled with a phenological correction derived from Sentinel-3. We validate the capacity of our method to reconstruct the phenological profile at a 10 m resolution over one rangeland area and one irrigated cropland area. The EFAST outperforms classical interpolation techniques over both rangeland (−72% in the mean absolute error, MAE) and agricultural areas (−43% MAE); it presents a performance comparable to the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) (+5% MAE in both test areas) while being 140 times faster. The computational efficiency of our approach and its temporal smoothing enable the creation of seamless and high-resolution phenology products on a regional to continental scale. [ABSTRACT FROM AUTHOR]

ABSTRACTRemote sensing provides us with an approach for the rapid identification and monitoring of spatiotemporal changes in the urban ecological environment at different scales. This study aimed to construct a remote sensing assessment index for urban ecological livability with continuous fine spatiotemporal resolution data from Landsat and MODIS to overcome the dilemma of single image-based, single-factor analysis, due to the limitations of atmospheric conditions or the revisit period of satellite platforms. The proposed Ecological Livability Index (ELI) covers five primary ecological indicators – greenness, temperature, dryness, water-wetness, and atmospheric turbidity – which are geometrically aggregated by non-equal weights based on an entropy method. Considering multisource time-series data of each indicator, the ELI can quickly and comprehensively reflect the characteristics of the Ecological Livability Quality (ELQ) and is also comparable at different time scales. Based on the proposed ELI, the urban ecological livability in the central urban area of Wuhan, China, from 2002 to 2017, in the different seasons was analyzed every 5 years. The ELQ of Wuhan was found to be generally at the medium level (ELI ≈0.6) and showed an initial trend of degradation but then improved. Moreover, the ecological livability in spring and autumn and near rivers and lakes was found to be better, whereas urban expansion has led to the outward ecological degradation of Wuhan, but urban afforestation has enhanced the environment. In general, this paper demonstrates that the ELI has an exemplary embodiment in urban ecological research, which will support urban ecological protection planning and construction.

Soil moisture (SM) plays an essential role in the hydrological cycle, drought monitoring, and water resources management. However, passive microwave remote sensing products offer coarse spatial resolutions (approximately 25–40 km), greatly limiting their applications. In this study, we considered vegetation memory and increased the amount of vegetation data by spatiotemporal fusion, integrated lagged vegetation index and fusion data as model inputs, and downscaled the spatial resolution of SMAP SM using the Random Forest (RF) model as well as multi-source remote sensing data. The downscaled SM was validated with 30 in situ SM measurements and daily precipitation data and explored the contributions of NDVIlagged and NDVIfused in downscaling. The results show that the downscaled SM is highly consistent with the in situ SM measurements. The correlation ranges from 0.628 to 0.849 with an average of 0.689. The root mean square error (RMSE) ranges from 0.024 to 0.094 m3/m3 with an average of 0.057 m3/m3. The proposed method leverages the strengths of the RF model, fusion vegetation data, and lagged vegetation data to improve high-resolution SM accuracy.

ABSTRACTMonitoring rapidly changing inland water bodies using remote sensing requires a temporal resolution of at least 3 days and a spatial resolution of at least 30 meters. However, the current satellite data falls short of meeting these monitoring requirements. Spatiotemporal fusion (STF) presents an effective method for obtaining continuous high-resolution data. This study explores five established STF algorithms, namely Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM), Robust Adaptive Spatial and Temporal Fusion Model (RASTFM), Flexible Spatiotemporal Data Fusion (FSDAF) and Unmixing-Based Data Fusion (UBDF), through experimentation across 45 lakes worldwide. Results are comprehensively compared and analyzed using two indicators: Root Mean Square Error (RMSE) and Robert’s edge (Edge) based on a dataset that contains 360 image pairs collected in 2021. The five algorithms underwent 950 random experiments. The results indicate that: (1) ESTARFM outperforms the other four algorithms in terms of visual effect and accuracy assessment. Its average RMSE of and EDGE are 0.0023 sr−1 and −0.0419, respectively. Followed by: FSDAF (RMSE: 0.0026 sr−1, EDGE: −0.0584), RASTFM (RMSE: 0.0029 sr−1, EDGE: −0.4128), STARFM (RMSE: 0.0027 sr−1, EDGE: −0.5207), and UBDF (RMSE: 0.0030 sr−1, EDGE: −0.6505) algorithms. (2) the algorithms have varied performance in different water monitoring missions: for water body extraction, the ESTARFM and FSDAF algorithms are recommended; for algae bloom detection, the FSDAF algorithm is recommended; for seasonal water body monitoring, the ESTARFM is recommended; for quantitative monitoring, the ESTARFM, FSDAF, and UBDF algorithms are recommended. (3) STARFM-like algorithms are sensitive to radiometric uncertainties and UBDF algorithm is sensitive to spatial uncertainties. The findings of this study can help generate high spatiotemporal resolution remote sensing datasets for water quality monitoring, waterbody detection, algae bloom tracking, and other water environmental monitoring missions.

Soil moisture is an important parameter to measure the material and energy exchange between soil and atmosphere. It is a key environmental factor in the fields of hydrology, meteorology, agriculture and so on. Remote sensing inversion and model simulation are two basic means to obtain global soil moisture data. Remote sensing inversion can obtain soil moisture data with relatively high resolution, but there are often time-space gaps while land surface model can simulate spatiotemporal continuous data, but the spatial resolution is often coarse. Both two data cannot meet the fine observation of soil moisture. Therefore, it is very important to develop an effective method to obtain high-resolution, spatiotemporal seamless soil moisture data. Methods: Focusing on their complementary advantages, we propose a global seamless soil moisture data generation method integrating microwave remote sensing and model simulation. Specifically, for the vacant area of 9 km soil moisture data of soil moisture active passive satellite, GLDAS Noah 0.25° model assimilation data is introduced to establish the spatiotemporal fusion model between them. Through this model, Noah data is downscaled to 9 km to fill the vacant area. As for some areas that have not been filled, Noah data is used for fitting, interpolation and filling, and the residual correction is further carried out based on Poisson equation method, and then the spatiotemporal seamless 9 km daily soil moisture data is generated. Results: We evaluated the experimental results by simulation experiment evaluation, site data evaluation and triple collocation evaluation. Experiments show that the global seamless soil moisture data produced in this paper performs well in all evaluation methods. Conclusions: This method can effectively combine the spatial resolution advantage of remote sensing observation with the spatiotemporal continuity advantage of model simulation, provide spatiotemporal seamless global soil moisture data, and better meet the needs of global scale water cycle monitoring and water resources management. [ABSTRACT FROM AUTHOR]

Due to budget and sensor technology constraints, a single sensor cannot simultaneously provide observational images with both a high spatial and temporal resolution. To solve the above problem, the spatiotemporal fusion (STF) method was proposed and proved to be an indispensable tool for monitoring land surface dynamics. There are relatively few systematic reviews of the STF method. Bibliometrics is a valuable method for analyzing the scientific literature, but it has not yet been applied to the comprehensive analysis of the STF method. Therefore, in this paper, we use bibliometrics and scientific mapping to analyze the 2967 citation data from the Web of Science from 1991 to 2023 in a metrological manner, covering the themes of STF, data fusion, multi-temporal analysis, and spatial analysis. The results of the literature analysis reveal that the number of articles displays a slow to rapid increase during the study period, but decreases significantly in 2023. Research institutions in China (1059 papers) and the United States (432 papers) are the top two contributors in the field. The keywords "Sentinel", "deep learning" (DL), and "LSTM" (Long Short-Term Memory) appeared most frequently in the past three years. In the future, remote sensing spatiotemporal fusion research can address more of the limitations of heterogeneous landscapes and climatic conditions to improve fused images' accuracy. [ABSTRACT FROM AUTHOR]

Spatiotemporal fusion can provide remote sensing images with both high temporal and high spatial resolution for earth observation applications. Most of the state-of-the-art models require three or even five images as input, which may lead to difficulties in practical applications due to bad weather or data missing. In this paper, the enhanced cross-paired wavelet based spatiotemporal fusion networks (ECPW-STFN) are proposed to improve the accuracy and quality of the fusions with fewer remote sensing images as inputs. Wavelet transform is introduced into spatiotemporal image fusion to achieve separate training of the high and low frequency components of the image to better extract features of different levels. In addition, a compound loss function containing wavelet loss is proposed to facilitate the preservation of details. Also, an enhancement module with convolutional block attention is put forward to further refine the prediction quality. Experiments were conducted to compare the proposed ECPW-STFN with five state-of-the-art methods on the public CIA and Daxing datasets. The results show ECPW-STFN is better than GAN-STFM which also requires two images, and not inferior to the methods requiring at least three inputs, even exceeds the optimal MLFF-GAN in some cases, proving its great superiority and potential. The code will be available at https://github.com/lixinghua5540/ECPW-STFN. [ABSTRACT FROM AUTHOR]

Spatiotemporal modeling is key for action recognition in videos. In this paper, we propose a Spatial features Compression and Temporal features Fusion (SCTF) block, including a Local Spatial features Compression (LSC) module and a Full Temporal features Fusion (FTF) module, we call the network equipped with SCTF block SCTF-NET, which is a human action recognition network more suitable for violent video detection. The spatial extraction and temporal fusions in previous works are typically achieved by stacking large numbers of convolution layers or adding some complex recurrent neural layers. In contrast, the SCTF module extracts the spatial information of video frames by LSC module, and the temporal sequence information of continuous frames is fused by FTF module, which can effectively conduct spatiotemporal modeling. Finally, our approach achieves good performance on action recognition benchmarks such as HMDB51 and UCF101. Meanwhile, it is more efficient in training and detection. What's more, experiments on violence datasets Hockey Fights, Movie Fight and Violent Flow show that, our proposed SCTF block is more suitable for violent action recognition. Our code is available at https://github.com/TAN-OpenLab/SCTF-Net. [ABSTRACT FROM AUTHOR]

Remote sensing provides us with an approach for the rapid identification and monitoring of spatiotemporal changes in the urban ecological environment at different scales. This study aimed to construct a remote sensing assessment index for urban ecological livability with continuous fine spatiotemporal resolution data from Landsat and MODIS to overcome the dilemma of single image-based, single-factor analysis, due to the limitations of atmospheric conditions or the revisit period of satellite platforms. The proposed Ecological Livability Index (ELI) covers five primary ecological indicators – greenness, temperature, dryness, water-wetness, and atmospheric turbidity – which are geometrically aggregated by non-equal weights based on an entropy method. Considering multisource time-series data of each indicator, the ELI can quickly and comprehensively reflect the characteristics of the Ecological Livability Quality (ELQ) and is also comparable at different time scales. Based on the proposed ELI, the urban ecological livability in the central urban area of Wuhan, China, from 2002 to 2017, in the different seasons was analyzed every 5 years. The ELQ of Wuhan was found to be generally at the medium level (ELI ≈0.6) and showed an initial trend of degradation but then improved. Moreover, the ecological livability in spring and autumn and near rivers and lakes was found to be better, whereas urban expansion has led to the outward ecological degradation of Wuhan, but urban afforestation has enhanced the environment. In general, this paper demonstrates that the ELI has an exemplary embodiment in urban ecological research, which will support urban ecological protection planning and construction. [ABSTRACT FROM AUTHOR]

Remote sensing data are becoming increasingly important for quantifying long-term changes in land surfaces. Optical sensors onboard satellite platforms face a tradeoff between temporal and spatial resolutions. Spatiotemporal fusion models can produce high spatiotemporal data, while existing models are not designed to produce moderate-spatial-resolution data, like Moderate-Resolution Imaging Spectroradiometer (MODIS), which has moderate spatial detail and frequent temporal coverage. This limitation arises from the challenge of combining coarse- and fine-spatial-resolution data, due to their large spatial resolution gap. This study presents a novel model, named multi-scale convolutional neural network for spatiotemporal fusion (MSCSTF), to generate MODIS-like data by addressing the large spatial-scale gap in blending the Advanced Very-High-Resolution Radiometer (AVHRR) and Landsat images. To mitigate the considerable biases between AVHRR and Landsat with MODIS images, an image correction module is included into the model using deep supervision. The outcomes show that the modeled MODIS-like images are consistent with the observed ones in five tested areas, as evidenced by the root mean square errors (RMSE) of 0.030, 0.022, 0.075, 0.036, and 0.045, respectively. The model makes reasonable predictions on reconstructing retrospective MODIS-like data when evaluating against Landsat data. The proposed MSCSTF model outperforms six other comparative models in accuracy, with regional average RMSE values being lower by 0.005, 0.007, 0.073, 0.062, 0.070, and 0.060, respectively, compared to the counterparts in the other models. The developed method does not rely on MODIS images as input, and it has the potential to reconstruct MODIS-like data prior to 2000 for retrospective studies and applications. [ABSTRACT FROM AUTHOR]

Medium- to high-resolution imagery is indispensable for various applications. Combining images from Landsat 8 and Sentinel-2 can improve the accuracy of observing dynamic changes on the Earth's surface. Many researchers use Sentinel-2 10 m resolution data in conjunction with Landsat 8 30 m resolution data to generate 10 m resolution data series. However, current fusion techniques have some algorithmic weaknesses, such as simple processing of coarse or fine images, which fail to extract image features to the fullest extent, especially in rapidly changing land cover areas. Facing the aforementioned limitations, we proposed a multiscale and attention mechanism-based residual spatiotemporal fusion network (MARSTFN) that utilizes Sentinel-2 10 m resolution data and Landsat 8 15 m resolution data as auxiliary data to upgrade Landsat 8 30 m resolution data to 10 m resolution. In this network, we utilized multiscale and attention mechanisms to extract features from coarse and fine images separately. Subsequently, the features outputted from all input branches are combined and further feature information is extracted through residual networks and skip connections. Finally, the features obtained from the residual network are merged with the feature information of the coarsely processed images from the multiscale mechanism to generate accurate prediction images. To assess the efficacy of our model, we compared it with existing models on two datasets. Results demonstrated that our fusion model outperformed baseline methods across various evaluation indicators, highlighting its ability to integrate Sentinel-2 and Landsat 8 data to produce 10 m resolution data. [ABSTRACT FROM AUTHOR]

The soil freeze/thaw (FT) state has emerged as a critical role in the ecosystem, hydrological, and biogeochemical processes, but obtaining representative soil FT state datasets with a long time sequence, fine spatial resolution, and high accuracy remains challenging. Therefore, we propose a decision-level spatiotemporal data fusion algorithm based on Convolutional Long Short-Term Memory networks (ConvLSTM) to expand the SMAP-enhanced L3 landscape freeze/thaw product (SMAP_E_FT) temporally. In the algorithm, the Freeze/Thaw Earth System Data Record product (ESDR_FT) is sucked in the ConvLSTM and fused with SMAP_E_FT at the decision level. Eight predictor datasets, i.e., soil temperature, snow depth, soil moisture, precipitation, terrain complexity index, area of open water data, latitude and longitude, are used to train the ConvLSTM. Direct validation using six dense observation networks located in the Genhe, Maqu, Naqu, Pali, Saihanba, and Shandian river shows that the fusion product (ConvLSTM_FT) effectively absorbs the high accuracy characteristics of ESDR_FT and expands SMAP_E_FT with an overall average improvement of 2.44% relative to SMAP_E_FT, especially in frozen seasons (averagely improved by 7.03%). The result from indirect validation based on categorical triple collocation also shows that ConvLSTM_FT performs stable regardless of land cover types, climate types, and terrain complexity. The findings, drawn from preliminary analyses on ConvLSTM_FT from 1980 to 2020 over China, suggest that with global warming, most parts of China suffer from different degrees of shortening of the frozen period. Moreover, in the Qinghai–Tibet region, the higher the permafrost thermal stability, the faster the degradation rate. [ABSTRACT FROM AUTHOR]

The coastal environment is characterized by high, multi-scale dynamics and the corresponding observations from a single remote sensing sensor are still facing challenges in achieving both high temporal and spatial resolution. This study proposed a spatiotemporal fusion model for coastal environments, which could fully enhance the efficiency of remote sensing data use and overcome the shortcomings of traditional spatiotemporal models that are insensitive to small-scale disturbances. The Enhanced Deep Super-Resolution Network (EDSR) was used to reconstruct spatial features in the lower spatial resolution GOCI-II data. The spatial features obtained instead of GOCI-II data were fed into the spatiotemporal fusion model, which enabled the fusion data to achieve an hour-by-hour observation of the water color and morphology information changes at 30 m resolution, including the changes in the spatial and temporal distributions of suspended particulate matter (SPM), the characterization of the vortex street caused by the bridge piers, the inundation process of the tidal flats, and coastline changes. In addition, this study analyzed the various factors affecting fusion accuracy, including spectral difference, errors in both temporal difference and location distance, and the structure of the EDSR model on the fusion accuracy. It is demonstrated that the location distance error and the spectral difference have the most significant impact on the fusion data, which may lead to the introduction of some ambiguous or erroneous spatial features. [ABSTRACT FROM AUTHOR]

The El Niño-Southern Oscillation (ENSO) is a global significant signal which exerts substantial climatic and socioeconomic impacts worldwide. However, the long-term prediction of ENSO persists as a challenge because of its diversity, irregularity, and asymmetry. Here, we develop a spatiotemporal fusion transformer network (STFTN), which designed a parallel encoder structure to effectively extract the spatiotemporal information from sea surface temperature anomaly and Niño3.4 index simultaneously, thereby enhancing the precision of Niño3.4 index forecasts. STFTN leverages the attention mechanism within its parallel encoder structure to extract global characteristics and establish remote dependencies on targets. With this structure, STFTN displays better prediction accuracy in different lead months. Furthermore, the activation map used in STFTN visualizes the contribution of the predictors to the output which helps to comprehend the factors contributing to ENSO events. The results highlight the potential of our model of ENSO forecasts and comprehension.

Deep learning methods have shown promising results in camera-based object detection. However, their effectiveness is significantly hindered in real-world scenes with poor illumination. In contrast, the trend of fusing millimeter-wave (mmWave) radar with camera for object detection is gaining momentum because radar can effectively compensate for the limitations of camera under poor illumination conditions. To this end, we propose ODSen, a lightweight, real-time, and robust fusion detection system. Specifically, ODSen presents several key advantages over existing sensor fusion methods: 1) despite fusing two sensing modalities in a deep learning-based approach, it requires only a small amount of multimodal data for new scenes to achieve real-time and robust object detection; 2) it employs a decoupled architecture that can switch between different image detectors to improve detection accuracy; 3) it performs spatiotemporal fusion of radar and camera features and employs a box refinement model to enhance computational efficiency, thereby ensuring real-time performance without compromising detection robustness. We collect a radar and camera dataset with diverse scenes on a university campus, and conduct extensive experiments, verifying that the proposed ODSen boosts over state-of-the-art methods in terms of average precision (AP), while yielding low computational cost.

Spatiotemporal fusion is a method of fusing high spatial resolution low temporal resolution remote sensing images and low spatial resolution high temporal resolution in order to obtain high spatiotemporal resolution remote sensing images, which can provide data support for temporal observation of fine objects, and plays an important role in the fields of Earth sciences, environmental monitoring, and so on. This article reveals an issue that is often overlooked in the field of deep learning-based spatiotemporal fusion: the discontinuity between image blocks and image blocks. This discontinuity may have an impact on the visualization of remote sensing images and subsequent applications. In this regard, this article proposes a spatially seamless stitching approach to optimize the spatiotemporal fusion model based on deep learning. By using this method, we successfully obtain high-quality fused remote sensing images with smoother transitions. The spatiotemporal fusion model used in the experiment is a generative adversarial network-based spatiotemporal fusion model (GAN-STFM) and the data are from the Beijing Gaofen-6 dataset (BJGF6). After our splicing method, the ratio of root-mean-square error (RMSE) at the splicing seam to the overall RMSE is reduced from 1.28 to 0.99, which effectively improves the continuity of the image. This new image splicing method has the potential to improve the utility of deep learning-based spatiotemporal fusion algorithms, which has application value for generating large-scale long time series remote sensing datasets with high temporal and high spatial resolution.

Precipitation nowcasting refers to the prediction of small-scale precipitation events at minute and kilometer scales within the upcoming 0 to 2 h, which significantly impacts both human activities and daily life. However, prevailing deep learning models have primarily focused on a single radar echo data source, limiting their ability to effectively capture intricate and rapidly evolving precipitation patterns. Thus, meteorological satellite is considered to supplement radar echo data. To achieve a comprehensive integration of multisource data with enhanced details, a two-stage fusion satellite and radar GAN-based prediction network (named FsrGAN) is proposed. In the first stage, we design a satellite-radar fusion prediction network known as FsrNet. This network employs an encoder-fusion-decoder architecture, where a novel spatial-channel attention (SCA) is proposed to enhance the filtering and fusion of multisource and multiscale features. In the second stage, we introduce a GAN-based network (FusionGAN) that also mines the complementary information of satellite images to sharpen the first-stage predicted radar maps with more details. Experiments are conducted on meteorological dataset in the Yangtze River Delta (YRD) region. The test results exhibit the notably superior performance of our model in terms of image quality and precipitation forecasting metrics in comparison to traditional optical flow-based methods and some well-known deep learning methods (ConvLSTM, ConvGRU, TrajGRU and PredRNN++). More importantly, our fusion model using satellite and radar data demonstrates the ability to predict convective initiation.

As the resolution of remote sensing quantitative inversion for albedo, clumping index, canopy height, and other biophysical parameter data is gradually refined, there is an increasing demand for higher spatial resolution bidirectional reflectance distribution function (BRDF) and angular reflectance data. In this article, we applied and evaluated flexible spatiotemporal data fusion to downscale angular reflectance using low-resolution BRDF data (MCD43A1) and high-resolution Sentinel-2 data as inputs. We selected Sentinel-2 orbital overlap regions with various anisotropic flat index (AFX) types and overpassing times as the study area as to obtain angular reflectance data under different anisotropic and solar-viewing geometry conditions. We found that the fusion method can downscale the angular reflectance with high accuracy (root mean square error < 0.04, bias < 0.02). The downscaling accuracy was higher in the cross-principle plane than in the principle plane, whereas there was no significant variation in accuracy under different AFX conditions. Owing to the lack of angular surface reflectance at larger viewing angles, validation was performed only within a view zenith angle of 10.3°. Further validation using satellites with larger viewing angles, such as Gaofen-6 or unmanned aerial vehicles, can be used to evaluate the accuracy of this method. This article presents a strategy for downscaling angular information, preliminary validation results, and satellite-based evidence of scale effects between the 500–10 m scales of the BRDF.

Snow cover is a sensitive indicator of climate change. Normalized difference snow index (NDSI) acquired from optical remote sensing data is usually used for monitoring snow cover, but the existing data are limited in spatiotemporal resolution. Here, we compared two blending strategies, blend-then-index (BI) and index-then-blend (IB), for generating high spatiotemporal resolution NDSI (daily, 20 m), and designed two groups of experiments (simulated and real) under three different snow cover periods over the Tibetan Plateau (TP). The flexible spatiotemporal data fusion (FSDAF) model was used as the fusion model. MODIS (daily, 500 m) and Sentinel-2A/B (2–5 days, 20 m) data were used as the inputs. The accuracy of the fused NDSI was evaluated from both spectral [root mean square error (RMSE), correlation coefficient (R), and average difference (AD)] and spatial (Robert's edge and local binary pattern) dimensions. Our results showed that the IB strategy produced more accurate NDSI results, with lower RMSE, higher R, and AD closer to zero compared to the BI strategy. In addition, there was no obvious difference in terms of texture between the two fusion strategies. Generally, the IB strategy is a better choice for generating a high spatiotemporal resolution NDSI through the FSDAF model under different snow cover periods on the TP. This study can provide effective guidelines for producing better high-resolution NDSI time series on the TP.

The investigation of the evolution of intertidal zones is a significant and extensively discussed subject in estuary and coastal research, both nationally and internationally. Recent advances in real-time global satellite products, such as GOCI, provide a potential approach for monitoring intertidal zones by utilizing waterline inversion based on spatial distribution processes. While these products offer the potential for monitoring intertidal zones through waterline inversion based on spatial distribution processes, they often suffer from coarse spatial resolution that smooths critical spatial heterogeneity. To overcome this limitation, a flexible spatiotemporal fusion model was employed to generate hourly time-series images with a spatial resolution of 10 m. This was achieved by combining Sentinel-2 satellite data (10 m spatial resolution; five-day revisit frequency) and GOCI-II data (500 m spatial resolution; 1-h revisit frequency). The resulting fusion images were used to construct an intertidal pseudodigital elevation model (DEM) by extracting waterlines and incorporating tidal-level information. The accuracy of the DEM was validated using surveyed real-time kinematic and drone data, with a root-mean-square error of 0.28 m. The analysis of the annual accretion and erosion evolution in the intertidal zone for the year 2023 revealed significant erosion in the central part of the zone, with a maximum erosion depth of 1 m at the bottom. This study contributes to the understanding of the response processes and mechanisms of the intertidal zone to natural and human disturbances, thus supporting coastal planning projects related to the intertidal zone.

Low spatial resolution is one of the main bottlenecks restricting the fine application of remote sensing parameters. Spatial downscaling is an effective way to improve their spatial resolution and application capabilities. Researchers have developed various downscaling methods for different remote sensing parameters. However, a unified and general method classification system has not yet been formed. Based on the in-depth commonality analysis of the downscaling of various parameters, this paper systematically summarizes the spatial downscaling methods from the perspective of data fusion, taking into account the required complementary information. Four types of downscaling methods are summarized: Multi-parameter fusion, spatiotemporal fusion, remote sensing-model fusion, and super-resolution reconstruction implicit fusion. The advantages, disadvantages, and applicable scenarios of various methods are analyzed, and the development trend of spatial downscaling method research is discussed. It can provide theoretical and technical support for improving the refined application ability of remote sensing data. [ABSTRACT FROM AUTHOR]

Video action recognition is vital in the research area of computer vision. In this paper, we develop a novel model, named Separable ConvNet Spatiotemporal Mixer (SCSM). Our goal is to develop an efficient and lightweight action recognition backbone that can be applied to multi-task models to increase the accuracy and processing speed. The SCSM model uses a new hierarchical spatial compression, employing the spatiotemporal fusion method, consisting of a spatial domain and a temporal domain. The SCSM model maintains the independence of each frame in the spatial domain for feature extraction and fuses the spatiotemporal features in the temporal domain. The architecture can be adapted to different frame rate requirements due to its high scalability. It is suitable to serve as a backbone for multi-task video feature extraction or industrial applications with its low prediction and training costs. According to the experimental results, SCSM has a low number of parameters and low computational complexity, making it highly scalable with strong transfer learning capabilities. The model achieves video action recognition accuracy comparable to state-of-the-art models with a smaller parameter size and fewer computational requirements. [ABSTRACT FROM AUTHOR]

Crop mapping is vital in ensuring food production security and informing governmental decision-making. The satellite-normalized difference vegetation index (NDVI) obtained during periods of vigorous crop growth is important for crop species identification. Sentinel-2 images with spatial resolutions of 10, 20, and 60 m are widely used in crop mapping. However, the images obtained during periods of vigorous crop growth are often covered by clouds. In contrast, time-series moderate-resolution imaging spectrometer (MODIS) images can usually capture crop phenology but with coarse resolution. Therefore, a time-series-based spatiotemporal fusion network (TSSTFN) was designed to generate TSSTFN-NDVI during critical phenological periods for finer-scale crop mapping. This network leverages multi-temporal MODIS-Sentinel-2 NDVI pairs from previous years as a reference to enhance the precision of crop mapping. The long short-term memory module was used to acquire data about the time-series change pattern to achieve this. The UNet structure was employed to manage the spatial mapping relationship between MODIS and Sentinel-2 images. The time distribution of the image sequences in different years was inconsistent, and time alignment strategies were used to process the reference data. The results demonstrate that incorporating the predicted critical phenological period NDVI consistently yields better crop classification performance. Moreover, the predicted NDVI trained with time-consistent data achieved a higher classification accuracy than the predicted NDVI trained with the original NDVI. [ABSTRACT FROM AUTHOR]

Spatiotemporal fusion (STF) is crucial for reconciling the conflict between temporal and spatial resolutions of remote sensing observations. However, fusing images in heterogeneous areas remains challenging under continuous missing values. Moreover, most current STF methods only consider temporal errors and disregard the spatial scale error in time variation. Therefore, we proposed a Quick Spatiotemporal Fusion with Coarse- and Fine-Resolution Scale Transformation Errors and Pixel-Based Synthesis Base Image Pair (STEPSBI). First, the optimal pixel-based image synthesis strategy was designed using all available fine- and coarse-resolution images. Then, the scale transformation error (STE) of coarse-resolution downscaling to fine-resolution in temporal variation was quantified. And a residual term was introduced to reduce the prediction error from the temporal variation. Finally, the spatial scale and temporal errors were corrected using the results of super-pixel segmentation as spatial weights. This model has two strengths: (1) Pixel-based image synthesis alleviates the absence of base images under continuous missing values; and (2) STE correction restores spatial details of heterogeneous areas in rapid land cover change. In scenarios of continuous missing images, abrupt land cover changes, and high-resolution heterogeneity, we evaluated STEPSBI, the Gap Filling and Savitzky–Golay filtering method (GF-SG), the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), the Flexible Spatiotemporal DAta Fusion (FSDAF), the cross-attention-based adaptive weighting fusion network (CAFE), and the Multi-scene Spatiotemporal Fusion Network (MUSTFN). The results indicate that STEPSBI yields better overall performance than other models in cropland, woodland, grassland, and other land cover types. Furthermore, ablation experiments demonstrated that each component improved the model's performance. In addition, STEPSBI had higher fusion efficiency because it was developed on the Google Earth Engine cloud computing platform. Therefore, STEPSBI was feasible for advancing fine monitoring of spatiotemporal image fusion under continuous missing values and heterogeneous surfaces. The program of STEPBSI is freely available at: https://code.earthengine.google.com/684844aa42f64fa6b4eebe3bc0dd6483. [ABSTRACT FROM AUTHOR]

Spatiotemporal fusion (STF) can provide dense satellite image series with high spatial resolution. However, most spatiotemporal fusion approaches are time-consuming, which seriously limits their applicability in large-scale areas. To address this problem, some efforts have been paid for accelerating STF approaches with help of graphics processing units (GPUs), whose effect is dramatic. However, this strategy is hardware dependent, which may not be always satisfied. In this paper, we develop a hardware independent accelerating strategy, named AcSTF. The proposed AcSTF consists of two steps, which are medium resolution STF (MSTF) and local normalization-based fast fusion (LNFM). The MSTF utilizes STF methods to improve the coarse spatial resolution images to a medium spatial resolution, while the LNFM further refines the medium spatial resolution images to provide fine spatial resolution images. To test the AcSTF, the experiments are conducted using five commonly used STF approaches on two public Landsat-MODIS datasets. The experimental results indicate that AcSTF can not only reduce 87%–95% running time of current STF approaches, but also preserve their qualitative and quantitative performance well. After that, we apply the AcSTF to produce an intact 30 m image of the whole Ukraine mainland. Without any hardware which can speed up computing,the time for reconstructing the 30 m image is 5.42 h just using an unremarkable central processing unit (CPU). Compared to the real Landsat image, the reconstructed image achieves remarkable qualitative and quantitative performance, which demonstrates the practicability of the AcSTF.

Due to hardware technology limitations, satellite sensors are unable to capture images with high temporal, spatial, and spectral resolutions simultaneously. However, the Gaofen-1 satellite overcomes this challenge by incorporating 2-meter panchromatic, 8-meter multispectral, and 16-meter wide-field cameras, allowing for the integration of images from these sensors. To address this issue, we propose a study on the spatio-temporal-spectral fusion method for Gaofen-1 images, aiming to achieve more comprehensive structures. Inspired by the diffusion model, which learns the data distribution of the target image, we propose a new network utilizing an enhanced diffusion framework. The network incorporates both structural and spectral constraints to guide the fusion process. This work represents the first application of the diffusion model to spatio-temporal-spectral fusion, specifically synthesizing the 2-meter multispectral images with dense temporal resolution. To assess fusion quality, we have developed a benchmark dataset. During the validation stage, we evaluate the radiometric deviation, structural similarity, and spectral fidelity between the fused 2-meter multispectral images and the reference images. Both visual and quantitative assessments demonstrate that our newly proposed method work well for the Gaofen-1 fusion.

Over recent decades, Landsat satellite data has evolved into a highly valuable resource across diverse fields. Long-term satellite data records with integrity and consistency, such as the Landsat series, provide indispensable data for many applications. However, the malfunction of the Scan Line Corrector (SLC) on the Landsat 7 satellite in 2003 resulted in stripping in subsequent images, compromising the temporal consistency and data quality of Landsat time-series data. While various methods have been proposed to improve the quality of Landsat 7 SLC-off data, existing gap-filling methods fail to enhance the temporal resolution of reconstructed images, and spatiotemporal fusion methods encounter challenges in managing large-scale datasets. Therefore, we propose a method for reconstructing dense time series from SLC-off data. This method utilizes the Neighborhood Similar Pixel Interpolator to fill in missing values and leverages the time-series information to reconstruct high-resolution images. Taking the blue band as an example, the surface reflectance verification results show that the Mean Absolute Error (MAE) and BIAS reach minimum values of 0.0069 and 0.0014, respectively, with the Correlation Coefficient (CC) and Structural Similarity Index Metric (SSIM) reaching 0.93 and 0.94. The proposed method exhibits advantages in repairing SLC-off data and reconstructing dense time-series data, enabling enhanced remote sensing applications and reliable Earth’s surface reflectance data reconstruction.

Sea surface temperature (SST) prediction plays an important role in scientific research, environmental protection, and other marine-related fields. However, most of the current prediction methods are not effective enough to utilize the spatial correlation of SSTs, which limits the improvement of SST prediction accuracy. Therefore, this paper first explores spatial correlation mining methods, including regular boundary division, convolutional sliding translation, and clustering neural networks. Then, spatial correlation mining through a graph convolutional neural network (GCN) is proposed, which solves the problem of the dependency on regular Euclidian space and the lack of spatial correlation around the boundary of groups for the above three methods. Based on that, this paper combines the spatial advantages of the GCN and the temporal advantages of the long short-term memory network (LSTM) and proposes a spatiotemporal fusion model (GCN-LSTM) for SST prediction. The proposed model can capture SST features in both the spatial and temporal dimensions more effectively and complete the SST prediction by spatiotemporal fusion. The experiments prove that the proposed model greatly improves the prediction accuracy and is an effective model for SST prediction.

The previous studies have demonstrated that the use of deep learning algorithms can make personality prediction based on two-dimensional image information, and the emergence of video provides more possibilities for exploring personality prediction. Compared to image-based personality prediction, using video can provide more information than static images. But videos contain hundreds of frames, not all of which are useful, and processing these images requires a lot of computation. This paper proposes to apply video analysis algorithms to the task of personality prediction and propose the use of LSTM to fuse image feature information. The best prediction effect is confirmed by experiments when the fusion frame number is 16 frames. This paper is based on 3D-ConvNet to build an end-to-end video analysis network and solve the network over fitting problem by pre-training and data augmentation. Experiments show that the accuracy of character prediction can be improved by using 3D-ConvNet to fuse the spatio-temporal information of videos. [ABSTRACT FROM AUTHOR]

Surface urban heat islands (SUHIs) are mostly an urban ecological issue. There is a growing demand for the quantification of the SUHI effect, and for its optimization to mitigate the increasing possible hazards caused by SUHI. Satellite-derived land surface temperature (LST) is an important indicator for quantifying SUHIs with frequent coverage. Current LST data with high spatiotemporal resolution is still lacking due to no single satellite sensor that can resolve the trade-off between spatial and temporal resolutions and this greatly limits its applications. To address this issue, we propose a multiscale geographically weighted regression (MGWR) coupling the comprehensive, flexible, spatiotemporal data fusion (CFSDAF) method to generate a high-spatiotemporal-resolution LST dataset. We then analyzed the SUHI intensity (SUHII) in Chengdu City, a typical cloudy and rainy city in China, from 2002 to 2022. Finally, we selected thirteen potential driving factors of SUHIs and analyzed the relation between these thirteen influential drivers and SUHIIs. Results show that: (1) an MGWR outperforms classic methods for downscaling LST, namely geographically weighted regression (GWR) and thermal image sharpening (TsHARP); (2) compared to classic spatiotemporal fusion methods, our method produces more accurate predicted LST images (R2, RMSE, AAD values were in the range of 0.8103 to 0.9476, 1.0601 to 1.4974, 0.8455 to 1.3380); (3) the average summer daytime SUHII increased form 2.08 °C (suburban area as 50% of the urban area) and 2.32 °C (suburban area as 100% of the urban area) in 2002 to 4.93 °C and 5.07 °C, respectively, in 2022 over Chengdu City; and (4) the anthropogenic activity drivers have a higher relative influence on SUHII than other drivers. Therefore, anthropogenic activity driving factors should be considered with CO2 emissions and land use changes for urban planning to mitigate the SUHI effect. [ABSTRACT FROM AUTHOR]

As an important part of remote sensing data, weather radar plays an important role in convective weather forecasts to reduce extreme precipitation disasters. The existing radar echo extrapolation methods do not utilize the local natural characteristics of the radar echo effectively but only roughly extract the whole characteristics of the radar echo. To address these challenges, we design a spatiotemporal difference and generative adversarial fusion model (STDGAN). Specifically, a spatiotemporal difference module (STD) is designed to extract local weather patterns and model them in detail. In our model, spatiotemporal difference information and spatiotemporal features captured by the model itself are fused together. In addition, our model is trained in a generative adversarial network (GAN) framework; it helps to generate a clearer map of future radar echoes at the image level. The discriminator consists of multi-scale feature extractors, which can simulate weather models of various scales more completely. Finally, extrapolation experiments were conducted using actual radar echo data from Shijiazhuang and Nanjing. The experiments have shown that our model has a more accurate prediction performance for predicting local weather patterns and overall echo change trajectories compared with previous research models. Our model achieved MSE, PSNE, and SSIM values of 132.22, 37.87, and 0.796, respectively, on the Shijiazhuang radar echo dataset. In addition, our model also showed better performance results on the Nanjing radar echo dataset. The results show that the MSE was 49.570, the PSNR was 0.714, and the SSIM was 30.633. The CC value was 0.855. [ABSTRACT FROM AUTHOR]

The monitoring of rapidly changing land surface processes requires remote sensing images with high spatiotemporal resolution. As remote sensing satellites have different satellite orbits, satellite orbital velocities, and sensors, it is challenging to acquire remote sensing images with high resolution and dense time series within a reasonable temporal interval. Remote sensing spatiotemporal fusion is one of the effective ways to acquire high-resolution images with long time series. Most of the existing STF methods use artificially specified fusion strategies, resulting in blurry images and poor generalization ability. Additionally, some methods lack continuous time change information, leading to poor performance in capturing sharp changes in land covers. In this paper, we propose an adaptive multiscale network for spatiotemporal fusion (AMS-STF) based on a generative adversarial network (GAN). AMS-STF reconstructs high-resolution images by leveraging the temporal and spatial features of the input data through multiple adaptive modules and multiscale features. In AMS-STF, for the first time, deformable convolution is used for the STF task to solve the shape adaptation problem, allowing for adaptive adjustment of the convolution kernel based on the different shapes and types of land use. Additionally, an adaptive attention module is introduced in the networks to enhance the ability to perceive temporal changes. We conducted experiments comparing AMS-STF to the most widely used and innovative models currently available on three Landsat-MODIS datasets, as well as ablation experiments to evaluate some innovative modules. The results demonstrate that the adaptive modules significantly improve the fusion effect of land covers and enhance the clarity of their boundaries, which proves the effectiveness of AMS-STF. [ABSTRACT FROM AUTHOR]

• SED that suitably combines the kernel- and fusion-based methods was proposed for LST downscaling. • SED uses a Landsat image to downscale MODIS LST by statistical regression. • SED has an average RMSE of 1.29 K over 50 diverse study areas worldwide. • SED is superior to kernel-based and fusion-based methods and their combinations. Kernel-based and fusion-based methods have been widely used to downscale satellite-derived land surface temperatures (LSTs) for obtaining LSTs with high spatiotemporal resolution. The integration of these two methods can potentially provide a more effective method for LST downscaling. However, a downscaling algorithm that appropriately integrates the advantages of these two methods in accuracy and stability, yet is characterized by adequate simplicity, effectiveness, and global applicability, is still lacking. To address this issue, here we proposed the Simple and Effective Downscaling (SED) algorithm, which integrates kernel- and fusion-based methods suitably and uses a single adjacent Landsat image to downscale MODIS LSTs by statistical regression. The SED was tested in 50 different study areas worldwide, and its performance was compared with four benchmark downscaling algorithms, including a kernel-based algorithm (the RFT21), a fusion-based algorithm (the FSDAF), and the simple average (the Hybrid 1) and weighted average (the Hybrid 2) of the RFT21 and the FSDAF. Our evaluations show that the SED performs better in almost all study areas and has finer global applicability when compared with these four benchmark algorithms. The mean RMSE of the SED is 1.29 K over these 50 study regions, with an accuracy improvement of over 13 % relative to the best benchmark algorithm. Particularly, the SED exhibits higher accuracies in tropical and temperate regions, over urban areas and savannas, and when using Landsat images closer to MODIS LSTs. We believe that the SED can contribute to the generation of high-quality and high spatiotemporal resolution LSTs over global lands. [ABSTRACT FROM AUTHOR]

Accurate spatial information for agricultural field parcels is important for agricultural production management and understanding agro-industrialization and intensification. However, traditional remote sensing methods that rely on single-modal or single-date data struggle to identify heterogeneous field parcels, particularly in regions dominated by smallholder farming systems. To address this challenge, we proposed a Dual branch Spatiotemporal Fusion Network (DSTFNet) that integrated very high-resolution (VHR) images and medium-resolution satellite image time series (MRSITS) to extract agricultural field parcels over various landscapes. The DSTFNet consisted of two branches: a spatial branch that extracted spatial features from VHR images and a temporal branch that explored seasonal spectral dynamics from MRSITS data by using ConvLSTM units and an attention module. We evaluated the DSTFNet in four regions across China by using GF-2 and Sentinel-2 data. The results showed that DSTFNet performed well in delineating agricultural field parcels, achieving the highest Matthew's correlation coefficient (MCC) = 0.823 for the field extent, the highest F1-score of edge (F edge) = 0.865 for field boundary, and the lowest errors of segmentation evaluation index (SEI) = 0.191 for the vectorized field parcels in Hubei province. In addition, DSTFNet significantly outperformed two single-branch models that used VHR or MRSITS alone, as well as existing BsiNet, ResUNet_a, UNet and RAUNet models. DSTFNet also showed good spatial transferability in distinct regions without training data (on average, MCC = 0.728, F edge = 0.729, and SEI = 0.281 for three target regions). Using limited training data to fine-tune the DSTFNet can further improve its ability to delineate field parcels over complex regions. The visualization analysis of temporal attention weights demonstrated that DSTFNet can well capture cropland spectral dynamics, making it advantageous in extracting diverse cropland parcels. By exploiting important spectral, spatial and temporal information from multimodal satellite data, DSTFNet provided an effective, robust, and transferable solution for accurately delineating agricultural field parcels across heterogeneous farming systems. [ABSTRACT FROM AUTHOR]

Satellite-derived aerosol optical depth (AOD) has been extensively utilized for retrieving ground-level PM2.5 distributions. However, the presence of non-random missing data gaps in AOD poses a challenge to directly obtaining the gap-free AOD-derived PM2.5, thereby impeding accurate exposure risk assessment. Here, this study presents a novel and flexible framework that couples stacking and flexible spatiotemporal data fusion (FSDAF) approaches. By integrating multiple models and data sources, this framework aims to generate hourly (24-h) gap-free PM2.5 estimates for the Beijing–Tianjin–Hebei (BTH) region in 2018. This study effectively reconstructed data at least three times more effectively than the original AOD-derived PM2.5, achieving the Pearson coefficient (r), the coefficient determination (R2), root mean squared error (RMSE), and mean absolute error (MAE) values of 0.91, 0.84, 19.38 µg/m3, and 12.17 µg/m3, respectively, based on entire samples. Such strong predictive performance was also exhibited in spatial-based (r: 0.92–0.93, R2: 0.85–0.87, RMSE: 18.13 µg/m3–20.18 µg/m3, and MAE: 11.21 µg/m3–12.52 µg/m3) and temporal-based (r: 0.91–0.98, R2: 0.82–0.96, RMSE: 3.8 µg/m3–21.89 µg/m3, and MAE: 2.71 µg/m3–14.00 µg/m3) validations, indicating the robustness of this framework. Additionally, this framework enables the assessment of annual and seasonal PM2.5 concentrations and distributions, revealing that higher levels are experienced in the southern region, while lower levels prevail in the northern part. Winter exhibits the most severe levels, followed by spring and autumn, with comparatively lower levels in summer. Notably, the proposed framework effectively mitigates bias in calculating population-weighted exposure risk by filling data gaps with calculated values of 51.04 µg/m3, 54.17 µg/m3, 56.24 µg/m3, and 55.00 µg/m3 in Beijing, Tianjin, Hebei, and the BTH region, respectively. [ABSTRACT FROM AUTHOR]

In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels' convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies. [ABSTRACT FROM AUTHOR]

Remote sensing spatiotemporal fusion is currently a popular research field in remote sensing that can cost-effectively generate remote sensing images with high spatiotemporal resolution. The deep neural network-based approach has strong feature extraction capability and has achieved great success in the fields of signal and image processing, but the deep network lacks interpretability. In this paper, a new interpretable deep network-based spatiotemporal fusion (UNSTF) method is proposed. The method proposes a new network model by first establishing a concise a priori formulation using sparse representation, constructing the proposed network by unfolding the proximal gradient algorithm for solving the model, and carefully designing each basic network module in the model to have a reasonable physical meaning, making the whole network interpretable. In addition, the UNSTF network method proposes a new network iteration loss function, where the predicted images of each iteration stage of the network are constrained by the real images, and the final stage and the intermediate network stages conform well to their inherent prior structures, effectively improving the accuracy and reliability of model prediction. Through extensive experiments, it is shown that the proposed method outperforms existing fusion methods in terms of both subjective and objective metrics. [ABSTRACT FROM AUTHOR]

Temporal reasoning is crucial for action recognition tasks. The previous works use 3D CNNs to jointly capture spatiotemporal information, but it causes a lot of computational costs as well. To improve the above problems, we propose a general channel split spatiotemporal network (CSST-Net) to achieve effective spatiotemporal feature representation learning. The CSST module consists of the grouped spatiotemporal modeling (GSTM) module and the parameter-free feature fusion (PFFF) module. The GSTM module decomposes features into spatial and temporal parts along the channel dimension in parallel, which focuses on spatial and temporal clues, respectively. Meanwhile, we utilize the combination of group-wise convolution and point-wise convolution to reduce the number of parameters in the temporal branch, thus alleviating the overfitting of 3D CNNs. Furthermore, for the problem of spatiotemporal feature fusion, the PFFF module performs the recalibration and fusion of spatial and temporal features by a soft attention mechanism, without introducing extra parameters, thus ensuring the correct network information flow effectively. Finally, extensive experiments on three benchmark databases (Sth-Sth V1, Sth-Sth V2, and Jester) indicate that the proposed CSST-Net can achieve competitive performance compared to existing methods, and significantly reduces the number of parameters and FLOPs of 3D CNNs baseline. [ABSTRACT FROM AUTHOR]

Spatiotemporal fusion (STF) is considered an effective way to address the mutual constraints of the spatiotemporal resolutions of the remote sensing images from a single satellite sensor. Although the deep learning (DL)-based STF methods have shown great potential so far, there are still some deficiencies. Supervised DL-based methods usually train the network on the original scale data with richer spatial information, but they utilize only the data at auxiliary dates to generate the fusion data at prediction dates, which may perform poorly when significant spatiotemporal changes occur between the auxiliary and prediction dates. For self-supervised DL-based methods, the training can focus more on the data at prediction dates, but the network is generally trained on the down-sampled data with less spatial information, causing insufficient spatial structures of the fusion results. Based on the above, we innovatively combine supervised and self-supervised learning and propose a dual-stage cascade STF framework. In the first stage, a model based on supervised learning is trained on the data at auxiliary dates to extract abundant spatial features and obtain the initial fusion results. In the second stage, we utilize a self-supervised strategy to excavate the spatiotemporal features of prediction dates based on the initial fusion results and the observed data at prediction dates. In addition, to alleviate the insufficient consideration of the temporal correlation in existing STF methods, we design a temporal consistency loss function to fully utilize the temporal correlation information between multiple prediction dates to generate more accurate fusion results. The proposed framework can not only realize the STF of remote sensing images but also be well applied to the STF of remote sensing products such as land surface temperature (LST). Based on the quantitative results of three datasets, the proposed method improves the quantitative indices of root mean square error (RMSE), structural similarity (SSIM), erreur relative global adimensionnelle de synthese (ERGAS), peak signal to noise ratio (PSNR), and spectral angle mapper (SAM) by up to 0.2014, 0.0279, 0.6797, 0.7629, and 0.2006 at most compared with the best comparison method, which fully shows the superiority of our method. [ABSTRACT FROM AUTHOR]

Remote sensing images obtained by a variety of sensors have been widely used in different Earth observation tasks. However, owing to budget and sensor technology constraints, a single sensor cannot simultaneously provide observational images with both high spatial and temporal resolution. This brings difficulties to remote sensing research which requires high spatial and temporal resolution data. To solve the above constraints, the spatiotemporal fusion (STF) method was proposed and has received widespread attention. The main challenge for remote sensing STF is to reconstruct both phenological and land cover type changes. To overcome this challenge, many STF methods have been proposed based on different principles and strategies. Since there are STF methods have been proposed recently, there is a need for new review to reflect the current research status. Therefore, in this review, we summarize the existing studies, discusses their basic principle and limitations, collect some recent applications, and provide a comprehensive overview of current advances. Furthermore, to facilitate and promote the research in this community, we also collect publicly available resources and introduce the most used quantitative metrics. Finally, we take a conversation about some open problems and challenges that require attention in the future.

Abstract Chlorophyll-a concentration for quantifying phytoplankton biomass is commonly used as an indicator for evaluating the trophic level of lakes and water quality. This research aimed to develop a high spatiotemporal-resolution model for the retrieval of chlorophyll-a in inland water. Firstly, the machine learning based models considering Sentinel-2 Multispectral Instrument and Sentinel-3 Ocean and Land Color Instrument (OLCI) images were applied to estimate chlorophyll-a concentrations (R 2 = 0.873 and 0.822, respectively). The spatiotemporal fusion was performed to fuse the OLCI and MSI chlorophyll-a images with low temporal resolution but fine spatial-resolution, and with high temporal resolution but coarse spatial-resolution. The random forest was applied to fuse images from two distinct sensors, and to refine the spatial resolution of OLCI estimations to be the same as those of Sentinel-2 MSI. Results showed that the spatiotemporal fusion can estimate dense-temporal 10 m spatial resolution chlorophyll-a concentration in the Tsengwen Reservoir (Root-Mean-Square Error, RMSE = 1.25–1.47 μg L−1). The spatiotemporal fusion model was effectively applied to determine high spatiotemporal-resolution chlorophyll-a measurements in the aquatic system.

Monitoring ecosystems at regional or continental scales is paramount for biodiversity conservation, climate change mitigation, and sustainable land management. Effective monitoring requires satellite imagery with both high spatial resolution and high temporal resolution. However, there is currently no single, freely available data source that fulfills these needs. A seamless fusion of data from the Sentinel-3 and Sentinel-2 optical sensors could meet these monitoring requirements as Sentinel-2 observes at the required spatial resolution (10 m) while Sentinel-3 observes at the required temporal resolution (daily). We introduce the Efficient Fusion Algorithm across Spatio-Temporal scales (EFAST), which interpolates Sentinel-2 data into smooth time series (both spatially and temporally). This interpolation is informed by Sentinel-3’s temporal profile such that the phenological changes occurring between two Sentinel-2 acquisitions at a 10 m resolution are assumed to mirror those observed at Sentinel-3’s resolution. The EFAST consists of a weighted sum of Sentinel-2 images (weighted by a distance-to-clouds score) coupled with a phenological correction derived from Sentinel-3. We validate the capacity of our method to reconstruct the phenological profile at a 10 m resolution over one rangeland area and one irrigated cropland area. The EFAST outperforms classical interpolation techniques over both rangeland (−72% in the mean absolute error, MAE) and agricultural areas (−43% MAE); it presents a performance comparable to the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) (+5% MAE in both test areas) while being 140 times faster. The computational efficiency of our approach and its temporal smoothing enable the creation of seamless and high-resolution phenology products on a regional to continental scale.