Search

This paper describes the technical development and accuracy assessment of the most recent and improved version of the SoilGrids system at 250m resolution (June 2016 update). SoilGrids provides global predictions for standard numeric soil properties (organic carbon, bulk density, Cation Exchange Capacity (CEC), pH, soil texture fractions and coarse fragments) at seven standard depths (0, 5, 15, 30, 60, 100 and 200 cm), in addition to predictions of depth to bedrock and distribution of soil classes based on the World Reference Base (WRB) and USDA classification systems (ca. 280 raster layers in total). Predictions were based on ca. 150,000 soil profiles used for training and a stack of 158 remote sensing-based soil covariates (primarily derived from MODIS land products, SRTM DEM derivatives, climatic images and global landform and lithology maps), which were used to fit an ensemble of machine learning methods-random forest and gradient boosting and/or multinomial logistic regression-as implemented in the R packages ranger, xgboost, nnet and caret. The results of 10-fold cross-validation show that the ensemble models explain between 56% (coarse fragments) and 83% (pH) of variation with an overall average of 61%. Improvements in the relative accuracy considering the amount of variation explained, in comparison to the previous version of SoilGrids at 1 km spatial resolution, range from 60 to 230%. Improvements can be attributed to: (1) the use of machine learning instead of linear regression, (2) to considerable investments in preparing finer resolution covariate layers and (3) to insertion of additional soil profiles. Further development of SoilGrids could include refinement of methods to incorporate input uncertainties and derivation of posterior probability distributions (per pixel), and further automation of spatial modeling so that soil maps can be generated for potentially hundreds of soil variables. Another area of future research is the development of methods for multiscale merging of SoilGrids predictions with local and/or national gridded soil products (e.g. up to 50 m spatial resolution) so that increasingly more accurate, complete and consistent global soil information can be produced. SoilGrids are available under the Open Data Base License.

Interpretable Machine Learning (IML) has rapidly advanced in recent years, offering new opportunities to improve our understanding of the complex Earth system. IML goes beyond conventional machine learning by not only making predictions but also seeking to elucidate the reasoning behind those predictions. The combination of predictive power and enhanced transparency makes IML a promising approach for uncovering relationships in data that may be overlooked by traditional analysis. Despite its potential, the broader implications for the field have yet to be fully appreciated. Meanwhile, the rapid proliferation of IML, still in its early stages, has been accompanied by instances of careless application. In response to these challenges, this paper focuses on how IML can effectively and appropriately aid geoscientists in advancing process understanding—areas that are often underexplored in more technical discussions of IML. Specifically, we identify pragmatic application scenarios for IML in typical geoscientific studies, such as quantifying relationships in specific contexts, generating hypotheses about potential mechanisms, and evaluating process‐based models. Moreover, we present a general and practical workflow for using IML to address specific research questions. In particular, we identify several critical and common pitfalls in the use of IML that can lead to misleading conclusions, and propose corresponding good practices. Our goal is to facilitate a broader, yet more careful and thoughtful integration of IML into Earth science research, positioning it as a valuable data science tool capable of enhancing our current understanding of the Earth system. Plain Language Summary: Artificial Intelligence is a rapidly advancing field, in which Interpretable Machine Learning (IML) is seen as having the potential to significantly improve our understanding of Earth's complex environmental systems. IML goes beyond the predictive power of machine learning models, focusing instead on uncovering the relationships within the data that are revealed by the model's learning process. However, there is still a lack of straightforward, practical domain‐specific guidelines for geoscientists that facilitate both broader and more careful application in the field. In this paper, we aim to demonstrate the real‐world benefits of IML in typical geoscientific analysis. We provide a clear, step‐by‐step workflow that shows how IML can be used to address specific questions. We also point out some common pitfalls in using IML and offer solutions to avoid them. Our goal is to make IML more accessible and useful to a wider range of geoscientists, and we believe that IML, if used properly and thoughtfully, can become an essential and valuable tool to advance our understanding of complex Earth systems. Key Points: We demonstrate the broader relevance of Interpretable Machine Learning (IML) to most geoscientists and underexplored opportunities for its useWe describe a workflow for the effective use of IML while cautioning against potential and common pitfallsWe suggest good practices for its adoption and advocate for more careful application to ensure reliable and robust insights for the field [ABSTRACT FROM AUTHOR]

Land carbon fluxes play a critical role in ecosystems, and acquiring a comprehensive global database of carbon fluxes is essential for understanding the Earth's carbon cycle. The primary methods of obtaining the spatial distribution of land carbon fluxes include utilizing machine learning models based on in situ measurements, estimating through satellite remote sensing, and simulating ecosystem models. Recently, an innovative machine learning product known as the Global Carbon Flux Dataset (GCFD) has been released. In this study, we assessed the reliability of the GCFD by comparing it with existing data products, including two machine learning products (FLUXCOM and NIES (National Institute for Environmental Studies)), two ecosystem model products (TRENDY and EC-LUE (eddy covariance–light use efficiency model)), and one remote sensing product (Global Land Surface Satellite), on both site and global scales. Our findings indicate that, in terms of average absolute difference, the spatial distribution of the GCFD is most similar to the NIES product, albeit with slightly larger discrepancies compared to the other two types of products. When using site observations as the benchmark, gross primary production (GPP), respiration of ecosystem (RECO), and net ecosystem exchange of machine learning products exhibit higher R2 (ranging from 0.57 to 0.85, 0.53–0.79, and 0.31–0.70, respectively) compared to model products and remote sensing products. Furthermore, we analyzed the spatial and temporal distribution characteristics of carbon fluxes in various regions. The results demonstrate an upward trend in both GPP and RECO over the past two decades, while NEE exhibits an opposite trend. This trend is particularly pronounced in tropical regions, where higher GPP is observed in tropical, subtropical, and oceanic climate zones. Additionally, two remote sensing variables that influence changes in carbon fluxes, i.e., fraction absorbed photosynthetically active radiation and leaf area index, exhibit relatively consistent spatial and temporal characteristics. Overall, our study can provide valuable insights into different types of carbon flux products and contribute to understanding the general features of global carbon fluxes. [ABSTRACT FROM AUTHOR]

Global carbon fluxes describe the carbon exchange between land and atmosphere. However, already available global carbon fluxes datasets have not been adjusted by the available site data and deep learning tools. In this work, a global carbon fluxes dataset (named as GCFD) of gross primary productivity (GPP), terrestrial ecosystem respiration (RECO), and net ecosystem exchange (NEE) has been developed via a deep learning based convolutional neural network (CNN) model. The dataset has a spatial resolution of 1 km at three time steps per month from January 1999 to June 2020. Flux measurements were used as a training target while remote sensing of vegetation conditions and meteorological data were used as predictors. The results showed that CNN could outperform other commonly used machine learning methods such as random forest (RF) and artificial neural network (ANN) by leading to satisfactory performance with R2 values of the validation stage as 0.82, 0.72 and 0.62 for GPP, RECO, and NEE modelling, respectively. Thus, CNN trained using reanalysis meteorological data and remote sensing data was chosen to produce the global dataset. GCFD showed higher accuracy and more spatial details than some other global carbon flux datasets with reasonable spatial pattern and temporal variation. GCFD is also in accordance with vegetation conditions detected by remote sensing. Owing to the obtained results, GCFD can be a useful reference for various meteorological and ecological analyses and modelling, especially when high resolution carbon flux maps are required. [ABSTRACT FROM AUTHOR]

High-quality gridded soil moisture products are essential for many Earth system science applications, while the recent reanalysis and remote sensing soil moisture data are often available at coarse resolution and remote sensing data are only for the surface soil. Here, we present a 1 km resolution long-term dataset of soil moisture derived through machine learning trained by the in situ measurements of 1789 stations over China, named SMCI1.0 (Soil Moisture of China by in situ data, version 1.0). Random forest is used as a robust machine learning approach to predict soil moisture using ERA5-Land time series, leaf area index, land cover type, topography and soil properties as predictors. SMCI1.0 provides 10-layer soil moisture with 10 cm intervals up to 100 cm deep at daily resolution over the period 2000–2020. Using in situ soil moisture as the benchmark, two independent experiments were conducted to evaluate the estimation accuracy of SMCI1.0: year-to-year (ubRMSE ranges from 0.041 to 0.052 and R ranges from 0.883 to 0.919) and station-to-station experiments (ubRMSE ranges from 0.045 to 0.051 and R ranges from 0.866 to 0.893). SMCI1.0 generally has advantages over other gridded soil moisture products, including ERA5-Land, SMAP-L4, and SoMo.ml. However, the high errors of soil moisture are often located in the North China Monsoon Region. Overall, the highly accurate estimations of both the year-to-year and station-to-station experiments ensure the applicability of SMCI1.0 to study the spatial–temporal patterns. As SMCI1.0 is based on in situ data, it can be a useful complement to existing model-based and satellite-based soil moisture datasets for various hydrological, meteorological, and ecological analyses and models. The DOI link for the dataset is http://dx.doi.org/10.11888/Terre.tpdc.272415 (Shangguan et al., 2022). [ABSTRACT FROM AUTHOR]

• A location correction method is proposed for widespread legacy soil profiles. • Consistency between covariates and soil profile description is enhanced. • Location error is reduced, and soil mapping results are improved significantly. • Elevation was the primary controlling factor for correcting location error. Digital soil mapping relies on statistical relationships between soil profile observations and environmental covariates at the sample locations. However, inherent limitations of legacy soil profiles, such as inaccurate georeferencing, could frequently introduce location errors into these soil profiles that affect the quality of digital soil mapping. To address this challenge, this study focuses on reducing the location error of legacy soil profiles and evaluating the resulting impact on digital soil mapping. We improved the agreement between the detailed descriptive information of legacy soil profiles and the relatively accurate environmental covariates (such as elevation, slope, and land use) to reduce the location errors of the legacy soil profiles. Quantile regression forest models were constructed to predict soil properties and their uncertainty using legacy soil profiles before and after location error correction. Our results demonstrate that for the majority of soil variables, correcting positional errors in legacy soil profiles to some extent enhances the accuracy of the digital soil mapping. The largest improvement was found for soil organic carbon at 0–5 cm depth interval, with 21 % increase of MEC. The impact of reduced location error is particularly noteworthy in regions characterized by complex terrain. In addition, the details of the predicted maps of legacy soil profiles were improved after correcting for positional errors, which better represent the spatial variation of soil properties across China. Besides, we also found that elevation was the primary controlling factor for correcting location error of legacy soil profiles. This research presents a step towards producing high-resolution and high-quality spatial soil datasets, which can provide essential support for soil management and ensure future soil security. [ABSTRACT FROM AUTHOR]

Remote sensing soil moisture (SM) has been widely used in various earth science studies and applications, but their low resolution limits their usage and downscaling of them is needed. In this study, we proposed a spatial downscaling method for SM based on random forest considering soil moisture memory and mass conservation to improve downscaling performance. The lagged SM was added as a predictor to represent soil moisture memory, in addition to the regular predictors in previous downscaling studies. The Soil Moisture Active Passive (SMAP) SM data of the Pearl River Basin were used to test our downscaling method. The results show that the downscaling model obtained good performance on the test set (R2 = 0.848, ubRMSE = 0.034 m3/m3 and Bias = 0.008 m3/m3). The spatial and temporal performance of the RF downscaling model can be improved by adding lagged SM variables. Downscaled data obtained can retain the information of the original SMAP SM data well and show more spatial details, and mass conservation correction is considered to be useful to eliminate systematic bias of the downscaling model. Downscaled SM achieved acceptable performance in in situ validation, though it was inevitably limited by the performance of the original SMAP data. The proposed downscaling method can serve as a powerful tool for the development of high-resolution SM information. [ABSTRACT FROM AUTHOR]

The accurate prediction of surface soil moisture (SM) is crucial for understanding hydrological processes. Deep learning (DL) models such as the long short-term memory model (LSTM) provide a powerful method and have been widely used in SM prediction. However, few studies have notably high success rates due to lacking prior knowledge in forms such as causality. Here we present a new causality-structure-based LSTM model (CLSTM), which could learn time interdependency and causality information for hydrometeorological applications. We applied and compared LSTM and CLSTM methods for forecasting SM across 64 FLUXNET sites globally. The results showed that CLSTM dramatically increased the predictive performance compared with LSTM. The Nash–Sutcliffe efficiency (NSE) suggested that more than 67% of sites witnessed an improvement of SM simulation larger than 10%. It is highlighted that CLSTM had a much better generalization ability that can adapt to extreme soil conditions, such as SM response to drought and precipitation events. By incorporating causal relations, CLSTM increased predictive ability across different lead times compared to LSTM. We also highlighted the critical role of physical information in the form of causality structure to improve drought prediction. At the same time, CLSTM has the potential to improve predictions of other hydrometeorological variables. [ABSTRACT FROM AUTHOR]

Accurate spatiotemporal predictions of surface soil moisture (SM) are important for many critical applications. Machine learning models provide a powerful method for building an accurate and reliable predictive model of SM. However, the models used in recent studies have some limitations, including lack of spatial autocorrelation (SAC), vague representation of important features, and primarily focused on the one-step forecast. Thus, we proposed an attention-based convolutional long short-term memory model (AttConvLSTM) for multistep forecasting. The model includes three layers, spatial compression, axial attention, and encoder–decoder prediction, which are used for compressing spatial information, feature extraction, and multistep prediction, respectively. The model was trained using surface SM from the Soil Moisture Active Passive L4 product at 18-km spatial resolution over the United States. The results show that AttConvLSTM predicts 24 h ahead SM with mean R2 and RMSE is equal to 0.82 and 0.02, respectively. Compared with LSTM, AttConvLSTM improves the model performance over 73.6% of regions, with an improvement of 8.4% and 17.4% in R2 and RMSE, respectively. The performance of the model is mainly influenced by temporal autocorrelation (TAC). Moreover, we also highlight the importance of SAC on model performance, especially over regions with high SAC and low TAC. Our model is also competent for SM predictions from several hours to several days, which could be a useful tool for predicting all meteorological variables and forecasting extremes. [ABSTRACT FROM AUTHOR]

Soil moisture (SM) is a key variable of land surface‐atmosphere interactions. Data‐driven methods have been used to predict SM, but the predictability of SM has not been well evaluated. This study investigated what variables and methods can be used to better predict SM for leading times of 7 days or longer with a global coverage of FLUXNET site data for the first time. Three machine‐learning models, that is, Bayesian linear regression, random forest, and gradient boosting regression tree, are used for the prediction. Variables including atmospheric forcing, surface soil temperature, time variables (year, day of year, and hour), the Fourier transformation of time variables, and lagged SM (7‐ to 14‐day lagged) were sequentially added into models. A framework with five experiments is designed for factorial exploration of SM predictability. A stepwise method was used to build the best models for each site. The performance of regression models became better when adding more explaining variables in most cases. The results showed that from 50 to 95% of variation of the best models can be explained. The important explaining variables are lagged surface SM, followed by day of year, year, soil temperature, and atmospheric forcing. The predictability of SM depends highly on SM memory characteristics and the persistence of seasonality. The effect of SM memory characteristics on SM prediction as an initial condition question has been widely discussed in this paper. Our results also provide an insight that mechanisms of seasonality effects on SM should be also paid more attention to. [ABSTRACT FROM AUTHOR]

With the increasing application of machine learning (ML) in hydrometeorology, we face the urge to demystify the ML's black-box nature because it usually does not provide physically interpretable information to users. Here, we demonstrate multiple post-hoc interpretation methods to evaluate feature effects for hydrometeorological prediction. These methods were integrated and applied to soil moisture (SM) as an example, and random forest was used to establish a prediction model based on a FLUXNET site in Haibei, China. Different views of interpretability, feature importance, Shapley values, partial dependence plot, individual conditional expectation, and accumulated local effect were used to investigate how features affect the prediction. The result shows that a comprehensive understanding can be achieved of the relationship between predicted SM and affecting variables, including lagged SM, precipitation, and soil temperature. Thus, we advocated integrated interpretation tools to enhance the practicability of ML for hydrologists and other physical scientists. A toolbox named ExplainAI is provided. • The black-box nature limits machine learning for hydrometeorological forecast. • Random forest to predict soil moisture is interpreted by multi tools. • Shapley value, ICE, PDP, 1D ALE and 2D ALE are used to explore feature effects. • The interpretations accord with the hydrometeorological knowledge mostly. • The interpretation tools are advocated to enhance practicability of machine learning. [ABSTRACT FROM AUTHOR]

Depth to bedrock serves as the lower boundary of land surface models, which controls hydrologic and biogeochemical processes. This paper presents a framework for global estimation of depth to bedrock (DTB). Observations were extracted from a global compilation of soil profile data (ca. 1,30,000 locations) and borehole data (ca. 1.6 million locations). Additional pseudo-observations generated by expert knowledge were added to fill in large sampling gaps. The model training points were then overlaid on a stack of 155 covariates including DEM-based hydrological and morphological derivatives, lithologic units, MODIS surface reflectance bands and vegetation indices derived from the MODIS land products. Global spatial prediction models were developed using random forest and Gradient Boosting Tree algorithms. The final predictions were generated at the spatial resolution of 250 m as an ensemble prediction of the two independently fitted models. The 10-fold cross-validation shows that the models explain 59% for absolute DTB and 34% for censored DTB (depths deep than 200 cm are predicted as 200 cm). The model for occurrence of R horizon (bedrock) within 200 cm does a good job. Visual comparisons of predictions in the study areas where more detailed maps of depth to bedrock exist show that there is a general match with spatial patterns from similar local studies. Limitation of the data set and extrapolation in data spare areas should not be ignored in applications. To improve accuracy of spatial prediction, more borehole drilling logs will need to be added to supplement the existing training points in under-represented areas. [ABSTRACT FROM AUTHOR]

• Incorporating meta -learning into LSTM model reduces the overall PBIAS by 8.68%, improving the overall KGE by 7% compared to standard LSTM model. • Meta-LSTM model improves extreme runoff prediction, enhancing FLV by 2.73% and FHV by 11.04%, aiding in better flood management strategies. • The proposed meta -learning framework enhances performance across various deep learning models, demonstrating its versatility in runoff prediction tasks. In the field of hydrology, deep learning has become a prevalent tool for runoff simulation. However, the limitations stem from their primary focus on normality, which fails to accurately capture rare events. This study introduces a novel Long Short-Term Memory (LSTM) model, termed Meta-LSTM, which is based on the Model-Agnostic Meta-Learning (MAML) framework. The Meta-LSTM model is capable of dynamically fine-tuning its parameters to ensure effective adaptation to various runoff scenarios. From this Meta-LSTM model, the Meta-enhanced paradigm is subsequently proposed, ensuring effective adaptation to diverse models and datasets. We conducted experiments with various models on the CAMELS and CAMELS-AUS datasets. Compared to conventional deep learning models for runoff simulation, Meta-enhanced models exhibit substantial improvements. Specifically, the Meta-LSTM model reduces the PBIAS from −13.86 % to −5.18 %, increasing the Kling-Gupta Efficiency (KGE) by 7 % on the CAMELS dataset. For the Meta-GR4J-CNN model, the PBIAS decreases by 21 %, and the Nash-Sutcliffe Efficiency (NSE) increases by 2 % on the CAMELS-AUS dataset. This demonstrates the ability of the Meta-enhanced model to more accurately represent observed data without significant additional time costs. Our approach overcomes the problem of uniform training in simulations of complex scenarios and promises to significantly improve the accuracy of future hydrologic studies. [ABSTRACT FROM AUTHOR]

• Skillful prediction of soil moisture (SM) is useful but great challenges exist. • A novel encoder-decoder LSTM model with residual learning (EDR-LSTM) is developed. • EDR-LSTM improved about 20% in SM prediction at the lead time of 3, 5 and 10 days. • Both encoder-decoder and residual learning are useful in improving SM prediction. • The predictability of SM over various conditions was widely investigated. The skillful prediction of soil moisture can provide much help for many practical applications including ecosystem management and precision agriculture. It presents great challenges because the future variation of soil moisture has much uncertainty. Therefore, a novel encoder-decoder deep learning model with residual learning based on Long Short-Term Memory (EDT-LSTM) is developed in this study as an alternative data-intelligence tool. This model includes two layers: the encoder-decoder LSTM layer and LSTM with a fully connected layer, which is used to enhance the prediction ability by considering the intermediate time-series data between the input time step and the predictive time step. We tested EDT-LSTM for soil moisture prediction at the lead time of 1, 3, 5, 7 and 10 days by using data from FLUXNET sites. The result shows that the improvements brought by EDT-LSTM were about 7.95% (1 day), 10.10% (3 days), 12.68% (5 days), 15.49% (7 days) and 19.71% (10 days) in average according to the R2 taking LSTM as the baseline. Furthermore, the predictability of soil moisture over various conditions (i.e., different hyper-parameters in EDT-LSTM, different predictive models, different climate regions and different sites) has been widely discussed for the understanding of models' behavior in this paper. The proposed EDT-LSTM offered a new tool to predict soil moisture better. The code of EDT-LSTM is publicly available at https://github.com/ljz1228/CLM-LSTM-soil-moisture-prediction. [ABSTRACT FROM AUTHOR]

• Deep learning models predicted soil moisture well with limited SMAP samples. • Transfer learning improved predictions with additional samples from ERA5-land. • Transfer ConvLSTM performed the best with over 90% variation explained. • The predictive ability of different factors was widely investigated. • Transfer learning is advocated for datasets with limited samples like SMAP. The skillful soil moisture (SM) for the Soil Moisture Active Passive (SMAP) L4 product can provide substantial value for many practical applications including ecosystem management and precision agriculture. Deep learning (DL) models provide powerful methods for hydrologic variables' prediction such as SM. However, the sample size of daily SM in the SMAP product is quite small, which may lead to overfitting and further impact the accuracy of DL models. From this, we first tested whether excellent predictive performance can be achieved with limited SMAP samples by the Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and Convolutional LSTM (ConvLSTM) models, which are frequent used for hydrologic prediction. Then we pre-trained the DL models in the source domain (ERA5-land) and fine-tuned them in the target domain (SMAP). The results show that the transfer ConvLSTM model had the highest R2 ranging from 0.909 to 0.916 and the lowest RMSE ranging from 0.0239 to 0.0247 for the lead time of 3, 5 and 7 days, and the regression lines between the predicted and the observed SM were closer to the ideal line (y = x) than all the other DL models. All the performances of transfer DL models were better than those of their corresponding DL models without transfer learning and some regions witnessed an increased explained variation over 20%. The predictive ability of different factors (i.e., lagged SM, soil temperature, season, and precipitation) has been widely discussed in this paper. According the results, we advocate for applying cross-source transfer learning with DL models for SM prediction in newly built datasets. [ABSTRACT FROM AUTHOR]

• A hybrid deep learning model is proposed. • The hybrid model performs very well in hourly streamflow prediction. • Significance of model inputs are investigated by quantifying their contributions to predictions. Process-based streamflow prediction is subjected to large uncertainties in model parameters and parameterizations related to the complex processes involved in streamflow generation. The data-driven models offer efficient alternatives without considering the physical processes, but their applications are limited by non-stationarity existing in observations. In this study, we propose a hybrid model, namely the DIFF-FFNN-LSTM model, to predict hourly streamflow. The model comprises three components, namely the first-order difference (DIFF), feedforward neural network (FFNN), and long short-term memory network (LSTM). When applied to the Andun basin of China, the proposed DIFF-FFNN-LSTM model performs very well in hourly streamflow prediction, with a RMSE of 9.31 m3/s with average streamflow rate of 54 m3/s and a MAE of 3.63 m3/s for all the flood events in the testing period. The comparison with five other machine learning models (of similar complexity or model structure) and four statistical models show superiority of our proposed DIFF-FFNN-LSTM model. The Shapley Additive exPlanations was used to quantify the contribution of each model input to the prediction skill. Streamflow at the previous hour was identified as the most important input, and streamflow generally contribute more than precipitation. Inputs closer to the prediction time do not necessarily have a greater impact on the model prediction. The study highlights the power of the combining of different data-driven methods and the promising prospect of our hybrid model in hydrological predictions. [ABSTRACT FROM AUTHOR]