Abstract
Accurate soil moisture (SM) prediction is critical for understanding hydrological processes. Physics-based (PB) models exhibit large uncertainties in SM predictions arising from uncertain parameterizations and insufficient representation of land-surface processes. In addition to PB models, deep learning (DL) models have been widely used in SM predictions recently. However, few pure DL models have notably high success rates due to lacking physical information. Thus, we developed hybrid models to effectively integrate the outputs of PB models into DL models to improve SM predictions. To this end, we first developed a hybrid model based on the attention mechanism to take advantage of PB models at each forecast time scale (attention model). We further built an ensemble model that combined the advantages of different hybrid schemes (ensemble model). We utilized SM forecasts from the Global Forecast System to enhance the convolutional long short-term memory (ConvLSTM) model for 1–16 days of SM predictions. The performances of the proposed hybrid models were investigated and compared with two existing hybrid models. The results showed that the attention model could leverage benefits of PB models and achieved the best predictability of drought events among the different hybrid models. Moreover, the ensemble model performed best among all hybrid models at all forecast time scales and different soil conditions. It is highlighted that the ensemble model outperformed the pure DL model over 79.5% of in situ stations for 16-day predictions. These findings suggest that our proposed hybrid models can adequately exploit the benefits of PB model outputs to aid DL models in making SM predictions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Beck, H. E., and Coauthors, 2021: Evaluation of 18 satellite-and model-based soil moisture products using in situ measurements from 826 sensors. Hydrology and Earth System Sciences, 25, 17–40, https://doi.org/10.5194/hess-25-17-2021.
Brooks, P. D., J. Chorover, Y. Fan, S. E. Godsey, R. M. Maxwell, J. P. McNamara, and C. Tague, 2015: Hydrological partitioning in the critical zone: Recent advances and opportunities for developing transferable understanding of water cycle dynamics. WaterResourse Research., 51, 6973–6987, https://doi.org/10.1002/2015WR017039.
Cai, Y. L., P. R. Fan, S. Lang, M. Y. Li, Y. Muhammad, and A. X. Liu, 2022: Downscaling of SMAP soil moisture data by using a deep belief network. Remote Sensing, 14, 5681, https://doi.org/10.3390/rs14225681.
Cho, K., B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, 2014: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406.1078, https://doi.org/10.48550/arXiv.1406.1078.
Crow, W. T., F. Chen, R. H. Reichle, Y. Xia, and Q. Liu, 2018: Exploiting soil moisture, precipitation, and streamflow observations to evaluate soil moisture/runoff coupling in land surface models. Geophysical Research Letter, 45, 4869–4878, https://doi.org/10.1029/2018GL077193.
Cui, Z., Y. L. Zhou, S. L. Guo, J. Wang, and C. Y. Xu, 2022: Effective improvement of multi-step-ahead flood forecasting accuracy through encoder-decoder with an exogenous input structure. Journal of Hydrology, 609, 127764, https://doi.org/10.1016/j.jhydrol.2022.127764.
Daw, A., A. Karpatne, W. D. Watkins, J. S. Read, and V. Kumar, 2022: Physics-guided neural networks (PGNN): An application in lake temperature modeling. Knowledge Guided Machine Learning, Chapman and Hall/CRC, 353–372.
de Rosnay, P., J. Munoz-Sabater, C. Albergel, L. Isaksen, S. English, M. Drusch, and J. P. Wigneron, 2020: SMOS brightness temperature forward modelling and long term monitoring at ECMWF. Remote Sensing of Environment, 237, 111424, https://doi.org/10.1016/j.rse.2019.111424.
Dharssi, I., K. J. Bovis, B. Macpherson, and C. P. Jones, 2011: Operational assimilation of ASCAT surface soil wetness at the Met Office. Hydrology and Earth System Sciences, 15, 2729–2746, https://doi.org/10.5194/hess-15-2729-2011.
Dorigo, W., and Coauthors, 2017: ESA CCI Soil Moisture for improved Earth system understanding: State-of-the art and future directions. Remote Sensing of Environment, 203, 185–215, https://doi.org/10.1016/j.rse.2017.07.001.
Dorigo, W. A., and Coauthors, 2013: Global automated quality control of in situ soil moisture data from the international soil moisture network. Vadose Zone Journal, 12, 1–21, https://doi.org/10.2136/vzj2012.0097.
ElSaadani, M., E. Habib, A. M. Abdelhameed, and M. Bayoumi, 2021: Assessment of a spatiotemporal deep learning approach for soil moisture prediction and filling the gaps in between soil moisture observations. Frontiers in Artificial Intelligence, 4, 362344, https://doi.org/10.3899/frai.2021.636234.
Entekhabi, D., R. H. Reichle, R. D. Koster, and W. T. Crow, 2010: Performance metrics for soil moisture retrievals and application requirements. Journal of Hydrometeorology, 11, 832–840, https://doi.org/10.1175/2010JHM1223.1.
Esit, M., S. Kumar, A. Pandey, D. M. Lawrence, I. Rangwala, and S. Yeager, 2021: Seasonal to multi-year soil moisture drought forecasting. npj Climate and Atmospheric Science, 4, 16, https://doi.org/10.1038/s41612-021-00172-z.
Fan, Y., and H. van den Dool, 2011: Bias correction and forecast skill of NCEP GFS ensemble week-1 and week-2 precipitation, 2-m surface air temperature, and soil moisture forecasts. Weather and Forecasting, 26, 355–370, https://doi.org/10.1175/WAF-D-10-05028.1.
Fang, K., and C. P. Shen, 2020: Near-real-time forecast of satellite-based soil moisture using long short-term memory with an adaptive data integration kernel. Journal of Hydrometeorology, 21, 399–413, https://doi.org/10.1175/JHM-D-19-0169.1.
Fang, K., M. Pan, and C. P. Shen, 2019: The value of SMAP for long-term soil moisture estimation with the help of deep learning. IEEE Transactions on Geoscience and Remote Sensing, 57, 2221–2233, https://doi.org/10.1109/TGRS.2018.2872131.
Fang, K., C. P. Shen, D. Kifer, and X. Yang, 2017: Prolongation of SMAP to spatiotemporally seamless coverage of continental U.S. using a deep learning neural network. Geophys. Res. Lett., 44, 11030–11039, https://doi.org/10.1002/2017GL075619.
Feng, D. P., J. T. Liu, K. Lawson, and C. P. Shen, 2022: Differentiable, learnable, regionalized process-based models with multiphysical outputs can approach state-of-the-art hydrologic prediction accuracy. Water Resour. Res., 58, e2022WR032404, https://doi.org/10.1029/2022WR032404.
Ford, T. W., and S. M. Quiring, 2019: Comparison of contemporary in situ, model, and satellite remote sensing soil moisture with a focus on drought monitoring. Water Resour. Res., 55, 1565–1582, https://doi.org/10.1029/2018WR024039.
Gruber, A., C. H. Su, S. Zwieback, W. Crow, W. Dorigo, and W. Wagner, 2016: Recent advances in (soil moisture) triple collocation analysis. International Journal of Applied Earth Observation and Geoinformation, 45, 200–211, https://doi.org/10.1016/j.jag.2015.09.002.
Heimhuber, V., M. G. Tulbure, and M. Broich, 2017: Modeling multidecadal surface water inundation dynamics and key drivers on large river basin scale using multiple time series of earth-observation and river flow data. Water Resour. Res., 53, 1251–1269, https://doi.org/10.1002/2016WR019858.
Huang, F. N., W. Shangguan, Q. L. Li, L. Li, and Y. Zhang, 2023: Beyond prediction: An integrated post-hoc approach to interpret complex model in hydrometeorology. Environmental Modelling & Software, 167, 105762, https://doi.org/10.1016/j.envsoft.2023.105762.
Kanamitsu, M., C.-H. Lu, J. Schemm, and W. Ebisuzaki, 2003: The predictability of soil moisture and near-surface temperature in Hindcasts of the NCEP seasonal forecast model. J. Climate, 16, 510–521, https://doi.org/10.1175/1520-0442(2003)016<0510:TPOSMA>2.0.CO;2.
Kannan, A., G. Tsagkatakis, R. Akbar, D. Selva, V. Ravindra, R. Levinson, S. Nag, and M. Moghaddam, 2022: Forecasting soil moisture using a deep learning model integrated with passive microwave retrieval. Preprints, IGARSS 2022–2022 IEEE International Geoscience and Remote Sensing Symposium, Kuala Lumpur, Malaysia, IEEE, 6112–6114, https://doi.org/10.1109/IGARSS46834.2022.9883245.
Karthikeyan, L., and A. K. Mishra, 2021: Multi-layer high-resolution soil moisture estimation using machine learning over the United States. Remote Sensing of Environment, 266, 112706, https://doi.org/10.1016/j.rse.2021.112706.
Kim, H., and Coauthors, 2020: Global scale error assessments of soil moisture estimates from microwave-based active and passive satellites and land surface models over forest and mixed irrigated/dryland agriculture regions. Remote Sensing of Environment, 511, 110522, https://doi.org/10.1016/j.rse.2020.112052.
Kingma, D. P., and J. Ba, 2017: Adam: A method for stochastic optimization. arXiv:1412.6980, https://doi.org/10.48550/arXiv.1412.6980.
Klocek, S., and Coauthors, 2022: MS-nowcasting: Operational precipitation nowcasting with convolutional LSTMs at Microsoft weather. arXiv:2111.09954, https://doi.org/10.48550/arXiv.2111.09954.
Lawston, P. M., J. A. Santanello Jr., and S. V. Kumar, 2017: Irrigation signals detected from SMAP soil moisture retrievals. Geophys. Res. Lett., 44, 11860–11867, https://doi.org/10.1002/2017GL075733.
Lee, J., S. Park, J. Im, C. Yoo, and E. Seo, 2022: Improved soil moisture estimation: Synergistic use of satellite observations and land surface models over CONUS based on machine learning. J. Hydrol., 609, 127749, https://doi.org/10.1016/j.jhydrol.2022.127749.
Li, L., Y. J. Dai, W. Shangguan, N. Wei, Z. W. Wei, and S. Gupta, 2022a: Multistep forecasting of soil moisture using spatiotemporal deep encoder-decoder networks. Journal of Hydrometeorology, 23, 337–350, https://doi.org/10.1175/JHM-D-21-0131.1.
Li, L., Y. J. Dai, W. Shangguan, Z. W. Wei, N. Wei, and Q. L. Li, 2022b: Causality-structured deep learning for soil moisture predictions. Journal of Hydrometeorology, 23, 1315–1331, https://doi.org/10.1175/JHM-D-21-0206.1.
Li, Q. L., Z. Y. Wang, W. Shangguan, L. Li, Y. F. Yao, and F. H. Yu, 2021: Improved daily SMAP satellite soil moisture prediction over China using deep learning model with transfer learning. J. Hydrol., 600, 126698, https://doi.org/10.1016/j.jhydrol.2021.126698.
Li, Y., S. Grimaldi, V. R. N. Pauwels, and J. P. Walker, 2018: Hydrologic model calibration using remotely sensed soil moisture and discharge measurements: The impact on predictions at gauged and ungauged locations. J. Hydrol., 557, 897–909, https://doi.org/10.1016/j.jhydrol.2018.01.013.
Liu, L. C., and Coauthors, 2022: KGML-ag: A modeling framework of knowledge-guided machine learning to simulate agroecosystems: A case study of estimating N2O emission using data from mesocosm experiments. Geoscientific Model Development, 15, 2839–2858, https://doi.org/10.5194/gmd-15-2839-2022.
Liu, W. B., T. Yang, F. B. Sun, H. Wang, Y. Feng, and M. Y. Du, 2021: Observation-constrained projection of global flood magnitudes with anthropogenic warming. Water Resour. Res., 55, e2020WR028830, https://doi.org/10.1029/2020WR028830.
Luo, L. F., E. F. Wood, and M. Pan, 2007: Bayesian merging of multiple climate model forecasts for seasonal hydrological predictions. J. Geophys. Res.: Atmos., 112, D10102, https://doi.org/10.1029/2006JD007655.
Maggioni, V., E. N. Anagnostou, and R. H. Reichle, 2012: The impact of model and rainfall forcing errors on characterizing soil moisture uncertainty in land surface modeling. Hydrology and Earth System Sciences, 16, 3499–3515, https://doi.org/10.5194/hess-16-3499-2012.
Martínez-Fernández, J., A. González-Zamora, N. Sánchez, and A. Gumuzzio, 2015: A soil water based index as a suitable agricultural drought indicator. J. Hydrol., 522, 265–273, https://doi.org/10.1016/j.jhydrol.2014.12.051.
Mishra, A., T. Vu, A. V. Veettil, and D. Entekhabi, 2017: Drought monitoring with soil moisture active passive (SMAP) measurements. J. Hydrol., 552, 620–632, https://doi.org/10.1016/j.jhydrol.2017.07.033.
Muñoz-Sabater, J., H. Lawrence, C. Albergel, P. Rosnay, L. Isaksen, S. Mecklenburg, Y. Kerr, and M. Drusch, 2019: Assimilation of SMOS brightness temperatures in the ECMWF integrated forecasting system. Quart. J. Roy. Meteor. Soc., 145, 2524–2548, https://doi.org/10.1002/qj.3577.
Muñoz-Sabater, J., and Coauthors, 2021: ERA5-Land: A state-of-the-art global reanalysis dataset for land applications. Earth System Science Data, 13, 4349–4383, https://doi.org/10.5194/essd-13-4349-2021.
O, S., and R. Orth, 2021: Global soil moisture data derived through machine learning trained with in-situ measurements. Scientific Data, 8, 170, https://doi.org/10.1038/s41597-021-00964-1.
Peng, J., and Coauthors, 2021: A roadmap for high-resolution satellite soil moisture applications–confronting product characteristics with user requirements. Remote Sensing of Environment, 252, 112162, https://doi.org/10.1016/j.rse.2020.112162.
Read, J. S., and Coauthors, 2019: Process-guided deep learning predictions of lake water temperature. Water Resour. Res., 55, 9173–9190, https://doi.org/10.1029/2019WR024922.
Reichle, R. H., and Coauthors, 2017: Assessment of the SMAP Level-4 surface and root-zone soil moisture product using in situ measurements. Journal of Hydrometeorology, 18, 2621–2645, https://doi.org/10.1175/JHM-D-17-0063.1.
Santanello, J. A. Jr., P. Lawston, S. Kumar, and E. Dennis, 2019: Understanding the impacts of soil moisture initial conditions on NWP in the context of land-atmosphere coupling. Journal of Hydrometeorology, 20, 793–819, https://doi.org/10.1175/JHM-D-18-0186.1.
Seneviratne, S. I., T. Corti, E. L. Davin, M. Hirschi, E. B. Jaeger, I. Lehner, B. Orlowsky, and A. J. Teuling, 2010: Investigating soil moisture-climate interactions in a changing climate: A review. Earth-Science Reviews, 99, 125–161, https://doi.org/10.1016/j.earscirev.2010.02.004.
Slater, L. J., and Coauthors, 2023: Hybrid forecasting: Blending climate predictions with AI models. Hydrology and Earth System Sciences, 27, 1865–1889, https://doi.org/10.5194/hess-27-1865-2023.
Speight, L. J., M. D. Cranston, C. J. White, and L. Kelly, 2021: Operational and emerging capabilities for surface water flood forecasting. WIREs Water, 8, e1517, https://doi.org/10.1002/wat2.1517.
Stoffelen, A., 1998: Toward the true near-surface wind speed: Error modeling and calibration using triple collocation. J. Geophys. Res.: Oceans, 103, 7755–7766, https://doi.org/10.1029/97JC03180.
Wigneron, J. P., and Coauthors, 2018: SMOS-IC: Current status and overview of soil moisture and VOD applications. Preprints, IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symp., Valencia, Spain, IEEE, 1451–1453, https://doi.org/10.1109/IGARSS.2018.8519382.
Willard, J., X. W. Jia, S. M. Xu, M. Steinbach, and V. Kumar, 2022a: Integrating scientific knowledge with machine learning for engineering and environmental systems. ACM Computing Surveys, ACM Computing Surveys, 55, 66, https://doi.org/10.1145/3514228.
Wood, A. W., and D. P. Lettenmaier, 2006: A test bed for new seasonal hydrologic forecasting approaches in the western United States. Bull. Amer. Meteor. Soc., 87, 1699–1712, https://doi.org/10.1175/BAMS-87-12-1699.
Xia, Y. L., J. Sheffield, M. B. Ek, J. R. Dong, N. Chaney, H. L. Wei, J. Meng, and E. F. Wood, 2014: Evaluation of multimodel simulated soil moisture in NLDAS-2. J. Hydrol., 512, 107–125, https://doi.org/10.1016/j.jhydrol.2014.02.027.
Yamazaki, D., and Coauthors, 2017: A high-accuracy map of global terrain elevations. Geophys. Res. Lett., 44, 5844–5853, https://doi.org/10.1002/2017GL072874.
Yang, H. C., H. X. Wang, G. B. Fu, H. M. Yan, P. P. Zhao, and M. H. Ma, 2017: A modified soil water deficit index (MSWDI) for agricultural drought monitoring: Case study of Songnen Plain, China. Agricultural Water Management, 194, 125–138, https://doi.org/10.1016/j.agwat.2017.07.022.
Yin, J. F., C. R. Hain, X. W. Zhan, J. R. Dong, and M. Ek, 2019: Improvements in the forecasts of near-surface variables in the Global Forecast System (GFS) via assimilating ASCAT soil moisture retrievals. J. Hydrol., 578, 124018, https://doi.org/10.1016/j.jhydrol.2019.124018.
Zhang, R. Q., and Coauthors, 2021: Assessment of agricultural drought using soil water deficit index based on ERA5-land soil moisture data in four southern provinces of China. Agriculture, 11, 411, https://doi.org/10.3390/agriculture11050411.
Acknowledgements
Lu LI was supported by the Natural Science Foundation of China (Grant Nos. 42088101 and 42205149); Zhongwang WEI was supported by the Natural Science Foundation of China (Grant No. 42075158); Wei SHANGGUAN was supported by the Natural Science Foundation of China (Grant No. 41975122); and Yonggen ZHANG was supported by the National Natural Science Foundation of Tianjin (Grant No. 20JCQNJC01660). All data, source codes and example codes are available at https://github.com/leelew/HybridHydro.
Author information
Authors and Affiliations
Corresponding author
Additional information
Article Highlights
• A hybrid scheme is proposed to exploit the benefits of physics-based and deep learning models.
• An ensemble hybrid model is proposed to combine the advantages of different hybrid models for improving soil moisture predictions.
• The proposed hybrid models outperformed pure deep learning models and other hybrid models.
This paper is a contribution to the special issue on AI Applications in Atmospheric and Oceanic Science: Pioneering the Future.
Electronic Supplementary Material to
Rights and permissions
About this article
Cite this article
Li, L., Dai, Y., Wei, Z. et al. Enhancing Deep Learning Soil Moisture Forecasting Models by Integrating Physics-based Models. Adv. Atmos. Sci. (2024). https://doi.org/10.1007/s00376-023-3181-8
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00376-023-3181-8