Evaluating lossy data compression on climate simulation data within a large ensemble
-
Published:2016-12-07
Issue:12
Volume:9
Page:4381-4403
-
ISSN:1991-9603
-
Container-title:Geoscientific Model Development
-
language:en
-
Short-container-title:Geosci. Model Dev.
Author:
Baker Allison H.ORCID, Hammerling Dorit M.ORCID, Mickelson Sheri A.ORCID, Xu Haiying, Stolpe Martin B.ORCID, Naveau PhillipeORCID, Sanderson BenORCID, Ebert-Uphoff ImmeORCID, Samarasinghe Savini, De Simone Francesco, Carbone Francesco, Gencarelli Christian N.ORCID, Dennis John M., Kay Jennifer E., Lindstrom PeterORCID
Abstract
Abstract. High-resolution Earth system model simulations generate enormous data volumes, and retaining the data from these simulations often strains institutional storage resources. Further, these exceedingly large storage requirements negatively impact science objectives, for example, by forcing reductions in data output frequency, simulation length, or ensemble size. To lessen data volumes from the Community Earth System Model (CESM), we advocate the use of lossy data compression techniques. While lossy data compression does not exactly preserve the original data (as lossless compression does), lossy techniques have an advantage in terms of smaller storage requirements. To preserve the integrity of the scientific simulation data, the effects of lossy data compression on the original data should, at a minimum, not be statistically distinguishable from the natural variability of the climate system, and previous preliminary work with data from CESM has shown this goal to be attainable. However, to ultimately convince climate scientists that it is acceptable to use lossy data compression, we provide climate scientists with access to publicly available climate data that have undergone lossy data compression. In particular, we report on the results of a lossy data compression experiment with output from the CESM Large Ensemble (CESM-LE) Community Project, in which we challenge climate scientists to examine features of the data relevant to their interests, and attempt to identify which of the ensemble members have been compressed and reconstructed. We find that while detecting distinguishing features is certainly possible, the compression effects noticeable in these features are often unimportant or disappear in post-processing analyses. In addition, we perform several analyses that directly compare the original data to the reconstructed data to investigate the preservation, or lack thereof, of specific features critical to climate science. Overall, we conclude that applying lossy data compression to climate simulation data is both advantageous in terms of data reduction and generally acceptable in terms of effects on scientific results.
Publisher
Copernicus GmbH
Reference56 articles.
1. Ana, F. and de Haan, L.: On the block maxima method in extreme value theory, Ann. Stat., 43, 276–298, 2015. 2. Baker, A., Xu, H., Dennis, J., Levy, M., Nychka, D., Mickelson, S., Edwards, J., Vertenstein, M., and Wegener, A.: A Methodology for Evaluating the Impact of Data Compression on Climate Simulation Data, in: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing, HPDC '14, 23–27 June 2014, Vancouver, Canada, 203–214, 2014. 3. Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., and Williamson, D.: A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0), Geosci. Model Dev., 8, 2829–2840, https://doi.org/10.5194/gmd-8-2829-2015, 2015. 4. Beirlant, J., Goegebeur, Y., Segers, J., and Teugels, J.: Statistics of Extremes: Theory and Applications, Wiley Series in Probability and Statistics, Hoboken, USA, 2004. 5. Bicer, T., Yin, J., Chiu, D., Agrawal, G., and Schuchardt, K.: Integrating online compression to accelerate large-scale data analytics applications. IEEE International Symposium on Parallel and Distributed Processing (IPDPS), 20–24 May 2013, Boston, Massachusetts, USA, 1205–1216, https://doi.org/10.1109/IPDPS.2013.81, 2013.
Cited by
46 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Impact of Lossy Compression Errors on Passive Seismic Data Analyses;Seismological Research Letters;2024-01-02 2. cuSZp: An Ultra-fast GPU Error-bounded Lossy Compression Framework with Optimized End-to-End Performance;Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis;2023-11-11 3. ADT-FSE: A New Encoder for SZ;Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis;2023-11-11 4. Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach;Journal of Agricultural, Biological and Environmental Statistics;2023-05-11 5. Discussion on “Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach”;Journal of Agricultural, Biological and Environmental Statistics;2023-05-11
|
|