Abstract
AbstractHundreds of petabytes are produced annually at weather and climate forecast centers worldwide. Compression is essential to reduce storage and to facilitate data sharing. Current techniques do not distinguish the real from the false information in data, leaving the level of meaningful precision unassessed. Here we define the bitwise real information content from information theory for the Copernicus Atmospheric Monitoring Service (CAMS). Most variables contain fewer than 7 bits of real information per value and are highly compressible due to spatio-temporal correlation. Rounding bits without real information to zero facilitates lossless compression algorithms and encodes the uncertainty within the data itself. All CAMS data are 17× compressed relative to 64-bit floats, while preserving 99% of real information. Combined with four-dimensional compression, factors beyond 60× are achieved. A data compression Turing test is proposed to optimize compressibility while minimizing information loss for the end use of weather and climate forecast data.
Publisher
Springer Science and Business Media LLC
Cited by
13 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Quantized compression of SAR data: Bounds on signal fidelity, InSAR PS candidates identification and surface motion accuracy;International Journal of Applied Earth Observation and Geoinformation;2023-12
2. Automatic Search Guided Code Optimization Framework for Mixed-Precision Scientific Applications;Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis;2023-11-12
3. Black-box statistical prediction of lossy compression ratios for scientific data;The International Journal of High Performance Computing Applications;2023-06-14
4. Change a Bit to Save Bytes: Compression for Floating Point Time-Series Data;ICC 2023 - IEEE International Conference on Communications;2023-05-28
5. Discussion on “Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach”;Journal of Agricultural, Biological and Environmental Statistics;2023-05-11