Affiliation:
1. Department of Computers and Informatics, Faculty of Electrical Engineering and Informatics , Technical University of Košice , Letná 9 , Košice , Slovak Republic , Tel. +421 55 602 3175
Abstract
Abstract
A lot of scientific work is dedicated to the analysis of data. Most of the analyzed data, like data from space missions, are structured. The choice of data format can affect various characteristics - read/write speed of standard files, read/write speed of small files and read/write speed of compressed data formats. In this paper, we analyze binary data formats, proposed types of the tests and testing methods, and compare their performance with human-readable text format. We also discuss compressed and uncompressed modes available for data formats like HDF5 and netCDF. When disregarding precision, the best data format from the size perspective is lossy HDF5 without compression. Losless HDF5 without compression show the best speed performance. Lossy HDF5 without compression is the best balance between size reduction and speed. However, for specific criteria and types of files, there might be better candidates as detailed in the conclusion.
Reference15 articles.
1. [1] Van den EYNDEN, V. – CORTI, L. – WOOLLARD, M. – BISHOP, L. – HORTON, L.: Managing and sharing data; a best practice guide for researchers, UK Data Archive, No. () 2011
2. [2] Space Physics Data Facility NASA / Goddard Space Flight Center: CDF User’s Guide,, No. 3.8.1 (2021)
3. [3] REW, R. – DAVIS, G.: NetCDF: an interface for scientific data access, IEEE computer graphics and applications 10, No. 4 (1990) 76–82
4. [4] KOZIOL, Q. – ROBINSON, D. et all.: Hdf5, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), No. (2018)
5. [5] GECÁŠEK, D. – BOBIK, P. – GENCI, J.: Transparency of magnetosphere for cosmic rays in last two millennia, Proceedings of 37th International Cosmic Ray Conference — PoS(ICRC2021) 395, No. - (2021) 133510.22323/1.395.1335