A Novel Compression Method of Spectral Data Matrix Based on the Low-Rank Approximation and the Fast Fourier Transform of the Singular Vectors

Abstract

Storage, processing, and transfer of huge matrices are becoming challenging tasks in the process analytical technology and scientific research. Matrix compression can solve these problems successfully. We developed a novel compression method of spectral data matrix based on its low-rank approximation and the fast Fourier transform of the singular vectors. This method differs from the known ones in that it does not require restoring the low-rank approximated matrix for further Fourier processing. Therefore, the compression ratio increases. A compromise between the losses of the accuracy of the data matrix restoring and the compression ratio was achieved by selecting the processing parameters. The method was applied to multivariate chemometrics analysis of the cow milk for determining fat and protein content using two data matrices (the file sizes were 5.7 and 12.0 MB) restored from their compressed form. The corresponding compression ratios were about 52 and 114, while the loss of accuracy of the analysis was less than 1% compared with processing of the non-compressed matrix. A huge, simulated matrix, compressed from 400 MB to 1.9 MB, was successfully used for multivariate calibration and segment cross-validation. The data set simulated a large matrix of 10 000 low-noise infrared spectra, measured in the range 4000–400 cm−1 with a resolution of 0.5 cm−1. The corresponding file was compressed from 262.8 MB to 19.8 MB. The discrepancies between original and restored spectra were less than the standard deviation of the noise. The method developed in the article clearly demonstrated its potential for future applications to chemometrics-enhanced spectrometric analysis with limited options of memory size and data transfer rate. The algorithm used the standard routines of Matlab software.