Data Dispersal for Large Storage Systems

Abstract

Data dispersal mechanism is a group of basic working protocols for oceans of data processing and it plays an important role in enhancing system error correction and avoiding storage device failure. The data recovery availability of the client is the main design goal of data dispersal algorithms. The system storage and computing costs are also important factors for a data dispersal algorithm. In this paper, we discussed how to disperse data with the special properties as well as with acceptable system costs. We developed a universal approach to measure storage node availability based on state variables. After analyzed the existing data processing algorithms, we presented a new hybrid data dispersal algorithm using the above approach in detail evaluation. Two redundancy policies are adapted to storage node with different node availability. According to our analysis, this algorithm has its advantages in system storing and communication bandwidth costs.