Kulla‐RIV: A composing model with integrity verification for efficient and reliable data processing services

Abstract

AbstractThis article presents the design and implementation of a reliable computing virtual container‐based model with integrity verification for data processing strategies named the reliability and integrity verification (RIV) scheme. It has been integrated into a system construction model as well as existing workflow engines (e.g., Kulla and Makeflow) for composing in‐memory systems. In the RIV scheme, the reliability (R) component is in charge of providing an implicit fault tolerance mechanism for the processes of data acquisition and storage that take place in a data processing system. The integrity verification (IV) component is in charge of ensuring that data transmitted/received between two processing stages are correct and are not modified during the transmission process. To show the feasibility of using the RIV scheme, real‐world applications were created by using different distributed and parallel systems to solve use cases of satellite and medical imagery processing. This evaluation revealed encouraging results as some solutions that assumed the cost (overhead) of using the RIV scheme, for example, Kulla (the Kulla‐RIV solution), achieve better response times than others without the RIV scheme (e.g., Makeflow) that remain exposed to the risks caused by to the lack of RIV strategies.