Recognizing Threats From Unknown Real-Time Big Data System Faults

Abstract

Processing big data in real time creates threats to the validity of the knowledge produced. This chapter discusses problems that may occur within the real-time data and the risks to the knowledge pyramid and decisions made based upon the knowledge gleaned from the volumes of data processed in real-time environments. The authors hypothesize that not yet encountered faults may require fault handling, analytic models and an architectural framework to manage the faults and mitigate the risks of correlating or integrating otherwise uncorrelated big data and to ensure the source pedigree, quality, set integrity, freshness, and validity of the data. This chapter provides a number of examples to support the hypothesis. The objectives of the designers of these knowledge management systems must be to mitigate the faults resulting from real-time streaming processes while ensuring that variables such as synchronization, redundancy, and latency are addressed. This chapter concludes that with improved designs, real-time big data systems may continuously deliver the value of streaming big data.