Big Data in E&P: Real-Time Adaptive Analytics and Data-Flow Architecture

Abstract

Abstract Big Data Analytics are most effective when data-in-motion are combined with data-at-rest. Stream computing is a new way of analyzing high-frequency data for real-time complex-event-processing (CEP) and for scoring data against a physics-based or empirical model for predictive analytics, without having to store the data. Hadoop Map/Reduce and other NoSQL approaches are a new way of analyzing massive volumes of data whether semistructured or unstructured, which can be used to support the E&P industry's many physics-based methods in modeling and simulation, over a wide range of disciplines from geology & geophysics to reservoir, production, & facilities engineering. High-volume data of many different types can be landed in Hadoop and analyzed without extensive transformation into a relational database model. Combining Stream computing with Hadoop Map/Reduce or massively parallel processing relational data warehousing (MPP DW) enables the analysis of high-frequency data scored against continuously updated models, known as "Real-Time Adaptive Analytics." This design pattern provides a low-latency "Real-Time Data-Flow Architecture," which enables decisions during operational events and also at the speed of the overall business. Big Data Analytics also depends on tooling for building modeling and simulation applications and for performing analytics, as much as it depends on infrastructure. Augmenting the industry's many decades of progress in physics-based modeling and simulation, powerful empirical models can be developed with tools for statistical analytics, text analytics, AI, and machine-learning and then used as scoring and data-validation models by implementing stream computing on-site where the data are generated in oilfield operations. This paper will explore RealTime Adaptive Analytics and the Real-Time Data-Flow Architecture combining stream computing, Hadoop/NoSQL, and MPP data warehousing. Several compelling use cases in drilling and in production will be reviewed.