Orchestrating Big Data Analysis Workflows in the Cloud-Reference-Cited by-同舟云学术

Orchestrating Big Data Analysis Workflows in the Cloud

Published:2020-09-30 Issue:5 Volume:52 Page:1-41
ISSN:0360-0300
Container-title:ACM Computing Surveys
language:en
Short-container-title:ACM Comput. Surv.

Author:

Barika Mutaz¹^ORCID,Garg Saurabh¹,Zomaya Albert Y.²,Wang Lizhe³,Moorsel Aad Van⁴,Ranjan Rajiv⁵

Affiliation:

1. University of Tasmania, Tasmania, Australia

2. University of Sydney, New South Wales, Australia

3. China University of Geoscience (Wuhan), Wuhan, P. R China

4. Newcastle University, United Kingdom

5. China University of Geoscience (Wuhan) and Newcastle University, United Kingdom

Abstract

Interest in processing big data has increased rapidly to gain insights that can transform businesses, government policies, and research outcomes. This has led to advancement in communication, programming, and processing technologies, including cloud computing services and technologies such as Hadoop, Spark, and Storm. This trend also affects the needs of analytical applications, which are no longer monolithic but composed of several individual analytical steps running in the form of a workflow. These big data workflows are vastly different in nature from traditional workflows. Researchers are currently facing the challenge of how to orchestrate and manage the execution of such workflows. In this article, we discuss in detail orchestration requirements of these workflows as well as the challenges in achieving these requirements. We also survey current trends and research that supports orchestration of big data workflows and identify open research challenges to guide future developments in this area.

Funder

Natural Environment Research Council

Publisher

Association for Computing Machinery (ACM)

Subject

General Computer Science,Theoretical Computer Science

Link

https://dl.acm.org/doi/pdf/10.1145/3332301

Reference151 articles.

1. {n.d.}. Chapter 15 - A taxonomy and survey of fault-tolerant workflow manag. sys. in cloud and dist. computing env. In Software Architecture for Big Data and the Cloud Ivan Mistrik Rami Bahsoon Nour Ali Maritta Heisel and Bruce Maxim (Eds.). Morgan Kaufmann. {n.d.}. Chapter 15 - A taxonomy and survey of fault-tolerant workflow manag. sys. in cloud and dist. computing env. In Software Architecture for Big Data and the Cloud Ivan Mistrik Rami Bahsoon Nour Ali Maritta Heisel and Bruce Maxim (Eds.). Morgan Kaufmann.

2. 2015. Anomaly Detection over Sensor Data Streams. Retrieved from http://wiki.clommunity-project.eu/pilots:and. 2015. Anomaly Detection over Sensor Data Streams. Retrieved from http://wiki.clommunity-project.eu/pilots:and.

3. Adamu et al. 2016. A Survey on Big Data Indexing Strategies. Technical Report. SLAC National Accelerator Lab. Menlo Park CA. Adamu et al. 2016. A Survey on Big Data Indexing Strategies. Technical Report. SLAC National Accelerator Lab. Menlo Park CA.

4. Data-Intensive Workflow Optimization Based on Application Task Graph Partitioning in Heterogeneous Computing Systems

5. Optimization of data-intensive workflows in stream-based data processing models

Cited by 36 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. A Taxonomy for Cloud Storage Cost;Communications in Computer and Information Science;2024

2. Anomaly Detection in Cloud Computing using Knowledge Graph Embedding and Machine Learning Mechanisms;Journal of Grid Computing;2023-12-29

3. Passenger flow prediction and management method of urban public transport based on SDAE model and improved Bi-LSTM neural network;Journal of Intelligent & Fuzzy Systems;2023-12-02

4. A New Classification for Data Placement Techniques in Cloud Computing;2023 IEEE 6th International Conference on Cloud Computing and Artificial Intelligence: Technologies and Applications (CloudTech);2023-11-21

5. Container-Based Data Pipelines on the Computing Continuum for Remote Patient Monitoring;Computer;2023-10