Author:
Peyravi Narges,Moeini Ali
Abstract
AbstractHadoop MapReduce is a framework to process vast amounts of data in the cluster of machines in a reliable and fault-tolerant manner. Since being aware of the runtime of a job is crucial to subsequent decisions of this platform and being better management, in this paper we propose a new method to estimate the runtime of a job. For this purpose, after analysis the anatomy of processing a job in Hadoop MapReduce precisely, we consider two cases: when a job runs for the first time or a job has run previously. In the first case, by considering essential and efficient parameters that higher impact on runtime we formulate each phase of the Hadoop execution pipeline and state them by mathematical expressions to calculate runtime of a job. In the second case, by referring to the profile or history of a job in the database and use a weighting system the runtime is estimated. The results show the average error rate is less than 12% in the estimation of runtime for the first run and less than 8.5% when the profile or history of the job has existed.
Publisher
Springer Science and Business Media LLC
Subject
Information Systems and Management,Computer Networks and Communications,Hardware and Architecture,Information Systems
Reference28 articles.
1. White T. Hadoop: the definitive guide. 4th ed. Newton: O’Reilly Media, Inc; 2015.
2. Perera S. Hadoop MapReduce Cookbook. Birmingham: Packt Publishing Ltd; 2013.
3. Alapati SR. Expert Hadoop administration: managing, tuning, and securing spark, YARN, and HDFS. Boston: Addison-Wesley Professional; 2016.
4. Heidari S, Alborzi M, Radfar R, Afsharkazemi MA, Ghatari AR. Big data clustering with varied density based on MapReduce. J Big Data. 2019;6(1):77.
5. Singh R, Kaur PJ. Analyzing performance of Apache Tez and MapReduce with hadoop multinode cluster on Amazon cloud. J Big Data. 2016;3(1):19.
Cited by
9 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Predictive modelling of MapReduce job performance in cloud environments using machine learning techniques;Journal of Big Data;2024-07-23
2. Runtime Estimation Using Linear Regression Method in Hadoop MapReduce;Proceedings of the 7th International Conference on Networking, Intelligent Systems and Security;2024-04-18
3. An Optimal Method for Testing Jobs' Execution in MapReduce Based Systems;2023 9th International Conference on Control, Decision and Information Technologies (CoDIT);2023-07-03
4. Comparative Analysis of K-means and Hierarchical Clustering in Bigdata Environment;2022 6th International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS);2022-12-21
5. Real-Time Scheduling with Predictions;2022 IEEE Real-Time Systems Symposium (RTSS);2022-12