Performance Optimization System for Hadoop and Spark Frameworks-Reference-Cited by-同舟云学术

Performance Optimization System for Hadoop and Spark Frameworks

Published:2020-12-01 Issue:6 Volume:20 Page:5-17
ISSN:1314-4081
Container-title:Cybernetics and Information Technologies
language:en
Short-container-title:

Author:

Astsatryan Hrachya¹,Kocharyan Aram²,Hagimont Daniel²,Lalayan Arthur³

Affiliation:

1. Institute for Informatics and Automation Problems of the National Academy of Sciences of the Republic of Armenia , Yerevan 0014 , Armenia

2. Université Fédérale Toulouse Midi-Pyrénées , Toulouse Cedex 7 , France

3. National Polytechnic University of Armenia , Yerevan 0009 , Armenia

Abstract

Abstract The optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows splitting large data sets into a set of blocks distributed on several machines. Data compression reduces data size and transfer time between disks and memory but requires additional processing. Therefore, finding an optimal tradeoff is a challenge, as a high compression factor may underload Input/Output but overload the processor. The paper aims to present a system enabling the selection of the compression tools and tuning the compression factor to reach the best performance in Apache Hadoop and Spark infrastructures based on simulation analyzes.

Publisher

Walter de Gruyter GmbH

Subject

General Computer Science

Reference28 articles.

1. 1. Chen, J., Y. Chen, X. Du, C. Li, J. Lu, S. Zhao, X. Zhou. Big Data Challenge: A Data Management Perspective. – Frontiers of Computer Science, Vol. 7, 2013, No 2, pp. 157-164.

2. 2. Lublinsky, B., K. T. Smith, A. Yakubovich. Professional Hadoop Solutions. Indiana, USA, John Wiley & Sons, 2013, p. 504.

3. 3. Zaharia, M., R. S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M. J. Franklin, A. Ghodsi. Apache Spark: A Unified Engine for Big Data Processing. – Communications of the ACM, Vol. 59, 2016, No 11, pp. 56-65.

4. 4. Cheng, D., X. Zhou, P. Lama, J. Mike, C. Jiang. Energy Efficiency Aware Task Assignment with DVFS in Heterogeneous Hadoop Clusters. – IEEE Transactions on Parallel and Distributed Systems, Vol. 29, 2017, No 1, pp. 70-82.

5. 5. Nitu, V., A. Kocharyan, H. Yaya, A. Tchana, D. Hagimont, H. Astsatryan. Working Set Size Estimation Techniques in Virtualized Environments: One Size Does Not Fit All – ACM Meas. Anal. Comput. Syst., Vol. 2, 2018, pp. 1-21.

Cited by 5 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Optimized Multi-Modular Services: Empowering Earth Observation Data Processing;Informatica;2024

2. About the possibility of executing tasks with a waiting time restriction in a multiprocessor system;PROCEEDINGS OF THE 1ST INTERNATIONAL CONFERENCE ON FRONTIER OF DIGITAL TECHNOLOGY TOWARDS A SUSTAINABLE SOCIETY;2023

3. Network Public Opinion Analysis Technology and Governance Measures in the New Media Era;Application of Intelligent Systems in Multi-modal Information Analytics;2022

4. Performance-efficient Recommendation and Prediction Service for Big Data frameworks focusing on Data Compression and In-memory Data Storage Indicators;Scalable Computing: Practice and Experience;2021-11-26

5. Simulation-Based Optimisation for Autonomous Transportation Systems Using a Parallel Real-Coded Genetic Algorithm with Scalable Nonuniform Mutation;Cybernetics and Information Technologies;2021-09-01