Affiliation:
1. Cyprus University of Technology, Limassol, Cyprus
2. University of Helsinki, Helsinki, Finland
Abstract
Big data processing systems (e.g., Hadoop, Spark, Storm) contain a vast number of configuration parameters controlling parallelism, I/O behavior, memory settings, and compression. Improper parameter settings can cause significant performance degradation and stability issues. However, regular users and even expert administrators grapple with understanding and tuning them to achieve good performance. We investigate existing approaches on parameter tuning for both batch and stream data processing systems and classify them into six categories: rule-based, cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. We summarize the pros and cons of each approach and raise some open research problems for automatic parameter tuning.
Publisher
Association for Computing Machinery (ACM)
Subject
General Computer Science,Theoretical Computer Science
Reference133 articles.
1. AMD Hadoop Tuning. 2012. AMD Hadoop Performance Tuning Guide. Retrieved from https://developer.amd.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf. AMD Hadoop Tuning. 2012. AMD Hadoop Performance Tuning Guide. Retrieved from https://developer.amd.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf.
2. Apache Flink. 2019. Apache Flink. Retrieved from https://flink.apache.org/. Apache Flink. 2019. Apache Flink. Retrieved from https://flink.apache.org/.
3. Apache Hadoop. 2019. Apache Hadoop. Retrieved from https://hadoop.apache.org/. Apache Hadoop. 2019. Apache Hadoop. Retrieved from https://hadoop.apache.org/.
4. Apache Samza. 2019. Apache Samza. Retrieved from http://samza.apache.org/. Apache Samza. 2019. Apache Samza. Retrieved from http://samza.apache.org/.
Cited by
57 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献