Author:
Elhmadany Mohammed,Elmadah Islam,Abdelmunim Hossam E.
Abstract
AbstractDistributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters.
Publisher
Springer Science and Business Media LLC
Subject
Information Systems and Management,Computer Networks and Communications,Hardware and Architecture,Information Systems
Reference46 articles.
1. Ng, Andrew, Machine learning yearning. deeplearning. ai. URL: https://www.deeplearning.ai, https://github.com/ajaymache/machine-learning-yearning 2018.
2. Teerapittayanon S, McDanel B, Kung HT. Distributed deep neural networks over the cloud, the edge, and end devices. 2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, 2017.
3. Krichevsky N, St Louis R, Guo T. Quantifying and improving performance of distributed deep learning with cloud storage. 2021 IEEE International Conference on Cloud Engineering (IC2E). IEEE, 2021.
4. 10 Best Machine Learning Software, unite.ai. Alex McFarland, 2022. [Online]. Available: https://www.unite.ai/10-best-machine-learning-software/ Accessed on: 2022.
5. Dai JJ, Wang Y, Qiu X, Ding D, Zhang Y, Wang Y, Jia X, Zhang CL, Wan Y, Li Z, Wang J et al. Bigdl: a distributed deep learning framework for big data. Proceedings of the ACM Symposium on Cloud Computing, 2019;50–60.