Navigating the Landscape of Distributed Computing Frameworks for Machine and Deep Learning

Abstract

For a number of reasons, distributed computing is crucial to machine learning and deep learning models. In the beginning, it makes it possible to train big models that won't fit in a single machine's memory. Second, by distributing the burden over several machines, it expedites the training process. Thirdly, it enables the management of vast amounts of data that may be dispersed across multiple devices or kept remotely. The system can continue processing data even if one machine fails because of distributed computing, which further improves fault tolerance. This chapter summarizes major frameworks Tensorflow, Pytorch, Apache spark Hadoop, and Horovod that are enabling developers to design and implement distributed computing models using large datasets. Some of the challenges faced by the distributed computing models are communication overhead, fault tolerance, load balancing, scalability and security, and the solutions are proposed to overcome the abovementioned challenges.