Differentiable programming in machine learning

Abstract

This paper explains automatic differentiation, discussing two primary modes - forward and backward - and their respective implementation methods. In the context of issues encountered in machine learning and deep learning, the forward mode is deemed more suitable as it efficiently differentiates functions with numerous inputs compared to outputs. Given Python's pivotal role in the ML landscape, the paper elaborates on two widely used deep learning libraries-PyTorch and TensorFlow. While both these libraries support automatic differentiation, they adopt distinct approaches, each carrying its unique strengths and weaknesses.