Affiliation:
1. Purdue University, USA
Abstract
Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests crucially on gradient-descent optimization and the ability to “learn” parameters of a neural network by backpropagating observed errors. However, neural network architectures are growing increasingly sophisticated and diverse, which motivates an emerging quest for even more general forms of differentiable programming, where arbitrary parameterized computations can be trained by gradient descent. In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes backpropagation in neural networks. We uncover a tight connection between reverse-mode AD and delimited continuations, which permits implementing reverse-mode AD purely via operator overloading and without managing any auxiliary data structures. We further show how this formulation of AD can be fruitfully combined with multi-stage programming (staging), leading to an efficient implementation that combines the performance benefits of deep learning frameworks based on explicit reified computation graphs (e.g., TensorFlow) with the expressiveness of pure library approaches (e.g., PyTorch).
Publisher
Association for Computing Machinery (ACM)
Subject
Safety, Risk, Reliability and Quality,Software
Reference82 articles.
1. A computational model for TensorFlow: an introduction
2. L. M. Beda L. N. Korolev N. V. Sukkikh and T. S. Frolova. 1959. Programs for automatic differentiation for the machine BESM. Technical Report. Institute for Precise Mechanics and Computation Techniques Academy of Science Moscow USSR. (In Russian). L. M. Beda L. N. Korolev N. V. Sukkikh and T. S. Frolova. 1959. Programs for automatic differentiation for the machine BESM. Technical Report. Institute for Precise Mechanics and Computation Techniques Academy of Science Moscow USSR. (In Russian).
3. Improving binding times without explicit CPS-conversion
Cited by
31 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Efficient CHAD;Proceedings of the ACM on Programming Languages;2024-01-05
2. Forward- or reverse-mode automatic differentiation: What's the difference?;Science of Computer Programming;2024-01
3. Differentiable Quantum Programming with Unbounded Loops;ACM Transactions on Software Engineering and Methodology;2023-11-23
4. Verifying an Effect-Handler-Based Define-By-Run Reverse-Mode AD Library;Logical Methods in Computer Science;2023-10-23
5. Partial Evaluation of Automatic Differentiation for Differential-Algebraic Equations Solvers;Proceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences;2023-10-22