Author:
Alt Tobias,Schrader Karl,Augustin Matthias,Peter Pascal,Weickert Joachim
Abstract
AbstractWe investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights, we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.
Funder
european research council
Publisher
Springer Science and Business Media LLC
Subject
Applied Mathematics,Geometry and Topology,Computer Vision and Pattern Recognition,Condensed Matter Physics,Modeling and Simulation,Statistics and Probability
Reference113 articles.
1. Lecture Notes in Computer Science;T Alt,2021
2. Alt, T., Weickert, J.: Learning integrodifferential models for denoising. In: Proc. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2045–2049. IEEE Computer Society Press, Toronto, Canada (2021)
3. Alt, T., Weickert, J., Peter, P.: Translating diffusion, wavelets, and regularisation into residual networks. arXiv:2002.02753v3 [cs.LG] (2020)
4. Andreu, F., Ballester, C., Caselles, V., Mazón, J.M.: Minimizing total variation flow. Differ. Integral Equations 14(3), 321–360 (2001)
5. Arridge, S., Hauptmann, A.: Networks for nonlinear diffusion problems in imaging. J. Math. Imaging Vis. 62, 471–487 (2020)
Cited by
6 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献