Compiler Technologies in Deep Learning Co-Design: A Survey

Abstract

With the rapid development of deep learning applications, general-purpose processors no longer suffice for deep learning workloads because of the dying of Moore’s Law. Thus, computer architecture innovation has entered a golden age for domain-specific design, which has led to a demand for new compilation technologies to facilitate cross-layer optimization. Historically, hardware and software have been collaboratively designed. Today, these co-design ideas still benefit the deep learning field in both academia and industry, encompassing additional aspects and layers. In this study, we elaborate on past and recent works on deep learning compilers and co-design while focusing on the combination of these two technologies, which we believe is the trend in the new deep learning era. After summarizing the existing compilation technologies and co-design approaches, we propose a domain-specific compilation framework, the Buddy Compiler, for a typical deep learning co-design system.