Algorithm-System-Hardware Co-Design for Efficient 3D Deep Learning

Abstract

3D point cloud deep learning has received significant attention due to its wide applications, such as augmented/mixed reality, autonomous vehicles/drones, and robots. Despite its remarkable accuracy, the computational cost and sparse nature of 3D point cloud deep learning models greatly hinder their use in real-world, latency-sensitive applications. In this paper, we develop efficient algorithms, systems, and hardware for 3D deep learning to overcome the computational challenges (especially sparsity) of 3D point cloud data, making it more applicable in a broader range of real-world scenarios. From the algorithm perspective, we introduce novel 3D building blocks (PVCNN and SPVNAS∗, https://pvnas.mit.edu ) that are tailor-made for point cloud data to reduce sparse and irregular overheads. On the system level, we develop a high-performance library (TorchSparse∗, https://torchsparse.mit.edu ) that can handle sparse and irregular workloads on general-purpose hardware, which is typically optimized for dense and regular workloads. Furthermore, we develop a specialized hardware accelerator (PointAcc∗, https://pointacc.mit.edu ) that can support various types of point cloud deep learning primitives and mitigate memory bottlenecks for sparse point cloud computing.