Affiliation:
1. Academia Sinica, Nankang, Taipei, Taiwan
2. National Taiwan University, Taipei, Taiwan
Abstract
Video captioning is a core technology to many important applications, such as AI-assisted medical diagnosis, video question answering, storytelling through videos, and lip-reading. Video captioning employs a hybrid CNN + RNN model. Accelerating such a hybrid model on a heterogeneous system is challenging for two reasons. First, CNN and RNN exhibit very different computing behaviors, making the mapping between computation and heterogeneous devices difficult. Second, data dependency exists between the CNN and RNN within a video frame and between adjacent RNNs across video frames. These data dependencies prohibit the full parallelization of the hybrid model. The issues also include the utilization of accelerator resources, which is critical to maximizing the performance. In this work, we propose a fine-grained scheduling scheme for mapping computation and devices within a video frame, and a pipeline scheduling scheme for exploiting maximum parallelism between the execution of the video frames. In addition, we propose two capacity-guided scheduling methods. On the server, the concurrent kernel execution mechanism is exploited for improving GPU utilization. On the edge platform, we rearrange CNN computation among the CPU and EdgeTPUs guided by the EdgeTPU’s SRAM capacity so that balanced computation is achieved and off-chip memory overhead is minimized. Experimental results show that our scheduling scheme improves video captioning performance by up to 3.24
\( \times \)
with CPU + GPU collaboration over the GPU-only execution. On an edge platform with an ARM CPU and two EdgeTPUs, our CPU + EdgeTPU scheduling exhibits outstanding performance, which achieves up to 54.9
\( \times \)
speedup compared to using ARM CPU only and can perform video captioning of 59 frames per second.
Funder
Ministry of Science and Technology
Publisher
Association for Computing Machinery (ACM)
Subject
Hardware and Architecture,Information Systems,Software
Reference68 articles.
1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the USENIX Conference on Operating Systems Design and Implementation. 265–283.
2. Fused-layer CNN accelerators
3. Andrew Anderson and David Gregg. 2018. Optimal DNN primitive selection with partitioned Boolean quadratic programming. In Proceedings of the International Symposium on Code Generation and Optimization. 340–351.
4. Optimizing performance of recurrent neural networks on GPUs;Appleyard Jeremy;arXiv:1604.01946,2016
5. Efficient Implementation of Recurrent Neural Network Transducer in Tensorflow
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Accelerating Convolutional Neural Networks via Inter-operator Scheduling;2022 IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS);2023-01