Affiliation:
1. Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India
2. Computer and Communication Sciences Division, Indian Statistical Institute, Kolkata, India
Abstract
Low-power edge devices equipped with Graphics Processing Units (GPUs) are a popular target platform for real-time scheduling of inference pipelines. Such application-architecture combinations are popular in Advanced Driver-Assistance Systems (ADAS) for aiding in the real-time decision-making of automotive controllers. However, the real-time throughput sustainable by such inference pipelines is limited by resource constraints of the target edge devices. Modern GPUs, both in edge devices and workstation variants, support the facility of concurrent execution of computation kernels and data transfers using the primitive of
streams
, also allowing for the assignment of priority to these streams. This opens up the possibility of executing computation layers of inference pipelines within a multi-priority, multi-stream environment on the GPU. However, manually co-scheduling such applications while satisfying their throughput requirement and platform memory budget may require an unmanageable number of profiling runs. In this work, we propose a Deep Reinforcement Learning (DRL) based method for deciding the start time of various operations in each pipeline layer while optimizing the latency of execution of inference pipelines as well as memory consumption. Experimental results demonstrate the promising efficacy of the proposed DRL approach in comparison with the baseline methods, particularly in terms of real-time performance enhancements, schedulability ratio, and memory savings. We have additionally assessed the effectiveness of the proposed DRL approach using a real-time traffic simulation tool IPG CarMaker.
Publisher
Association for Computing Machinery (ACM)
Reference59 articles.
1. AlexeyAB. 2019. Yolo-mark. https://github.com/AlexeyAB/Yolo_mark.
2. GPU Scheduling on the NVIDIA TX2: Hidden Details Revealed
3. Stanley Bak, Gang Yao, Rodolfo Pellizzoni, and Marco Caccamo. 2012. Memory-aware scheduling of multicore task sets for real-time systems. In RTCSA. IEEE, 300–309.
4. Soroush Bateni et al. 2020. Co-optimizing performance and memory footprint via integrated CPU/GPU memory management, an implementation on autonomous driving platform. In RTAS. IEEE, Sydney, Australia, 310–323.
5. Developing Real-Time Scheduling Policy by Deep Reinforcement Learning