Affiliation:
1. National Technical University of Athens, Zografou, Greece
2. Samsung AI Center-Cambridge, London, United Kingdom of Great Britain and Northern Ireland
3. National Technical University of Athens, Zografou Greece
Abstract
The relentless expansion of deep learning applications in recent years has prompted a pivotal shift toward on-device execution, driven by the urgent need for real-time processing, heightened privacy concerns, and reduced latency across diverse domains. This article addresses the challenges inherent in optimising the execution of deep neural networks (DNNs) on mobile devices, with a focus on device heterogeneity, multi-DNN execution, and dynamic runtime adaptation. We introduce
CARIn
, a novel framework designed for the optimised deployment of both single- and multi-DNN applications under user-defined service-level objectives. Leveraging an expressive multi-objective optimisation framework and a runtime-aware sorting and search algorithm (
RASS
) as the MOO solver,
CARIn
facilitates efficient adaptation to dynamic conditions while addressing resource contention issues associated with multi-DNN execution. Notably,
RASS
generates a set of configurations, anticipating subsequent runtime adaptation, ensuring rapid, low-overhead adjustments in response to environmental fluctuations. Extensive evaluation across diverse tasks, including text classification, scene recognition, and face analysis, showcases the versatility of
CARIn
across various model architectures, such as Convolutional Neural Networks and Transformers, and realistic use cases. We observe a substantial enhancement in the fair treatment of the problem’s objectives, reaching 1.92× when compared to single-model designs and up to 10.69× in contrast to the state-of-the-art OODIn framework. Additionally, we achieve a significant gain of up to 4.06× over hardware-unaware designs in multi-DNN applications. Finally, our framework sustains its performance while effectively eliminating the time overhead associated with identifying the optimal design in response to environmental challenges.
Funder
Hellenic Foundation for Research and Innovation (HFRI) under the 3rd Call for HFRI PhD Fellowships
Publisher
Association for Computing Machinery (ACM)
Reference84 articles.
1. EmBench
2. Smart at what cost?
3. AOWS: Adaptive and Optimal Network Width Search With Latency Constraints
4. HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scaling
5. Language models are few-shot learners;Brown Tom;Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS'20),2020