LMD-DARTS: Low-Memory, Densely Connected, Differentiable Architecture Search

Abstract

Neural network architecture search (NAS) technology is pivotal for designing lightweight convolutional neural networks (CNNs), facilitating the automatic search for network structures without requiring extensive prior knowledge. However, NAS is resource-intensive, consuming significant computational power and time due to the evaluation of numerous candidate architectures. To address the issues of high memory usage and slow search speed in traditional NAS algorithms, we propose the Low-Memory, Densely Connected, Differentiable Architecture Search (LMD-DARTS) algorithm. To expedite the updating speed of the optional operation weights during the search process, LMD-DARTS introduces a continuous strategy based on weight redistribution. Furthermore, to mitigate the influence of low-weight operations on classification results and reduce the number of searches, LMD-DARTS employs a dynamic sampler to prune underperforming operations during the search process, thereby lowering memory consumption and simplifying the complexity of individual searches. Additionally, to sparsify the dense connection matrix and mitigate redundant connections while maintaining optimal network performance, we introduce an adaptive downsampling search algorithm. Our experimental results show that the proposed LMD-DARTS achieves a remarkable 20% reduction in search time, along with a significant decrease in memory utilization within NAS process. Notably, the lightweight CNNs derived through this algorithm exhibit commendable classification accuracy, underscoring their effectiveness and efficiency for practical applications.