Hyperparameter optimization method based on dynamic Bayesian with sliding balance mechanism in neural network for cloud computing

Abstract

AbstractHyperparameter optimization (HPO) of deep neural networks plays an important role of performance and efficiency of detection networks. Especially for cloud computing, automatic HPO can greatly reduce the network deployment cost by taking advantage of the computing power. Benefiting from its global-optimal search ability and simple requirements, Bayesian optimization has become the mainstream optimization method in recent years. However, in a non-ideal environment, Bayesian method still suffers from the following shortcomings: (1) when search resource is limited, it can only achieve inferior suboptimal results; (2) the acquisition mechanism cannot effectively balance the exploration of parameter space and the exploitation of historical data in different search stages. In this paper, we focused on the limited resources and big data provided by the cloud computing platform, took the anchor boxes of target detection networks as the research object, employed search resource as a restraint condition, and designed a dynamic Bayesian HPO method based on sliding balance mechanism. The dynamism of our method is mainly reflected in two aspects: (1) A dynamic evaluation model is proposed which uses the cross-validation mechanism to evaluate the surrogate model library and select the best model in real time; (2) A sliding balance mechanism is designed based on resource constraints to seek a balance between exploration and exploitation. We firstly augment the recommended samples of probability of improvement acquisition function by using k-nearest neighbor method, then introduce Hausdorff distance to measure the exploration value and match sampling strategy with resource utilization, which makes it slide smoothly with resource consumption to establish a dynamic balance of exploration to exploitation. The provided experiments show that our method can quickly and stably obtain better results under the same resource constraints compared with mature methods like BOHB. Graphical Abstract