Q-Learning based Load Balancing in Heterogeneous Networks with Human and Machine Type Communication Co-existence

Abstract

Abstract A heterogeneous network (HetNet) is a network comprised of many different wireless network nodes with varying capabilities and features deployed within the coverage area of cellular service. Low power nodes such as pico-cells are deployed within the coverage area of a large macro-cell to cover areas with high user density or areas not well-covered by the macro eNB. In this paper, we focus on a model comprised of macro eNBs and pico eNBs serving both human-to-human (H2H) and machine-to-machine (M2M) devices that have different quality of service (QoS) requirements. We propose a new Q-learning based scheme for cell association and network load balancing for both types of devices. The scheme is comprised of two independent algorithms: an algorithm applied at the M2M devices that uses Q-learning to associate the device with the eNB that best meets its QoS requirements and a second algorithm applied at pico eNBs that uses Q-learning to tune the parameters of the cell range expansion to balance the load between the macro-cell and pico-cells. To evaluate the proposed scheme performance, we compare the H2H and M2M blocking probability and the M2M uplink transmission power with the traditional method and a scheme that uses Q-learning at the UE devices to assist the load balancing. The results indicate that the proposed scheme reduces the blocking probability by about 10% for both M2M and H2H devices and also reduces the uplink transmission power for M2M devices by 50% even under high load conditions.