Shah, M.A., Zhang, S., Maple, C. (2013). Cognitive radio networks for Internet of Things: Applications, challenges and future. In 19th International Conference on Automation & Computing. IEEE, London.
Shaikh, F.K., Zeadally, S., Exposito, E. (2017). Enabling technologies for green Internet of Things. IEEE Systems Journal, 11(2), 983–994.
Shrestha, R., Bajracharya, R., Nam, S.Y. (2018). Challenges of future VANET and cloud-based approaches. Wireless Communications and Mobile Computing. doi:10.1155/2018/5603518.
Singh, K.D., Rawat, P., Bonnin, J. (2014). Cognitive radio for vehicular ad hoc networks (CR-VANETs): Approaches and challenges. Journal on Wireless Communications and Networking, 49(2014). doi:10.1186/1687-1499-2014-49.
Song, Y. and Xie, J. (2012). ProSpect: A proactive spectrum handoff framework for cognitive radio ad hoc networks without common control channel. IEEE Transactions on Mobile Computing, 11(7), 1127–1139.
Tarek, A., Benslimane, M., Darwish, M. (2020). Survey on spectrum sharing/allocation for cognitive radio networks Internet of Things. Egyptian Informatics Journal. doi:10.1016/j.eij.2020.02.003.
Tuysuz, M.F. and Trestian, R. (2017). Energy-efficient vertical handover parameters, classification and solutions over wireless heterogeneous networks: A comprehensive survey. Wireless Personal Communications, 97, 1155.
Vlacheas, P., Giaffreda, R., Stavroulaki, V., Kelaidonis, D., Foteinos, V., Poulios, G., Demestichas, P., Somov, A., Biswas, A.R., Moessner, K. (2013). Enabling smart cities through a cognitive management framework for the Internet of Things. IEEE Communications Magazine, 51(6), 102–111.
Wang, L.-C. and Wang, C.-W. (2008). Spectrum handoff for cognitive radio networks: Reactive-sensing or proactive-sensing? In Proceedings of IEEE International in Performance, Computing and Communications Conference (IPCCC 2008), Austin, TX, 343–348. 10.1109/PCCC.2008.4745128.
Wang, Y.-H., Huang, G.-R., Tung, Y.-C. (2014). A handover prediction mechanism based on LTE-A UE history information. In International Conference on Computer, Information and Telecommunication Systems (CITS). IEEE, Jeju.
Wu, Q., Ding, G., Xu, Y., Feng, S., Du, Z., Wang, J., Long, K. (2014a). Cognitive Internet of Things: A new paradigm beyond connection. IEEE Internet of Things Journal, 1(2), 129–143. doi:10.1109/JIOT.2014.2311513.
Wu, Y., Hu, F., Kumar, S., Zhu, Y., Talari, A., Rahnavard, N., Matyjas, J.D. (2014b). A learning-based QoE-driven spectrum handoff scheme for multimedia transmissions over cognitive radio networks. IEEE Journal on Selected Areas in Communications, 32(11), 2134–2148.
Xenakis, D., Passas, N., Di Gregorio, L., Verikoukis, C. (2011). A context aware vertical handover framework towards energy-efficiency. In Proceedings of the IEEE Vehicular Technology Conference (VTC).
Xiao, K. and Li, C. (2018). Vertical handoff decision algorithm for heterogeneous wireless networks based on entropy and improved TOPSIS. In 2018 IEEE 18th International Conference on Communication Technology (ICCT). IEEE, Chongqing.
Xu, L.D., He, W., Li, S. (2014). Internet of Things in industries: A survey. IEEE Transactions on Industrial Informatics, 10(4), 2233–2243.
Zekri, M., Jouaber, B., Zeghlache, D. (2012). A review on mobility management and vertical handover solutions over heterogeneous wireless networks. Computer Communications, 35, 2055–2068.
Zhu, C., Leung, V.C., Shu, L., Ngai, E.C.H. (2015). Green IoT for smart world. Special Section on Challenges for Smart Worlds. doi:10.1109/ACCESS.2015.2497312.
Zou, L., Javed, A., Muntean, G. (2017). Smart mobile device power consumption measurement for video streaming in wireless environments: WiFi vs. LTE. In 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB). IEEE, Cagliari.
2
Using Reinforcement Learning to Manage Massive Access in NB-IoT Networks
Yassine HADJADJ-AOUL and Soraya AIT-CHELLOUCHE
Inria, CNRS, IRISA, University of Rennes 1, France
2.1. Introduction
Communications between objects in the Internet of Things (IoT) and particularly machine-to-machine (M2M) communications are considered as one of the most important evolutions of the Internet. Supporting these devices is, however, one of the most significant challenges that network operators need to overcome (Lin et al. 2016). In fact, the considerable number of these devices that might attempt to access a network at the same time could lead to heavy congestion, or even a total saturation, with all the consequences this might cause. In fact, as can be seen in Figure 2.1, a very limited number of devices trying simultaneously to access the network can reduce the network’s performance to zero, independently of the access opportunities available (Bouzouita et al. 2016). In these circumstances, it seems clear that effective access control mechanisms are needed to maintain a reasonable number of access attempts.
The Third Generation Partnership Project (3GPP) group has identified the overload on the Random Access Network (RAN) as a priority at a very early stage and has suggested several solutions. Among the approaches suggested, the Access Class Barring (ACB), suggested in version 8, and its extension, the Extended Access Barring (EAB), suggested in version 11, are certainly the most effective strategies (3GPP 2011). In fact, these approaches tackle the problem at its roots by preventing any attempts to access the network. However, these approaches merely provide a framework for congestion management, without providing a ready-made solution.
Figure 2.1. Impact on performance of the number of IoT devices simultaneously attempting access
The idea introduced in this chapter is quite simple, because it involves calculating a blocking factor. Nevertheless, a good implementation would require a good knowledge of the number of terminals ready to attempt access, so as to deduce from this the probability of the optimal blocking. This information is unfortunately not available in the network.
To solve this problem, two significant challenges should be taken into account: (1) designing an access control strategy for dynamic generation of the blocking factor and (2) estimating the number of devices simultaneously attempting to access the network.
In this chapter, we tackle these questions using an estimator that was suggested in earlier research (Bouzouita et al. 2019). Since this estimation is very noisy, we exploit the potential of the most advanced reinforcement learning techniques, to take account of this complex reality (the state of the network is not observable) and deduce a sub-optimal control strategy. More especially in this chapter, we use the deep reinforcement learning algorithm Twin Delayed Deep Deterministic policy gradient algorithm (TD3) (Fujimoto et al. 2018) to produce the optimal blocking factor from these past estimations (Hadjadj-Aoul and Ait-Chellouche 2020).
The remainder of this chapter is organized as follows. Section 2.2 briefly presents the NB-IoT standard considered in our proposal. Section 2.3 gives an overview of the main congestion control techniques in IoT networks, and more especially cellular IoT networks. Section 2.4 describes the model for IoT terminals to access the network. Section 2.5 describes the suggested control solution, based on the TD3 algorithm, adapted to solve the problem of calculating