Applications of Reinforcement Learning to Transportation and Traffic Engineering

Applications in traffic signal control

• Multi-agent control

• Leveraging data from connected-vehicle environments

• Deep-reinforcement learning approaches

• Co-learning of vehicles and controllers

• Distributed control

Within a connected environment allowing vehicle-to-infrastructure and vehicle-to-vehicle communications, reinforcement learning (RL) provides an efficient way develop data-driven signal control algorithms. Advanced sensing and communications can be leveraged to develop RL-based control for a network of signalized intersections. The academic efforts on RL-based control development have addressed several challenges such as scalability of the state-space aka “curse of dimensionality”, convergence of optimal policies, and computational expenses regarding training and tuning the hyperparameters. (Mannion, Duggan, and Howley 2016; Bazzan 2009) provides a systematic review of RL-based signal control. Recent works include,(Aslani, Mesgari, and Wiering 2017)(Aziz, Zhu, and Ukkusuri 2017)(El-Tantawy, Abdulhai, and Abdelgawad 2013)(Medina and Benekohal 2012a)(Prashanth and Bhatnagar 2011)(Cai, Wong, and Heydecker 2009)(Balaji, German, and Srinivasan 2010). Dimensions addressed by the works spans cooperative multi-agent control (Bazzan, de Oliveira, and da Silva 2010; Zhu et al. 2015; Wiering 2000; Kok and Vlassis 2006; Houli, Zhiheng, and Yi 2010; Sunehag et al. 2017; Arel et al. 2010; Kuyer et al. 2008), multi-policy consideration (Khamis and Gomaa 2014; Moffaert et al. 2014; Dusparic and Cahill 2010; Zhao, Chen, and Hu 2010), and decentralized methods for scalability(Medina and Benekohal 2012b; Zhu et al. 2015; Sunehag et al. 2017; Bazzan 2017; 2005). A few recent works also applied deep learning concepts within the context of RL-based traffic control ((Gao et al. 2017; Hensher and Greene 2001; Yu and Nagpal 2011)(Le, Vien, and Chung 2018; Liang et al. 2019; Tan et al. 2019)).

Most algorithms described above assume a perfect observation state for the underlying RL system. In a real-world setting, this might not be the case. Even for a connected-vehicle environment the traffic state may not be fully observed because of the less than “perfect” market share of connected vehicles. It is important to find the bounds of estimated benefits—reduction of delay, number of stops for RL-based control—accounting for the partially observed state. Regarding implementation, LiDaR based detection and 360 degree fish-eye cameras can be handy for the level of resolution needed for a successful RL implementation. This is still a progressing research area and we expect it to be evolving as the automated vehicles are being adapted (and trusted) and disruption takes place within the connected and automated vehicle technologies landscape.

Besides traffic signal control reinforcement learning found its way in many traffic engineering related application including dynamic speed limit control(Zhu and Ukkusuri 2014), driving strategies for autonomous vehicles(Ngai and Yung 2011; Mousa, Mousa, and Ishak 2018; Desjardins and Chaib-draa 2011), shared-used mobility(Wen, Zhao, and Jaillet 2018), fuel/energy optimization (Islam et al. 2018)(Qi et al. 2019; 2016; Liu et al. 2017) and routing (Bazzan and Grunitzki 2016; Grunitzki, Ramos, and Bazzan 2014).