Combining deep reinforcement learning and computational fluid dynamics for efficient navigation in turbulent flows

Autonomous underwater vehicles (AUVs) face significant challenges when navigating in turbulent environments, particularly when carrying out tasks such as inspecting offshore structures that generate large turbulent wakes. These environments increase the risk of collision and damage, and decrease the success rate of recorded video frames, but carrying out such inspections with AUV offers large potential cost savings and reduced risk to human operators. Reinforcement learning (RL) combined with computational fluid dynamics (CFD) can help develop control strategies suitable for handling such complex navigation problems. The objective of this study is to assess the feasibility of such approach. To this end, two versions of the soft actor-critic algorithm are tested: one relying on the estimated vehicle position and velocity and the other augmented with pressure surface measurements obtained from fitting the vehicle with simulated pressure transducers. Both RL agents successfully navigate in turbulent flows, but the agent provided with force estimates deduced from the surface pressure has significantly improved performance. This improvement is seen in the quality of individual episodes as well as in the training robustness and speed. Therefore, this study demonstrates the potential of using RL agents to assimilate additional information for developing robust control strategies and shows the usefulness of training RL agents in high-fidelity environments, such as CFD simulations.