Intelligent controller for unmanned surface vehicles by deep reinforcement learning

With the development of the applications of unmanned surface vehicles (USVs), USV automation technologies are attracting increasing attention. In the industry, through the subtask division, it is generally believed that course-keeping is a critical basic sub-system in a series of complex automation systems and affects USV automation performance to a great extent. By course-keeping, we mean USV adjusts its angle to the desired angle and keeps it. In recent decades, course-keeping has been mainly achieved through classical first principles technologies, such as proportion–integral–differential (PID) controllers, leading to extremely laborious parameter tuning, especially in changeable wave environments. With the emergence and extensive application of data-driven technologies, deep reinforcement learning is conspicuous in sequential decision-making tasks, but it introduces a lack of explainability and physical meaning. To take full advantage of the data-driven and first principles paradigm and easily extend to the industry, in this paper, we propose an intelligent adaptive PID controller enhanced by proximal policy optimization (PPO) to achieve USV high-level automation. We then further verify its performance in path-following tasks compared with the PID controller. The results demonstrate that the proposed controller inherits the merits of explainability from PID and excellent sequential decision making from PPO and possesses excellent disturbance rejection performance when facing the disturbance of a changeable wave environment.