Sim-to-Real Transfer with Action Mapping and State Prediction for Robot Motion Control

Deep reinforcement learning (DRL) has been proved to be a very promising method for robot motion control. However, it usually needs a large number of samples in training, which restricts its application in real-world robots. Sim-to-Real means transferring training strategies in simulation to reality, and it has become one of the hottest research areas in recent years. Embodiment of DRL algorithms is a necessary step in Sim-to-Real, which is faced with many challenges, such as poor sample efficiency, wear and tear of robots, safety, etc. In this paper, we present a new algorithm called action mapping and state prediction (AMSP), which considers three main factors in training including inaccurate parameters, unmodeled action damping and action delay. This method includes model error compensation based on action mapping, and delay compensation based on state prediction. The method in this paper is demonstrated in OpenAI inverted pendulum environment, and the strategy trained in the ideal environments with no action damping and no action delay is successfully transferred in the form of zero-shot to the artificial simulation environment with action damping and action delay, which shows the effectiveness of AMSP.