In this paper, the dynamic behavior of the van der Pol-Rayleigh system is studied by using the fast–slow analysis method and the transformation phase portrait method. Firstly, the stability and bifurcation behavior of the equilibrium point of the system are analyzed. We find that the system has no fold bifurcation, but has Hopf bifurcation. By calculating the first Lyapunov coefficient, the bifurcation direction and stability of the Hopf bifurcation are obtained. Moreover, the bifurcation diagram of the system with respect to the external excitation is drawn. Then, the fast subsystem is simulated numerically and analyzed with or without external excitation. Finally, the vibration behavior and its generation mechanism of the system in different modes are analyzed. The vibration mode of the system is affected by both the fast and slow varying processes. The mechanisms of different modes of vibration of the system are revealed by the transformation phase portrait method, because the system trajectory will encounter different types of attractors in the fast subsystem. [ABSTRACT FROM AUTHOR]