In the transmission process of gear transmission system of three-cylinder engine, there are great vibration and noise problems. Therefore, a rigid-flexible coupling model of gear transmission system with damping ring was proposed, and the motion characteristics and vibration suppression performance of flexible gear structure were studied experimentally. Firstly, a 6-degree-of-freedom bending and torsional model of the gear transmission system with additional damping ring was established by using the lumped mass method. Ishikawa formula was used to calculate the time-varying meshing stiffness of the flexible gear. The vibration response of the gear system with and without damping ring was compared. Then a special gear transmission system test equipment was built to test the acceleration and noise peak of steel gear and flexible gear at different speeds. Finally, a laser displacement test bench was designed to compare and analyze the micro-jitter of the gears under different rotational speeds, and the vibration reduction effect of the flexible gears was verified, and the theoretical model was verified. The esearch results show that the maximum acceleration signals of steel gear and flexible gear are respectively 35. 58 m/ s 2 and 20. 37 m/ s 2, when the speed is 1 000 r/ min. When the gear speed is 4 000 r/ min, the damping effect of the flexible gear is the best, the peak acceleration is reduced by 45. 23%, and the peak noise is reduced by 39. 06% . The micro-jitter amplitude of steel gear is obviously higher than that of flexible gear, and the change trend of both is similar. The flexible gear can effectively suppress the unsmooth and burr phenomenon of the micro-jitter amplitude curve of the gear teeth, thus enhancing the stability of the balance shafting of the three-cylinder engine. [ABSTRACT FROM AUTHOR]