The vibration signals of gearbox often contain complex components, such as the gear engagement, shafts rotation, bearings, and vibration of the box, and they brings a big challenge to fault diagnosis of gearbox. Furthermore, there are many fault types of gearbox, and the fault characteristics of them are possibly the same. For example, the pass frequency of the missing tooth or root crack in the gear on the input shaft are both equal to the rotation frequency of input shaft. In thus circumstance, traditional diagnosis methods are inapplicable, because we can’t determine which type of faults. In this paper, we proposed a dictionary learning with the shift-invariant dictionary to extract the fault features in a sparse way, and afterwards hidden Markov model (HMM) is utilized to identify the fault type from extractive features. We used the proposed method to diagnose the experimental data from a benchmark two-stage gearbox and there are nine different health conditions introduced to the pinion on the input shaft, and the diagnosis result verifies the good performance of the novel scheme in the fault diagnosis of gearbox.