Male-male competition and female mate choice based on ornamentation and genetic traits are the main drivers of animal sexual selection and group spawning. If male-male competition is intense, males with specific phenotypes should have advantages in breeding success or occupying superior mating positions. If female choice is important, females should have preferences for mate relatedness or males with good genes or optimal genetic compatibility against themselves. To detect the intensities of male-male competition and female choice and test the good genes and genetic compatibility hypotheses, we observed breeding behaviors, measured individual body lengths and breeding success indicators, and calculated male genetic heterozygosity, male–female relatedness, and genetic dissimilarity in an Omei treefrog (Rhacophorus omeimontis) population in Badagongshan, China. Our analyses showed that larger males obtained larger mates, had more mating opportunities, occupied better amplectant positions, and produced more offspring. However, females showed no inbreeding/outbreeding bias in mate choice, and the good genes and genetic compatibility hypotheses were not supported in female selection on mates and amplectant positions. We considered male-male competition as the main driver of sexual selection and group spawning in this prolonged mating species because the cost of choosing mates with specific genetic traits may be high for females. Male-male competition and female mate choice are the main drivers of animal sexual selection and mating behaviors. However, their impacts on the evolution of mating system are not yet quite clear. By studying the Omei treefrog, a Chinese endemic anuran species with common group spawning behaviors, this research posed the importance of male-male competition and female mate choice and discussed the mechanisms of sexual selection and multiple mating, which are the most essential and debated issues in evolutionary biology and behavioral ecology. We found that, in this prolonged mating and lek-patterned species, male-male competition is the main driver of sexual selection and group spawning. Larger males can get larger females, have more breeding opportunities, occupy better amplectant positions, and thus, obtain greater numbers of offspring. Whereas, females have no significant preferences on ornamentation and genetic traits of their males or mating positions, and the inferior males “make the best of a bad lot” by joining mating pairs to produce mating groups. Our study provides empirical evidence of reproductive mechanisms of amphibian species and could advance the understandings on the evolution of animals’ sexual selection and mating system.