Evolution of transmission modifiers under frequency-dependent selection and transmission in constant or fluctuating environments

Although the Reduction Principle for rates of mutation, migration, and recombination has been proved for large populations under constant selection, the fate of modifiers of these evolutionary forces under frequency-dependent or fluctuating selection is, in general, less well understood. Here we study modifiers of transmission, which include modifiers of mutation and oblique cultural transmission, under frequency-dependent and cyclically fluctuating selection, and develop models for which the Reduction Principle fails. We show that whether increased rates of transmission can evolve from an equilibrium at which there is zero transmission (for example, no mutation) depends on the number of alleles among which transmission is occurring. In addition, properties of the null-transmission state are clarified.