Decomposing the site frequency spectrum: the impact of tree topology on neutrality tests

bioRxiv preprint
We investigate the dependence of the site frequency spectrum on the topological structure of genealogical trees. We show that basic population genetic statistics, for instance, estimators of θ or neutrality tests such as Tajima’s D, can be decomposed into components of waiting times between coalescent events and of tree topology. Our results clarify the relative impact of the two components on these statistics. We provide a rigorous interpretation of positive or negative values of an important class of neutrality tests in terms of the underlying tree shape. In particular, we show that values of Tajima’s D and Fay and Wu’s H depend in a direct way on a peculiar measure of tree balance, which is mostly determined by the root balance of the tree. We present a new test for selection in the same class as Fay and Wu’s H and discuss its interpretation and power. Finally, we determine the trees corresponding to extreme expected values of these neutrality tests and present formulas for these extreme values as a function of sample size and number of segregating sites.
A.L. is funded by the United Kingdom National Institute for Health Research, Health Protection Research Unit on Modelling Methodology (grant HPRU-2012-10080). L.F. and G.A. acknowledge support from the grant ANR-12-JSV7-0007 from Agence Nationale de Recherche (France). G.A. acknowledges support from the grant ANR-12-BSV7-0012-04 from Agence Nationale de Recherche (France). T.W. acknowledges support from DFG-SPP1590 by the German Science Foundation. S.E.R.O. is supported by grants CGL2009-09346 (MICINN, Spain), AGL2013-41834-R (MEC, Spain), by the CERCA Programme/Generalitat de Catalunya and acknowledges financial support from the Spanish Ministry of Economy and Competitiveness, through the Severo Ochoa Programme for Centres of Excellence in R&D 2016-2019 (SEV‐2015‐0533).