1. Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology.
- Author
-
Gould E, Fraser HS, Parker TH, Nakagawa S, Griffith SC, Vesk PA, Fidler F, Hamilton DG, Abbey-Lee RN, Abbott JK, Aguirre LA, Alcaraz C, Aloni I, Altschul D, Arekar K, Atkins JW, Atkinson J, Baker CM, Barrett M, Bell K, Bello SK, Beltrán I, Berauer BJ, Bertram MG, Billman PD, Blake CK, Blake S, Bliard L, Bonisoli-Alquati A, Bonnet T, Bordes CNM, Bose APH, Botterill-James T, Boyd MA, Boyle SA, Bradfer-Lawrence T, Bradham J, Brand JA, Brengdahl MI, Bulla M, Bussière L, Camerlenghi E, Campbell SE, Campos LLF, Caravaggi A, Cardoso P, Carroll CJW, Catanach TA, Chen X, Chik HYJ, Choy ES, Christie AP, Chuang A, Chunco AJ, Clark BL, Contina A, Covernton GA, Cox MP, Cressman KA, Crotti M, Crouch CD, D'Amelio PB, de Sousa AA, Döbert TF, Dobler R, Dobson AJ, Doherty TS, Drobniak SM, Duffy AG, Duncan AB, Dunn RP, Dunning J, Dutta T, Eberhart-Hertel L, Elmore JA, Elsherif MM, English HM, Ensminger DC, Ernst UR, Ferguson SM, Fernandez-Juricic E, Ferreira-Arruda T, Fieberg J, Finch EA, Fiorenza EA, Fisher DN, Fontaine A, Forstmeier W, Fourcade Y, Frank GS, Freund CA, Fuentes-Lillo E, Gandy SL, Gannon DG, García-Cervigón AI, Garretson AC, Ge X, Geary WL, Géron C, Gilles M, Girndt A, Gliksman D, Goldspiel HB, Gomes DGE, Good MK, Goslee SC, Gosnell JS, Grames EM, Gratton P, Grebe NM, Greenler SM, Griffioen M, Griffith DM, Griffith FJ, Grossman JJ, Güncan A, Haesen S, Hagan JG, Hager HA, Harris JP, Harrison ND, Hasnain SS, Havird JC, Heaton AJ, Herrera-Chaustre ML, Howard TJ, Hsu BY, Iannarilli F, Iranzo EC, Iverson ENK, Jimoh SO, Johnson DH, Johnsson M, Jorna J, Jucker T, Jung M, Kačergytė I, Kaltz O, Ke A, Kelly CD, Keogan K, Keppeler FW, Killion AK, Kim D, Kochan DP, Korsten P, Kothari S, Kuppler J, Kusch JM, Lagisz M, Lalla KM, Larkin DJ, Larson CL, Lauck KS, Lauterbur ME, Law A, Léandri-Breton DJ, Lembrechts JJ, L'Herpiniere K, Lievens EJP, de Lima DO, Lindsay S, Luquet M, MacLeod R, Macphie KH, Magellan K, Mair MM, Malm LE, Mammola S, Mandeville CP, Manhart M, Manrique-Garzon LM, Mäntylä E, Marchand P, Marshall BM, Martin CA, Martin DA, Martin JM, Martinig AR, McCallum ES, McCauley M, McNew SM, Meiners SJ, Merkling T, Michelangeli M, Moiron M, Moreira B, Mortensen J, Mos B, Muraina TO, Murphy PW, Nelli L, Niemelä P, Nightingale J, Nilsonne G, Nolazco S, Nooten SS, Novotny JL, Olin AB, Organ CL, Ostevik KL, Palacio FX, Paquet M, Parker DJ, Pascall DJ, Pasquarella VJ, Paterson JH, Payo-Payo A, Pedersen KM, Perez G, Perry KI, Pottier P, Proulx MJ, Proulx R, Pruett JL, Ramananjato V, Randimbiarison FT, Razafindratsima OH, Rennison DJ, Riva F, Riyahi S, Roast MJ, Rocha FP, Roche DG, Román-Palacios C, Rosenberg MS, Ross J, Rowland FE, Rugemalila D, Russell AL, Ruuskanen S, Saccone P, Sadeh A, Salazar SM, Sales K, Salmón P, Sánchez-Tójar A, Santos LP, Santostefano F, Schilling HT, Schmidt M, Schmoll T, Schneider AC, Schrock AE, Schroeder J, Schtickzelle N, Schultz NL, Scott DA, Scroggie MP, Shapiro JT, Sharma N, Shearer CL, Simón D, Sitvarin MI, Skupien FL, Slinn HL, Smith GP, Smith JA, Sollmann R, Whitney KS, Still SM, Stuber EF, Sutton GF, Swallow B, Taff CC, Takola E, Tanentzap AJ, Tarjuelo R, Telford RJ, Thawley CJ, Thierry H, Thomson J, Tidau S, Tompkins EM, Tortorelli CM, Trlica A, Turnell BR, Urban L, Van de Vondel S, van der Wal JEM, Van Eeckhoven J, van Oordt F, Vanderwel KM, Vanderwel MC, Vanderwolf KJ, Vélez J, Vergara-Florez DC, Verrelli BC, Vieira MV, Villamil N, Vitali V, Vollering J, Walker J, Walker XJ, Walter JA, Waryszak P, Weaver RJ, Wedegärtner REM, Weller DL, Whelan S, White RL, Wolfson DW, Wood A, Yanco SW, Yen JDL, Youngflesh C, Zilio G, Zimmer C, Zimmerman GM, and Zitomer RA
- Subjects
- Animals, Passeriformes physiology, Eucalyptus growth & development, Ecology methods, Biological Evolution
- Abstract
Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future., Competing Interests: Declarations. Ethics approval and consent to particpate: We obtained permission to conduct this research from the Whitman College Institutional Review Board (IRB). As part of this permission, the IRB approved the consent form ( https://osf.io/xyp68/ ) that all participants completed prior to joining the study. The authors declare that they have no competing interests. Competing interests: The authors declare that they have no competing interests., (© 2025. The Author(s).)
- Published
- 2025
- Full Text
- View/download PDF