Search

Your search keyword '"Van Aert, Robbie C. M."' showing total 165 results
Start Over Author "Van Aert, Robbie C. M."
165 results on '"Van Aert, Robbie C. M."'

1. Meta-Analyzing Partial Correlation Coefficients Using Fisher's Z Transformation

Author

van Aert, Robbie C. M.

Abstract

The partial correlation coefficient (PCC) is used to quantify the linear relationship between two variables while taking into account/controlling for other variables. Researchers frequently synthesize PCCs in a meta-analysis, but two of the assumptions of the common equal-effect and random-effects meta-analysis model are by definition violated. First, the sampling variance of the PCC cannot assumed to be known, because the sampling variance is a function of the PCC. Second, the sampling distribution of each primary study's PCC is not normal since PCCs are bounded between -1 and 1. I advocate applying the Fisher's z transformation analogous to applying Fisher's z transformation for Pearson correlation coefficients, because the Fisher's z transformed PCC is independent of the sampling variance and its sampling distribution more closely follows a normal distribution. Reproducing a simulation study by Stanley and Doucouliagos and adding meta-analyses based on Fisher's z transformed PCCs shows that the meta-analysis based on Fisher's z transformed PCCs had lower bias and root mean square error than meta-analyzing PCCs. Hence, meta-analyzing Fisher's z transformed PCCs is a viable alternative to meta-analyzing PCCs, and I recommend to accompany any meta-analysis based on PCCs with one using Fisher's z transformed PCCs to assess the robustness of the results.

Published
2023
Full Text
View/download PDF

3. A Critical Reflection on Computing the Sampling Variance of the Partial Correlation Coefficient

Author

van Aert, Robbie C. M. and Goos, Cas

Abstract

The partial correlation coefficient quantifies the relationship between two variables while taking into account the effect of one or multiple control variables. Researchers often want to synthesize partial correlation coefficients in a meta-analysis since these can be readily computed based on the reported results of a linear regression analysis. The default inverse variance weights in standard meta-analysis models require researchers to compute not only the partial correlation coefficients of each study but also its corresponding sampling variance. The existing literature is diffuse on how to estimate this sampling variance, because two estimators exist that are both widely used. We critically reflect on both estimators, study their statistical properties, and provide recommendations for applied researchers. We also compute the sampling variances of studies using both estimators in a meta-analysis on the partial correlation between self-confidence and sports performance.

Published
2023
Full Text
View/download PDF

4. Publication Bias

Author

van Aert, Robbie C. M., Niemeyer, Helen, O'Donohue, William, editor, Masuda, Akihiko, editor, and Lilienfeld, Scott, editor

Published
2022
Full Text
View/download PDF

6. Statistical Properties of Methods Based on the 'Q'-Statistic for Constructing a Confidence Interval for the Between-Study Variance in Meta-Analysis

Author

van Aert, Robbie C. M., van Assen, Marcel A. L. M., and Viechtbauer, Wolfgang

Abstract

The effect sizes of studies included in a meta-analysis do often not share a common true effect size due to differences in for instance the design of the studies. Estimates of this so-called between-study variance are usually imprecise. Hence, reporting a confidence interval together with a point estimate of the amount of between-study variance facilitates interpretation of the meta-analytic results. Two methods that are recommended to be used for creating such a confidence interval are the "Q"-profile and generalized "Q"-statistic method that both make use of the "Q"-statistic. These methods are exact if the assumptions underlying the random-effects model hold, but these assumptions are usually violated in practice such that confidence intervals of the methods are approximate rather than exact confidence intervals. We illustrate by means of two Monte-Carlo simulation studies with odds ratio as effect size measure that coverage probabilities of both methods can be substantially below the nominal coverage rate in situations that are representative for meta-analyses in practice. We also show that these too low coverage probabilities are caused by violations of the assumptions of the random-effects model (ie, normal sampling distributions of the effect size measure and known sampling variances) and are especially prevalent if the sample sizes in the primary studies are small.

Published
2019
Full Text
View/download PDF

7. p-Values Less Than 0.05 in Psychology: What is Going on?

Author

Hartgerink, Chris H. J., van Aert, Robbie C. M., Nuijten, Michèle B., Wicherts, Jelte M., van Assen, Marcel A. L. M., Gruber, Craig W., Gruber, Craig W., Series Editor, Valsiner, Jaan, Series Editor, Clark, Matthew G., Series Editor, and Klempe, Sven Hroar, Series Editor

Published
2020
Full Text
View/download PDF

16. A many-analysts approach to the relation between religiosity and well-being

Author

Hoogeveen, Suzanne, Sarafoglou, Alexandra, Aczel, Balazs, Aditya, Yonathan, Alayan, Alexandra J., Allen, Peter J., Altay, Sacha, Alzahawi, Shilaan, Amir, Yulmaida, Anthony, Francis-Vincent, Appiah, Obed Kwame, Atkinson, Quentin D., Baimel, Adam, Balkaya-Ince, Merve, Balsamo, Michela, Banker, Sachin, Bartos, Frantisek, Becerra, Mario, Beffara, Bertrand, Beitner, Julia, Bendixen, Theiss, Berkessel, Jana B., Berniunas, Renatas, Billet, Matthew, Billingsley, Joseph, Bortolini, Tiago, Breitsohl, Heiko, Bret, Amelie, Brown, Faith L., Brown, Jennifer, Brumbaugh, Claudia C., Buczny, Jacek, Bulbulia, Joseph, Caballero, Saul, Carlucci, Leonardo, Carmichael, Cheryl L., Cattaneo, Marco E. G., Charles, Sarah J., Claessens, Scott, Panagopoulos, Maxinne C., Costa, Angelo Brandelli, Crone, Damien L., Czoschke, Stefan, Czymara, Christian, D'Urso, E. Damiano, Dahlstrom, Orjan, Dalla Rosa, Anna, Danielsson, Henrik, De Ron, Jill, de Vries, Ymkje Anna, Dean, Kristy K., Dik, Bryan J., Disabato, David J., Doherty, Jaclyn K., Draws, Tim, Drouhot, Lucas, Dujmovic, Marin, Dunham, Yarrow, Ebert, Tobias, Edelsbrunner, Peter A., Eerland, Anita, Elbaek, Christian T., Farahmand, Shole, Farahmand, Hooman, Farias, Miguel, Feliccia, Abrey A., Fischer, Kyle, Fischer, Ronald, Fisher-Thompson, Donna, Francis, Zoe, Frick, Susanne, Frisch, Lisa K., Geraldes, Diogo, Gerdin, Emily, Geven, Linda, Ghasemi, Omid, Gielens, Erwin, Gligoric, Vukasin, Hagel, Kristin, Hajdu, Nandor, Hamilton, Hannah R., Hamzah, Imaduddin, Hanel, Paul H. P., Hawk, Christopher E., Himawan, Karel K., Holding, Benjamin C., Homman, Lina E., Ingendahl, Moritz, Inkila, Hilla, Inman, Mary L., Islam, Chris-Gabriel, Isler, Ozan, Izydorczyk, David, Jaeger, Bastian, Johnson, Kathryn A., Jong, Jonathan, Karl, Johannes A., Kaszubowski, Erikson, Katz, Benjamin A., Keefer, Lucas A., Kelchtermans, Stijn, Kelly, John M., Klein, Richard A., Kleinberg, Bennett, Knowles, Megan L., Kolczynska, Marta, Koller, Dave, Krasko, Julia, Kritzler, Sarah, Krypotos, Angelos-Miltiadis, Kyritsis, Thanos, Landes, Todd L., Laukenmann, Ruben, Forsyth, Guy A. Lavender, Lazar, Aryeh, Lehman, Barbara J., Levy, Neil, Lo, Ronda F., Lodder, Paul, Lorenz, Jennifer, Lowicki, Pawel, Ly, Albert L., Maassen, Esther, Magyar-Russell, Gina M., Maier, Maximilian, Marsh, Dylan R., Martinez, Nuria, Martinie, Marcellin, Martoyo, Ihan, Mason, Susan E., Mauritsen, Anne Lundahl, McAleer, Phil, McCauley, Thomas, McCullough, Michael, McKay, Ryan, McMahon, Camilla M., McNamara, Amelia A., Means, Kira K., Mercier, Brett, Mitkidis, Panagiotis, Monin, Benoit, Moon, Jordan W., Moreau, David, Morgan, Jonathan, Murphy, James, Muscatt, George, Nagel, Christof, Nagy, Tamas, Nalborczyk, Ladislas, Nilsonne, Gustav, Noack, Pamina, Norenzayan, Ara, Nuijten, Michele B., Olsson-Collentine, Anton, Oviedo, Lluis, Pavlov, Yuri G., Pawelski, James O., Pearson, Hannah, Pedder, Hugo, Peetz, Hannah K., Pinus, Michael, Pirutinsky, Steven, Polito, Vince, Porubanova, Michaela, Poulin, Michael J., Prenoveau, Jason M., Prince, Mark A., Protzko, John, Pryor, Campbell, Purzycki, Benjamin G., Qiu, Lin, Putter, Julian Quevedo, Rabelo, Andre, Radell, Milen L., Ramsay, Jonathan E., Reid, Graham, Roberts, Andrew J., Luna, Lindsey M. Root, Ross, Robert M., Roszak, Piotr, Roy, Nirmal, Saarelainen, Suvi-Maria K., Sasaki, Joni Y., Schaumans, Catherine, Schivinski, Bruno, Schmitt, Marcel C., Schnitker, Sarah A., Schnuerch, Martin, Schreiner, Marcel R., Schuttengruber, Victoria, Sebben, Simone, Segerstrom, Suzanne C., Seryczynska, Berenika, Shjoedt, Uffe, Simsek, Muge, Sleegers, Willem W. A., Smith, Eliot R., Sowden, Walter J., Spath, Marion, Sporlein, Christoph, Stedden, William, Stoevenbelt, Andrea H., Stuber, Simon, Sulik, Justin, Suwartono, Christiany, Syropoulos, Stylianos, Szaszi, Barnabas, Szecsi, Peter, Tappin, Ben M., Tay, Louis, Thibault, Robert T., Thompson, Burt, Thurn, Christian M., Torralba, Josefa, Tuthill, Shelby D., Ullein, Ann-Marie, Van Aert, Robbie C. M., van Assen, Marcel A. L. M., Van Cappellen, Patty, van den Akker, Olmo R., Van der Cruyssen, Ine, Van der Noll, Jolanda, van Dongen, Noah N. N., Van Lissa, Caspar J., van Mulukom, Valerie, van Ravenzwaaij, Don, van Zyl, Casper J. J., Vaughn, Leigh Ann, Verschuere, Bruno, Vianello, Michelangelo, Vilanova, Felipe, Vishkin, Allon, Vogel, Vera, Vogelsmeier, Leonie V. D. E., Watanabe, Shoko, White, Cindel J. M., Wiebels, Kristina, Wiechert, Sera, Willett, Zachary Z., Witkowiak, Maciej, Witvliet, Charlotte V. O., Wiwad, Dylan, Wuyts, Robin, Xygalatas, Dimitris, Yang, Xin, Yeo, Darren J., Yilmaz, Onurcan, Zarzeczna, Natalia, Zhao, Yitong, Zijlmans, Josjan, van Elk, Michiel, Wagenmakers, Eric-Jan, Hoogeveen, Suzanne, Sarafoglou, Alexandra, Aczel, Balazs, Aditya, Yonathan, Alayan, Alexandra J., Allen, Peter J., Altay, Sacha, Alzahawi, Shilaan, Amir, Yulmaida, Anthony, Francis-Vincent, Appiah, Obed Kwame, Atkinson, Quentin D., Baimel, Adam, Balkaya-Ince, Merve, Balsamo, Michela, Banker, Sachin, Bartos, Frantisek, Becerra, Mario, Beffara, Bertrand, Beitner, Julia, Bendixen, Theiss, Berkessel, Jana B., Berniunas, Renatas, Billet, Matthew, Billingsley, Joseph, Bortolini, Tiago, Breitsohl, Heiko, Bret, Amelie, Brown, Faith L., Brown, Jennifer, Brumbaugh, Claudia C., Buczny, Jacek, Bulbulia, Joseph, Caballero, Saul, Carlucci, Leonardo, Carmichael, Cheryl L., Cattaneo, Marco E. G., Charles, Sarah J., Claessens, Scott, Panagopoulos, Maxinne C., Costa, Angelo Brandelli, Crone, Damien L., Czoschke, Stefan, Czymara, Christian, D'Urso, E. Damiano, Dahlstrom, Orjan, Dalla Rosa, Anna, Danielsson, Henrik, De Ron, Jill, de Vries, Ymkje Anna, Dean, Kristy K., Dik, Bryan J., Disabato, David J., Doherty, Jaclyn K., Draws, Tim, Drouhot, Lucas, Dujmovic, Marin, Dunham, Yarrow, Ebert, Tobias, Edelsbrunner, Peter A., Eerland, Anita, Elbaek, Christian T., Farahmand, Shole, Farahmand, Hooman, Farias, Miguel, Feliccia, Abrey A., Fischer, Kyle, Fischer, Ronald, Fisher-Thompson, Donna, Francis, Zoe, Frick, Susanne, Frisch, Lisa K., Geraldes, Diogo, Gerdin, Emily, Geven, Linda, Ghasemi, Omid, Gielens, Erwin, Gligoric, Vukasin, Hagel, Kristin, Hajdu, Nandor, Hamilton, Hannah R., Hamzah, Imaduddin, Hanel, Paul H. P., Hawk, Christopher E., Himawan, Karel K., Holding, Benjamin C., Homman, Lina E., Ingendahl, Moritz, Inkila, Hilla, Inman, Mary L., Islam, Chris-Gabriel, Isler, Ozan, Izydorczyk, David, Jaeger, Bastian, Johnson, Kathryn A., Jong, Jonathan, Karl, Johannes A., Kaszubowski, Erikson, Katz, Benjamin A., Keefer, Lucas A., Kelchtermans, Stijn, Kelly, John M., Klein, Richard A., Kleinberg, Bennett, Knowles, Megan L., Kolczynska, Marta, Koller, Dave, Krasko, Julia, Kritzler, Sarah, Krypotos, Angelos-Miltiadis, Kyritsis, Thanos, Landes, Todd L., Laukenmann, Ruben, Forsyth, Guy A. Lavender, Lazar, Aryeh, Lehman, Barbara J., Levy, Neil, Lo, Ronda F., Lodder, Paul, Lorenz, Jennifer, Lowicki, Pawel, Ly, Albert L., Maassen, Esther, Magyar-Russell, Gina M., Maier, Maximilian, Marsh, Dylan R., Martinez, Nuria, Martinie, Marcellin, Martoyo, Ihan, Mason, Susan E., Mauritsen, Anne Lundahl, McAleer, Phil, McCauley, Thomas, McCullough, Michael, McKay, Ryan, McMahon, Camilla M., McNamara, Amelia A., Means, Kira K., Mercier, Brett, Mitkidis, Panagiotis, Monin, Benoit, Moon, Jordan W., Moreau, David, Morgan, Jonathan, Murphy, James, Muscatt, George, Nagel, Christof, Nagy, Tamas, Nalborczyk, Ladislas, Nilsonne, Gustav, Noack, Pamina, Norenzayan, Ara, Nuijten, Michele B., Olsson-Collentine, Anton, Oviedo, Lluis, Pavlov, Yuri G., Pawelski, James O., Pearson, Hannah, Pedder, Hugo, Peetz, Hannah K., Pinus, Michael, Pirutinsky, Steven, Polito, Vince, Porubanova, Michaela, Poulin, Michael J., Prenoveau, Jason M., Prince, Mark A., Protzko, John, Pryor, Campbell, Purzycki, Benjamin G., Qiu, Lin, Putter, Julian Quevedo, Rabelo, Andre, Radell, Milen L., Ramsay, Jonathan E., Reid, Graham, Roberts, Andrew J., Luna, Lindsey M. Root, Ross, Robert M., Roszak, Piotr, Roy, Nirmal, Saarelainen, Suvi-Maria K., Sasaki, Joni Y., Schaumans, Catherine, Schivinski, Bruno, Schmitt, Marcel C., Schnitker, Sarah A., Schnuerch, Martin, Schreiner, Marcel R., Schuttengruber, Victoria, Sebben, Simone, Segerstrom, Suzanne C., Seryczynska, Berenika, Shjoedt, Uffe, Simsek, Muge, Sleegers, Willem W. A., Smith, Eliot R., Sowden, Walter J., Spath, Marion, Sporlein, Christoph, Stedden, William, Stoevenbelt, Andrea H., Stuber, Simon, Sulik, Justin, Suwartono, Christiany, Syropoulos, Stylianos, Szaszi, Barnabas, Szecsi, Peter, Tappin, Ben M., Tay, Louis, Thibault, Robert T., Thompson, Burt, Thurn, Christian M., Torralba, Josefa, Tuthill, Shelby D., Ullein, Ann-Marie, Van Aert, Robbie C. M., van Assen, Marcel A. L. M., Van Cappellen, Patty, van den Akker, Olmo R., Van der Cruyssen, Ine, Van der Noll, Jolanda, van Dongen, Noah N. N., Van Lissa, Caspar J., van Mulukom, Valerie, van Ravenzwaaij, Don, van Zyl, Casper J. J., Vaughn, Leigh Ann, Verschuere, Bruno, Vianello, Michelangelo, Vilanova, Felipe, Vishkin, Allon, Vogel, Vera, Vogelsmeier, Leonie V. D. E., Watanabe, Shoko, White, Cindel J. M., Wiebels, Kristina, Wiechert, Sera, Willett, Zachary Z., Witkowiak, Maciej, Witvliet, Charlotte V. O., Wiwad, Dylan, Wuyts, Robin, Xygalatas, Dimitris, Yang, Xin, Yeo, Darren J., Yilmaz, Onurcan, Zarzeczna, Natalia, Zhao, Yitong, Zijlmans, Josjan, van Elk, Michiel, and Wagenmakers, Eric-Jan

Abstract

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N = 10, 535 participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported beta = 0.120). For the second research question, this was the case for 65% of the teams (median reported beta = 0.039). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates.

Published
2023

17. Cognitive outcomes after multimodal treatment in adult glioma patients: A meta-analysis

Author

De Roeck, Laurien, Gillebert, R Céline, van Aert, Robbie C M, Vanmeenen, Amber, Klein, Martin, Taphoorn, Martin J B, Gehring, Karin, Lambrecht, Maarten, Sleurs, Charlotte, De Roeck, Laurien, Gillebert, R Céline, van Aert, Robbie C M, Vanmeenen, Amber, Klein, Martin, Taphoorn, Martin J B, Gehring, Karin, Lambrecht, Maarten, and Sleurs, Charlotte

Abstract

Purpose: Cognitive functioning is increasingly assessed as a secondary outcome in neuro-oncological trials. However, which cognitive domains or tests to assess, remains debatable. In this meta-analysis, we aimed to elucidate the longer-term test-specific cognitive outcomes in adult glioma patients.Methods: A systematic search yielded 7098 articles for screening. To investigate cognitive changes in glioma patients and differences between patients and controls ≥one-year follow-up, random-effects meta-analyses were conducted per cognitive test, separately for studies with a longitudinal and cross-sectional design. A meta-regression analysis with a moderator for interval testing (additional cognitive testing between baseline and one-year post-treatment) was performed to investigate the impact of practice in longitudinal designs.Results: Eighty-three studies were reviewed, of which 37 were analyzed in the meta-analysis, involving 4078 patients. In longitudinal designs, semantic fluency was the most sensitive test to detect cognitive decline over time. Cognitive performance on MMSE, digit span forward, phonemic and semantic fluency declined over time in patients who had no interval testing. In cross-sectional studies, patients performed worse than controls on the MMSE, digit span backward, semantic fluency, Stroop speed interference task, trail making test B and finger tapping.Conclusion: Cognitive performance of glioma patients one year after treatment is significantly lower compared to the norm, with specific tests potentially being more sensitive. Cognitive decline over time occurs as well, but can easily be overlooked in longitudinal designs due to practice effects (as a result of interval testing). It is warranted to sufficiently correct for practice effects in future longitudinal trials.

Published
2023

18. Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints

Author

Isager, Peder Mortvedt, van Aert, Robbie C. M., Bahnik, Stepan, Brandt, Mark J., DeSoto, K. Andrew, Giner-Sorolla, Roger, Krueger, Joachim, Perugini, Marco, Ropovik, Ivan, van 't Veer, Anna E., Vranka, Marek, Lakens, Daniel, Isager, Peder Mortvedt, van Aert, Robbie C. M., Bahnik, Stepan, Brandt, Mark J., DeSoto, K. Andrew, Giner-Sorolla, Roger, Krueger, Joachim, Perugini, Marco, Ropovik, Ivan, van 't Veer, Anna E., Vranka, Marek, and Lakens, Daniel

Abstract

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.Translational Abstract Replication-redoing a study using the same procedures-is an important part of checking the robustness of claims in the psychological literature. The practice of replicating original studies has been woefully devalued for many years, but this is now changing. Recent calls for improving the quality of research in psychology has generated a surge of interest in funding, conducting, and publishing replication studies. Because many studies have ne

Published
2023

19. Multistep estimators of the between‐study covariance matrix under the multivariate random‐effects model for meta‐analysis.

Author

Jackson, Dan, Viechtbauer, Wolfgang, and van Aert, Robbie C. M.

Subjects
COVARIANCE matrices, MOMENTS method (Statistics), STATISTICAL models, SENSITIVITY analysis
Abstract

A wide variety of methods are available to estimate the between‐study variance under the univariate random‐effects model for meta‐analysis. Some, but not all, of these estimators have been extended so that they can be used in the multivariate setting. We begin by extending the univariate generalised method of moments, which immediately provides a wider class of multivariate methods than was previously available. However, our main proposal is to use this new type of estimator to derive multivariate multistep estimators of the between‐study covariance matrix. We then use the connection between the univariate multistep and Paule–Mandel estimators to motivate taking the limit, where the number of steps tends toward infinity. We illustrate our methodology using two contrasting examples and investigate its properties in a simulation study. We conclude that the proposed methodology is a fully viable alternative to existing estimation methods, is well suited to sensitivity analyses that explore the use of alternative estimators, and should be used instead of the existing DerSimonian and Laird‐type moments based estimator in application areas where data are expected to be heterogeneous. However, multistep estimators do not seem to outperform the existing estimators when the data are more homogeneous. Advantages of the new multivariate multistep estimator include its semi‐parametric nature and that it is computationally feasible in high dimensions. Our proposed estimation methods are also applicable for multivariate random‐effects meta‐regression, where study‐level covariates are included in the model. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

22. The Meta-Plot

Author

van Assen, Marcel A. L. M., primary, van den Akker, Olmo R., additional, Augusteijn, Hilde E. M., additional, Bakker, Marjan, additional, Nuijten, Michèle B., additional, Olsson-Collentine, Anton, additional, Stoevenbelt, Andrea H., additional, Wicherts, Jelte M., additional, and van Aert, Robbie C. M., additional

Published
2023
Full Text
View/download PDF

23. A many-analysts approach to the relation between religiosity and well-being

Author

Hoogeveen, Suzanne, primary, Sarafoglou, Alexandra, additional, Aczel, Balazs, additional, Aditya, Yonathan, additional, Alayan, Alexandra J., additional, Allen, Peter J., additional, Altay, Sacha, additional, Alzahawi, Shilaan, additional, Amir, Yulmaida, additional, Anthony, Francis-Vincent, additional, Kwame Appiah, Obed, additional, Atkinson, Quentin D., additional, Baimel, Adam, additional, Balkaya-Ince, Merve, additional, Balsamo, Michela, additional, Banker, Sachin, additional, Bartoš, František, additional, Becerra, Mario, additional, Beffara, Bertrand, additional, Beitner, Julia, additional, Bendixen, Theiss, additional, Berkessel, Jana B., additional, Berniūnas, Renatas, additional, Billet, Matthew I., additional, Billingsley, Joseph, additional, Bortolini, Tiago, additional, Breitsohl, Heiko, additional, Bret, Amélie, additional, Brown, Faith L., additional, Brown, Jennifer, additional, Brumbaugh, Claudia C., additional, Buczny, Jacek, additional, Bulbulia, Joseph, additional, Caballero, Saúl, additional, Carlucci, Leonardo, additional, Carmichael, Cheryl L., additional, Cattaneo, Marco E. G. V., additional, Charles, Sarah J., additional, Claessens, Scott, additional, Panagopoulos, Maxinne C., additional, Costa, Angelo Brandelli, additional, Crone, Damien L., additional, Czoschke, Stefan, additional, Czymara, Christian, additional, D'Urso, E. Damiano, additional, Dahlström, Örjan, additional, Rosa, Anna Dalla, additional, Danielsson, Henrik, additional, De Ron, Jill, additional, de Vries, Ymkje Anna, additional, Dean, Kristy K., additional, Dik, Bryan J., additional, Disabato, David J., additional, Doherty, Jaclyn K., additional, Draws, Tim, additional, Drouhot, Lucas, additional, Dujmovic, Marin, additional, Dunham, Yarrow, additional, Ebert, Tobias, additional, Edelsbrunner, Peter A., additional, Eerland, Anita, additional, Elbaek, Christian T., additional, Farahmand, Shole, additional, Farahmand, Hooman, additional, Farias, Miguel, additional, Feliccia, Abrey A., additional, Fischer, Kyle, additional, Fischer, Ronald, additional, Fisher-Thompson, Donna, additional, Francis, Zoë, additional, Frick, Susanne, additional, Frisch, Lisa K., additional, Geraldes, Diogo, additional, Gerdin, Emily, additional, Geven, Linda, additional, Ghasemi, Omid, additional, Gielens, Erwin, additional, Gligorić, Vukašin, additional, Hagel, Kristin, additional, Hajdu, Nandor, additional, Hamilton, Hannah R., additional, Hamzah, Imaduddin, additional, Hanel, Paul H. P., additional, Hawk, Christopher E., additional, K. Himawan, Karel, additional, Holding, Benjamin C., additional, Homman, Lina E., additional, Ingendahl, Moritz, additional, Inkilä, Hilla, additional, Inman, Mary L., additional, Islam, Chris-Gabriel, additional, Isler, Ozan, additional, Izydorczyk, David, additional, Jaeger, Bastian, additional, Johnson, Kathryn A., additional, Jong, Jonathan, additional, Karl, Johannes A., additional, Kaszubowski, Erikson, additional, Katz, Benjamin A., additional, Keefer, Lucas A., additional, Kelchtermans, Stijn, additional, Kelly, John M., additional, Klein, Richard A., additional, Kleinberg, Bennett, additional, Knowles, Megan L., additional, Kołczyńska, Marta, additional, Koller, Dave, additional, Krasko, Julia, additional, Kritzler, Sarah, additional, Krypotos, Angelos-Miltiadis, additional, Kyritsis, Thanos, additional, L. Landes, Todd, additional, Laukenmann, Ruben, additional, Forsyth, Guy A. Lavender, additional, Lazar, Aryeh, additional, Lehman, Barbara J., additional, Levy, Neil, additional, Lo, Ronda F., additional, Lodder, Paul, additional, Lorenz, Jennifer, additional, Łowicki, Paweł, additional, Ly, Albert L., additional, Maassen, Esther, additional, Magyar-Russell, Gina M., additional, Maier, Maximilian, additional, Marsh, Dylan R., additional, Martinez, Nuria, additional, Martinie, Marcellin, additional, Martoyo, Ihan, additional, Mason, Susan E., additional, Mauritsen, Anne Lundahl, additional, McAleer, Phil, additional, McCauley, Thomas, additional, McCullough, Michael, additional, McKay, Ryan, additional, McMahon, Camilla M., additional, McNamara, Amelia A., additional, Means, Kira K., additional, Mercier, Brett, additional, Mitkidis, Panagiotis, additional, Monin, Benoît, additional, Moon, Jordan W., additional, Moreau, David, additional, Morgan, Jonathan, additional, Murphy, James, additional, Muscatt, George, additional, Nägel, Christof, additional, Nagy, Tamás, additional, Nalborczyk, Ladislas, additional, Nilsonne, Gustav, additional, Noack, Pamina, additional, Norenzayan, Ara, additional, Nuijten, Michèle B., additional, Olsson-Collentine, Anton, additional, Oviedo, Lluis, additional, Pavlov, Yuri G., additional, Pawelski, James O., additional, Pearson, Hannah I., additional, Pedder, Hugo, additional, Peetz, Hannah K., additional, Pinus, Michael, additional, Pirutinsky, Steven, additional, Polito, Vince, additional, Porubanova, Michaela, additional, Poulin, Michael J., additional, Prenoveau, Jason M., additional, Prince, Mark A., additional, Protzko, John, additional, Pryor, Campbell, additional, Purzycki, Benjamin G., additional, Qiu, Lin, additional, Pütter, Julian Quevedo, additional, Rabelo, André, additional, Radell, Milen L., additional, Ramsay, Jonathan E., additional, Reid, Graham, additional, J. Roberts, Andrew, additional, Luna, Lindsey M. Root, additional, Ross, Robert M., additional, Roszak, Piotr, additional, Roy, Nirmal, additional, Saarelainen, Suvi-Maria K., additional, Sasaki, Joni Y., additional, Schaumans, Catherine, additional, Schivinski, Bruno, additional, Schmitt, Marcel C., additional, Schnitker, Sarah A., additional, Schnuerch, Martin, additional, Schreiner, Marcel R., additional, Schüttengruber, Victoria, additional, Sebben, Simone, additional, Segerstrom, Suzanne C., additional, Seryczyńska, Berenika, additional, Shjoedt, Uffe, additional, Simsek, Müge, additional, Sleegers, Willem W. A., additional, Smith, Eliot R., additional, Sowden, Walter J., additional, Späth, Marion, additional, Spörlein, Christoph, additional, Stedden, William, additional, Stoevenbelt, Andrea H., additional, Stuber, Simon, additional, Sulik, Justin, additional, Suwartono, Christiany, additional, Syropoulos, Stylianos, additional, Szaszi, Barnabas, additional, Szecsi, Peter, additional, Tappin, Ben M., additional, Tay, Louis, additional, Thibault, Robert T., additional, Thompson, Burt, additional, Thurn, Christian M., additional, Torralba, Josefa, additional, Tuthill, Shelby D., additional, Ullein, Ann-Marie, additional, Van Aert, Robbie C. M., additional, van Assen, Marcel A. L. M., additional, Van Cappellen, Patty, additional, van den Akker, Olmo R., additional, Van der Cruyssen, Ine, additional, Van der Noll, Jolanda, additional, van Dongen, Noah N. N., additional, Van Lissa, Caspar J., additional, van Mulukom, Valerie, additional, van Ravenzwaaij, Don, additional, van Zyl, Casper J. J., additional, Ann Vaughn, Leigh, additional, Većkalov, Bojana, additional, Verschuere, Bruno, additional, Vianello, Michelangelo, additional, Vilanova, Felipe, additional, Vishkin, Allon, additional, Vogel, Vera, additional, Vogelsmeier, Leonie V. D. E., additional, Watanabe, Shoko, additional, White, Cindel J. M., additional, Wiebels, Kristina, additional, Wiechert, Sera, additional, Willett, Zachary Z., additional, Witkowiak, Maciej, additional, Witvliet, Charlotte V. O., additional, Wiwad, Dylan, additional, Wuyts, Robin, additional, Xygalatas, Dimitris, additional, Yang, Xin, additional, Yeo, Darren J., additional, Yilmaz, Onurcan, additional, Zarzeczna, Natalia, additional, Zhao, Yitong, additional, Zijlmans, Josjan, additional, van Elk, Michiel, additional, and Wagenmakers, Eric-Jan, additional

Published
2022
Full Text
View/download PDF

24. A many-analysts approach to the relation between religiosity and well-being

Author

Hoogeveen, Suzanne, Sarafoglou, Alexandra, Aczel, Balazs, Aditya, Yonathan, Alayan, Alexandra J., Allen, Peter J., Altay, Sacha, Alzahawi, Shilaan, Amir, Yulmaida, Anthony, Francis-Vincent, Appiah, Obed Kwame, Atkinson, Quentin D., Baimel, Adam, Balkaya-Ince, Merve, Balsamo, Michela, Banker, Sachin, Bartos, Frantisek, Becerra, Mario, Beffara, Bertrand, Beitner, Julia, Bendixen, Theiss, Berkessel, Jana B., Berniunas, Renatas, Billet, Matthew, Billingsley, Joseph, Bortolini, Tiago, Breitsohl, Heiko, Bret, Amelie, Brown, Faith L., Brown, Jennifer, Brumbaugh, Claudia C., Buczny, Jacek, Bulbulia, Joseph, Caballero, Saul, Carlucci, Leonardo, Carmichael, Cheryl L., Cattaneo, Marco E. G., Charles, Sarah J., Claessens, Scott, Panagopoulos, Maxinne C., Costa, Angelo Brandelli, Crone, Damien L., Czoschke, Stefan, Czymara, Christian, D'Urso, E. Damiano, Dahlstrom, Orjan, Dalla Rosa, Anna, Danielsson, Henrik, De Ron, Jill, de Vries, Ymkje Anna, Dean, Kristy K., Dik, Bryan J., Disabato, David J., Doherty, Jaclyn K., Draws, Tim, Drouhot, Lucas, Dujmovic, Marin, Dunham, Yarrow, Ebert, Tobias, Edelsbrunner, Peter A., Eerland, Anita, Elbaek, Christian T., Farahmand, Shole, Farahmand, Hooman, Farias, Miguel, Feliccia, Abrey A., Fischer, Kyle, Fischer, Ronald, Fisher-Thompson, Donna, Francis, Zoe, Frick, Susanne, Frisch, Lisa K., Geraldes, Diogo, Gerdin, Emily, Geven, Linda, Ghasemi, Omid, Gielens, Erwin, Gligoric, Vukasin, Hagel, Kristin, Hajdu, Nandor, Hamilton, Hannah R., Hamzah, Imaduddin, Hanel, Paul H. P., Hawk, Christopher E., Himawan, Karel K., Holding, Benjamin C., Homman, Lina E., Ingendahl, Moritz, Inkila, Hilla, Inman, Mary L., Islam, Chris-Gabriel, Isler, Ozan, Izydorczyk, David, Jaeger, Bastian, Johnson, Kathryn A., Jong, Jonathan, Karl, Johannes A., Kaszubowski, Erikson, Katz, Benjamin A., Keefer, Lucas A., Kelchtermans, Stijn, Kelly, John M., Klein, Richard A., Kleinberg, Bennett, Knowles, Megan L., Kolczynska, Marta, Koller, Dave, Krasko, Julia, Kritzler, Sarah, Krypotos, Angelos-Miltiadis, Kyritsis, Thanos, Landes, Todd L., Laukenmann, Ruben, Forsyth, Guy A. Lavender, Lazar, Aryeh, Lehman, Barbara J., Levy, Neil, Lo, Ronda F., Lodder, Paul, Lorenz, Jennifer, Lowicki, Pawel, Ly, Albert L., Maassen, Esther, Magyar-Russell, Gina M., Maier, Maximilian, Marsh, Dylan R., Martinez, Nuria, Martinie, Marcellin, Martoyo, Ihan, Mason, Susan E., Mauritsen, Anne Lundahl, McAleer, Phil, McCauley, Thomas, McCullough, Michael, McKay, Ryan, McMahon, Camilla M., McNamara, Amelia A., Means, Kira K., Mercier, Brett, Mitkidis, Panagiotis, Monin, Benoit, Moon, Jordan W., Moreau, David, Morgan, Jonathan, Murphy, James, Muscatt, George, Nagel, Christof, Nagy, Tamas, Nalborczyk, Ladislas, Nilsonne, Gustav, Noack, Pamina, Norenzayan, Ara, Nuijten, Michele B., Olsson-Collentine, Anton, Oviedo, Lluis, Pavlov, Yuri G., Pawelski, James O., Pearson, Hannah, Pedder, Hugo, Peetz, Hannah K., Pinus, Michael, Pirutinsky, Steven, Polito, Vince, Porubanova, Michaela, Poulin, Michael J., Prenoveau, Jason M., Prince, Mark A., Protzko, John, Pryor, Campbell, Purzycki, Benjamin G., Qiu, Lin, Putter, Julian Quevedo, Rabelo, Andre, Radell, Milen L., Ramsay, Jonathan E., Reid, Graham, Roberts, Andrew J., Luna, Lindsey M. Root, Ross, Robert M., Roszak, Piotr, Roy, Nirmal, Saarelainen, Suvi-Maria K., Sasaki, Joni Y., Schaumans, Catherine, Schivinski, Bruno, Schmitt, Marcel C., Schnitker, Sarah A., Schnuerch, Martin, Schreiner, Marcel R., Schuttengruber, Victoria, Sebben, Simone, Segerstrom, Suzanne C., Seryczynska, Berenika, Shjoedt, Uffe, Simsek, Muge, Sleegers, Willem W. A., Smith, Eliot R., Sowden, Walter J., Spath, Marion, Sporlein, Christoph, Stedden, William, Stoevenbelt, Andrea H., Stuber, Simon, Sulik, Justin, Suwartono, Christiany, Syropoulos, Stylianos, Szaszi, Barnabas, Szecsi, Peter, Tappin, Ben M., Tay, Louis, Thibault, Robert T., Thompson, Burt, Thurn, Christian M., Torralba, Josefa, Tuthill, Shelby D., Ullein, Ann-Marie, Van Aert, Robbie C. M., van Assen, Marcel A. L. M., Van Cappellen, Patty, van den Akker, Olmo R., Van der Cruyssen, Ine, Van der Noll, Jolanda, van Dongen, Noah N. N., Van Lissa, Caspar J., van Mulukom, Valerie, van Ravenzwaaij, Don, van Zyl, Casper J. J., Vaughn, Leigh Ann, Verschuere, Bruno, Vianello, Michelangelo, Vilanova, Felipe, Vishkin, Allon, Vogel, Vera, Vogelsmeier, Leonie V. D. E., Watanabe, Shoko, White, Cindel J. M., Wiebels, Kristina, Wiechert, Sera, Willett, Zachary Z., Witkowiak, Maciej, Witvliet, Charlotte V. O., Wiwad, Dylan, Wuyts, Robin, Xygalatas, Dimitris, Yang, Xin, Yeo, Darren J., Yilmaz, Onurcan, Zarzeczna, Natalia, Zhao, Yitong, Zijlmans, Josjan, van Elk, Michiel, Wagenmakers, Eric-Jan, Hoogeveen, Suzanne, Sarafoglou, Alexandra, Aczel, Balazs, Aditya, Yonathan, Alayan, Alexandra J., Allen, Peter J., Altay, Sacha, Alzahawi, Shilaan, Amir, Yulmaida, Anthony, Francis-Vincent, Appiah, Obed Kwame, Atkinson, Quentin D., Baimel, Adam, Balkaya-Ince, Merve, Balsamo, Michela, Banker, Sachin, Bartos, Frantisek, Becerra, Mario, Beffara, Bertrand, Beitner, Julia, Bendixen, Theiss, Berkessel, Jana B., Berniunas, Renatas, Billet, Matthew, Billingsley, Joseph, Bortolini, Tiago, Breitsohl, Heiko, Bret, Amelie, Brown, Faith L., Brown, Jennifer, Brumbaugh, Claudia C., Buczny, Jacek, Bulbulia, Joseph, Caballero, Saul, Carlucci, Leonardo, Carmichael, Cheryl L., Cattaneo, Marco E. G., Charles, Sarah J., Claessens, Scott, Panagopoulos, Maxinne C., Costa, Angelo Brandelli, Crone, Damien L., Czoschke, Stefan, Czymara, Christian, D'Urso, E. Damiano, Dahlstrom, Orjan, Dalla Rosa, Anna, Danielsson, Henrik, De Ron, Jill, de Vries, Ymkje Anna, Dean, Kristy K., Dik, Bryan J., Disabato, David J., Doherty, Jaclyn K., Draws, Tim, Drouhot, Lucas, Dujmovic, Marin, Dunham, Yarrow, Ebert, Tobias, Edelsbrunner, Peter A., Eerland, Anita, Elbaek, Christian T., Farahmand, Shole, Farahmand, Hooman, Farias, Miguel, Feliccia, Abrey A., Fischer, Kyle, Fischer, Ronald, Fisher-Thompson, Donna, Francis, Zoe, Frick, Susanne, Frisch, Lisa K., Geraldes, Diogo, Gerdin, Emily, Geven, Linda, Ghasemi, Omid, Gielens, Erwin, Gligoric, Vukasin, Hagel, Kristin, Hajdu, Nandor, Hamilton, Hannah R., Hamzah, Imaduddin, Hanel, Paul H. P., Hawk, Christopher E., Himawan, Karel K., Holding, Benjamin C., Homman, Lina E., Ingendahl, Moritz, Inkila, Hilla, Inman, Mary L., Islam, Chris-Gabriel, Isler, Ozan, Izydorczyk, David, Jaeger, Bastian, Johnson, Kathryn A., Jong, Jonathan, Karl, Johannes A., Kaszubowski, Erikson, Katz, Benjamin A., Keefer, Lucas A., Kelchtermans, Stijn, Kelly, John M., Klein, Richard A., Kleinberg, Bennett, Knowles, Megan L., Kolczynska, Marta, Koller, Dave, Krasko, Julia, Kritzler, Sarah, Krypotos, Angelos-Miltiadis, Kyritsis, Thanos, Landes, Todd L., Laukenmann, Ruben, Forsyth, Guy A. Lavender, Lazar, Aryeh, Lehman, Barbara J., Levy, Neil, Lo, Ronda F., Lodder, Paul, Lorenz, Jennifer, Lowicki, Pawel, Ly, Albert L., Maassen, Esther, Magyar-Russell, Gina M., Maier, Maximilian, Marsh, Dylan R., Martinez, Nuria, Martinie, Marcellin, Martoyo, Ihan, Mason, Susan E., Mauritsen, Anne Lundahl, McAleer, Phil, McCauley, Thomas, McCullough, Michael, McKay, Ryan, McMahon, Camilla M., McNamara, Amelia A., Means, Kira K., Mercier, Brett, Mitkidis, Panagiotis, Monin, Benoit, Moon, Jordan W., Moreau, David, Morgan, Jonathan, Murphy, James, Muscatt, George, Nagel, Christof, Nagy, Tamas, Nalborczyk, Ladislas, Nilsonne, Gustav, Noack, Pamina, Norenzayan, Ara, Nuijten, Michele B., Olsson-Collentine, Anton, Oviedo, Lluis, Pavlov, Yuri G., Pawelski, James O., Pearson, Hannah, Pedder, Hugo, Peetz, Hannah K., Pinus, Michael, Pirutinsky, Steven, Polito, Vince, Porubanova, Michaela, Poulin, Michael J., Prenoveau, Jason M., Prince, Mark A., Protzko, John, Pryor, Campbell, Purzycki, Benjamin G., Qiu, Lin, Putter, Julian Quevedo, Rabelo, Andre, Radell, Milen L., Ramsay, Jonathan E., Reid, Graham, Roberts, Andrew J., Luna, Lindsey M. Root, Ross, Robert M., Roszak, Piotr, Roy, Nirmal, Saarelainen, Suvi-Maria K., Sasaki, Joni Y., Schaumans, Catherine, Schivinski, Bruno, Schmitt, Marcel C., Schnitker, Sarah A., Schnuerch, Martin, Schreiner, Marcel R., Schuttengruber, Victoria, Sebben, Simone, Segerstrom, Suzanne C., Seryczynska, Berenika, Shjoedt, Uffe, Simsek, Muge, Sleegers, Willem W. A., Smith, Eliot R., Sowden, Walter J., Spath, Marion, Sporlein, Christoph, Stedden, William, Stoevenbelt, Andrea H., Stuber, Simon, Sulik, Justin, Suwartono, Christiany, Syropoulos, Stylianos, Szaszi, Barnabas, Szecsi, Peter, Tappin, Ben M., Tay, Louis, Thibault, Robert T., Thompson, Burt, Thurn, Christian M., Torralba, Josefa, Tuthill, Shelby D., Ullein, Ann-Marie, Van Aert, Robbie C. M., van Assen, Marcel A. L. M., Van Cappellen, Patty, van den Akker, Olmo R., Van der Cruyssen, Ine, Van der Noll, Jolanda, van Dongen, Noah N. N., Van Lissa, Caspar J., van Mulukom, Valerie, van Ravenzwaaij, Don, van Zyl, Casper J. J., Vaughn, Leigh Ann, Verschuere, Bruno, Vianello, Michelangelo, Vilanova, Felipe, Vishkin, Allon, Vogel, Vera, Vogelsmeier, Leonie V. D. E., Watanabe, Shoko, White, Cindel J. M., Wiebels, Kristina, Wiechert, Sera, Willett, Zachary Z., Witkowiak, Maciej, Witvliet, Charlotte V. O., Wiwad, Dylan, Wuyts, Robin, Xygalatas, Dimitris, Yang, Xin, Yeo, Darren J., Yilmaz, Onurcan, Zarzeczna, Natalia, Zhao, Yitong, Zijlmans, Josjan, van Elk, Michiel, and Wagenmakers, Eric-Jan

Abstract

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N = 10, 535 participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported beta = 0.120). For the second research question, this was the case for 65% of the teams (median reported beta = 0.039). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates.

Published
2022

25. Analyzing data of a Multilab replication project with individual participant data meta-analysis: A tutorial

Author

van Aert, Robbie C. M. and van Aert, Robbie C. M.

Abstract

Multilab replication projects such as Registered Replication Reports (RRR) and Many Labs projects are used to replicate an effect in different tabs. Data of these projects are usually analyzed using conventional meta-analysis methods. This is certainly not the best approach because it does not make optimal use of the available data as a summary rather than participant data are analyzed. I propose to analyze data of multilab replication projects with individual participant data (IPD) meta-analysis where the participant data are analyzed directly. The prominent advantages of IPD meta-analysis are that it generally has larger statistical power to detect moderator effects and allows drawing conclusions at the participant and lab level. However, a disadvantage is that IPD meta-analysis is more complex than conventional meta-analysis. In this tutorial, I illustrate IPD meta-analysis using the RRR by McCarthy and colleagues, and 1 provide R code and recommendations to facilitate researchers to apply these methods.

Published
2022

26. Reanalysis Project Psychology

Author

Plonski, Paul E., Wicherts, Jelte, Volz, Leonhard, Grüning, David, Van Aert, Robbie C. M., Bloom, Paul Alexander, Oliveira, Catia Margarida, Cruz, Rick Anthony, Moreau, David, Stoevenbelt, Andrea H., Pronizius, Ekaterina, Herbers, Judith, Nuijten, Michèle B., Elsherif, Mahmoud, Van Assen, Marcel A.L.M., Flournoy, John, Ross, Robert M, Azevedo, Flavio, Schmidt, Kathleen, Ghasemi, Omid, Klein, Richard A., and Rein, Manuel T.

Abstract

Landing page with information about the Reanalysis Project Psychology

Published
2022
Full Text
View/download PDF

30. Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Author

Isager, Peder Mortvedt, primary, van Aert, Robbie C. M., additional, Bahník, Štěpán, additional, Brandt, Mark J., additional, DeSoto, K. Andrew, additional, Giner-Sorolla, Roger, additional, Krueger, Joachim I., additional, Perugini, Marco, additional, Ropovik, Ivan, additional, van 't Veer, Anna E., additional, Vranka, Marek, additional, and Lakens, Daniël, additional

Published
2021
Full Text
View/download PDF

31. Do Behavioral Observations Make People Catch the Goal? A Meta-Analysis on Goal Contagion

Author

Brohmer, Hilmar, Eckerstorfer, Lisa V., van Aert, Robbie C. M., Corcoran, Katja, and Department of Methodology and Statistics

Subjects
Open science, Social Psychology, media_common.quotation_subject, POWER, MOTIVES, MODELS, Control (management), lcsh:BF1-990, 050109 social psychology, CONDUCTING METAANALYSES, 050105 experimental psychology, Presentation, Selection (linguistics), 0501 psychology and cognitive sciences, media_common, publication bias, OBJECT, goal contagion, goal pursuit, automatic inference, meta-analysis, ESTIMATING EFFECT SIZE, 05 social sciences, Contrast (statistics), MOTIVATION, PURSUIT, Publication bias, Moderation, lcsh:Psychology, Meta-analysis, REPLICATION, social psychology, social cognition, Psychology, Cognitive psychology
Abstract

Goal contagion is a social-cognitive approach to understanding how other people's behavior influences one's goal pursuit: An observation of goal-directed behavior leads to an automatic inference and activation of the goal before it can be adopted and pursued thereafter by the observer. We conducted a meta-analysis focusing on experimental studies with a goal condition, depicting goal-directed behavior and a control condition. We searched four databases (PsychInfo, Web of Science, ScienceDirect, and JSTOR) and the citing literature on Google Scholar, and eventually included e = 48 effects from published studies, unpublished studies and registered reports based on 4751 participants. The meta-analytic summary effect was small - g = 0.30, 95%CI [0.21; 0.40], tau(2) = 0.05, 95%CI [0.03, 0.13] - implying that goal contagion might occur for some people, compared to when this goal is not perceived in behavior. However, the original effect seemed to be biased through the current publication system. As shown by several publication-bias tests, the effect could rather be half the size, for example, selection model: g = 0.15, 95%CI [-0.02; 0.32]. Further, we could not detect any potential moderator (such as the presentation of the manipulation and the contrast of the control condition). We suggest that future research on goal contagion makes use of open science practices to advance research in this domain.

Published
2021

32. Same data, different conclusions : Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Author

Schweinsberg, Martin, Feldman, Michael, Staub, Nicola, van den Akker, Olmo R., van Aert, Robbie C. M., van Assen, Marcel A. L. M., Liu, Yang, Althoff, Tim, Heer, Jeffrey, Kale, Alex, Mohamed, Zainab, Amireh, Hashem, Prasad, Vaishali Venkatesh, Bernstein, Abraham, Robinson, Emily, Snellman, Kaisa, Sommer, S. Amy, Otner, Sarah M. G., Robinson, David, Madan, Nikhil, Silberzahn, Raphael, Goldstein, Pavel, Tierney, Warren, Murase, Toshio, Mandl, Benjamin, Viganola, Domenico, Strobl, Carolin, Schaumans, Catherine B. C., Kelchtermans, Stijn, Naseeb, Chan, Garrison, S. Mason, Yarkoni, Tal, Chan, C. S. Richard, Adie, Prestone, Alaburda, Paulius, Albers, Casper, Alspaugh, Sara, Alstott, Jeff, Nelson, Andrew A., de la Rubia, Eduardo Arinno, Arzi, Adbi, Bahnik, Stepan, Baik, Jason, Balling, Laura Winther, Banker, Sachin, Baranger, David A. A., Barr, Dale J., Barros-Rivera, Brenda, Bauer, Matt, Blaise, Enuh, Boelen, Lisa, Carbonell, Katerina Bohle, Briers, Robert A., Burkhard, Oliver, Canela, Miguel-Angel, Castrillo, Laura, Catlett, Timothy, Chen, Olivia, Clark, Michael, Cohn, Brent, Coppock, Alex, Cuguero-Escofet, Natalia, Curran, Paul G., Cyrus-Lai, Wilson, Dai, David, Dalla Riva, Giulio Valentino, Danielsson, Henrik, Russo, Rosaria de F. S. M., de Silva, Niko, Derungs, Curdin, Dondelinger, Frank, de Souza, Carolina Duarte, Dube, B. Tyson, Dubova, Marina, Dunn, Ben Mark, Edelsbrunner, Peter Adriaan, Finley, Sara, Fox, Nick, Gnambs, Timo, Gong, Yuanyuan, Grand, Erin, Greenawalt, Brandon, Han, Dan, Hanel, Paul H. P., Hong, Antony B., Hood, David, Hsueh, Justin, Huang, Lilian, Hui, Kent N., Hultman, Keith A., Javaid, Azka, Jiang, Lily Ji, Jong, Jonathan, Kamdar, Jash, Kane, David, Kappler, Gregor, Kaszubowski, Erikson, Kavanagh, Christopher M., Khabsa, Madian, Kleinberg, Bennett, Kouros, Jens, Krause, Heather, Krypotos, Angelos-Miltiadis, Lavbic, Dejan, Lee, Rui Ling, Leffel, Timothy, Lim, Wei Yang, Liverani, Silvia, Loh, Bianca, Lonsmann, Dorte, Low, Jia Wei, Lu, Alton, MacDonald, Kyle, Madan, Christopher R., Madsen, Lasse Hjorth, Maimone, Christina, Mangold, Alexandra, Marshall, Adrienne, Matskewich, Helena Ester, Mavon, Kimia, McLain, Katherine L., McNamara, Amelia A., McNeill, Mhairi, Mertens, Ulf, Miller, David, Moore, Ben, Moore, Andrew, Nantz, Eric, Nasrullah, Ziauddin, Nejkovic, Valentina, Nell, Colleen S., Nelson, Andrew Arthur, Nilsonne, Gustav, Nolan, Rory, O'Brien, Christopher E., O'Neil, Patrick, O'Shea, Kieran, Olita, Toto, Otterbacher, Jahna, Palsetia, Diana, Pereira, Bianca, Pozdniakov, Ivan, Protzko, John, Reyt, Jean-Nicolas, Riddle, Travis, Ali, Amal (Akmal) Ridhwan Omar, Ropovik, Ivan, Rosenberg, Joshua M., Rothen, Stephane, Schulte-Mecklenbeck, Michael, Sharma, Nirek, Shotwell, Gordon, Skarzynski, Martin, Stedden, William, Stodden, Victoria, Stoffel, Martin A., Stoltzman, Scott, Subbaiah, Subashini, Tatman, Rachael, Thibodeau, Paul H., Tomkins, Sabina, Valdivia, Ana, van de Woestijne, Gerrieke B., Viana, Laura, Villeseche, Florence, Wadsworth, W. Duncan, Wanders, Florian, Watts, Krista, Wells, Jason D., Whelpley, Christopher E., Won, Andy, Wu, Lawrence, Yip, Arthur, Youngflesh, Casey, Yu, Ju-Chi, Zandian, Arash, Zhang, Leilei, Zibman, Chava, Uhlmann, Eric Luis, Schweinsberg, Martin, Feldman, Michael, Staub, Nicola, van den Akker, Olmo R., van Aert, Robbie C. M., van Assen, Marcel A. L. M., Liu, Yang, Althoff, Tim, Heer, Jeffrey, Kale, Alex, Mohamed, Zainab, Amireh, Hashem, Prasad, Vaishali Venkatesh, Bernstein, Abraham, Robinson, Emily, Snellman, Kaisa, Sommer, S. Amy, Otner, Sarah M. G., Robinson, David, Madan, Nikhil, Silberzahn, Raphael, Goldstein, Pavel, Tierney, Warren, Murase, Toshio, Mandl, Benjamin, Viganola, Domenico, Strobl, Carolin, Schaumans, Catherine B. C., Kelchtermans, Stijn, Naseeb, Chan, Garrison, S. Mason, Yarkoni, Tal, Chan, C. S. Richard, Adie, Prestone, Alaburda, Paulius, Albers, Casper, Alspaugh, Sara, Alstott, Jeff, Nelson, Andrew A., de la Rubia, Eduardo Arinno, Arzi, Adbi, Bahnik, Stepan, Baik, Jason, Balling, Laura Winther, Banker, Sachin, Baranger, David A. A., Barr, Dale J., Barros-Rivera, Brenda, Bauer, Matt, Blaise, Enuh, Boelen, Lisa, Carbonell, Katerina Bohle, Briers, Robert A., Burkhard, Oliver, Canela, Miguel-Angel, Castrillo, Laura, Catlett, Timothy, Chen, Olivia, Clark, Michael, Cohn, Brent, Coppock, Alex, Cuguero-Escofet, Natalia, Curran, Paul G., Cyrus-Lai, Wilson, Dai, David, Dalla Riva, Giulio Valentino, Danielsson, Henrik, Russo, Rosaria de F. S. M., de Silva, Niko, Derungs, Curdin, Dondelinger, Frank, de Souza, Carolina Duarte, Dube, B. Tyson, Dubova, Marina, Dunn, Ben Mark, Edelsbrunner, Peter Adriaan, Finley, Sara, Fox, Nick, Gnambs, Timo, Gong, Yuanyuan, Grand, Erin, Greenawalt, Brandon, Han, Dan, Hanel, Paul H. P., Hong, Antony B., Hood, David, Hsueh, Justin, Huang, Lilian, Hui, Kent N., Hultman, Keith A., Javaid, Azka, Jiang, Lily Ji, Jong, Jonathan, Kamdar, Jash, Kane, David, Kappler, Gregor, Kaszubowski, Erikson, Kavanagh, Christopher M., Khabsa, Madian, Kleinberg, Bennett, Kouros, Jens, Krause, Heather, Krypotos, Angelos-Miltiadis, Lavbic, Dejan, Lee, Rui Ling, Leffel, Timothy, Lim, Wei Yang, Liverani, Silvia, Loh, Bianca, Lonsmann, Dorte, Low, Jia Wei, Lu, Alton, MacDonald, Kyle, Madan, Christopher R., Madsen, Lasse Hjorth, Maimone, Christina, Mangold, Alexandra, Marshall, Adrienne, Matskewich, Helena Ester, Mavon, Kimia, McLain, Katherine L., McNamara, Amelia A., McNeill, Mhairi, Mertens, Ulf, Miller, David, Moore, Ben, Moore, Andrew, Nantz, Eric, Nasrullah, Ziauddin, Nejkovic, Valentina, Nell, Colleen S., Nelson, Andrew Arthur, Nilsonne, Gustav, Nolan, Rory, O'Brien, Christopher E., O'Neil, Patrick, O'Shea, Kieran, Olita, Toto, Otterbacher, Jahna, Palsetia, Diana, Pereira, Bianca, Pozdniakov, Ivan, Protzko, John, Reyt, Jean-Nicolas, Riddle, Travis, Ali, Amal (Akmal) Ridhwan Omar, Ropovik, Ivan, Rosenberg, Joshua M., Rothen, Stephane, Schulte-Mecklenbeck, Michael, Sharma, Nirek, Shotwell, Gordon, Skarzynski, Martin, Stedden, William, Stodden, Victoria, Stoffel, Martin A., Stoltzman, Scott, Subbaiah, Subashini, Tatman, Rachael, Thibodeau, Paul H., Tomkins, Sabina, Valdivia, Ana, van de Woestijne, Gerrieke B., Viana, Laura, Villeseche, Florence, Wadsworth, W. Duncan, Wanders, Florian, Watts, Krista, Wells, Jason D., Whelpley, Christopher E., Won, Andy, Wu, Lawrence, Yip, Arthur, Youngflesh, Casey, Yu, Ju-Chi, Zandian, Arash, Zhang, Leilei, Zibman, Chava, and Uhlmann, Eric Luis

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists' gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for orga-nizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

Published
2021
Full Text
View/download PDF

33. The role of meta-analysis and preregistration in assessing the evidence for cleansing effects

Author

Ross, Robert M., van Aert, Robbie C. M., van den Akker, Olmo R., van Elk, Michiel, Ross, Robert M., van Aert, Robbie C. M., van den Akker, Olmo R., and van Elk, Michiel

Abstract

Lee and Schwarz interpret meta-analytic research and replication studies as providing evidence for the robustness of cleansing effects. We argue that the currently available evidence is unconvincing because (a) publication bias and the opportunistic use of researcher degrees of freedom appear to have inflated meta-analytic effect size estimates, and (b) preregistered replications failed to find any evidence of cleansing effects.

Published
2021

37. Comparison of the performance of methods to assess publication bias in real data

Author

Niemeyer, Helen, Van Aert, Robbie C. M., Schmid, Sebastian, Uelsmann, Dominik, Knaevelsrud, Christine, and Schulte-Herbrueggen, Olaf

Abstract

Background: Publication bias is widespread within many scientific disciplines and also within psychology. If only published studies are included in a meta-analysis of psychotherapy research, the efficacy of interventions may be overestimated. However, the treatments in evidence-based psychotherapy are mainly selected based on published rather than unpublished research. The presence and impact of publication bias in psychotherapy research remains largely unknown. Posttraumatic stress disorder (PTSD) is a highly distressing and common condition, and various forms of psychological interventions for treating PTSD have been investigated in a large number of studies. A comprehensive statistical assessment of publication bias in meta-analyses of psychotherapeutic treatments for PTSD has not been conducted, even though a considerable number of statistical methods to investigate the presence and impact of publication bias have been developed in recent years. Objectives: We compare the performance of six state-of-the-art publication bias methods on a large-scale data set by re-analyzing all meta-analyses which investigate the efficacy of psychotherapeutic interventions for PTSD. Research question: We aim at investigating the amount of publication bias in all meta-analyses on the efficacy of psychotherapeutic treatment for PTSD and to compare the performance of methods to assess publication bias. A comparison on real data is not as straightforward as in simulation studies, since the true amount of publication bias is unknown in real data. Hence, the performance of the methods will be examined by comparing them to the other included methods. Method: We screened the databases PsycINFO, Psyndex, PubMed, and the Cochrane Database for all published and unpublished meta-analyses in English or German up to 5th September 2015. In addition, a snowball search system was used. Meta-analyses were required to meet the following inclusion criteria: 1) A psychotherapeutic intervention was evaluated. 2) The intervention aimed at reducing subclinical or clinical PTSD. 3) A summary effect size was provided. We included only data sets where the null hypothesis of homogeneous true effect size was not rejected, because the statistical methods become biased if the true effect sizes are heterogeneous. This hypothesis was tested by means of the Q-test and quantified by I2. Moreover, we excluded all data sets of a meta-analysis that included five or fewer trials, as the methods to detect publication bias are underpowered if the number of studies is too small. We included the following methods to test whether publication bias was present in a meta-analysis: Egger’s regression test, rank-correlation test, TES, and p-uniform’s publication bias test. Four different methods were included to estimate the effect size and test the null hypothesis of no effect: traditional meta-analysis, trim and fill, PET-PEESE, and p-uniform. The degree of agreement among the methods was examined by means of Loevinger’s H, because the publication bias tests and tests of the null hypothesis of no effect resulted in a dichotomous decision (statistically significant or not). Results: The literature search resulted in 7,647 hits including duplicates. The screening process reduced this number to 502 meta-analyses, of which 83 dealt with the efficacy of psychotherapeutic interventions for PTSD and were included. The meta-analyses included a total number of 2,131 data sets, of which 93 data sets from 24 meta-analyses fulfilled all inclusion criteria. They included a median number of 7 studies. Since publication bias tests have low statistical power if the number of effect sizes is small in a meta-analysis, the characteristics of many of the data sets are not well-suited for methods of detecting publication bias. The median number of statistically significant effect sizes in the data sets was 3. Of all methods Egger’s regression test detected publication bias the most, i.e. in 17 data sets (18.3%). At most two methods detected publication bias test in the same data set, which occurred in 4 data sets (4.3%). Loevinger’s H varied between -.075 and 1. For the test of no effect, Loevingers´s H varied between .668 and 1. When estimating effect sizes corrected for publication bias, results show that especially estimates of PET-PEESE were closer to zero than traditional meta-analysis and that the standard deviation of the estimates of PET-PEESE and p-uniform was larger than traditional meta-analysis and trim and fill. The mean of the difference in effect size estimate between PET-PEESE and the traditional meta-analytic estimate was -0.108 (SD = 0.886). P-uniform was applied to a subset of 72 data sets, because at least one study in a data set has to be statistically significant for this method. The mean of the difference in effect size estimate of p-uniform and traditional meta-analysis was 0.002 (SD = 0.355). Estimates of PET-PEESE were especially unrealistic if there was a small number of effect sizes in a data set in combination with small variation in the standard errors of the primary studies. P-uniform’s estimates were unrealistically large or small when a small number of statistically significant effect sizes were observed with p-values just below the α-level. Conclusions: Our study is the first to apply a multitude of publication bias methods to a large-scale real data set. Publication bias tests did not result in the same conclusion in the majority of the data sets which is unlikely if extreme publication bias was present. No clear indications for overestimated effect sizes were observed when comparing effect size estimates of traditional meta-analysis with methods to correct for publication bias. However, the assessments of publication bias in psychotherapy research may have lacked statistical power to detect publication bias. Moreover, the conclusion regarding statistical significance of the test of no effect often changed when correcting for publication bias with PET-PEESE and p-uniform compared to traditional meta-analysis. This is at least partly caused by the less precise effect size estimates of these methods since the effect size estimates corrected for publication bias did not provide strong evidence for overestimation caused by publication bias. Future research is needed to study the convergence and divergence of publication bias tests as a function of publication bias and the number of primary studies in a meta-analysis.

Published
2019
Full Text
View/download PDF

39. The effect of publication bias on the Q test and assessment of heterogeneity

Author

Augusteijn, Hilde E M, van Aert, Robbie C M, van Assen, M.A.L.M., Augusteijn, Hilde E M, van Aert, Robbie C M, and van Assen, M.A.L.M.

Abstract

One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main contributions and is relevant to all researchers conducting meta-analysis. First, we show when and how publication bias affects the assessment of heterogeneity. The expected values of heterogeneity measures H² and I² were analytically derived, and the power and Type I error rate of the Q test were examined in a Monte Carlo simulation study. Our results show that the effect of publication bias on the Q test and assessment of heterogeneity is large, complex, and nonlinear. Publication bias can both dramatically decrease and increase heterogeneity in true effect size, particularly if the number of studies is large and population effect size is small. We therefore conclude that the Q test of homogeneity and heterogeneity measures H² and I² are generally not valid when publication bias is present. Our second contribution is that we introduce a web application, Q-sense, which can be used to determine the impact of publication bias on the assessment of heterogeneity within a certain meta-analysis and to assess the robustness of the meta-analytic estimate to publication bias. Furthermore, we apply Q-sense to 2 published meta-analyses, showing how publication bias can result in invalid estimates of effect size and heterogeneity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Published
2019

40. Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

Author

Van Aert, Robbie C. M., Wicherts, Jelte M., Van Assen, Marcel A. L. M., Van Aert, Robbie C. M., Wicherts, Jelte M., and Van Assen, Marcel A. L. M.

Abstract

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medici

Published
2019

41. Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication

Author

Van Aert, Robbie C M, van Assen, Marcel A.L.M., Leerstoel Buskens, Social Networks, Solidarity and Inequality, and Department of Methodology and Statistics

Subjects
Open science, Computer science, Replication, 050109 social psychology, Experimental and Cognitive Psychology, Reproducibility Project, 050105 experimental psychology, Article, Arts and Humanities (miscellaneous), Meta-Analysis as Topic, Statistical significance, IMPLICIT, Statistics, Replication (statistics), Developmental and Educational Psychology, Humans, Psychology, 0501 psychology and cognitive sciences, p-Uniform, General Psychology, EFFECT SIZE, METAANALYSIS, CONFLICT, Estimation, 05 social sciences, MEMORY, ATTENTION, Reproducibility of Results, DECISION, Confidence interval, Reproducibility, Meta-analysis, BIAS, EXCESS, SINGLE, Research Design, Data Interpretation, Statistical, Psychology (miscellaneous), Null hypothesis
Abstract

The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method. Electronic supplementary material The online version of this article (10.3758/s13428-017-0967-6) contains supplementary material, which is available to authorized users.

Published
2018

42. Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication

Author

Van Aert, Robbie C M, van Assen, Marcel A.L.M., Leerstoel Buskens, and Social Networks, Solidarity and Inequality

Subjects
Meta-analysis, Replication, p-Uniform, Reproducibility
Abstract

The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study’s effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application (https://rvanaert.shinyapps.io/hybrid) for applying the hybrid method.

Published
2017

44. Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Author

Klein, Richard A., primary, Vianello, Michelangelo, additional, Hasselman, Fred, additional, Adams, Byron G., additional, Adams, Reginald B., additional, Alper, Sinan, additional, Aveyard, Mark, additional, Axt, Jordan R., additional, Babalola, Mayowa T., additional, Bahník, Štěpán, additional, Batra, Rishtee, additional, Berkics, Mihály, additional, Bernstein, Michael J., additional, Berry, Daniel R., additional, Bialobrzeska, Olga, additional, Binan, Evans Dami, additional, Bocian, Konrad, additional, Brandt, Mark J., additional, Busching, Robert, additional, Rédei, Anna Cabak, additional, Cai, Huajian, additional, Cambier, Fanny, additional, Cantarero, Katarzyna, additional, Carmichael, Cheryl L., additional, Ceric, Francisco, additional, Chandler, Jesse, additional, Chang, Jen-Ho, additional, Chatard, Armand, additional, Chen, Eva E., additional, Cheong, Winnee, additional, Cicero, David C., additional, Coen, Sharon, additional, Coleman, Jennifer A., additional, Collisson, Brian, additional, Conway, Morgan A., additional, Corker, Katherine S., additional, Curran, Paul G., additional, Cushman, Fiery, additional, Dagona, Zubairu K., additional, Dalgar, Ilker, additional, Dalla Rosa, Anna, additional, Davis, William E., additional, de Bruijn, Maaike, additional, De Schutter, Leander, additional, Devos, Thierry, additional, de Vries, Marieke, additional, Doğulu, Canay, additional, Dozo, Nerisa, additional, Dukes, Kristin Nicole, additional, Dunham, Yarrow, additional, Durrheim, Kevin, additional, Ebersole, Charles R., additional, Edlund, John E., additional, Eller, Anja, additional, English, Alexander Scott, additional, Finck, Carolyn, additional, Frankowska, Natalia, additional, Freyre, Miguel-Ángel, additional, Friedman, Mike, additional, Galliani, Elisa Maria, additional, Gandi, Joshua C., additional, Ghoshal, Tanuka, additional, Giessner, Steffen R., additional, Gill, Tripat, additional, Gnambs, Timo, additional, Gómez, Ángel, additional, González, Roberto, additional, Graham, Jesse, additional, Grahe, Jon E., additional, Grahek, Ivan, additional, Green, Eva G. T., additional, Hai, Kakul, additional, Haigh, Matthew, additional, Haines, Elizabeth L., additional, Hall, Michael P., additional, Heffernan, Marie E., additional, Hicks, Joshua A., additional, Houdek, Petr, additional, Huntsinger, Jeffrey R., additional, Huynh, Ho Phi, additional, IJzerman, Hans, additional, Inbar, Yoel, additional, Innes-Ker, Åse H., additional, Jiménez-Leal, William, additional, John, Melissa-Sue, additional, Joy-Gaba, Jennifer A., additional, Kamiloğlu, Roza G., additional, Kappes, Heather Barry, additional, Karabati, Serdar, additional, Karick, Haruna, additional, Keller, Victor N., additional, Kende, Anna, additional, Kervyn, Nicolas, additional, Knežević, Goran, additional, Kovacs, Carrie, additional, Krueger, Lacy E., additional, Kurapov, German, additional, Kurtz, Jamie, additional, Lakens, Daniël, additional, Lazarević, Ljiljana B., additional, Levitan, Carmel A., additional, Lewis, Neil A., additional, Lins, Samuel, additional, Lipsey, Nikolette P., additional, Losee, Joy E., additional, Maassen, Esther, additional, Maitner, Angela T., additional, Malingumu, Winfrida, additional, Mallett, Robyn K., additional, Marotta, Satia A., additional, Međedović, Janko, additional, Mena-Pacheco, Fernando, additional, Milfont, Taciano L., additional, Morris, Wendy L., additional, Murphy, Sean C., additional, Myachykov, Andriy, additional, Neave, Nick, additional, Neijenhuijs, Koen, additional, Nelson, Anthony J., additional, Neto, Félix, additional, Lee Nichols, Austin, additional, Ocampo, Aaron, additional, O’Donnell, Susan L., additional, Oikawa, Haruka, additional, Oikawa, Masanori, additional, Ong, Elsie, additional, Orosz, Gábor, additional, Osowiecka, Malgorzata, additional, Packard, Grant, additional, Pérez-Sánchez, Rolando, additional, Petrović, Boban, additional, Pilati, Ronaldo, additional, Pinter, Brad, additional, Podesta, Lysandra, additional, Pogge, Gabrielle, additional, Pollmann, Monique M. H., additional, Rutchick, Abraham M., additional, Saavedra, Patricio, additional, Saeri, Alexander K., additional, Salomon, Erika, additional, Schmidt, Kathleen, additional, Schönbrodt, Felix D., additional, Sekerdej, Maciej B., additional, Sirlopú, David, additional, Skorinko, Jeanine L. M., additional, Smith, Michael A., additional, Smith-Castro, Vanessa, additional, Smolders, Karin C. H. J., additional, Sobkow, Agata, additional, Sowden, Walter, additional, Spachtholz, Philipp, additional, Srivastava, Manini, additional, Steiner, Troy G., additional, Stouten, Jeroen, additional, Street, Chris N. H., additional, Sundfelt, Oskar K., additional, Szeto, Stephanie, additional, Szumowska, Ewa, additional, Tang, Andrew C. W., additional, Tanzer, Norbert, additional, Tear, Morgan J., additional, Theriault, Jordan, additional, Thomae, Manuela, additional, Torres, David, additional, Traczyk, Jakub, additional, Tybur, Joshua M., additional, Ujhelyi, Adrienn, additional, van Aert, Robbie C. M., additional, van Assen, Marcel A. L. M., additional, van der Hulst, Marije, additional, van Lange, Paul A. M., additional, van ’t Veer, Anna Elisabeth, additional, Vásquez- Echeverría, Alejandro, additional, Ann Vaughn, Leigh, additional, Vázquez, Alexandra, additional, Vega, Luis Diego, additional, Verniers, Catherine, additional, Verschoor, Mark, additional, Voermans, Ingrid P. J., additional, Vranka, Marek A., additional, Welch, Cheryl, additional, Wichman, Aaron L., additional, Williams, Lisa A., additional, Wood, Michael, additional, Woodzicka, Julie A., additional, Wronska, Marta K., additional, Young, Liane, additional, Zelenski, John M., additional, Zhijia, Zeng, additional, and Nosek, Brian A., additional

Published
2018
Full Text
View/download PDF

46. Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Author

Klein, Richard A., Vianello, Michelangelo, Hasselman, Fred, Adams, Byron G., Adams, Reginald B., Alper, Sinan, Aveyard, Mark, Axt, Jordan R., Babalola, Mayowa T., Bahník, Štěpán, Batra, Rishtee, Berkics, Mihály, Bernstein, Michael J., Berry, Daniel R., Bialobrzeska, Olga, Binan, Evans Dami, Bocian, Konrad, Brandt, Mark J., Busching, Robert, Rédei, Anna Cabak, Cai, Huajian, Cambier, Fanny, Cantarero, Katarzyna, Carmichael, Cheryl L., Ceric, Francisco, Chandler, Jesse, Chang, Jen-Ho, Chatard, Armand, Chen, Eva E., Cheong, Winnee, Cicero, David C., Coen, Sharon, Coleman, Jennifer A., Collisson, Brian, Conway, Morgan A., Corker, Katherine S., Curran, Paul G., Cushman, Fiery, Dagona, Zubairu K., Dalgar, Ilker, Dalla Rosa, Anna, Davis, William E., de Bruijn, Maaike, De Schutter, Leander, Devos, Thierry, de Vries, Marieke, Doğulu, Canay, Dozo, Nerisa, Dukes, Kristin Nicole, Dunham, Yarrow, Durrheim, Kevin, Ebersole, Charles R., Edlund, John E., Eller, Anja, English, Alexander Scott, Finck, Carolyn, Frankowska, Natalia, Freyre, Miguel-ángel, Friedman, Mike, Galliani, Elisa Maria, Gandi, Joshua C., Ghoshal, Tanuka, Giessner, Steffen R., Gill, Tripat, Gnambs, Timo, Gómez, Ángel, González, Roberto, Graham, Jesse, Grahe, Jon E., Grahek, Ivan, Green, Eva G.T., Hai, Kakul, Haigh, Matthew, Haines, Elizabeth L., Hall, Michael P., Heffernan, Marie E., Hicks, Joshua A., Houdek, Petr, Huntsinger, Jeffrey R., Huynh, Ho Phi, Ijzerman, Hans, Inbar, Yoel, Innes-ker, Åse H., Jiménez-leal, William, John, Melissa-sue, Joy-gaba, Jennifer A., Kamiloğlu, Roza G., Kappes, Heather Barry, Karabati, Serdar, Karick, Haruna, Keller, Victor N., Kende, Anna, Kervyn, Nicolas, Knežević, Goran, Kovacs, Carrie, Krueger, Lacy E., Kurapov, German, Kurtz, Jamie, Lakens, Daniël, Lazarević, Ljiljana B., Levitan, Carmel A., Lewis, Neil A., Lins, Samuel, Lipsey, Nikolette P., Losee, Joy E., Maassen, Esther, Maitner, Angela T., Malingumu, Winfrida, Mallett, Robyn K., Marotta, Satia A., Međedović, Janko, Mena-pacheco, Fernando, Milfont, Taciano L., Morris, Wendy L., Murphy, Sean C., Myachykov, Andriy, Neave, Nick, Neijenhuijs, Koen, Nelson, Anthony J., Neto, Félix, Lee Nichols, Austin, Ocampo, Aaron, O’donnell, Susan L., Oikawa, Haruka, Oikawa, Masanori, Ong, Elsie, Orosz, Gábor, Osowiecka, Malgorzata, Packard, Grant, Pérez-sánchez, Rolando, Petrović, Boban, Pilati, Ronaldo, Pinter, Brad, Podesta, Lysandra, Pogge, Gabrielle, Pollmann, Monique M. H., Rutchick, Abraham M., Saavedra, Patricio, Saeri, Alexander K., Salomon, Erika, Schmidt, Kathleen, Schönbrodt, Felix D., Sekerdej, Maciej B., Sirlopú, David, Skorinko, Jeanine L. M., Smith, Michael A., Smith-castro, Vanessa, Smolders, Karin C.H.J., Sobkow, Agata, Sowden, Walter, Spachtholz, Philipp, Srivastava, Manini, Steiner, Troy G., Stouten, Jeroen, Street, Chris N.H., Sundfelt, Oskar K., Szeto, Stephanie, Szumowska, Ewa, Tang, Andrew C.W., Tanzer, Norbert, Tear, Morgan J., Theriault, Jordan, Thomae, Manuela, Torres, David, Traczyk, Jakub, Tybur, Joshua M., Ujhelyi, Adrienn, Van Aert, Robbie C. M., Van Assen, Marcel A. L. M., Van Der Hulst, Marije, Van Lange, Paul A. M., Van ’t Veer, Anna Elisabeth, Vásquez- Echeverría, Alejandro, Ann Vaughn, Leigh, Vázquez, Alexandra, Vega, Luis Diego, Verniers, Catherine, Verschoor, Mark, Voermans, Ingrid P. J., Vranka, Marek A., Welch, Cheryl, Wichman, Aaron L., Williams, Lisa A., Wood, Michael, Woodzicka, Julie A., Wronska, Marta K., Young, Liane, Zelenski, John M., Zhijia, Zeng, Nosek, Brian A., Klein, Richard A., Vianello, Michelangelo, Hasselman, Fred, Adams, Byron G., Adams, Reginald B., Alper, Sinan, Aveyard, Mark, Axt, Jordan R., Babalola, Mayowa T., Bahník, Štěpán, Batra, Rishtee, Berkics, Mihály, Bernstein, Michael J., Berry, Daniel R., Bialobrzeska, Olga, Binan, Evans Dami, Bocian, Konrad, Brandt, Mark J., Busching, Robert, Rédei, Anna Cabak, Cai, Huajian, Cambier, Fanny, Cantarero, Katarzyna, Carmichael, Cheryl L., Ceric, Francisco, Chandler, Jesse, Chang, Jen-Ho, Chatard, Armand, Chen, Eva E., Cheong, Winnee, Cicero, David C., Coen, Sharon, Coleman, Jennifer A., Collisson, Brian, Conway, Morgan A., Corker, Katherine S., Curran, Paul G., Cushman, Fiery, Dagona, Zubairu K., Dalgar, Ilker, Dalla Rosa, Anna, Davis, William E., de Bruijn, Maaike, De Schutter, Leander, Devos, Thierry, de Vries, Marieke, Doğulu, Canay, Dozo, Nerisa, Dukes, Kristin Nicole, Dunham, Yarrow, Durrheim, Kevin, Ebersole, Charles R., Edlund, John E., Eller, Anja, English, Alexander Scott, Finck, Carolyn, Frankowska, Natalia, Freyre, Miguel-ángel, Friedman, Mike, Galliani, Elisa Maria, Gandi, Joshua C., Ghoshal, Tanuka, Giessner, Steffen R., Gill, Tripat, Gnambs, Timo, Gómez, Ángel, González, Roberto, Graham, Jesse, Grahe, Jon E., Grahek, Ivan, Green, Eva G.T., Hai, Kakul, Haigh, Matthew, Haines, Elizabeth L., Hall, Michael P., Heffernan, Marie E., Hicks, Joshua A., Houdek, Petr, Huntsinger, Jeffrey R., Huynh, Ho Phi, Ijzerman, Hans, Inbar, Yoel, Innes-ker, Åse H., Jiménez-leal, William, John, Melissa-sue, Joy-gaba, Jennifer A., Kamiloğlu, Roza G., Kappes, Heather Barry, Karabati, Serdar, Karick, Haruna, Keller, Victor N., Kende, Anna, Kervyn, Nicolas, Knežević, Goran, Kovacs, Carrie, Krueger, Lacy E., Kurapov, German, Kurtz, Jamie, Lakens, Daniël, Lazarević, Ljiljana B., Levitan, Carmel A., Lewis, Neil A., Lins, Samuel, Lipsey, Nikolette P., Losee, Joy E., Maassen, Esther, Maitner, Angela T., Malingumu, Winfrida, Mallett, Robyn K., Marotta, Satia A., Međedović, Janko, Mena-pacheco, Fernando, Milfont, Taciano L., Morris, Wendy L., Murphy, Sean C., Myachykov, Andriy, Neave, Nick, Neijenhuijs, Koen, Nelson, Anthony J., Neto, Félix, Lee Nichols, Austin, Ocampo, Aaron, O’donnell, Susan L., Oikawa, Haruka, Oikawa, Masanori, Ong, Elsie, Orosz, Gábor, Osowiecka, Malgorzata, Packard, Grant, Pérez-sánchez, Rolando, Petrović, Boban, Pilati, Ronaldo, Pinter, Brad, Podesta, Lysandra, Pogge, Gabrielle, Pollmann, Monique M. H., Rutchick, Abraham M., Saavedra, Patricio, Saeri, Alexander K., Salomon, Erika, Schmidt, Kathleen, Schönbrodt, Felix D., Sekerdej, Maciej B., Sirlopú, David, Skorinko, Jeanine L. M., Smith, Michael A., Smith-castro, Vanessa, Smolders, Karin C.H.J., Sobkow, Agata, Sowden, Walter, Spachtholz, Philipp, Srivastava, Manini, Steiner, Troy G., Stouten, Jeroen, Street, Chris N.H., Sundfelt, Oskar K., Szeto, Stephanie, Szumowska, Ewa, Tang, Andrew C.W., Tanzer, Norbert, Tear, Morgan J., Theriault, Jordan, Thomae, Manuela, Torres, David, Traczyk, Jakub, Tybur, Joshua M., Ujhelyi, Adrienn, Van Aert, Robbie C. M., Van Assen, Marcel A. L. M., Van Der Hulst, Marije, Van Lange, Paul A. M., Van ’t Veer, Anna Elisabeth, Vásquez- Echeverría, Alejandro, Ann Vaughn, Leigh, Vázquez, Alexandra, Vega, Luis Diego, Verniers, Catherine, Verschoor, Mark, Voermans, Ingrid P. J., Vranka, Marek A., Welch, Cheryl, Wichman, Aaron L., Williams, Lisa A., Wood, Michael, Woodzicka, Julie A., Wronska, Marta K., Young, Liane, Zelenski, John M., Zhijia, Zeng, and Nosek, Brian A.

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relati

Published
2018

47. Data and Results (per site, etc.)

Author

Huajian Cai, Vázquez, Alexandra, Voermans, Ingrid, Spachtholz, Philipp, English, Alexander Scott, Adams, Byron Gregory, Conway, Morgan, Stouten, Jeroen, Ruggs, Enrica N., Bocian, Konrad, Cicero, David C, Pinter, Brad, Štěpán Bahník, Van Aert, Robbie C. M., Chatard, Armand, Sacco, Airi, Kende, Anna, Aveyard, Mark, Vranka, Marek A., Ujhelyi, Adrienn, Zelenski, John M., Patricio Saavedra, M., Young, Liane, Bernstein, Michael Jason, Theriault, Jordan, Neto, Felix, De Bruijn, Maaike, Lewis, Neil, Berry, Daniel R., Gnambs, Timo, Busching, Robert, Tanuka Ghoshal, Houdek, Petr, Kappes, Andreas, Kakul Hai, Vaughn, Leigh Ann, Pilati, Ronaldo, Giessner, Steffen Robert, Green, Eva, Malgorzata Osowiecka, Mayowa Babaloia, Mihaly Berkics, Krueger, Lacy Elise, Joy-Gaba, Jennifer, Melissa-Sue John, Verniers, Catherine, Yarrow Dunham, Vásquez-Echeverría, Alejandro, Winfrida Malingumu, Coen, Sharon, Woodzicka, Julie A., Morris, Wendy, Ilker Dalgar, Van Assen, Marcel A.L.M., Tanzer, Norbert K., Collisson, Brian, Dukes, Kristin, Bialobrzeska, Olga, Durrheim, Kevin, Jen-Ho Chang, Welch, Cheryl, Orosz, Gabor, Huynh, Ho Phi, Thomae, Manuela, Friedman, Mike, Kappes, Heather, Jimenez-Leal, William, CHEN, Eva, Karabati, Serdar, Corker, Katherine S., Grahe, Jon, Pérez, Rolando, Alper, Sinan, Haines, Elizabeth L., Sundfelt, Oskar, De Vries, Marieke, Innes-Ker, Åse, Međedović, Janko, Eller, Anja, Curran, Paul G, Nerisa Dozo, Steiner, Troy G., Tanzer, Norbert, Karick, Haruna, Schmidt, Kathleen, Maitner, Angela, Davis, William E., Sowden, Walter J., Tripat Gill, Gonzalez, Roberto, Murphy, Sean, Doğulu, Canay, Wronska, Marta Katarzyna, Huntsinger, Jeffrey R., Lakens, Daniel, Graham, Jesse, Carmichael, Cheryl L, Kovacs, Carrie, IJzerman, Hans, Cemalcilar, Zeynep, O'Donnell, Susan L., Maciek Sekerdej, Maassen, Esther, Traczyk, Jakub, Tang, Andrew, Rosa, Anna Dalla, Smith-Castro, Vanessa, Tybur, Joshua M., Vega, Diego, Sobkow, Agata, Kervyn, Nicolas, Easterbrook, Matthew, Wichman, Aaron, Nosek, Brian A., Hall, Michael P., Nelson, Anthony J., Edlund, John, Williams, Lisa A., Finck, Carolyn, Myachykov, Andriy, Brandt, Mark, Heffernan, Marie E., Verschoor, Mark, Nichols, Austin Lee, Coleman, Jennifer A., Winnee Cheong, Petrovic, Boban, Inbar, Yoel, Van Der Hulst, Marije, Levitan, Carmel, Lazarevic, Lili, Galliani, Elisa Maria, Packard, Grant, McIntyre, Jason, Haigh, Matthew, Skorinko, Jeanine, Frankowska, Natalia, Szumowska, Ewa, Keller, Victor N., Podesta, Lysandra, Adams, Reginald B., Klein, Richard A., Knezevic, Goran, Saeri, Alexander K, Torres, David, Sirlopu, David Ricardo, Salomon, Erika, Anna Van 'T Veer, Tear, Morgan, Lins, Samuel, Neijenhuijs, Koen Ilja, Vianello, Michelangelo, Marotta, Satia, Street, Chris N. H., Mallett, Robyn, Kurapov, German, Pollmann, Monique, Kamiloglu, Roza, Ebersole, Charles R., Cambier, Fanny, Gomez, Angel, Smith, Michael A., Chandler, Jesse J., Kurtz, Jaime, Oikawa, Masanori, Devos, Thierry, Schönbrodt, Felix, Milfont, Taciano L., Miguel-Ángel Freyre, Rutchick, Abraham M., Axt, Jordan, Hasselman, Fred, Hicks, Joshua A., Grahek, Ivan, Cantarero, Katarzyna, and Cushman, Fiery

Abstract

Data from Many Labs 2 Replication Project

Published
2017
Full Text
View/download PDF

48. Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve

Author

van Aert, Robbie C M, Wicherts, Jelte M., van Assen, Marcel A L M, Leerstoel Buskens, and Social Networks, Solidarity and Inequality

Subjects
meta-analysis, p-uniform, p-hacking, p-curve, heterogeneity, Psychology(all)
Abstract

Because of overwhelming evidence of publication bias in psychology, techniques to correct meta-analytic estimates for such bias are greatly needed. The methodology on which the p-uniform and p-curve methods are based has great promise for providing accurate meta-analytic estimates in the presence of publication bias. However, in this article, we show that in some situations, p-curve behaves erratically, whereas p-uniform may yield implausible estimates of negative effect size. Moreover, we show that (and explain why) p-curve and p-uniform result in overestimation of effect size under moderate-to-large heterogeneity and may yield unpredictable bias when researchers employ p-hacking. We offer hands-on recommendations on applying and interpreting results of meta-analyses in general and p-uniform and p-curve in particular. Both methods as well as traditional methods are applied to a meta-analysis on the effect of weight on judgments of importance. We offer guidance for applying p-uniform or p-curve using R and a user-friendly web application for applying p-uniform.

Published
2016

49. Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Author

Wicherts, Jelte M., Veldkamp, Coosje L. S., Augusteijn, Hilde E. M., Bakker, Marjan, Van Aert, Robbie C. M., Van Assen, Marcel A. L. M., Leerstoel Buskens, Social Networks, Solidarity and Inequality, Department of Methodology and Statistics, Leerstoel Buskens, and Social Networks, Solidarity and Inequality

Subjects
experimental design, media_common.quotation_subject, Applied psychology, lcsh:BF1-990, Review, Experimental design (study designs), 050105 experimental psychology, 03 medical and health sciences, 0302 clinical medicine, Bias, Significance testing, 0502 economics and business, P-hacking, Psychology, 0501 psychology and cognitive sciences, Psychological testing, Quality (business), Statistical analysis, 050207 economics, General Psychology, media_common, 050208 finance, Psychological research, significance chasing, Degrees of freedom, 05 social sciences, Research methods education, questionable research practices, Checklist, lcsh:Psychology, p-hacking, Social psychology, 030217 neurology & neurosurgery
Abstract

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size estimates. In this review article, we present an extensive list of 34 degrees of freedom that researchers have in formulating hypotheses, and in designing, running, analyzing, and reporting of psychological research. The list can be used in research methods education, and as a checklist to assess the quality of preregistrations and to determine the potential for bias due to (arbitrary) choices in unregistered studies.

Published
2016
Full Text
View/download PDF

50. Distributions of p-values smaller than .05 in psychology

Author

Hartgerink, Chris H J, Van Aert, Robbie C M, Nuijten, Michèle B., Wicherts, Jelte M., Van Assen, Marcel A L M, Leerstoel Buskens, Social Networks, Solidarity and Inequality, Leerstoel Buskens, Social Networks, Solidarity and Inequality, and Department of Methodology and Statistics

Subjects
Data peeking, Social psychology (sociology), QRP, Science Policy, media_common.quotation_subject, Neuroscience(all), lcsh:Medicine, Psychiatry and Psychology, 050109 social psychology, Biochemistry, 050105 experimental psychology, General Biochemistry, Genetics and Molecular Biology, NHST, Personality, 0501 psychology and cognitive sciences, Caliper test, media_common, Medicine(all), Agricultural and Biological Sciences(all), Biochemistry, Genetics and Molecular Biology(all), General Neuroscience, Psychological literature, Statistics, 05 social sciences, lcsh:R, General Medicine, Data science, Test (assessment), p-values, General Agricultural and Biological Sciences, Psychology, Genetics and Molecular Biology(all), Cognitive psychology
Abstract

Previous studies provided mixed findings on pecularities inp-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence ofp-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reportedp-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculatedp-values. We found clear and direct evidence for the QRP “incorrect rounding ofp-value” (John, Loewenstein & Prelec, 2012) in all psychology journals. Finally, we also investigated monotonic excess ofp-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significantp-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling ofp-value distributions.

Published
2016
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results