Search

Your search keyword '"Montealegre, Andres"' showing total 31 results
Start Over Author "Montealegre, Andres"
31 results on '"Montealegre, Andres"'

2. Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Author

Ebersole, Charles R, Mathur, Maya B, Baranski, Erica, Bart-Plange, Diane-Jo, Buttrick, Nicholas R, Chartier, Christopher R, Corker, Katherine S, Corley, Martin, Hartshorne, Joshua K, IJzerman, Hans, Lazarević, Ljiljana B, Rabagliati, Hugh, Ropovik, Ivan, Aczel, Balazs, Aeschbach, Lena F, Andrighetto, Luca, Arnal, Jack D, Arrow, Holly, Babincak, Peter, Bakos, Bence E, Baník, Gabriel, Baskin, Ernest, Belopavlović, Radomir, Bernstein, Michael H, Białek, Michał, Bloxsom, Nicholas G, Bodroža, Bojana, Bonfiglio, Diane BV, Boucher, Leanne, Brühlmann, Florian, Brumbaugh, Claudia C, Casini, Erica, Chen, Yiling, Chiorri, Carlo, Chopik, William J, Christ, Oliver, Ciunci, Antonia M, Claypool, Heather M, Coary, Sean, Čolić, Marija V, Collins, W Matthew, Curran, Paul G, Day, Chris R, Dering, Benjamin, Dreber, Anna, Edlund, John E, Falcão, Filipe, Fedor, Anna, Feinberg, Lily, Ferguson, Ian R, Ford, Máire, Frank, Michael C, Fryberger, Emily, Garinther, Alexander, Gawryluk, Katarzyna, Ashbaugh, Kayla, Giacomantonio, Mauro, Giessner, Steffen R, Grahe, Jon E, Guadagno, Rosanna E, Hałasa, Ewa, Hancock, Peter JB, Hilliard, Rias A, Hüffmeier, Joachim, Hughes, Sean, Idzikowska, Katarzyna, Inzlicht, Michael, Jern, Alan, Jiménez-Leal, William, Johannesson, Magnus, Joy-Gaba, Jennifer A, Kauff, Mathias, Kellier, Danielle J, Kessinger, Grecia, Kidwell, Mallory C, Kimbrough, Amanda M, King, Josiah PJ, Kolb, Vanessa S, Kołodziej, Sabina, Kovacs, Marton, Krasuska, Karolina, Kraus, Sue, Krueger, Lacy E, Kuchno, Katarzyna, Lage, Caio Ambrosio, Langford, Eleanor V, Levitan, Carmel A, de Lima, Tiago Jessé Souza, Lin, Hause, Lins, Samuel, Loy, Jia E, Manfredi, Dylan, Markiewicz, Łukasz, Menon, Madhavi, Mercier, Brett, Metzger, Mitchell, Meyet, Venus, Millen, Ailsa E, Miller, Jeremy K, and Montealegre, Andres

Subjects
Psychology, Cognitive and Computational Psychology, Applied and Developmental Psychology, Behavioral and Social Science, replication, reproducibility, metascience, peer review, Registered Reports, open data, preregistered, Applied and developmental psychology, Cognitive and computational psychology
Abstract

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p

Published
2020

3. Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

Author

Landy, Justin F, Jia, Miaolei Liam, Ding, Isabel L, Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R, Gronau, Quentin F, Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M, Bauman, Christopher W, Brady, William J, Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M Brent, Hahn, Adam, Hall, Michael P, Jiménez-Leal, William, Johnson, David J, Lucas, Richard E, Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A, Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M, Van Bavel, Jay J, Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A, Schweinsberg, Martin, Urwitz, Molly, The Crowdsourcing Hypothesis Tests Collaboration, and Uhlmann, Eric L

Subjects
conceptual replications, crowdsourcing, forecasting, research robustness, scientific transparency, Crowdsourcing Hypothesis Tests Collaboration, Crowdsourcing, stimulus sampling, replication, Psychology, Cognitive Sciences, Marketing, Social Psychology
Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Published
2020

6. Scripts, IRB, etc

Author

Hartshorne, Joshua, Skorb, Lauren, Nave, Gideon, Jimenez-Leal, William, Montealegre, Andres, Schultze, Thomas, Schulz-Hardt, Stefan, Rentzsch, Katrin, Kauff, Mathias, Fedor, Anna, Ravid, Jon, Feinberg, Lily, and Manfredi, Dylan

Published
2022
Full Text
View/download PDF

7. Replications

Author

Hartshorne, Joshua, Skorb, Lauren, Nave, Gideon, Jimenez-Leal, William, Montealegre, Andres, Schultze, Thomas, Schulz-Hardt, Stefan, Rentzsch, Katrin, Kauff, Mathias, Fedor, Anna, Ravid, Jon, Feinberg, Lily, Manfredi, Dylan, Aczel, Balazs, Szaszi, Barnabas, Salamon, Janos, Muda, Rafal, Weissgerber, Sophia, and Tullett, Alexa

Published
2022
Full Text
View/download PDF

8. RPP Correspondence

Author

Hartshorne, Joshua, Skorb, Lauren, Nave, Gideon, Jimenez-Leal, William, Montealegre, Andres, Schultze, Thomas, Schulz-Hardt, Stefan, Rentzsch, Katrin, Kauff, Mathias, Fedor, Anna, and Ravid, Jon

Published
2022
Full Text
View/download PDF

10. Ejercicio ciencia abierta

Author

Montealegre, Andres, Jimenez-Leal, William, silva, luz, Camacho, Carlos, Rodríguez, Luis, Gutierrez, Diana, Poveda, Diana, Góngora, Maria, Urrego González, Sara, Suarez, Marisol, and Cardona-Betancourt, Verónica

Published
2022
Full Text
View/download PDF

13. Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008)

Author

Skorb, Lauren, primary, Aczel, Balazs, additional, Bakos, Bence E., additional, Feinberg, Lily, additional, Hałasa, Ewa, additional, Kauff, Mathias, additional, Kovacs, Marton, additional, Krasuska, Karolina, additional, Kuchno, Katarzyna, additional, Manfredi, Dylan, additional, Montealegre, Andres, additional, Pękala, Emilian, additional, Pieńkosz, Damian, additional, Ravid, Jonathan, additional, Rentzsch, Katrin, additional, Szaszi, Barnabas, additional, Schulz-Hardt, Stefan, additional, Sioma, Barbara, additional, Szecsi, Peter, additional, Szuts, Attila, additional, Szöke, Orsolya, additional, Christ, Oliver, additional, Fedor, Anna, additional, Jiménez-Leal, William, additional, Muda, Rafał, additional, Nave, Gideon, additional, Salamon, Janos, additional, Schultze, Thomas, additional, and Hartshorne, Joshua K., additional

Published
2020
Full Text
View/download PDF

15. Skorb_Hartshorne_Rev_Supplemental_Material – Supplemental material for Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008)

Author

Skorb, Lauren, Balazs Aczel, Bakos, Bence E., Feinberg, Lily, Hałasa, Ewa, Kauff, Mathias, Marton Kovacs, Krasuska, Karolina, Kuchno, Katarzyna, Manfredi, Dylan, Montealegre, Andres, Pękala, Emilian, Pieńkosz, Damian, Ravid, Jonathan, Rentzsch, Katrin, Szaszi, Barnabas, Schulz-Hardt, Stefan, Sioma, Barbara, Szecsi, Peter, Szuts, Attila, Szöke, Orsolya, Christ, Oliver, Fedor, Anna, Jiménez-Leal, William, Rafał Muda, Nave, Gideon, Janos Salamon, Schultze, Thomas, and Hartshorne, Joshua K.

Subjects
FOS: Psychology, FOS: Clinical medicine, 170199 Psychology not elsewhere classified, 110319 Psychiatry (incl. Psychotherapy)
Abstract

Supplemental material, Skorb_Hartshorne_Rev_Supplemental_Material for Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008) by Lauren Skorb, Balazs Aczel, Bence E. Bakos, Lily Feinberg, Ewa Hałasa, Mathias Kauff, Marton Kovacs, Karolina Krasuska, Katarzyna Kuchno, Dylan Manfredi, Andres Montealegre, Emilian Pękala, Damian Pieńkosz, Jonathan Ravid, Katrin Rentzsch, Barnabas Szaszi, Stefan Schulz-Hardt, Barbara Sioma, Peter Szecsi, Attila Szuts, Orsolya Szöke, Oliver Christ, Anna Fedor, William Jiménez-Leal, Rafał Muda, Gideon Nave, Janos Salamon, Thomas Schultze and Joshua K. Hartshorne in Advances in Methods and Practices in Psychological Science

Published
2020
Full Text
View/download PDF

16. Hartshorne_AMPPSOpenPracticesDisclosure-v1-0 – Supplemental material for Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008)

Author

Skorb, Lauren, Balazs Aczel, Bakos, Bence E., Feinberg, Lily, Hałasa, Ewa, Kauff, Mathias, Marton Kovacs, Krasuska, Karolina, Kuchno, Katarzyna, Manfredi, Dylan, Montealegre, Andres, Pękala, Emilian, Pieńkosz, Damian, Ravid, Jonathan, Rentzsch, Katrin, Szaszi, Barnabas, Schulz-Hardt, Stefan, Sioma, Barbara, Szecsi, Peter, Szuts, Attila, Szöke, Orsolya, Christ, Oliver, Fedor, Anna, Jiménez-Leal, William, Rafał Muda, Nave, Gideon, Janos Salamon, Schultze, Thomas, and Hartshorne, Joshua K.

Subjects
FOS: Psychology, FOS: Clinical medicine, 170199 Psychology not elsewhere classified, 110319 Psychiatry (incl. Psychotherapy)
Abstract

Supplemental material, Hartshorne_AMPPSOpenPracticesDisclosure-v1-0 for Many Labs 5: Replication of van Dijk, van Kleef, Steinel, and van Beest (2008) by Lauren Skorb, Balazs Aczel, Bence E. Bakos, Lily Feinberg, Ewa Hałasa, Mathias Kauff, Marton Kovacs, Karolina Krasuska, Katarzyna Kuchno, Dylan Manfredi, Andres Montealegre, Emilian Pękala, Damian Pieńkosz, Jonathan Ravid, Katrin Rentzsch, Barnabas Szaszi, Stefan Schulz-Hardt, Barbara Sioma, Peter Szecsi, Attila Szuts, Orsolya Szöke, Oliver Christ, Anna Fedor, William Jiménez-Leal, Rafał Muda, Gideon Nave, Janos Salamon, Thomas Schultze and Joshua K. Hartshorne in Advances in Methods and Practices in Psychological Science

Published
2020
Full Text
View/download PDF

17. ML5_Overview_Open_Practices_Disclosure_Rev – Supplemental material for Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Author

Ebersole, Charles R., Mathur, Maya B., Baranski, Erica, Diane-Jo Bart-Plange, Buttrick, Nicholas R., Chartier, Christopher R., Corker, Katherine S., Corley, Martin, Hartshorne, Joshua K., IJzerman, Hans, Lazarević, Ljiljana B., Rabagliati, Hugh, Ropovik, Ivan, Balazs Aczel, Aeschbach, Lena F., Andrighetto, Luca, Arnal, Jack D., Arrow, Holly, Babincak, Peter, Bakos, Bence E., Baník, Gabriel, Baskin, Ernest, Belopavlović, Radomir, Bernstein, Michael H., Michał Białek, Bloxsom, Nicholas G., Bodroža, Bojana, Bonfiglio, Diane B. V., Boucher, Leanne, Brühlmann, Florian, Brumbaugh, Claudia C., Casini, Erica, Yiling Chen, Chiorri, Carlo, Chopik, William J., Christ, Oliver, Ciunci, Antonia M., Claypool, Heather M., Coary, Sean, Čolić, Marija V., W. Matthew Collins, Curran, Paul G., Day, Chris R., Dering, Benjamin, Dreber, Anna, Edlund, John E., Falcão, Filipe, Fedor, Anna, Feinberg, Lily, Ferguson, Ian R., Ford, Máire, Frank, Michael C., Fryberger, Emily, Garinther, Alexander, Gawryluk, Katarzyna, Ashbaugh, Kayla, Giacomantonio, Mauro, Giessner, Steffen R., Grahe, Jon E., Guadagno, Rosanna E., Hałasa, Ewa, Hancock, Peter J. B., Rias A. Hilliard, Hüffmeier, Joachim, Hughes, Sean, Idzikowska, Katarzyna, Inzlicht, Michael, Jern, Alan, Jiménez-Leal, William, Johannesson, Magnus, Joy-Gaba, Jennifer A., Kauff, Mathias, Kellier, Danielle J., Grecia Kessinger, Kidwell, Mallory C., Kimbrough, Amanda M., King, Josiah P. J., Kolb, Vanessa S., Kołodziej, Sabina, Marton Kovacs, Krasuska, Karolina, Kraus, Sue, Krueger, Lacy E., Kuchno, Katarzyna, Lage, Caio Ambrosio, Langford, Eleanor V., Levitan, Carmel A., Lima, Tiago Jessé Souza De, Hause Lin, Lins, Samuel, Loy, Jia E., Manfredi, Dylan, Łukasz Markiewicz, Menon, Madhavi, Mercier, Brett, Metzger, Mitchell, Meyet, Venus, Millen, Ailsa E., Miller, Jeremy K., Montealegre, Andres, Moore, Don A., Rafał Muda, Nave, Gideon, Nichols, Austin Lee, Novak, Sarah A., Nunnally, Christian, Orlić, Ana, Palinkas, Anna, Panno, Angelo, Parks, Kimberly P., Pedović, Ivana, Pękala, Emilian, Penner, Matthew R., Pessers, Sebastiaan, Petrović, Boban, Pfeiffer, Thomas, Pieńkosz, Damian, Preti, Emanuele, Purić, Danka, Ramos, Tiago, Ravid, Jonathan, Razza, Timothy S., Rentzsch, Katrin, Richetin, Juliette, Rife, Sean C., Rosa, Anna Dalla, Kaylis Hase Rudy, Janos Salamon, Saunders, Blair, Przemysław Sawicki, Schmidt, Kathleen, Schuepfer, Kurt, Schultze, Thomas, Schulz-Hardt, Stefan, Schütz, Astrid, Shabazian, Ani N., Shubella, Rachel L., Siegel, Adam, Rúben Silva, Sioma, Barbara, Skorb, Lauren, Souza, Luana Elayne Cunha De, Steegen, Sara, L. A. R. Stein, R. Weylin Sternglanz, Stojilović, Darko, Storage, Daniel, Sullivan, Gavin Brent, Szaszi, Barnabas, Szecsi, Peter, Szöke, Orsolya, Szuts, Attila, Thomae, Manuela, Tidwell, Natasha D., Tocco, Carly, Ann-Kathrin Torka, Tuerlinckx, Francis, Vanpaemel, Wolf, Vaughn, Leigh Ann, Vianello, Michelangelo, Viganola, Domenico, Vlachou, Maria, Walker, Ryan J., Weissgerber, Sophia C., Wichman, Aaron L., Wiggins, Bradford J., Wolf, Daniel, Wood, Michael J., Zealley, David, Žeželj, Iris, Zrubka, Mark, and Nosek, Brian A.

Subjects
FOS: Psychology, FOS: Clinical medicine, 170199 Psychology not elsewhere classified, 110319 Psychiatry (incl. Psychotherapy)
Abstract

Supplemental material, ML5_Overview_Open_Practices_Disclosure_Rev for Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability by Charles R. Ebersole, Maya B. Mathur, Erica Baranski, Diane-Jo Bart-Plange, Nicholas R. Buttrick, Christopher R. Chartier, Katherine S. Corker, Martin Corley, Joshua K. Hartshorne, Hans IJzerman, Ljiljana B. Lazarević, Hugh Rabagliati, Ivan Ropovik, Balazs Aczel, Lena F. Aeschbach, Luca Andrighetto, Jack D. Arnal, Holly Arrow, Peter Babincak, Bence E. Bakos, Gabriel Baník, Ernest Baskin, Radomir Belopavlović, Michael H. Bernstein, Michał Białek, Nicholas G. Bloxsom, Bojana Bodroža, Diane B. V. Bonfiglio, Leanne Boucher, Florian Brühlmann, Claudia C. Brumbaugh, Erica Casini, Yiling Chen, Carlo Chiorri, William J. Chopik, Oliver Christ, Antonia M. Ciunci, Heather M. Claypool, Sean Coary, Marija V. Čolić, W. Matthew Collins, Paul G. Curran, Chris R. Day, Benjamin Dering, Anna Dreber, John E. Edlund, Filipe Falcão, Anna Fedor, Lily Feinberg, Ian R. Ferguson, Máire Ford, Michael C. Frank, Emily Fryberger, Alexander Garinther, Katarzyna Gawryluk, Kayla Ashbaugh, Mauro Giacomantonio, Steffen R. Giessner, Jon E. Grahe, Rosanna E. Guadagno, Ewa Hałasa, Peter J. B. Hancock, Rias A. Hilliard, Joachim Hüffmeier, Sean Hughes, Katarzyna Idzikowska, Michael Inzlicht, Alan Jern, William Jiménez-Leal, Magnus Johannesson, Jennifer A. Joy-Gaba, Mathias Kauff, Danielle J. Kellier, Grecia Kessinger, Mallory C. Kidwell, Amanda M. Kimbrough, Josiah P. J. King, Vanessa S. Kolb, Sabina Kołodziej, Marton Kovacs, Karolina Krasuska, Sue Kraus, Lacy E. Krueger, Katarzyna Kuchno, Caio Ambrosio Lage, Eleanor V. Langford, Carmel A. Levitan, Tiago Jessé Souza de Lima, Hause Lin, Samuel Lins, Jia E. Loy, Dylan Manfredi, Łukasz Markiewicz, Madhavi Menon, Brett Mercier, Mitchell Metzger, Venus Meyet, Ailsa E. Millen, Jeremy K. Miller, Andres Montealegre, Don A. Moore, Rafał Muda, Gideon Nave, Austin Lee Nichols, Sarah A. Novak, Christian Nunnally, Ana Orlić, Anna Palinkas, Angelo Panno, Kimberly P. Parks, Ivana Pedović, Emilian Pękala, Matthew R. Penner, Sebastiaan Pessers, Boban Petrović, Thomas Pfeiffer, Damian Pieńkosz, Emanuele Preti, Danka Purić, Tiago Ramos, Jonathan Ravid, Timothy S. Razza, Katrin Rentzsch, Juliette Richetin, Sean C. Rife, Anna Dalla Rosa, Kaylis Hase Rudy, Janos Salamon, Blair Saunders, Przemysław Sawicki, Kathleen Schmidt, Kurt Schuepfer, Thomas Schultze, Stefan Schulz-Hardt, Astrid Schütz, Ani N. Shabazian, Rachel L. Shubella, Adam Siegel, Rúben Silva, Barbara Sioma, Lauren Skorb, Luana Elayne Cunha de Souza, Sara Steegen, L. A. R. Stein, R. Weylin Sternglanz, Darko Stojilović, Daniel Storage, Gavin Brent Sullivan, Barnabas Szaszi, Peter Szecsi, Orsolya Szöke, Attila Szuts, Manuela Thomae, Natasha D. Tidwell, Carly Tocco, Ann-Kathrin Torka, Francis Tuerlinckx, Wolf Vanpaemel, Leigh Ann Vaughn, Michelangelo Vianello, Domenico Viganola, Maria Vlachou, Ryan J. Walker, Sophia C. Weissgerber, Aaron L. Wichman, Bradford J. Wiggins, Daniel Wolf, Michael J. Wood, David Zealley, Iris Žeželj, Mark Zrubka and Brian A. Nosek in Advances in Methods and Practices in Psychological Science

Published
2020
Full Text
View/download PDF

18. Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability (results-blind manuscript)

Author

Pessers, Sebastiaan, Petrovic , Boban, Nichols , Austin, Metzger, Mitchell, Manfredi , Dylan, Mercier , Brett, Montealegre , Andres, Kuchno, Katarzyna, Millen , Ailsa, Levitan , Carmel, Nave , Gideon, Lage, Caio, Pedović, Ivana, Krueger , Lacy, Krasuska, Karolina, Kraus, Sue, Kovacs , Marton, Inzlicht , Michael, Hałasa, Ewa, Idzikowska, Katarzyna, Kołodziej, Sabina, Kimbrough, Amanda, Kauff , Mathias, Hancock , Peter, Jern, Alan, Kellier, Danielle, Leal, Wlliam, Hüffmeier, Joachim, Hughes , Sean, King , Josiah, Kidwell , Mallory, Johannesson , Magnus, Joy-Gaba , Jennifer, PęKala, Emilian, Penner, Matthew, Moore , Don, Menon, Madhavi, Langford, Eleanor, Panno, Angelo, Lima , Tiago, Lins , Samuel, Parks, Kimberly, Miller, Jeremy, Novak, Sarah, Markiewicz, Lukasz, Loy , J, Orlic, Ana, Lin , Hause, Guadagno, Rosanna, Grahe , Jon, Giessner , Steffen, Giacomantonio , Mauro, Gawryluk, Katarzyna, Garinther, Alexander, Fryberger, Emily, Frank , Michael, Ford, Máire, Ferguson, Ian, Feinberg, Lily, Fedor , Anna, Falcão, Filipe, Edlund , John, Dreber , Anna, Dering , Benjamin, Day, Chris, Curran , Paul, Collins, W., Čolić , Marija, Coary, Sean, Claypool, Heather, Christ , Oliver, Chopik , William, Chiorri, Carlo, Chen, Yiling, Casini, Erica, Brumbaugh , Claudia, Brühlmann, Florian, Boucher, Leanne, Bonfiglio, Diane, Bodroža, Bojana, Bialek , Michal, Bloxsom , Nicholas, Bernstein, Michael, Belopavlović, Radomir, Baskin, Ernest, Baník , Gabriel, Bakos, Bence, Babincak , Peter, Arrow, Holly, Arnal , Jack, Andrighetto, Luca, Aeschbach, Lena, Ropovik , Ivan, Lazarevic , Ljiljana, Aczel , Balazs, Ijzerman , Hans, Hartshorne , Joshua, Corker , Katherine, Corley , Martin, Buttrick , Nick, Baranski , Erica, Bart-Plange, Diane-Jo, Mathur , Maya, Chartier , Christopher, Rabagliati , Hugh, Campbell , Lorne, Ebersole , Charles, Salamon, Janos, Rife , Sean, Stojilović , Darko, Szecsi , Peter, Schulz-Hardt , Stefan, Weissgerber , Sophia, Tocco, Carly, Wichman , Aaron, Zrubka , Mark, Sullivan, Gavin, Schuepfer, Kurt, Rosa , Anna, Schultze , Thomas, Storage , Daniel, Schütz , Astrid, Stein, Lar, Schmidt , Kathleen, Razza, Timothy, Wolf, Daniel, Vaughn , Leigh, Saunders , Blair, Richetin , Juliette, Jimenez-Leal , William, Silva, Rúben, Sioma, Barbara, Viganola, Domenico, Vlachou, Maria, Szaszi , Barnabas, Rentzsch, Katrin, Vianello , Michelangelo, Szoke, Orsolya, Nosek , Brian, Ravid, Jon, Torka, Ann-Kathrin, Walker, Ryan, Skorb , Lauren, Sternglanz, R., Szuts , Attila, Tidwell, Natasha, Thomae , Manuela, De Souza, Luana, Zealley, David, Tuerlinckx , Francis, Steegen, Sara, Shabazian, Ani, Zezelj , Iris, Preti , Emanuele, Purić , Danka, Ramos, Tiago, Pieńkosz , Damian, Pfeiffer, Thomas, Wood, Michael, Sawicki, Przemyslaw, Wiggins , Bradford, and Vanpaemel , Wolf

Subjects
PsyArXiv|Social and Behavioral Sciences, bepress|Social and Behavioral Sciences, bepress|Social and Behavioral Sciences|Psychology|Personality and Social Contexts, PsyArXiv|Social and Behavioral Sciences|Social and Personality Psychology
Abstract

Replication efforts in psychological science sometimes fail to replicate prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the replication protocol rather than a challenge to the original finding. Formal pre-data collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replications from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) in which the original authors had expressed concerns about the replication designs before data collection and only one of which was “statistically significant” (p < .05). Commenters on RP:P suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these failed to replicate (Gilbert et al., 2016). We revised the replication protocols and received formal peer review prior to conducting new replications. We administered the RP:P and Revised replication protocols in multiple laboratories (Median number of laboratories per original study = XX; Range XX to YY; Median total sample = XX; Range XX to YY) for high-powered tests of each original finding with both protocols. Overall, XX of 10 RP:P protocols and XX of 10 Revised protocols showed significant evidence in the same direction as the original finding (p < .05), compared to an expected XX. The median effect size was [larger/smaller/similar] for Revised protocols (ES = .XX) compared to RP:P protocols (ES = .XX), and [larger/smaller/similar] compared to the original studies (ES = .XX) and [larger/smaller/similar] compared to the original RP:P replications (ES = .XX). Overall, Revised protocols produced [much larger/somewhat larger/similar] effect sizes compared to RP:P protocols (ES = .XX). We also elicited peer beliefs about the replications through prediction markets and surveys of a group of researchers in psychology. The peer researchers predicted that the Revised protocols would [decrease/not affect/increase] the replication rate, [consistent with/not consistent with] the observed replication results. The results suggest that the lack of replicability of these findings observed in RP:P was [partly/completely/not] due to discrepancies in the RP:P protocols that could be resolved with expert peer review.

Published
2019

19. Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

Author

Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M. Brent, Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, The Crowdsourcing Hypothesis Tests Collaboration, Uhlmann, Eric L., Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M. Brent, Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, The Crowdsourcing Hypothesis Tests Collaboration, and Uhlmann, Eric L.

Published
2020

20. Crowdsourcing hypothesis tests: Making transparent how design choices shape research results

Author

Landy, J, Jia, M, Ding, I, Viganola, D, Tierney, W, Dreber, A, Johannesson, M, Pfeiffer, T, Ebersole, C, Gronau, Q, Ly, A, van den Bergh, D, Marsman, M, Derks, K, Wagenmakers, E, Proctor, A, Bartels, D, Bauman, C, Brady, W, Cheung, F, Cimpian, A, Dohle, S, Donnellan, M, Hahn, A, Hall, M, Jiménez-Leal, W, Johnson, D, Lucas, R, Monin, B, Montealegre, A, Mullen, E, Pang, J, Ray, J, Reinero, D, Reynolds, J, Sowden, W, Storage, D, Su, R, Tworek, C, Van Bavel, J, Walco, D, Wills, J, Xu, X, Yam, K, Yang, X, Cunningham, W, Schweinsberg, M, Urwitz, M, The Crowdsourcing Hypothesis Tests, C, Uhlmann, E, Mari, S, Landy, Justin F, Jia, Miaolei Liam, Ding, Isabel L, Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R, Gronau, Quentin F, Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M, Bauman, Christopher W, Brady, William J, Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M Brent, Hahn, Adam, Hall, Michael P, Jiménez-Leal, William, Johnson, David J, Lucas, Richard E, Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A, Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M, Van Bavel, Jay J, Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A, Schweinsberg, Martin, Urwitz, Molly, The Crowdsourcing Hypothesis Tests Collaboration, Uhlmann, Eric L, Landy, J, Jia, M, Ding, I, Viganola, D, Tierney, W, Dreber, A, Johannesson, M, Pfeiffer, T, Ebersole, C, Gronau, Q, Ly, A, van den Bergh, D, Marsman, M, Derks, K, Wagenmakers, E, Proctor, A, Bartels, D, Bauman, C, Brady, W, Cheung, F, Cimpian, A, Dohle, S, Donnellan, M, Hahn, A, Hall, M, Jiménez-Leal, W, Johnson, D, Lucas, R, Monin, B, Montealegre, A, Mullen, E, Pang, J, Ray, J, Reinero, D, Reynolds, J, Sowden, W, Storage, D, Su, R, Tworek, C, Van Bavel, J, Walco, D, Wills, J, Xu, X, Yam, K, Yang, X, Cunningham, W, Schweinsberg, M, Urwitz, M, The Crowdsourcing Hypothesis Tests, C, Uhlmann, E, Mari, S, Landy, Justin F, Jia, Miaolei Liam, Ding, Isabel L, Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R, Gronau, Quentin F, Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M, Bauman, Christopher W, Brady, William J, Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M Brent, Hahn, Adam, Hall, Michael P, Jiménez-Leal, William, Johnson, David J, Lucas, Richard E, Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A, Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M, Van Bavel, Jay J, Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A, Schweinsberg, Martin, Urwitz, Molly, The Crowdsourcing Hypothesis Tests Collaboration, and Uhlmann, Eric L

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

Published
2020

21. Crowd-sourcing Hypothesis Tests: Making Transparent How Design Choices Shape Research Results

Author

Landy, Justin, primary, Jia, Miaolei, additional, Ding, Isabel, additional, Viganola, Domenico, additional, Tierney, Warren, additional, Dreber, Anna, additional, Johanneson, Magnus, additional, Pfeiffer, Thomas, additional, Ebersole, Charles, additional, Gronau, Quentin, additional, Ly, Alexander, additional, van den Bergh, Don, additional, Marsman, Maarten, additional, Derks, Koen, additional, Wagenmakers, Eric-Jan, additional, Proctor, Andrew, additional, Bartels, Daniel M., additional, Bauman, Christopher W., additional, Brady, William J., additional, Cheung, Felix, additional, Cimpian, Andrei, additional, Dohle, Simone, additional, Donnellan, M. Brent, additional, Hahn, Adam, additional, Hall, Michael P., additional, Jiménez-Leal, William, additional, Johnson, David J., additional, Lucas, Richard E., additional, Monin, Benoit, additional, Montealegre, Andres, additional, Mullen, Elizabeth, additional, Pang, Jun, additional, Ray, Jennifer, additional, Reinero, Diego A., additional, Reynolds, Jesse, additional, Sowden, Walter, additional, Storage, Daniel, additional, Su, Runkun, additional, Tworek, Christina M., additional, Van Bavel, Jay, additional, Walco, Daniel, additional, Wills, Julian, additional, XU, Xiaobing, additional, Yam, Kai Chi, additional, Yang, Xiaoyu, additional, Cunningham, William A., additional, Schweinsberg, Martin, additional, Urwitz, Molly, additional, and Uhlmann, Eric Luis, additional

Published
2020
Full Text
View/download PDF

22. Many Labs 5: Testing pre-data collection peer review as an intervention to increase replicability

Author

Ebersole, Charles R., primary, Mathur, Maya B, additional, Baranski, Erica, additional, Bart-Plange, Diane-Jo, additional, Buttrick, Nick, additional, Chartier, Christopher R., additional, Corker, Katherine S., additional, Corley, Martin, additional, Hartshorne, Joshua K., additional, IJzerman, Hans, additional, Lazarevic, Ljiljana B., additional, Rabagliati, Hugh, additional, Ropovik, Ivan, additional, Aczel, Balazs, additional, Aeschbach, Lena Fanya, additional, Andrighetto, Luca, additional, Arnal, Jack Dennis, additional, Arrow, Holly, additional, Babincak, Peter, additional, Bakos, Bence Endre, additional, Baník, Gabriel, additional, Baskin, Ernest, additional, Belopavlović, Radomir, additional, Bernstein, Michael, additional, Bialek, Michal, additional, Bloxsom, Nicholas, additional, Bodroža, Bojana, additional, Bonfiglio, Diane B. V., additional, Boucher, Leanne, additional, Brühlmann, Florian, additional, Brumbaugh, Claudia Chloe, additional, Casini, Erica, additional, Chen, Yiling, additional, Chiorri, Carlo, additional, Chopik, William J., additional, Christ, Oliver, additional, Claypool, Heather M., additional, coary, Sean, additional, Čolić, Marija V., additional, Collins, W. Matthew, additional, Curran, Paul G, additional, Day, Chris, additional, Dering, Benjamin, additional, Dreber, Anna, additional, Edlund, John, additional, Falcão, Filipe, additional, Fedor, Anna, additional, Feinberg, Lily, additional, Ferguson, Ian, additional, Ford, Máire, additional, Frank, Michael C., additional, Fryberger, Emily, additional, Garinther, Alexander, additional, Gawryluk, Katarzyna, additional, Giacomantonio, Mauro, additional, Giessner, Steffen Robert, additional, Grahe, Jon E., additional, Guadagno, Rosanna Elizabeth, additional, Hałasa, Ewa, additional, Hancock, Peter, additional, Hüffmeier, Joachim, additional, Hughes, Sean, additional, Idzikowska, Katarzyna, additional, Inzlicht, Michael, additional, Jern, Alan, additional, Jimenez-Leal, William, additional, Johannesson, Magnus, additional, Joy-Gaba, Jennifer Alana, additional, Kauff, Mathias, additional, Kellier, Danielle, additional, Kidwell, Mallory, additional, Kimbrough, Amanda, additional, King, Josiah, additional, Kołodziej, Sabina, additional, Kovacs, Marton, additional, Krasuska, Karolina, additional, Kraus, Sue, additional, Krueger, Lacy Elise, additional, Kuchno, Katarzyna, additional, Lage, Caio Ambrosio, additional, Langford, Eleanor V., additional, Levitan, Carmel, additional, Lima, Tiago Jessé Souza, additional, Lin, Hause, additional, Lins, Samuel, additional, Loy, J E, additional, Manfredi, Dylan, additional, Markiewicz, Lukasz, additional, Menon, Madhavi, additional, Mercier, Brett, additional, Metzger, Mitchell, additional, Millen, Ailsa E, additional, Miller, Jeremy K., additional, Montealegre, Andres, additional, Moore, Don A, additional, Nave, Gideon, additional, Nichols, Austin Lee, additional, Novak, Sarah Ann, additional, Orlic, Ana, additional, Panno, Angelo, additional, Parks, Kimberly P., additional, Pedović, Ivana, additional, Pękala, Emilian, additional, Penner, Matthew R., additional, Pessers, Sebastiaan, additional, Petrovic, Boban, additional, Pfeiffer, Thomas, additional, Pieńkosz, Damian, additional, Preti, Emanuele, additional, Purić, Danka, additional, Ramos, Tiago Silva, additional, Ravid, Jon, additional, Razza, Timothy, additional, Rentzsch, Katrin, additional, Richetin, Juliette, additional, Rife, Sean Chandler, additional, Dalla Rosa, Anna, additional, Salamon, Janos, additional, Saunders, Blair, additional, Sawicki, Przemyslaw, additional, Schmidt, Kathleen, additional, Schuepfer, Kurt, additional, Schultze, Thomas, additional, Schulz-Hardt, Stefan, additional, Schütz, Astrid, additional, Shabazian, Ani, additional, Silva, Rúben Filipe Lopes, additional, Sioma, Barbara, additional, Skorb, Lauren, additional, Souza, Luana Elayne Cunha, additional, steegen, sara, additional, Stein, LAR, additional, Sternglanz, R. Weylin, additional, Stojilović, Darko, additional, Storage, Daniel, additional, Sullivan, Gavin Brent, additional, Szaszi, Barnabas, additional, Szecsi, Peter, additional, Szoke, Orsolya, additional, Szuts, Attila, additional, Thomae, Manuela, additional, Tidwell, Natasha Davis, additional, tocco, Carly, additional, Torka, Ann-Kathrin, additional, tuerlinckx, francis, additional, vanpaemel, wolf, additional, Vaughn, Leigh Ann, additional, Vianello, Michelangelo, additional, Viganola, Domenico, additional, Vlachou, Maria, additional, Walker, Ryan J., additional, Weissgerber, Sophia Christin, additional, Wichman, Aaron Lee, additional, Wiggins, Bradford Jay, additional, Wolf, Daniel, additional, Wood, Michael James, additional, Zealley, David A., additional, Zezelj, Iris, additional, Zrubka, Mark, additional, and Nosek, Brian A., additional

Published
2019
Full Text
View/download PDF

23. Many Labs 5: Replication of Van Dijk, Van Kleef, Steinel, &amp; Van Beest (2008). A social functional approach to emotions in bargaining

Author

Skorb, Lauren, primary, Aczel, Balazs, additional, Bakos, Bence Endre, additional, Feinberg, Lily, additional, Hałasa, Ewa, additional, Kauff, Mathias, additional, Kovacs, Marton, additional, Krasuska, Karolina, additional, Kuchno, Katarzyna, additional, Manfredi, Dylan, additional, Montealegre, Andres, additional, Pękala, Emilian, additional, Pieńkosz, Damian, additional, Ravid, Jon, additional, Rentzsch, Katrin, additional, Szaszi, Barnabas, additional, Schulz-Hardt, Stefan, additional, Sioma, Barbara, additional, Szecsi, Peter, additional, Szuts, Attila, additional, Szoke, Orsolya, additional, Christ, Oliver, additional, Fedor, Anna, additional, Jimenez-Leal, William, additional, Muda, Rafal, additional, Nave, Gideon, additional, Salamon, Janos, additional, Schultze, Thomas, additional, and Hartshorne, Joshua K., additional

Published
2019
Full Text
View/download PDF

28. Significados elaborados en la construcción del contrato psicológico en el grupo administrativo de Vicpar S.A., sede Bogotá, Colombia

Author

Buitrago Arévalo, Luz Esperanza / Asesora, Cardona Flórez, Sandra Patricia, Hoyos Cortes, Felix Alexander, Murillo Romero, Rainer Leonardo, Riaño Montealegre, Andres Felipe, Buitrago Arévalo, Luz Esperanza / Asesora, Cardona Flórez, Sandra Patricia, Hoyos Cortes, Felix Alexander, Murillo Romero, Rainer Leonardo, and Riaño Montealegre, Andres Felipe

Abstract

La presente investigación tiene como objetivo fundamental describir los significados asociados a la construcción del contrato psicológico en la división administrativa de una empresa privada colombiana a partir de la relación laboral establecida con la organización. Para este efecto se realizaron seis escenarios conversacionales, llevando a cabo un análisis de contenido de tipo categorial a través de matrices reconociendo las expectativas y compromisos bidireccionales en torno a la relación laboral. Los resultados de la investigación demuestran que el grupo establece tanto contratos transaccionales como relacionales a partir del tipo de vinculación laboral (formal o por recomendación). Las categorías emergentes en torno a la significación del contrato se describen en términos de capacitación y desarrollo, beneficios, comunicación, compensación económica, percepción actual y futura de la organización, This research aims to describe key meanings associated with the construction of the psychological contract in the administrative division of a Colombian state from the settled up working relationship with the organization. To do that there were performed six conversational scenarios, doing a categorical content analysis through matrices recognizing bidirectional expectations and commitments regarding the employment relationship. The research results show that the group provides both transactional and relational contracts from the type of employment relationship (formal or recommendation). The categories emerging about the significance of the contract are described in terms of training and development, benefits, communication, compensation, current and future perception of the organization

Published
2013

29. Crowdsourcing hypothesis tests: making transparent how design choices shape research results

Author

Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maaten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, Brent M., Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoit, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Chi Kai, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, Uhlmann, Eric Luis, Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maaten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, Brent M., Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoit, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Chi Kai, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, and Uhlmann, Eric Luis

Abstract

peer-reviewed, To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

30. Crowdsourcing hypothesis tests: making transparent how design choices shape research results

Author

Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maaten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, Brent M., Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoit, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Chi Kai, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, Uhlmann, Eric Luis, Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maaten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, Brent M., Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoit, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Chi Kai, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, and Uhlmann, Eric Luis

Abstract

peer-reviewed, To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

31. Crowdsourcing hypothesis tests: making transparent how design choices shape research results

Author

Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maaten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, Brent M., Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoit, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Chi Kai, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, Uhlmann, Eric Luis, Landy, Justin F., Jia, Miaolei (Liam), Ding, Isabel L., Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maaten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, Brent M., Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoit, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Chi Kai, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly, and Uhlmann, Eric Luis

Abstract

peer-reviewed, To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

Catalog

Books, media, physical & digital resources
See catalog results