Search

The field of traditional, complementary, and integrative medicine (TCIM) has garnered increasing attention due to its holistic approach to health and well-being. While the quantity of published research about TCIM has increased exponentially, critics have argued that the field faces challenges related to methodological rigour, reproducibility, and overall quality. This article proposes meta-research as one approach to evaluating and improving the quality of TCIM research. Meta-research, also known as research about research, can be defined as “the study of research itself: its methods, reporting, reproducibility, evaluation, and incentives”. By systematically evaluating methodological rigour, identifying biases, and promoting transparency, meta-research can enhance the reliability and credibility of TCIM research. Specific topics of interest that are discussed in this article include the following: 1) study design and research methodology, 2) reporting of research, 3) research ethics, integrity, and misconduct, 4) replicability and reproducibility, 5) peer review and journal editorial practices, 6) research funding: grants and awards, and 7) hiring, promotion, and tenure. For each topic, we provide case examples to illustrate meta-research applications in TCIM. We argue that meta-research initiatives can contribute to maintaining public trust, safeguarding research integrity, and advancing evidence based TCIM practice, while challenges include navigating methodological complexities, biases, and disparities in funding and academic recognition. Future directions involve tailored research methodologies, interdisciplinary collaboration, policy implications, and capacity building in meta-research.

Science and knowledge are studied by researchers across many disciplines, examining how they are developed, what their current boundaries are and how we can advance them. By integrating evidence across disparate disciplines, the holistic field of science of science can address these foundational questions. This field illustrates how science is shaped by many interconnected factors: the cognitive processes of scientists, the historical evolution of science, economic incentives, institutional influences, computational approaches, statistical, mathematical and instrumental foundations of scientific inference, scientometric measures, philosophical and ethical dimensions of scientific concepts, among other influences. Achieving a comprehensive overview of a multifaceted field like the science of science requires pulling together evidence from the many sub-fields studying science across the natural and social sciences and humanities. This enables developing an interdisciplinary perspective of scientific practice, a more holistic understanding of scientific processes and outcomes, and more nuanced perspectives to how scientific research is conducted, influenced and evolves. It enables leveraging the strengths of various disciplines to create a holistic view of the foundations of science. Different researchers study science from their own disciplinary perspective and use their own methods, and there is a large divide between quantitative and qualitative researchers as they commonly do not read or cite research using other methodological approaches. A broader, synthesizing paper employing a qualitative approach can however help provide a bridge between disciplines by pulling together aspects of science (economic, scientometric, psychological, philosophical etc.). Such an approach enables identifying, across the range of fields, the powerful role of our scientific methods and instruments in shaping most aspects of our knowledge and science, whereas economic, social and historical influences help shape what knowledge we pursue. A unifying theory is then outlined for science of science – the new-methods-drive-science theory.

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

We outline essential considerations for any study of partial randomisation of research funding, and consider scenarios in which randomised controlled trials (RCTs) would be feasible and appropriate. We highlight the interdependence of target outcomes, sample availability and statistical power for determining the cost and feasibility of a trial. For many choices of target outcome, RCTs may be less practical and more expensive than they at first appear (in large part due to issues pertaining to sample size and statistical power). As such, we briefly discuss alternatives to RCTs. It is worth noting that many of the considerations relevant to experiments on partial randomisation may also apply to other potential experiments on funding processes (as described in The Experimental Research Funder’s Handbook. RoRI, June 2022).

Metascience scholars have long been concerned with tracking the use of rhetorical language in scientific discourse, oftentimes to analyze the legitimacy and validity of scientific claim-making. Psychology, however, has only recently become the explicit target of such metascientific scholarship, much of which has been in response to the recent crises surrounding replicability of quantitative research findings and questionable research practices. The focus of this paper is on the rhetoric of psychological measurement and validity scholarship, in both the theoretical and methodological and empirical literatures. We examine various discourse practices in published psychological measurement and validity literature, including: (a) clear instances of rhetoric (i.e., persuasion or performance); (b) common or rote expressions and tropes (e.g., perfunctory claims or declarations); (c) metaphors and other “literary” styles; and (d) ambiguous, confusing, or unjustifiable claims. The methodological approach we use is informed by a combination of conceptual analysis and exploratory grounded theory, the latter of which we used to identify relevant themes within the published psychological discourse. Examples of both constructive and useful or misleading and potentially harmful discourse practices will be given. Our objectives are both to contribute to the critical methodological literature on psychological measurement and connect metascience in psychology to broader interdisciplinary examinations of science discourse.

The reality that volumes of published biomedical research are not reproducible is an increasingly recognized problem. Spurious results reduce trustworthiness of reported science, increasing research waste. While science should be self-correcting from a philosophical perspective, that in insolation yields no information on efforts required to nullify suspect findings or factors shaping how quickly science may be corrected. There is also a paucity of information on how perverse incentives in the publishing ecosystem favouring novel positive findings over null results shape the ability of published science to self-correct. Knowledge of factors shaping self-correction of science remain obscure, limiting our ability to mitigate harms. This modelling study introduces a simple model to capture dynamics of the publication ecosystem, exploring factors influencing research waste, trustworthiness, corrective effort and time to correction. Results from this work indicate that research waste and corrective effort are highly dependent on field-specific false positive rates and time delays to corrective results to spurious findings are propagated. The model also suggests conditions under which biomedical science is self-correcting and those under which publication of correctives alone cannot stem propagation of untrustworthy results. Finally, this work models a variety of potential mitigation strategies, including researcher- and publisher-driven interventions.

Open science is an increasingly important topic for research, politics and funding agencies. However, the discourse on open science is heavily influenced by certain research fields and paradigms, leading to the risk of generalizing what counts as openness to other research fields, regardless of its applicability. In our paper, we provide evidence that researchers perceive different profiles in the potential to apply open science practices to their projects, making a one-size-fits-all approach unsuitable. In a pilot study, we first systematized the breadth of open science practices. The subsequent survey study examined the perceived applicability of 13 open science practices across completed research projects in a broad variety of research disciplines. We were able to identify four different profiles in the perceived applicability of open science practices. For researchers conducting qualitative-empirical research projects, comprehensively implementing the breadth of open science practices is tendentially not feasible. Further, research projects from some disciplines tended to fit a profile with little opportunity for public participation. Yet, disciplines and research paradigms appear not to be the key factors in predicting the perceived applicability of open science practices. Our findings underscore the case for considering project-related conditions when implementing open science practices. This has implications for the establishment of policies, guidelines and standards concerning open science.

In biomedical science, it is a reality that many published results do not withstand deeper investigation, and there is growing concern over a replicability crisis in science. Recently, Ellipse of Insignificance (EOI) analysis was introduced as a tool to allow researchers to gauge the robustness of reported results in dichotomous outcome design trials, giving precise deterministic values for the degree of miscoding between events and non-events tolerable simultaneously in both control and experimental arms (Grimes, 2022). While this is useful for situations where potential miscoding might transpire, it does not account for situations where apparently significant findings might result from accidental or deliberate data redaction in either the control or experimental arms of an experiment, or from missing data or systematic redaction. To address these scenarios, we introduce Region of Attainable Redaction (ROAR), a tool that extends EOI analysis to account for situations of potential data redaction. This produces a bounded cubic curve rather than an ellipse, and we outline how this can be used to identify potential redaction through an approach analogous to EOI. Applications are illustrated, and source code, including a web-based implementation that performs EOI and ROAR analysis in tandem for dichotomous outcome trials is provided.

How do prior beliefs affect the interpretation of scientific results? I discuss a hypothetical scenario where researchers publish results that could either support a theory they believe in, or refute that theory, and ask if the two instances carry the same weight. More colloquially, I ask if we should overweigh scientific results supporting a given theory and reported by a researcher, or a team, that initially did not support that theory. I illustrate the challenge using two examples from psychology: evidence accumulation models, and extra sensory perception.

The introduction of ChatGPT has fuelled a public debate on the appropriateness of using Generative AI (large language models; LLMs) in work, including a debate on how they might be used (and abused) by researchers. In the current work, we test whether delegating parts of the research process to LLMs leads people to distrust researchers and devalues their scientific work. Participants (N = 402) considered a researcher who delegates elements of the research process to a PhD student or LLM and rated three aspects of such delegation. Firstly, they rated whether it is morally appropriate to do so. Secondly, they judged whether—after deciding to delegate the research process—they would trust the scientist (that decided to delegate) to oversee future projects. Thirdly, they rated the expected accuracy and quality of the output from the delegated research process. Our results show that people judged delegating to an LLM as less morally acceptable than delegating to a human (d = –0.78). Delegation to an LLM also decreased trust to oversee future research projects (d = –0.80), and people thought the results would be less accurate and of lower quality (d = −0.85). We discuss how this devaluation might transfer into the underreporting of Generative AI use.

Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of [Formula: see text]. We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.

In recent years, the reliability and validity of psychology measurement practices has been called into question, as part of an ongoing reappraisal of the robustness, reproducibility, and transparency of psychological research. While useful progress has been made, to date, the majority of discussions surrounding psychology’s measurement crisis have involved technical, quantitative investigations into the validity, reliability, and statistical robustness of psychological measures. This registered report offers a seldom-heard qualitative perspective on these ongoing debates, critically exploring members of the general public’s (i.e., non-experts) lay perceptions of widely used measures in psychology. Using a combination of cognitive interviews and a think aloud study protocol, participants (n = 23) completed one of three popular psychology measures. Participants reflected on each of the measures, discussed the contents, and provided perceptions of what the measures are designed to test. Coding of the think aloud protocols showed that participants across the measures had issues in interpreting and responding to items. Thematic analysis of the cognitive interviews identified three dominant themes that each relate to lay perceptions of psychology measurements. These were: (1) participants’ grappling with attempting to ‘capture their multiple selves’ in the questionnaires, (2) participants perceiving the questionnaire method as generally ‘missing nuance and richness’ and (3) exposing the ‘hidden labour of questionnaires’. These findings are discussed in the context of psychology’s measurement reform.

Two distinct modes of data governance have emerged in accessing and reusing viral data pertaining to COVID-19: an unrestricted model, espoused by data repositories part of the International Nucleotide Sequence Database Collaboration and a regulated model promoted by the Global Initiative on Sharing All Influenza data. In this paper, we focus on publications mentioning either infrastructure in the period between January 2020 and January 2023, thus capturing a period of acute response to the COVID-19 pandemic. Through a variety of bibliometric and network science methods, we compare the extent to which either data infrastructure facilitated collaboration from different countries around the globe to understand how data reuse can enhance forms of diversity between institutions, countries, and funding groups. Our findings reveal disparities in representation and usage between the two data infrastructures. We conclude that both approaches offer useful lessons, with the unrestricted model providing insights into complex data linkage and the regulated model demonstrating the importance of global representation.

Hypotheses are frequently the starting point when undertaking the empirical portion of the scientific process. They state something that the scientific process will attempt to evaluate, corroborate, verify or falsify. Their purpose is to guide the types of data we collect, analyses we conduct, and inferences we would like to make. Over the last decade, metascience has advocated for hypotheses being in preregistrations or registered reports, but how to formulate these hypotheses has received less attention. Here, we argue that hypotheses can vary in specificity along at least three independent dimensions: the relationship, the variables, and the pipeline. Together, these dimensions form the scope of the hypothesis. We demonstrate how narrowing the scope of a hypothesis in any of these three ways reduces the hypothesis space and that this reduction is a type of novelty. Finally, we discuss how this formulation of hypotheses can guide researchers to formulate the appropriate scope for their hypotheses and should aim for neither too broad nor too narrow a scope. This framework can guide hypothesis-makers when formulating their hypotheses by helping clarify what is being tested, chaining results to previous known findings, and demarcating what is explicitly tested in the hypothesis.

We outline essential considerations for any study of partial randomisation of research funding, and consider scenarios in which randomised controlled trials (RCTs) would be feasible and appropriate. We highlight the interdependence of target outcomes, sample availability and statistical power for determining the cost and feasibility of a trial. For many choices of target outcome, RCTs may be less practical and more expensive than they at first appear (in large part due to issues pertaining to sample size and statistical power). As such, we briefly discuss alternatives to RCTs. It is worth noting that many of the considerations relevant to experiments on partial randomisation may also apply to other potential experiments on funding processes (as described in The Experimental Research Funder’s Handbook. RoRI, June 2022).

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as ‘effect size’ and ‘reputation’ (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.

The scientific reform movement has proposed openness as a potential remedy to the putative reproducibility or replication crisis. However, the conceptual relationship among openness, replication experiments and results reproducibility has been obscure. We analyse the logical structure of experiments, define the mathematical notion of idealized experiment and use this notion to advance a theory of reproducibility. Idealized experiments clearly delineate the concepts of replication and results reproducibility, and capture key differences with precision, allowing us to study the relationship among them. We show how results reproducibility varies as a function of the elements of an idealized experiment, the true data-generating mechanism, and the closeness of the replication experiment to an original experiment. We clarify how openness of experiments is related to designing informative replication experiments and to obtaining reproducible results. With formal backing and evidence, we argue that the current ‘crisis’ reflects inadequate attention to a theoretical understanding of results reproducibility.

Comparative cognitive and behavior research aims to investigate cognitive evolution by comparing performance in different species to understand how these abilities have evolved. Ideally, this requires large and diverse samples; however, these can be difficult to obtain by single labs or institutions, leading to potential reproducibility and generalization issues with small, less representative samples. To help mitigate these issues, we are establishing a multi-site collaborative Open Science approach called ManyBirds, with the aim of providing new insight into the evolution of avian cognition and behavior through large-scale comparative studies, following the lead of exemplary ManyPrimates, ManyBabies and ManyDogs projects. Here, we outline a) the replicability crisis and why we should study birds, including the origin of modern birds, avian brains and convergent evolution of cognition; b) the current state of the avian cognition field, including a ‘snapshot’ review; c) the ManyBirds project, with plans, infrastructure, limitations, implications and future directions. In sharing this process, we hope that this may be useful for other researchers in devising similar projects in other taxa, like non-avian reptiles or mammals, and to encourage further collaborations with ManyBirds and related ManyX projects. Ultimately, we hope to promote collaboration between ManyX projects to allow for wider investigation of the evolution of cognition across all animals, including potentially humans.

Shortcomings in the rigour and reproducibility of research have become well-known issues and persist despite repeated calls for improvement. A coordinated effort among researchers, institutions, funders, publishers, learned societies, and regulators may be the most effective way of tackling these issues. The UK Reproducibility Network (UKRN) has fostered collaboration across various stakeholders in research and are creating the infrastructure necessary to advance rigorous and reproducible research practices across the United Kingdom. Other Reproducibility Networks, modelled on UKRN, are now emerging in other countries. Canada could benefit from a comparable network to unify the voices around research quality and maximize the value of Canadian research.

The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100, 407–425. (doi:10.1037/a0021524)) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.

Researchers routinely face choices throughout the data analysis process. It is often opaque to readers how these choices are made, how they affect the findings, and whether or not data analysis results are unduly influenced by subjective decisions. This concern is spurring numerous investigations into the variability of data analysis results. The findings demonstrate that different teams analyzing the same data may reach different conclusions. This is the “many-analysts” problem. Previous research on the many-analysts problem focused on demonstrating its existence, without identifying specific practices for solving it. We address this gap by identifying three pitfalls that have contributed to the variability observed in many-analysts publications and providing suggestions on how to avoid them.

This study reports an independent replication of the findings presented by Smaldino and McElreath (Smaldino, McElreath 2016 R. Soc. Open Sci. 3, 160384 (doi:10.1098/rsos.160384)). The replication was successful with one exception. We find that selection acting on scientist’s propensity for replication frequency caused a brief period of exuberant replication not observed in the original paper due to a coding error. This difference does not, however, change the authors’ original conclusions. We call for more replication studies for simulations as unique contributions to scientific quality assurance.

As research becomes more interdisciplinary, fast-paced, data-intensive, and collaborative, there is an increasing need to share data and other research products in accordance with Open Science principles. In response to this need, we created an Open Science & Data Collaborations (OSDC) program at the Carnegie Mellon University Libraries that provides Open Science tools, training, collaboration opportunities, and community-building events to support Open Research and Open Science adoption. This program presents a unique end-to-end model for Open Science programs because it extends open science support beyond open repositories and open access publishing to the entire research lifecycle. We developed a logic model and a preliminary assessment metrics framework to evaluate the impact of the program activities based on existing data collected through event and workshop registrations and platform usage. The combination of these evaluation instruments has provided initial insight into our service productivity and impact. It will further help to answer more in-depth questions regarding the program impact, launch targeted surveys, and identify priority service areas and interesting Open Science projects.

Abstract How do scientists generate and weight candidate queries for hypothesis testing, and how does learning from observations or experimental data impact query selection? Field sciences offer a compelling context to ask these questions because query selection and adaptation involves consideration of the spatiotemporal arrangement of data, and therefore closely parallels classic search and foraging behavior. Here we conduct a novel simulated data foraging study—and a complementary real-world case study—to determine how spatiotemporal data collection decisions are made in field sciences, and how search is adapted in response to in-situ data. Expert geoscientists evaluated a hypothesis by collecting environmental data using a mobile robot. At any point, participants were able to stop the robot and change their search strategy or make a conclusion about the hypothesis. We identified spatiotemporal reasoning heuristics, to which scientists strongly anchored, displaying limited adaptation to new data. We analyzed two key decision factors: variable-space coverage, and fitting error to the hypothesis. We found that, despite varied search strategies, the majority of scientists made a conclusion as the fitting error converged. Scientists who made premature conclusions, due to insufficient variable-space coverage or before the fitting error stabilized, were more prone to incorrect conclusions. We found that novice undergraduates used the same heuristics as expert geoscientists in a simplified version of the scenario. We believe the findings from this study could be used to improve field science training in data foraging, and aid in the development of technologies to support data collection decisions.

There is a confidence crisis in many scientific disciplines, in particular disciplines researching human behavior, as many effects of original experiments have not been replicated successfully in large-scale replication studies. While human-robot interaction (HRI) is an interdisciplinary research field, the study of human behavior, cognition and emotion in HRI plays also a vital part. Are HRI user studies facing the same problems as other fields and if so, what can be done to overcome them? In this article, we first give a short overview of the replicability crisis in behavioral sciences and its causes. In a second step, we estimate the replicability of HRI user studies mainly 1) by structural comparison of HRI research processes and practices with those of other disciplines with replicability issues, 2) by systematically reviewing meta-analyses of HRI user studies to identify parameters that are known to affect replicability, and 3) by summarizing first replication studies in HRI as direct evidence. Our findings suggest that HRI user studies often exhibit the same problems that caused the replicability crisis in many behavioral sciences, such as small sample sizes, lack of theory, or missing information in reported data. In order to improve the stability of future HRI research, we propose some statistical, methodological and social reforms. This article aims to provide a basis for further discussion and a potential outline for improvements in the field.

Disagreement is essential to scientific progress but the extent of disagreement in science, its evolution over time, and the fields in which it happens remain poorly understood. Here we report the development of an approach based on cue phrases that can identify instances of disagreement in scientific articles. These instances are sentences in an article that cite other articles. Applying this approach to a collection of more than four million English-language articles published between 2000 and 2015 period, we determine the level of disagreement in five broad fields within the scientific literature (biomedical and health sciences; life and earth sciences; mathematics and computer science; physical sciences and engineering; and social sciences and humanities) and 817 meso-level fields. Overall, the level of disagreement is highest in the social sciences and humanities, and lowest in mathematics and computer science. However, there is considerable heterogeneity across the meso-level fields, revealing the importance of local disciplinary cultures and the epistemic characteristics of disagreement. Analysis at the level of individual articles reveals notable episodes of disagreement in science, and illustrates how methodological artifacts can confound analyses of scientific texts.

In three years, two tailings dam failures have profoundly changed the vision on mining in Brazil. More than three hundred deaths and one thousand kilometers of destroyed rivers are nowadays study objects of almost all scientific fields. Although, many gaps still exist. To comprehend the actual scenario of the academic production on these technological disasters, we did a systematic review of the literature by analytical procedures that are rigorous, explicit, and reproducible. A set of 326 scientific documents was identified, interpreted, and classified by their socio-academic relevance. The results show a greater participation of Geosciences and Science of Law, but there is little dialogue between publications, which can be seen for the low number of citations in most documents.

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Adapting studies typically run in the lab, preschool, or museum to online data collection presents a variety of challenges. The solutions to those challenges depend heavily on the specific questions pursued, the methods used, and the constraints imposed by available technology. We present a partial sample of solutions, discussing approaches we have developed for adapting studies targeting a range of different developmental populations, from infants to school-aged children, and utilizing various online methods such as high-framerate video presentation, having participants interact with a display on their own computer, having the experimenter interact with both the participant and an actor, recording free-play with physical objects, recording infant looking times both offline and live, and more. We also raise issues and solutions regarding recruitment and representativeness in online samples. By identifying the concrete needs of a given approach, tools that meet each of those individual needs, and interfaces between those tools, we have been able to implement many (but not all) of our studies using online data collection during the COVID-19 pandemic. This systematic review aligning available tools and approaches with different methods can inform the design of future studies, in and outside of the lab.

Philosophical interests of Joseph Życiński (1948-2011) in the domain of the philosophy of science were focused on the debate concerning the nature of science and philosophy of science that followed the Einstein-Planck revolution in science. The unexpected discovery of the philosophical, extra-scientific presuppositions in science, as well as of the extra-rational factors determining the way these presuppositions are accepted in science were to be explained within the meta-scientific framework. It is the aim of this paper to present ˙ Życiński’s diagnosis of this post-revolutionary situation in the philosophy of science as well as his critique of the metascientific answers to this challenge. The reasons will be given why all those answers are put under two dichotomous rubrics of internalism and externalism. It will be also explained how Życiński intends to supersede this false in his opinion opposition with a new concept of the doxatic rationality. However, the details of the metascientific proposal of Życiński will be given only in the subsequent paper. In order to perform the aim of the paper the metatheoretic tools set out by Popper (1979) will be used.

The present work contains a personal perspective on what sport and exercise psychology (SEP) is today. It is a global synthesis of research about psychological aspects related to the context and practice of sport and exercise. The intended impact was to positively influence teaching SEP to students, to promote interdisciplinary research and practice, and to assist the development of SEP as an applied science by helping experts develop a more holistic view of the field. Over 650 theoretical and review articles about psychological concepts in connection to sport and exercise were read in the process of creating a conceptual model that reflects the essence of SEP and leads to a conceptualization of SEP based on research topics. The result was a knowledge map of SEP made up of four main research clusters: biopsychological descriptors, external variables, psychological skills, and applied SEP practice. In terms of interdisciplinarity, the present perspective on SEP suggests that sport and exercise can be used as a research paradigm or natural laboratory to study psychological aspects relevant to various scientific fields, and that sport and exercise can be used as a therapeutic framework in response to challenges that researchers and practitioners in these fields are typically addressing.

Research in primate cognition improves our understanding of the evolutionary history of these mental abilities. Large samples from a wide diversity of species are essential to achieve such objectives. However, research on primate cognition usually suffers from very small samples limited to a handful of species, which prevents reliable evolutionary inferences. The ManyPrimates project offers to overcome some of these problems by providing a large-scale collaborative framework for comparative studies on primate cognition. Building on similar initiatives in other areas of psychology, this consortium has already produced a pilot study on short-term memory gathering one of the largest, i.e. 176 individuals, and most diverse, i.e. twelve species from eleven different sites, primate samples to date. Other projects on short-term memory, inhibitory abilities and on inference by exclusion are in progress on large and diverse data sets and represent exciting opportunities to answer still open questions on the origin of the extraordinary cognitive abilities of primates. This paper describes these projects but also the coordination and communication of this collaborative infrastructure and the different ways to contribute to it.

Science is self-correcting, or so the adage goes, but to what extent is that indeed the case? Answering this question requires careful consideration of the various approaches to achieve the collective goal of self-correction. One of the most straightforward mechanisms is individual self-correction: researchers rectifying their own mistakes by publishing a correction notice. Although it offers an efficient route to correcting the scientific record, it has received little to no attention from a metascientific point of view. We aim to fill this void by analysing the content of correction notices published from 2010 until 2018 in the three psychology journals featuring the highest number of corrections over that timespan based on the Scopus database (i.e. Psychological Science with N = 58, Frontiers in Psychology with N = 99 and Journal of Affective Disorders with N = 57). More concretely, we examined which aspects of the original papers were affected (e.g. hypotheses, data-analyses, metadata such as author order, affiliations, funding information etc.) as well as the perceived implications for the papers’ main findings. Our exploratory analyses showed that many corrections involved inconsequential errors. Furthermore, authors rarely revised their conclusions, even though several corrections concerned changes to the results. We conclude with a discussion of current policies, and suggest ways to improve upon the present situation by (i) preventing mistakes, and (ii) transparently rectifying those mistakes that do find their way into the literature.

As part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Phelps et al., 2016) that described how we intended to replicate selected experiments from the paper ‘Coding-independent regulation of the tumor suppressor PTEN by competing endogenous mRNAs’ (Tay et al., 2011). Here, we report the results. We found depletion of putative PTEN competing endogenous mRNAs (ceRNAs) in DU145 cells did not impact PTEN 3’UTR regulation using a reporter, while the original study reported decreased activity when SERINC1, VAPA, and CNOT6L were depleted (Figure 3C; Tay et al., 2011). Using the same reporter, we found decreased activity when ceRNA 3’UTRs were overexpressed, while the original study reported increased activity (Figure 3D; Tay et al., 2011). In HCT116 cells, ceRNA depletion resulted in decreased PTEN protein levels, a result similar to the findings reported in the original study (Figure 3G,H; Tay et al., 2011); however, while the original study reported an attenuated ceRNA effect in microRNA deficient (DicerEx5) HCT116 cells, we observed increased PTEN protein levels. Further, we found depletion of the ceRNAs VAPA or CNOT6L did not statistically impact DU145, wild-type HCT116, or DicerEx5 HCT116 cell proliferation. The original study reported increased DU145 and wild-type HCT116 cell proliferation when these ceRNAs were depleted, which was attenuated in the DicerEx5 HCT116 cells (Figure 5B; Tay et al., 2011). Differences between the original study and this replication attempt, such as variance between biological repeats, are factors that might have influenced the results. Finally, we report meta-analyses for each result.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Khan et al., 2015), that described how we intended to replicate selected experiments from the paper "A coding-independent function of gene and pseudogene mRNAs regulates tumour biology" (Poliseno et al., 2010). Here we report the results. We found PTEN depletion in the prostate cancer cell line DU145 did not detectably impact expression of the corresponding pseudogene PTENP1. Similarly, depletion of PTENP1 did not impact PTEN mRNA levels. The original study reported PTEN or PTENP1 depletion statistically reduced the corresponding pseudogene or gene (Figure 2G; Poliseno et al., 2010). PTEN and/or PTENP1 depletion in DU145 cells decreased PTEN protein expression, which was similar to the original study (Figure 2H; Poliseno et al., 2010). Further, depletion of PTEN and/or PTENP1 increased DU145 proliferation compared to non-targeting siRNA, which was in the same direction as the original study (Figure 2F; Poliseno et al., 2010), but not statistically significant. We found PTEN 3'UTR overexpression in DU145 cells did not impact PTENP1 expression, while the original study reported PTEN 3'UTR increased PTENP1 levels (Figure 4A; Poliseno et al., 2010). Overexpression of PTEN 3'UTR also statistically decreased DU145 proliferation compared to controls, which was similar to the findings reported in the original study (Figure 4A; Poliseno et al., 2010). Differences between the original study and this replication attempt, such as level of knockdown efficiency and cellular confluence, are factors that might have influenced the results. Finally, where possible, we report meta-analyses for each result.

The partial success of a study to reproduce experiments that linked pseudogenes and cancer proves that understanding RNA networks is more complicated than expected.

What explanation is there when teams of researchers are unable to successfully replicate already established ‘canonical’ findings? One suggestion that has been put forward, but left largely untested, is that those researchers who fail to replicate prior studies are of low ‘expertise and diligence’ and lack the skill necessary to successfully replicate the conditions of the original experiment. Here we examine the replication success of 100 scientists of differing ‘expertise and diligence’ who attempted to replicate five different studies. Using a bibliometric tool (h-index) as our indicator of researcher ‘expertise and diligence’, we examine whether this was predictive of replication success. Although there was substantial variability in replication success and in the h-factor of the investigators, we find no relationship between these variables. The present results provide no evidence for the hypothesis that systematic replications fail because of low ‘expertise and diligence’ among replicators.

The aim of this paper is an analysis of the interaction of philosophy and natural sciences in Byzantine Empire. The cornerstone for this will be the clarification of the function of philosophy as a metascience (μεταεπιστήμη) in Greek thought, which grappled the order of things in its functional unity of macroscopic and microscopic observation of the world. This macroscopic observation was never at odds with microscopic research but was a universal science (καθόλου ἐπιστήμη) or science about scientifically knowable phenomena (ἡ μάλιστα ἐπιστήμη τοῦ μάλιστα ἐπιστητοῦ), that is, philosophy. Despite philosophy being an activity closely linked to natural sciences, it is not a science in itself, even though it is often mistaken for such as a term. Philosophy as knowledge, or a function that leads to knowledge, is called science (ἐπιστήμη), but this does not make it equal to scientific disciplines. In the unity of all scientific disciplines, which are hierarchically ordered into a pyramid, philosophy always stands on top, it being a science about scientifically knowable phenomena. Out of all the scientific disciplines, philosophy is the one that leads to knowledge of first beginnings and causes. It is a general observation of reality, which secures the unity of scientifically knowable facts and phenomena. Philosophical knowledge as a "universal science" (καθόλου ἐπιστήμη) concentrates all partial findings into a unified whole. Philosophy is continually attempting to reach a global understanding, while science focuses on a microscopic observation of a specific part of the system.

Over the past century, advancements in psychiatric treatments have freed countless individuals from the burden of life-long, incapacitating mental illness. These treatments have largely been discovered by chance. Theory has driven advancement in the natural sciences and other branches of medicine, but psychiatry remains a field in its “infancy”. The targets for healing in psychiatry lie within the realm of the mind’s subjective experience and thought, which we cannot yet describe in terms of their biological underpinnings in the brain. Our technology is sufficiently advanced to study brain neurons and their interactions on an electrophysiological and molecular level, but we cannot say how these form a single feeling or thought. While psychiatry waits for its “Copernican Revolution”, we continue the work in developing theories and associated experiments based on our existing diagnostic systems, for example, the Diagnostic and Statistical Manual of Mental Disorders (DSM), International Classification of Diseases (ICD), or the more newly introduced Research Domain Criteria (RDoC) framework. Understanding the subjective reality of the mind in biological terms would doubtless lead to huge advances in psychiatry, as well as to ethical dilemmas, from which we are spared for the time being.

The article considers the status of technology as a branch of science. Technology is understood as the totality of all technical theories accepted by the scientific community. The dominant trend in understanding the status of technology is that a demarcation line is drawn between, on the one hand, science and, on the other hand, technology. In fact, there is a kind of alienation of technology from science. Researchers provide various arguments to prove the legitimacy of this alienation. As a matter of fact, there are no arguments against the separation technology from science. This asymmetry is subjected to critical analysis in the article. The author, being guided by the theory of conceptual transduction developed by him, proves that scientifically sound arguments testify to the inclusion of technology in science on the rights of a full-fledged branch in all respects. The alienation of technology from science is baseless. All arguments that downgrade the status of technology as a branch of science are erroneous. The known features characterizing technology, namely, its appeal to the phenomenon of obligation, practical orientation, consideration of the methods for inventing and handling artifacts, in no way do not call into question the scientific status of technology. The decisive mistake of those who support the separation of science from technology is that they do not consider the conceptual and methodological structure of existing branches of science. If one looks at it, it turns out that it is the same for all branches of science, including technology. First, in all cases, scientific theories are ways to manage the concepts of principles, laws and variables by methods of deduction, adduction, induction and abduction. Second, the progress of science is realized as a transition from less developed to more developed theories. Third, each branch of science develops its own potential not in isolation from other branches of science but through its interdisciplinary relations with them. These three features indicate the status of technology as a branch of science. A proper understanding of the status of technology is a factor contributing to its development.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Fiering et al., 2015) that described how we intended to replicate selected experiments from the paper ‘Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis’ (Goetz et al., 2011). Here we report the results. Primary mouse embryonic fibroblasts (pMEFs) expressing caveolin 1 (Cav1WT) demonstrated increased extracellular matrix remodeling in vitro compared to Cav1 deficient (Cav1KO) pMEFs, similar to the original study (Goetz et al., 2011). In vivo, we found higher levels of intratumoral stroma remodeling, determined by fibronectin fiber orientation, in tumors from cancer cells co-injected with Cav1WT pMEFs compared to cancer cells only or cancer cells plus Cav1KO pMEFs, which were in the same direction as the original study (Supplemental Figure S7C; Goetz et al., 2011), but not statistically significant. Primary tumor growth was similar between conditions, like the original study (Supplemental Figure S7Ca; Goetz et al., 2011). We found metastatic burden was similar between Cav1WT and Cav1KO pMEFs, while the original study found increased metastases with Cav1WT (Figure 7C; Goetz et al., 2011); however, the duration of our in vivo experiments (45 days) were much shorter than in the study by Goetz et al. (2011) (75 days). This makes it difficult to interpret the difference between the studies as it is possible that the cells required more time to manifest the difference between treatments observed by Goetz et al. We also found a statistically significant negative correlation of intratumoral remodeling with metastatic burden, while the original study found a statistically significant positive correlation (Figure 7Cd; Goetz et al., 2011), but again there were differences between the studies in terms of the duration of the metastasis studies and the imaging approaches that could have impacted the outcomes. Finally, we report meta-analyses for each result.

The partial success of an attempt to repeat findings in cancer biology highlights the need to improve study designs for preclinical research into metastasis and the targeting of cancer cells.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Evans et al., 2015), that described how we intended to replicate selected experiments from the paper ‘Wnt activity defines colon cancer stem cells and is regulated by the microenvironment’ (Vermeulen et al., 2010). Here, we report the results. Using three independent primary spheroidal colon cancer cultures that expressed a Wnt reporter construct we observed high Wnt activity was associated with the cell surface markers CD133, CD166, and CD29, but not CD24 and CD44, while the original study found all five markers were correlated with high Wnt activity (Figure 2F; Vermeulen et al., 2010). Clonogenicity was highest in cells with high Wnt activity and clonogenic potential of cells with low Wnt activity were increased by myofibroblast-secreted factors, including HGF. While the effects were in the same direction as the original study (Figure 6D; Vermeulen et al., 2010) whether statistical significance was reached among the different conditions varied. When tested in vivo, we did not find a difference in tumorigenicity between high and low Wnt activity, while the original study found cells with high Wnt activity were more effective in inducing tumors (Figure 7E; Vermeulen et al., 2010). Tumorigenicity, however, was increased with myofibroblast-secreted factors, which was in the same direction as the original study (Figure 7E; Vermeulen et al., 2010), but not statistically significant. Finally, we report meta-analyses for each results where possible.

Incentive structures shape scientists' research practices. One incentive in particular, rewarding priority of publication, is hypothesized to harm scientific reliability by promoting rushed, low-quality research. Here, we develop a laboratory experiment to test whether competition affects information sampling and guessing accuracy in a game that mirrors aspects of scientific investigation. In our experiment, individuals gather data in order to guess true states of the world and face a tradeoff between guessing quickly and increasing accuracy by acquiring more information. To test whether competition affects accuracy, we compare a treatment in which individuals are rewarded for each correct guess to a treatment where individuals face the possibility of being ‘scooped’ by a competitor. In a second set of conditions, we make information acquisition contingent on solving arithmetic problems to test whether competition increases individual effort (i.e. arithmetic-problem solving speed). We find that competition causes individuals to make guesses using less information, thereby reducing their accuracy (H1a and H1b confirmed). We find no evidence that competition increases individual effort (H2, inconclusive evidence). Our experiment provides proof of concept that rewarding priority of publication can incentivize individuals to acquire less information, producing lower-quality research as a consequence.

As part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Li et al., 2015), that described how we intended to replicate selected experiments from the paper ‘The microRNA miR-34a inhibits prostate cancer stem cells and metastasis by directly repressing CD44’ (Liu et al., 2011). Here we report the results. We found the microRNA, miR-34a, was expressed at twice the level in CD44+ prostate cancer cells purified from xenograft tumors (LAPC4 cells) compared to CD44- LAPC4 cells, whereas the original study reported miR-34a was underexpressed in CD44+ LAPC4 cells (Figure 1B; Liu et al., 2011). When LAPC4 cells engineered to express miR-34a were injected into mice, we did not observe changes in tumor growth or CD44 expression; however, unexpectedly miR-34a expression was lost in vivo. In the original study, LAPC4 cells expressing miR-34a had a statistically significant reduction in tumor regeneration and reduced CD44 expression compared to control (Figure 4A and Supplemental Figures 4A,B and 5C; Liu et al., 2011). Furthermore, when we tested if miR-34a regulated CD44 through binding sites in the 3’UTR we did not find a statistically significant difference, whereas the original study reported miR-34a decreased CD44 expression that was partially abrogated by mutation of the binding sites in the CD44 3’UTR (Figure 4D; Liu et al., 2011). Finally, where possible, we report meta-analyses for each result.

It is noted that achievements by metascience are insufficiently used in the development of the theory of knowledge management including nuclear knowledge. The phenomenon of theory deserves paying special attention because knowledge means the theories mastered by people. There is no such knowledge about subjects and objects, processes or phenomena that would not be the representation of theories. That is why the main provisions of metascience are first disclosed by the author in the present paper and following this they are applied for characterization of 15 important problematic issues of the nuclear knowledge management theory. Management of intratheoretical concepts, i.e. the principles, laws and variables, is implemented using four methods, namely, deduction, experiment, induction and correction of the original underlying principles. Management of theories is implemented by the use of three methods: problematization, innovation and interpretation. Multiplication of theory management cycles results in the generation of series of interpretative theories. Subject matter of each separate theory is exposed in the composition of the series from the viewpoint of the most well developed concept. Conclusion is made that series of interpretative theories constitute the basic element (unit) of knowledge. Significant place is occupied in the characterization of outstanding problems of nuclear knowledge management by the correlation between the articulated and practical, as well as between tacit and explicit knowledge. Mechanism is examined for implementation of the discourse leading to the development of the group knowledge. Characteristic is given of the status of competence, skills and hands-on experience of carriers of nuclear knowledge. Approaches to the preservation and development of theories are discussed.

Most efforts to estimate the reproducibility of published findings have focused on specific areas of research, even though science is usually assessed and funded on a regional or national basis. Here we describe a project to assess the reproducibility of findings in biomedical science published by researchers based in Brazil. The Brazilian Reproducibility Initiative is a systematic, multicenter effort to repeat between 60 and 100 experiments: the project will focus on a set of common methods, repeating each experiment in three different laboratories from a countrywide network. The results, due in 2021, will allow us to estimate the level of reproducibility of biomedical science in Brazil, and to investigate what aspects of the published literature might help to predict whether a finding is reproducible.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Lesnik et al., 2016) that described how we intended to replicate selected experiments from the paper ‘Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET’ (Peinado et al., 2012). Here we report the results. We regenerated tumor cells stably expressing a short hairpin to reduce Met expression (shMet) using the same highly metastatic mouse melanoma cell line (B16-F10) as the original study, which efficiently downregulated Met in B16F10 cells similar to the original study (Supplementary Figure 5A; Peinado et al., 2012). Exosomes from control cells expressed Met, which was reduced in exosomes from shMet cells; however, we were unable to reliably detect phosphorylated Met in exosomes. We tested the effect of exosome-dependent Met signaling on primary tumor growth and metastasis. Similar to the results in the original study, we did not find a statistically significant change in primary tumor growth. Measuring lung and femur metastases, we found a small increase in metastatic burden with exosomes from control cells that was diminished when Met expression was reduced; however, while the effects were in the same direction as the original study (Figure 4E; Peinado et al., 2012), they were not statistically significant. Differences between the original study and this replication attempt, such as level of knockdown efficiency, cell line genetic drift, sample sizes, study endpoints, and variability of observed metastatic burden, are factors that might have influenced the outcomes. Finally, we report meta-analyses for each result.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Eaton et al., 2015) that described how we intended to replicate selected experiments from the paper “Intestinal Inflammation Targets Cancer-Inducing Activity of the Microbiota” (Arthur et al., 2012). Here we report the results. We observed no impact on bacterial growth or colonization capacity when the polyketide synthase (pks) genotoxic island was deleted from E. coli NC101, similar to the original study (Supplementary Figure 7; Arthur et al., 2012). However, for the experiment that compared inflammation, invasion, and neoplasia in azoxymethane (AOM)-treated interleukin-10-deficient mice mono-associated with NC101 or NC101[Formula: see text] pks the experimental timing of the replication attempt was longer than that of the original study. This difference was because in the original study the methodology was not clearly stated and likely led to the increased mortality and severity of inflammation observed in this replication attempt. Additionally, early death occurred during AOM treatment with higher mortality observed in NC101[Formula: see text] pks mono-associated mice compared to NC101, which was in the same direction, but more severe than the original study (Suppleme1ntal Figure 10; Arthur et al., 2012). A meta-analysis suggests that mice mono-associated with NC101[Formula: see text] pks have higher mortality compared to NC101. While these data were unable to address whether, under the conditions of the original study, NC101 and NC101[Formula: see text] pks differ in inflammation, invasion, and neoplasia this replication attempt demonstrates that clear description of experimental methods is essential to ensure accurate reproduction of experimental studies.