Search

Background:Changing the traditional p value significance level from 0.05 to 0.005 has been proposed as one remedy to the common ill understanding and use of the p values (1-2) especially leading to poor reproducibility. It was then reported that this change would decrease the number of statistically significant primary end points in randomized controlled trials (RCT) reported during 2017 in 3 main general medical journals by 1/3 (3).Objectives:We reasoned that the same change would affect in the right direction the significance conclusions more in observational studies (OS) in that OS generally have retrospective or cross-sectional designs, looser study inclusion criteria, many potential confounders and post – hoc analyses.Methods:We surveyed how the 0.05 to 0.005 change would affect the OS in the same 3 journals in reference 3 for the same year (2017) and in another year (2002), included to better assess the validity of our findings. RCT were also surveyed for both time periods. Two authors (MO, ZTD) surveyed all studies. Disagreements were settled by HY. Differing from the previous study (3) we analyzed the p values as related to the primary end points only in the abstracts. The RCT and OS between January 1, 2002 – December 31, 2002 and January 1, 2017 -December 31, 2017 in N Engl J Med, Lancet and JAMA were included.Results:Among the RCT in 2017 the decrease in the number of significant p values was quite similar between what was reported (3) and the current study (Table) (29.3 vs 30.0%) supporting the validity of the slightly different methodology we used. We were however surprised to see that changing the significance level to 0.005 did not decrease the number of significant p values in OS as compared to the RCT. In fact the percentage decrease in RCT was statistically significantly less in 2002 and numerically in 2017 (Table).Table.OS and RCT in 2002 and 2017OS (2002)RCT (2002)Number of articles305211Total number of p values related to primary outcome(s)323322p260(80.4)240(74.5)p190(58.8)121(37.5)Reductionin significant number of p values* (%)70(26.9)P=0.011119(49.5)OS (2017)RTC (2017)Number of articles167200Total number of p values related to primary outcome(s)187257p162 (86.6)173(67.3)p125(66.8%)121(47.0%)Reductionin significant number of p values* (%)37(22.8)P=0.5552(30.0)RCT:Randomized Controlled TrialsOS: Observational Studies*Number of p values Conclusion:Contrary to our initial reasoning we now realize that changing the traditional p threshold will not decrease the number of significant p values more in OS than in RCT. A limitation of our work was that we surveyed only 3, high impact factor, general medicine journals. We envisage that in studies with smaller sample sizes more commonly published in journals with lower impact factors, the proposed change in the significance threshold would be more useful. Smaller sample sizes lead to more unstable, fickle p (4) values and changing the threshold would conceivably more or less stabilize this fickleness both in OS and RCT.References:[1]Ioannidis JPA. The proposal to lower p value thresholds to .005. JAMA. 2018;319:1429-30.[2]Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers EJ, Berk R,et al. Redefine statistical significance. Nat Hum Behav. 2018;2:6-10.[3]Wayant C, Scott J, Vassar M. Evaluation of lowering the p value threshold for statistical significance from .05 to .005 in previously published randomized clinical trials in major medical journals. JAMA. 2018;320:1813-15.[4]Halsey, Lewis G., et al. The fickle P value generates irreproducible results. Nature methods. 2015;12:179-185.Disclosure of Interests:None declared

Objectives To assess quality of reporting of sample size calculation, ascertain accuracy of calculations, and determine the relevance of assumptions made when calculating sample size in randomised controlled trials. Design Review. Data sources We searched MEDLINE for all primary reports of two arm parallel group randomised controlled trials of superiority with a single primary outcome published in six high impact factor general medical journals between 1 January 2005 and 31 December 2006. All extra material related to design of trials (other articles, online material, online trial registration) was systematically assessed. Data extracted by use of a standardised form included parameters required for sample size calculation and corresponding data reported in results sections of articles. We checked completeness of reporting of the sample size calculation, systematically replicated the sample size calculation to assess its accuracy, then quantified discrepancies between a priori hypothesised parameters necessary for calculation and a posteriori estimates. Results Of the 215 selected articles, 10 (5%) did not report any sample size calculation and 92 (43%) did not report all the required parameters. The difference between the sample size reported in the article and the replicated sample size calculation was greater than 10% in 47 (30%) of the 157 reports that gave enough data to recalculate the sample size. The difference between the assumptions for the control group and the observed data was greater than 30% in 31% (n=45) of articles and greater than 50% in 17% (n=24). Only 73 trials (34%) reported all data required to calculate the sample size, had an accurate calculation, and used accurate assumptions for the control group. Conclusions Sample size calculation is still inadequately reported, often erroneous, and based on assumptions that are frequently inaccurate. Such a situation raises questions about how sample size is calculated in randomised controlled trials.

In a recent issue, we published a list of the 100 most cited authors who have published work relevant to tobacco control.1 We also listed the 50 most cited papers in our field. That exercise produced lists of authors and papers dominated by “big epi” work: papers mainly establishing the contribution of tobacco use to disease. Such papers are often published in high impact factor journals and tend to be cited in the introductory sections of other papers. Constructing the lists by citations alone resulted in some authors appearing who would not normally be considered leaders in tobacco control. They were primarily researchers working in epidemiology—often multi-risk factor epidemiology—whose work involved them in looking at the relation of smoking to disease. There were very few whose work involved tobacco control . Similarly, the 50 most cited papers were also dominated by epidemiological studies. The lists were decidedly narrow in showing the breadth of research scholarship examining all aspects of tobacco control policy, programmes, and the science underpinning these. Throughout my own career, I have often noted papers that struck me as in some way seminal or as having made a research contribution that changed …