Search

Many peer-review processes involve reviewers submitting their independent reviews, followed by a discussion between the reviewers of each paper. A common question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this question by conducting a randomized controlled trial at the Conference on Uncertainty in Artificial Intelligence (UAI) 2022 conference where reviewer discussions were conducted over a typed forum. We randomly split the reviewers and papers into two conditions–one with anonymous discussions and the other with non-anonymous discussions. We also conduct an anonymous survey of all reviewers to understand their experience and opinions. We compare the two conditions in terms of the amount of discussion, influence of seniority on the final decisions, politeness, reviewers' self-reported experiences and preferences. Overall, this experiment finds small, significant differences favoring the anonymous discussion setup based on the evaluation criteria considered in this work. [ABSTRACT FROM AUTHOR]

Objective: Peer review frequently follows a process where reviewers first provide initial reviews, authors respond to these reviews, then reviewers update their reviews based on the authors' response. There is mixed evidence regarding whether this process is useful, including frequent anecdotal complaints that reviewers insufficiently update their scores. In this study, we aim to investigate whether reviewers anchor to their original scores when updating their reviews, which serves as a potential explanation for the lack of updates in reviewer scores. Design: We design a novel randomized controlled trial to test if reviewers exhibit anchoring. In the experimental condition, participants initially see a flawed version of a paper that is corrected after they submit their initial review, while in the control condition, participants only see the correct version. We take various measures to ensure that in the absence of anchoring, reviewers in the experimental group should revise their scores to be identically distributed to the scores from the control group. Furthermore, we construct the reviewed paper to maximize the difference between the flawed and corrected versions, and employ deception to hide the true experiment purpose. Results: Our randomized controlled trial consists of 108 researchers as participants. First, we find that our intervention was successful at creating a difference in perceived paper quality between the flawed and corrected versions: Using a permutation test with the Mann-Whitney U statistic, we find that the experimental group's initial scores are lower than the control group's scores in both the Evaluation category (Vargha-Delaney A = 0.64, p = 0.0096) and Overall score (A = 0.59, p = 0.058). Next, we test for anchoring by comparing the experimental group's revised scores with the control group's scores. We find no significant evidence of anchoring in either the Overall (A = 0.50, p = 0.61) or Evaluation category (A = 0.49, p = 0.61). The Mann-Whitney U represents the number of individual pairwise comparisons across groups in which the value from the specified group is stochastically greater, while the Vargha-Delaney A is the normalized version in [0, 1]. [ABSTRACT FROM AUTHOR]

Background: Pain is multifactorial, and pain intensity has been shown to be influenced by patients' thoughts. The Negative Pain Thoughts Questionnaire Short Form (NPTQ-SF) can be used to quantify unhelpful negative cognitive biases about pain, but the relationship between negative pain thoughts and orthopaedic surgery outcomes is not known. Purpose: To evaluate the prevalence of negative pain thoughts in patients undergoing arthroscopic meniscectomy using the NPTQ-SF survey and assess the relationship these thoughts have to knee function, general health, pain, and satisfaction before and after surgery. Study Design: Case series; Level of evidence, 4. Methods: In total, 146 patients undergoing arthroscopic meniscectomy were administered the 4-item NPTQ-SF, 12-item Short Form Survey (SF-12), International Knee Documentation Committee (IKDC) questionnaire, and visual analog scale pain survey preoperatively between July 2021 and August 2022. The same surveys were completed at a minimum of 3 months and no later than 1 year postoperatively by 92 patients confirmed to have undergone meniscectomy. Results: NPTQ-SF scores were correlated with IKDC, SF-12, and satisfaction score preoperatively and at least 3 months postoperatively (mean, 108.5 ± 43.7 days). Preoperative NPTQ-SF scores were significantly negatively correlated with postoperative IKDC (R = −0.284), SF-12 (R = −0.266 and −0.328), and visual analog scale pain (R = 0.294) scores, while a relationship with postoperative satisfaction did not reach statistical significance (P =.067). Patients with a preoperative NPTQ-SF score >8 were less likely to achieve a Patient Acceptable Symptom State on the postoperative IKDC questionnaire (39% vs 63%; P =.03). Patients with a history of a psychiatric or chronic pain diagnoses have worse NPTQ-SF, SF-12, and IKDC scores pre- and postoperatively. Conclusion: The level of negative pain thoughts in patients undergoing meniscectomy is related to knee function, general health, and pain. Patients with a high level of negative pain thoughts are less likely to achieve a favorable outcome from meniscectomy, with a score ≥8 representing a clinically significant threshold for preoperative screening. [ABSTRACT FROM AUTHOR]

Background: Pain and pain perception are influenced by patients' thoughts. The short form Negative Pain Thoughts Questionnaire (NPTQ-SF) can be used to quantify unhelpful negative cognitive biases about pain, but the relationship between NPTQ-SF scores and orthopaedic surgery outcomes is not known. Purpose/Hypothesis: The purpose was to assess the relationship between negative pain thoughts, as measured by the NPTQ-SF, and patient-reported outcomes in patients undergoing arthroscopic rotator cuff repair, as well as to compare NPTQ-SF scores and outcomes between patients with and without a history of chronic pain and psychiatric history. It was hypothesized that patients with worse negative pain thoughts would have worse patient-reported outcomes. Study Design: Cohort study; Level of evidence, 2. Methods: In total, 109 patients undergoing arthroscopic rotator cuff repair were administered the 4-item NPTQ-SF, 12-item Short Form Health Survey (SF-12), American Shoulder and Elbow Surgeons (ASES) Shoulder Evaluation Form, and visual analog scale pain survey preoperatively between July 2021 and August 2022. The same surveys were completed ≥6 months postoperatively by 74 patients confirmed to have undergone arthroscopic rotator cuff repair. Results: Preoperative NPTQ-SF scores did not show any correlation with the postoperative patient-reported outcomes measured in this study. Postoperative NPTQ-SF scores were statistically significantly negatively correlated with postoperative SF-12 Physical Health Score, SF-12 Mental Health Score, ASES, and satisfaction scores (P <.05). Postoperative NPTQ-SF scores were statistically significantly positively correlated with postoperative visual analog scale scores (P <.001). Moreover, postoperative NPTQ-SF scores were statistically significantly negatively correlated with achieving a Patient Acceptable Symptom State and the minimal clinically important difference on the postoperative ASES form (P <.001 and P =.009, respectively). Conclusion: Postoperative patient thought patterns and their perception of pain are correlated with postoperative outcomes after rotator cuff repair. This correlation suggests a role for counseling and expectation management in the postoperative setting. Conversely, preoperative thought patterns regarding pain, as measured by the NPTQ-SF, do not correlate with postoperative patient-reported outcome measures. Therefore, the NPTQ-SF should not be used as a preoperative tool to aid the prediction of outcomes after rotator cuff repair. [ABSTRACT FROM AUTHOR]

How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we surveyed the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors had roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction was 70% for an approximately 25% acceptance rate. (2) Female authors exhibited a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers were similarly calibrated, but better than authors who were not invited to review. (3) Authors' relative ranking of scientific contribution of two submissions they made generally agreed with their predicted acceptance probabilities (93% agreement), but there was a notable 7% responses where authors predicted a worse outcome for their better paper. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate—about a third of the time. (5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review. [ABSTRACT FROM AUTHOR]