Liliane Reis Teixeira, Daniele Mandrioli, Kathrine Sørensen, Lode Godderis, Daniela Pachito, Kai N. Streicher, Tom Loney, Daria Sgargi, Vivi Schlünssen, Diana Gagliardi, Paul T.J. Scheepers, Alberto Modenese, Grace Sembajwe, Alexis Descatha, Frank Pega, Denise Torres da Silva, Thomas Tenkate, Rebecca L. Morgan, Natalie C. Momen, Marília Silva Paulo, Monique H. W. Frings-Dresen, WHO-TDR Suisse, Escola Nacional de Saude Publica Sergio Arouca / Sergio Arouca National School of Public Health [Rio de Janeiro] (ENSP), Fundação Oswaldo Cruz (FIOCRUZ), Réseau International des Instituts Pasteur (RIIP)-Réseau International des Instituts Pasteur (RIIP), École des Hautes Études en Santé Publique [EHESP] (EHESP), Institut de recherche en santé, environnement et travail (Irset), Structure Fédérative de Recherche en Biologie et Santé de Rennes ( Biosit : Biologie - Santé - Innovation Technologique )-Institut National de la Santé et de la Recherche Médicale (INSERM)-École des Hautes Études en Santé Publique [EHESP] (EHESP)-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Université d'Angers (UA), Service de médecine physique et de réadaptation [ CHU Raymond-Poincaré], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Hôpital Raymond Poincaré [AP-HP], Cohortes épidémiologiques en population (CONSTANCES), Université de Versailles Saint-Quentin-en-Yvelines (UVSQ)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris-Saclay-Université de Paris (UP), Vieillissement et Maladies chroniques : approches épidémiologique et de santé publique (VIMA), Université de Versailles Saint-Quentin-en-Yvelines (UVSQ)-Institut National de la Santé et de la Recherche Médicale (INSERM), Amsterdam UMC, Istituto Nazionale per l’Assicurazione contro gli Infortuni sul Lavoro [Italian Workers Compensation Authority] (INAIL), Catholic University of Leuven - Katholieke Universiteit Leuven (KU Leuven), Mohammed Bin Rashid University of Medicine and Health Sciences (MBRU), Università degli Studi di Modena e Reggio Emilia (UNIMORE), McMaster University [Hamilton, Ontario], Fundacao Getulio Vargas [Rio de Janeiro] (FGV), Radboud University Medical Center [Nijmegen], United Arab Emirates University (UAEU), Aarhus University [Aarhus], City University of New York [New York] (CUNY), Ryerson University [Toronto], World Health Organization from National Institute for Occupational Safety and Health of the Centres for Disease Control and Prevention of the United States of America [1E11OH0010676-02, 6NE11OH010461-02-01, 5NE11OH010461-03-00], German Federal Ministry of Health (BMG Germany) under the BMG-WHO Collaboration Programme 2020-2023 (WHO specified award) [70672], Spanish Agency for International Cooperation (AECID) [71208], Université d'Angers (UA)-Université de Rennes (UR)-École des Hautes Études en Santé Publique [EHESP] (EHESP)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Structure Fédérative de Recherche en Biologie et Santé de Rennes ( Biosit : Biologie - Santé - Innovation Technologique ), Université de Versailles Saint-Quentin-en-Yvelines (UVSQ)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris-Saclay-Université Paris Cité (UPCité), Université de Rennes (UR), Amsterdam UMC - Amsterdam University Medical Center, Università degli Studi di Modena e Reggio Emilia = University of Modena and Reggio Emilia (UNIMORE), National Research Centre for the Working Environment (NRCWE), Chard-Hutchinson, Xavier, Fundação Oswaldo Cruz / Oswaldo Cruz Foundation (FIOCRUZ), Public and occupational health, APH - Quality of Care, and APH - Societal Participation & Health
Graphical abstract, Highlights • Risk of bias (RoB) assessment is a critical step of a systematic review. • RoB-SPEO assesses RoB in prevalence studies of occupational risk factor exposure. • Inter-rater agreement for its domains ranged from proportions of 0.54–0.82. • Median time taken to complete one assessment using RoB-SPEO was 40 min. • User experience was positive but called for an online tool and further instructions., Background As part of the development of the World Health Organization (WHO)/International Labour Organization (ILO) Joint Estimates of the Work-related Burden of Disease and Injury, WHO and ILO carried out several systematic reviews to determine the prevalence of exposure to selected occupational risk factors. Risk of bias assessment for individual studies is a critical step of a systematic review. No tool existed for assessing the risk of bias in prevalence studies of exposure to occupational risk factors, so WHO and ILO developed and pilot tested the RoB-SPEO tool for this purpose. Here, we investigate the assessor burden, inter-rater agreement, and user experience of this new instrument, based on the abovementioned WHO/ILO systematic reviews. Methods Twenty-seven individual experts applied RoB-SPEO to assess risk of bias. Four systematic reviews provided a total of 283 individual assessments, carried out for 137 studies. For each study, two or more assessors independently assessed risk of bias across the eight RoB-SPEO domains selecting one of RoB-SPEO’s six ratings (i.e., “low”, “probably low”, “probably high”, “high”, “unclear” or “cannot be determined”). Assessors were asked to report time taken (i.e. indicator of assessor burden) to complete each assessment and describe their user experience. To gauge assessor burden, we calculated the median and inter-quartile range of times taken per individual risk of bias assessment. To assess inter-rater reliability, we calculated a raw measure of inter-rater agreement (Pi) for each RoB-SPEO domain, between Pi = 0.00, indicating no agreement and Pi = 1.00, indicating perfect agreement. As subgroup analyses, Pi was also disaggregated by systematic review, assessor experience with RoB-SPEO (≤10 assessments versus > 10 assessments), and assessment time (tertiles: ≤25 min versus 26–66 min versus ≥ 67 min). To describe user experience, we synthesised the assessors’ comments and recommendations. Results Assessors reported a median of 40 min to complete one assessment (interquartile range 21–120 min). For all domains, raw inter-rater agreement ranged from 0.54 to 0.82. Agreement varied by systematic review and assessor experience with RoB-SPEO between domains, and increased with increasing assessment time. A small number of users recommended further development of instructions for selected RoB-SPEO domains, especially bias in selection of participants into the study (domain 1) and bias due to differences in numerator and denominator (domain 7). Discussion Overall, our results indicated good agreement across the eight domains of the RoB-SPEO tool. The median assessment time was comparable to that of other risk of bias tools, indicating comparable assessor burden. However, there was considerable variation in time taken to complete assessments. Additional time spent on assessments may improve inter-rater agreement. Further development of the RoB-SPEO tool could focus on refining instructions for selected RoB-SPEO domains and additional testing to assess agreement for different topic areas and with a wider range of assessors from different research backgrounds.