The inter-rater reliability and individual reviewer performance of the 2012 world heart federation guidelines for the echocardiographic diagnosis of latent rheumatic heart disease.

In 2012, the World Heart Federation (WHF) published guidelines for the echocardiographic diagnosis of rheumatic heart disease (RHD). This study assesses individual reviewer performance and inter-rater agreement and reliability on the presence of any RHD, as well classification of RHD based on the 2012 WHF criteria. Four cardiologists individually reviewed echocardiograms in the context of a randomized clinical trial (ClinicalTrials.gov : NCT03346525) and participated in a blinded adjudication panel. Panel decision was the reference standard for diagnosis. Performance of individual reviewers to panel adjudication was compared through sensitivity and specificity analyses and inter-rater reliability was assessed between individual panelists using Fleiss free marginal multirater kappa. Echocardiograms from 784 children had two independent reports and panel adjudication. The accuracy of independent reviewers for any RHD had high sensitivity (94%, 95% CI 93–95%) and moderate specificity (62%, 95% CI 53–70%). Sensitivity and specificity for definite RHD was 61.3 (95% CI, 55.3–67.1) and 93.1 (95% CI, 91.6–94.4), with 86.8 (84.7–88.7) and 65.8 (61.0–70.4) for borderline RHD. There was moderate inter-rater agreement (κ = 0.66) on the presence of any RHD while agreement for specific 2012 WHF classification was only fair (κ = 0.51). The 2012 WHF guidelines are moderately reproducible when used by expert cardiologists. More cases of RHD were diagnosed by an consensus panel than by individual reviewers. A revision to the criteria is now warranted to further increase the reliability of the WHF criteria. • Compared to the panel consensus, individuals were less likely to diagnose any RHD, as well as morphological abnormalities. • Inter-rater agreement, even amongst experts, for 2012 WHF classification is fair. • Considerable variability remains amongst reviewers utilizing the WHF criteria. • Our data support the use of a > 2 reviewer system for more diagnosis of RHD. • A revision to the 2012 WHF is now warranted. [ABSTRACT FROM AUTHOR]