Search

Background: The literature shows conflicting results regarding inter- and intra-rater reliability, even for the same movement screen. The purpose of this study was to assess inter- and intra-rater reliability of movement scores within and between sessions of expert assessors and the effects of body-shape on reliability during a movement screen using a custom online visualisation software. Methods: Kinematic data from 542 athletes performing seven movement tasks were used to create animations (i.e., avatar representations) using motion and shape capture from sparse markers (MoSh). For each task, assessors viewed a total of 90 animations. Using a custom developed visualisation tool, expert assessors completed two identical sessions where they rated each animation on a scale of 1–10. The arithmetic mean of weighted Cohen's kappa for each task and day were calculated to test reliability. Results: Across tasks, inter-rater reliability ranged from slight to fair agreement and intra-rater reliability had slightly better reliability with slight to moderate agreement. When looking at the average kappa values, intra-rater reliability within session with and without body manipulation and between sessions were 0.45, 0.37, and 0.35, respectively. Conclusions: Based on these results, supplementary or alternative methods should be explored and are likely required to increase scoring objectivity and reliability even within expert assessors. To help future research and practitioners, the custom visualisation software has been made available to the public. [ABSTRACT FROM AUTHOR]

Abstract Background The literature shows conflicting results regarding inter- and intra-rater reliability, even for the same movement screen. The purpose of this study was to assess inter- and intra-rater reliability of movement scores within and between sessions of expert assessors and the effects of body-shape on reliability during a movement screen using a custom online visualisation software. Methods Kinematic data from 542 athletes performing seven movement tasks were used to create animations (i.e., avatar representations) using motion and shape capture from sparse markers (MoSh). For each task, assessors viewed a total of 90 animations. Using a custom developed visualisation tool, expert assessors completed two identical sessions where they rated each animation on a scale of 1–10. The arithmetic mean of weighted Cohen’s kappa for each task and day were calculated to test reliability. Results Across tasks, inter-rater reliability ranged from slight to fair agreement and intra-rater reliability had slightly better reliability with slight to moderate agreement. When looking at the average kappa values, intra-rater reliability within session with and without body manipulation and between sessions were 0.45, 0.37, and 0.35, respectively. Conclusions Based on these results, supplementary or alternative methods should be explored and are likely required to increase scoring objectivity and reliability even within expert assessors. To help future research and practitioners, the custom visualisation software has been made available to the public.

Abstract Background The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses. Methods This paper evaluates a recent publication by Chen et al., featured in BMC Nephrology, aiming to introduce an alternative statistical approach to assessing IRR and discuss its statistical properties. The study underscores the crucial need for selecting appropriate Kappa statistics, emphasizing the accurate computation, interpretation, and reporting of commonly used IRR statistics between two raters. Results The Cohen’s Kappa statistic is typically used for two raters dealing with two categories or for unordered categorical variables having three or more categories. On the other hand, when assessing the concordance between two raters for ordered categorical variables with three or more categories, the commonly employed measure is the weighted Kappa. Conclusion Chen and colleagues might have underestimated the agreement between AU5800 and UN2000. Although the statistical approach adopted in Chen et al.’s research did not alter their findings, it is important to underscore the importance of researchers being discerning in their choice of statistical techniques to address their specific research inquiries.

Background: Numerous cardiovascular risk prediction models (RPM) have been developed, however, agreement studies between these models are scarce. We aimed to assess the inter-model agreement between eight RPMs: assessing cardiovascular risk using SIGN, the Australian CVD risk score (AusCVDRisk), the Framingham Risk Score for Hard Coronary Heart Disease, the Multi-Ethnic Study of Atherosclerosis risk score, the Pooled Cohort Equation (PCE), the QRISK3 cardiovascular risk calculator, the Reynolds Risk Score, and Systematic Coronary Risk Evaluation-2 (SCORE2). Methods: A cross-sectional study was conducted on 11,174 40–65-year-old individuals with diagnosed metabolic syndrome from a single tertiary university hospital in Lithuania. Cardiovascular risk was calculated using the eight RPMs, and the results were categorized into high, intermediate, and low-risk groups. Inter-model agreement was quantified using Cohen's Kappa coefficients. Results: The study revealed significant heterogeneity in risk categorizations with only 1.49% of cases where all models agree on the risk category. SCORE2 predominantly categorized participants as high-risk (67.39%), while the PCE identified the majority as low-risk (62.03%). Cohen's Kappa coefficients ranged from −0.09 to 0.64, indicating varying degrees of inter-model agreement. Conclusions: The choice of RPM can substantially influence clinical decision-making and patient management. The PCE and AusCVDRisk models exhibited the highest degree of agreement while the SCORE2 model consistently exhibited low agreement with other models. [ABSTRACT FROM AUTHOR]

Background: The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses. Methods: This paper evaluates a recent publication by Chen et al., featured in BMC Nephrology, aiming to introduce an alternative statistical approach to assessing IRR and discuss its statistical properties. The study underscores the crucial need for selecting appropriate Kappa statistics, emphasizing the accurate computation, interpretation, and reporting of commonly used IRR statistics between two raters. Results: The Cohen's Kappa statistic is typically used for two raters dealing with two categories or for unordered categorical variables having three or more categories. On the other hand, when assessing the concordance between two raters for ordered categorical variables with three or more categories, the commonly employed measure is the weighted Kappa. Conclusion: Chen and colleagues might have underestimated the agreement between AU5800 and UN2000. Although the statistical approach adopted in Chen et al.'s research did not alter their findings, it is important to underscore the importance of researchers being discerning in their choice of statistical techniques to address their specific research inquiries. [ABSTRACT FROM AUTHOR]

From 2010 to 2015, our school of education used the Performance Assessment for California Teachers (PACT), a summative assessment designed to assess preservice teacher competence. Candidate portfolios were uploaded to an evaluation portal, and trained evaluators assigned a final score of Pass or Fail to the work samples. Three consensus estimates of inter-rater reliability of 181 candidate portfolios that were either double- or triple-scored were computed. Two chance-corrected estimates of inter-rater reliability (Cohen's kappa and Gwet's AC1) and percent agreement were computed and calculated within five content areas: elementary math, secondary history/social science, math, science, and English language arts. An initial Pass or Fail score was not more likely to be followed by either a Pass or Fail score given by a subsequent evaluator. Inter-rater reliability was interpreted as being low across all content areas that were examined. None of the percent agreement coefficients attained the minimum standard of 0.700 for consensus agreement. Increasing research access to proprietary double-scored data would lead to an increased understanding of, and perhaps improvement in, teacher performance assessments. [ABSTRACT FROM AUTHOR]

In this article, we introduce the concepts of taxicab and uniform products in the context of dual curves associated with Cohen's kappa, primarily defined by a set of inflection curvatures of an ellipse and a circle using parallel asymptotes. The novel curve under scrutiny, denominated as the "Like-Bulb Filament" (LBF) curve, is delineated as the locus of dual vertices originating from a couple of conic curvatures. The emergence of LBF transpires through the orchestrated arrangement of line segments emanating from a predetermined central focal point upon an elliptical form concomitant with a circular entity possessing a radius equivalent to the ellipse's minor axis. The LBF’s curve is intricately choreographed through the dynamic interplay of a constant unit circle and three asymptotic lines. Notably, two of these asymptotes achieve tangential intersections with the LBF curve, while the third gracefully traverses its central core. Additionally, we embark on a comprehensive algebraic examination complemented by a geometrically informed construction methodology. In these instances, a consistent conic curvature of the uint circle and an elliptical structure assume pivotal roles in the genesis of the LBF’s curve. Also, a geometric connection is speculated between these curve configurations and their relevance to engineering processes across fields. However, the document acknowledges the need for more intensive study on the presented traits. Hence, it emphasizes addressing the existing research gap in subsequent investigations.

Copyright of Estudios sobre el Mensaje Periodistico is the property of Universidad Complutense de Madrid and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Abstract Background In research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses’ statistical power for hypothesis testing. Methods This article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved. Results The Cohen’s Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure. Conclusion Despite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research.

Through various strategies and learning models, efforts have been made to improve students' critical thinking skills in elementary and secondary school science classrooms. However, there are still a few research results that show efforts to improve critical thinking skills through Higher Other Thinking Skill (HOTS)-based science questions. Therefore, this study aims to test the effectiveness of students' habituation to HOTS-based science questions in improving students' critical thinking skills in science. In this study, students who were exposed to the HOTS-based science questions and who were not are being compared. The method used is an experimental method with a Pretest Posttest Non-Equivalent Control Group Design. This study involved 165 students from one of the Public Junior High Schools in Banda Aceh, Indonesia. To collect data related to critical thinking skills, a pretest, and posttest were administered, where each test used a long answer HOTS-based science questions. The data obtained from the test were then analyzed using descriptive and inferential statistical techniques. This study found that the habituation of HOTS-based science questions can effectively improve students' critical thinking, indicated by a significant difference in critical thinking skills scores in science between the control (n=81) and the experimental classes (n=84). The results of this study are expected to provide an overview and benefits for practitioners of science education to improve critical thinking skills in science using HOTS questions. [ABSTRACT FROM AUTHOR]

Several measures of agreement, such as the Perreault–Leigh coefficient, the AC 1 , and the recent coefficient of van Oest, are based on explicit models of how judges make their ratings. To handle such measures of agreement under a common umbrella, we propose a class of models called guessing models, which contains most models of how judges make their ratings. Every guessing model have an associated measure of agreement we call the knowledge coefficient. Under certain assumptions on the guessing models, the knowledge coefficient will be equal to the multi-rater Cohen's kappa, Fleiss' kappa, the Brennan–Prediger coefficient, or other less-established measures of agreement. We provide several sample estimators of the knowledge coefficient, valid under varying assumptions, and their asymptotic distributions. After a sensitivity analysis and a simulation study of confidence intervals, we find that the Brennan–Prediger coefficient typically outperforms the others, with much better coverage under unfavorable circumstances. [ABSTRACT FROM AUTHOR]

From 2010 to 2015, our school of education used the Performance Assessment for California Teachers (PACT), a summative assessment designed to assess preservice teacher competence. Candidate portfolios were uploaded to an evaluation portal, and trained evaluators assigned a final score of Pass or Fail to the work samples. Three consensus estimates of inter-rater reliability of 181 candidate portfolios that were either double- or triple-scored were computed. Two chance-corrected estimates of inter-rater reliability (Cohen’s kappa and Gwet’s AC1) and percent agreement were computed and calculated within five content areas: elementary math, secondary history/social science, math, science, and English language arts. An initial Pass or Fail score was not more likely to be followed by either a Pass or Fail score given by a subsequent evaluator. Inter-rater reliability was interpreted as being low across all content areas that were examined. None of the percent agreement coefficients attained the minimum standard of 0.700 for consensus agreement. Increasing research access to proprietary double-scored data would lead to an increased understanding of, and perhaps improvement in, teacher performance assessments.

There are many annotation methods for the English language based on adapting an argumentation model according to the study domain. However, as far as research has been done, there are no annotation methods for detecting argumentative content in Spanish, not only due to the complexity of identifying the evidence but also because of the lack of data available for this task. The research aims to present and evaluate an annotation method consisting of an adapted argumentation model, an annotation guide, and an annotation process based on Twitter data analysis. The Inter Annotator Agreement (IAA) study achieves 0.63 Fleiss Kappa for Argument/Non-Argument tagging, 0.35 Fleiss Kappa for Argument Component tagging, and 0.53 Fleiss Kappa for Non-Argument Component tagging, while the best Cohen's kappa (k) index achieved, was 0.73, 0.52 and 0.75 respectively. The results' assessment highlights the need to include linguistic segmentation rules for the second annotation task. It is crucial to use discourse markers for the claim and evidence detection. For the first annotation task, it determined that if the prevalence index and the bias index are very low, the prevalence index predominates over the bias index because k increases (0.52= 0.60, agreement and disagreement tables, and confusion matrices code are available on Mendeley Data Repository.

Background: In research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses' statistical power for hypothesis testing. Methods: This article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved. Results: The Cohen's Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure. Conclusion: Despite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research. [ABSTRACT FROM AUTHOR]

Scientific studies of the elements that influence the box office performance of Indian films have generally concentrated on post-production elements, such as those discovered after a film has been completed or released, and notably for Bollywood films. Only fewer studies have looked at regional film industries and pre-production factors, which are elements that are known before a decision to greenlight a film is made. This study looked at Indian films using natural language processing and machine learning approaches to see if they would be profitable in the pre-production stage. We extract movie data and English subtitles (as an approximation to the screenplay) for the top five Indian regional film industries: Bollywood, Kollywood, Tollywood, Mollywood, and Sandalwood, as they make up a major portion of the Indian film industry's revenue. Subtitle Vector (Sub2Vec), a Paragraph Vector model trained on English subtitles, was used to embed subtitle text into 50 and 100 dimensions. The proposed approach followed a two-stage pipeline. In the first stage, Return on Investment (ROI) was calculated using aggregated subtitle embeddings and associated movie data. Classification models used the ROI calculated in the first step to predicting a film's verdict in the second step. The optimal regressor–classifier pair was determined by evaluating classification models using F 1 -score and Cohen's Kappa scores on various hyperparameters. When compared to benchmark methods, our proposed methodology forecasts box office success more accurately. [ABSTRACT FROM AUTHOR]

Artificial Intelligence (AI)-based medical computer vision algorithm training and evaluations depend on annotations and labeling. However, variability between expert annotators introduces noise in training data that can adversely impact the performance of AI algorithms. This study aims to assess, illustrate and interpret the inter-annotator agreement among multiple expert annotators when segmenting the same lesion(s)/abnormalities on medical images. We propose the use of three metrics for the qualitative and quantitative assessment of inter-annotator agreement: 1) use of a common agreement heatmap and a ranking agreement heatmap; 2) use of the extended Cohen’s kappa and Fleiss’ kappa coefficients for a quantitative evaluation and interpretation of inter-annotator reliability; and 3) use of the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm, as a parallel step, to generate ground truth for training AI models and compute Intersection over Union (IoU), sensitivity, and specificity to assess the inter-annotator reliability and variability. Experiments are performed on two datasets, namely cervical colposcopy images from 30 patients and chest X-ray images from 336 tuberculosis (TB) patients, to demonstrate the consistency of inter-annotator reliability assessment and the importance of combining different metrics to avoid bias assessment.

Abstract Background In inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table. Methods In this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen’s kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone. Results The sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for $$3\times 3$$ 3 × 3 and $$4 \times 4$$ 4 × 4 tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies. Conclusions The accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables.

Cohen's and Fleiss's kappa are popular estimators for assessing agreement among two and multiple raters, respectively, for a binary response. While additional methods have been developed to account for multiple raters and covariates, they are not always applicable, rarely used, and none simplify to Cohen's kappa. Furthermore, there are no methods to simulate Bernoulli observations under the kappa agreement structure such that the developed methods could be adequately assessed. This manuscript overcomes these shortfalls. First, we developed a model‐based estimator for kappa that accommodates multiple raters and covariates through a generalized linear mixed model and encompasses Cohen's kappa as a special case. Second, we created a framework to simulate dependent Bernoulli observations that upholds all 2‐tuple pair of rater's kappa agreement structure and includes covariates. We used this framework to assess our method when kappa was nonzero. Simulations showed that Cohen's and Fleiss's kappa estimates were inflated unlike our model‐based kappa. We analyzed an Alzheimer's disease neuroimaging study and the classic cervical cancer pathology study. The proposed model‐based kappa and advancement in simulation methodology demonstrates that the popular approaches of Cohen's and Fleiss's kappa are poised to yield invalid conclusions while our work overcomes shortfalls, leading to improved inferences. [ABSTRACT FROM AUTHOR]

INTRODUCTION: The Workplace Safety Culture Questionnaire (WSCQ) was developed to assess the level of knowledge, attitudes and practices on Workplace Safety Culture (WSC) among public sector office workers in Nigeria. The main objective of this study was to determine the validity and reliability of the new Workplace Safety Culture Questionnaire (WSCQ) using the Information Motivation and Behavior (IMB) model. MATERIALS AND METHODS: The WSCQ questionnaire was completed twice by the respondents themselves, with an interval of ten days between sessions, to assess the accuracy of the initial results with the retest. This study involved 44 participants. RESULTS: Cronbach's alpha showed significant item consistency for each construct in the second pilot study. The information construct (Cronbach's a=0.929 and minimum corrected item-total correlation (CITC)=0.399). The motivation construct (Cronbach's a=0.932 and minimum corrected item-total correlation (CITC)=0.450). The behavior construct (Cronbach's a=0.812 and minimum corrected item-total correlation (CITC)= 0.401). In test-retest reliability, Cohen's kappa coefficient for construct of information for all items was almost 70% between (kappa k=0.689-1000, p<0.001). The intraclass correlation coefficient (ICC) for the construct of motivation for all items is 90% between (ICC=0.810-1000, p<0.001) and (kappa k=0.944-1000, p<0.001) for the construct of behavior was 71% for all items. CONCLUSION: This study introduces the WSCQ using the IMB model among public sector office workers as the first official validation of the WSCQ. [ABSTRACT FROM AUTHOR]

In our research, we investigated the surface cover of the Karcsa Lake, a relic of the former Tisza riverbed, by digitising and evaluating images from several dates (1966, 2000, 2005, 2007, 2011, 2020) over the past 50 years. In addition to monitoring the changes in land cover over the past decades, our results have been used to draw conclusions about the extent to which the lake environment may be pressuring the lake and the proportion of ecologically valuable areas. By evaluating the above, we can draw conclusions about the future of a valuable lake. [ABSTRACT FROM AUTHOR]

Copyright of Revista Signos is the property of Pontificia Universidad Catolica de Valparaiso and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Abstract Background The purpose of this article is to discuss the statistical methods for agreement analysis used in Richelle’s article (BMC Med Educ 22:335, 2022). The authors investigated the attitudes of final-year medical students regarding substance use during pregnancy and identified the factors that influence these attitudes. Methods We found that Cohen’s kappa value for measuring the agreement between these medical students’ attitudes towards drugs/alcohol use during pregnancy was questionable. In addition, we recommend using weighted kappa instead of Cohen’s kappa for agreement analysis at the presence of three categories. Results The agreement improved from “good” (Cohen’s kappa) to “very good” (weighted kappa) for medical students’ attitudes towards drugs/alcohol use during pregnancy. Conclusion To conclude, we recognize that this does not significantly alter the conclusions of the Richelle et al. paper, but it is necessary to ensure that the appropriate statistical tools are used.

Abstract Background Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters’ random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies. Methods We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested. Results The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r 2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott’s π, Cohen’s κ and Krippendorff’s α, underperformed all other indices, reporting directional r 2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet’s AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al’s S ranked behind AC1, and Perreault and Leigh’s Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating. Conclusion The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Background: In inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table. Methods: In this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone. Results: The sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for 3 × 3 and 4 × 4 tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies. Conclusions: The accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables. [ABSTRACT FROM AUTHOR]

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters. [ABSTRACT FROM AUTHOR]

To evaluate inter observer agreement between Radiologist and Otorhinolaryngologists in identifying important structures and pathologies on pre operative computed tomography (CT) scans of paranasal sinus (PNS) in chronic rhinosinusitis (CRS). A retrospective review of CT scans PNS of CRS patients. Archived CT scans were evaluated by 3 observers, similarly experienced radiologist and otorhinolaryngologist and a less experienced otorhinolarygologist. The degree of intra- as well as inter observer agreement was assessed by Cohen's kappa statistics. A moderate and a fair inter observer agreement was noted among experienced radiologist and otorhinolaryngologist; & experienced experts and less experienced otorhinolaryngologist respectively. The greatest disagreements among all observers were found in superior attachment of uncinate process, optic nerve's relationship with sphenoid sinus and dehiscence of anterior ethmoid canal. The present study demonstrated a considerable inter observer variability among radiologists and otorhinolaryngologists, irrespective of their experience in the field of speciality. A few critically important structures which showed an unsatisfactory level of agreement and thus need to be sought after with more training and practice, were also identified. These included Keros type, optic nerve's location, dehiscence of anterior ethmoid artery, vidian and maxillary nerves, lamina papyracia defect, sphenoid sinus pneumatization and attachment of uncinate process. The study also emphasized the importance of inter disciplinary conferences for pre operative evaluation of sinus CT scans of CRS patients. [ABSTRACT FROM AUTHOR]

Abstract Background The reliability of Criteria for Assessing Prescription Quality in Chinese Hospitals (CAPQCH) has never been rigorously verified. This study was designed to verify the reliability of the CAPQCH among pharmacists in China. Methods Fourteen pharmacists, 5 from hospitals and 9 from the communities were recruited. We randomly selected 200 prescriptions, and made the testing prescriptions including appropriate and inappropriate testing prescriptions. Pharmacists assessed these testing prescriptions according to criteria in CAPQCH. Three test sets (Set 1, Set 2, and Set 3) were evaluated at 6-month intervals. Before administration of Set 3, pharmacists were informed that achievement on Set 3 would be reflected in their performance appraisal. We also evaluated the performance based on prescription comments before and after combining several confusing criteria. Cohen’s Kappa statistic, Fleiss’ Kappa statistic, and accuracy were employed to evaluate reliability among pharmacists. Results Median values of Cohen’s Kappa were 0.61 in Set 1, 0.66 in Set 2, and 0.80 in Set 3; reliability is thus substantial. Our data indicate no significant differences between Set 1 and Set 2, whereas Set 3 indicates significantly improved performance. Moreover, combinations of confusing criteria contributed little to improvement of performance in prescription comments. Conclusion Our results verified the reliability of CAPQCH application by working pharmacists. Adding performance based on prescription comments to personal appraisals was effective in improving the quality of prescription comments. These findings may be useful when future modification of the CAPQCH is considered. Moreover, this study contributes to improving the understanding of the prescription assessment situation in China.

Abstract We read with great interest the article by Hendershot et al. (Parasit Vectors 14:473, 2021). The authors compared a PCR method for detecting Plasmodium vivax’s mitochondrial (mt) cytochrome oxidase I (COX-I) gene with the current “gold standard” circumsporozoite (CSP) ELISA for detecting circumsporozoite protein for identification of different life stages of Plasmodium vivax during development within Anopheles arabiensis. We found that Cohen’s kappa value for measuring the agreement between mt COX-I PCR and CSP ELISA was questionable. In addition, we recommend a more appropriate statistical method in this article. In short, any scientific conclusion requires support by the reasonable application of methodological and statistical methods.

The paper investigates changing levels of online concern about the Kashmiri Pandit migration of the 1990s on Twitter. Although decades old, this movement of people is an ongoing issue in India, with no current resolution. Analysing changing reactions to it on social media may shed light on trends in public attitudes to the event. Tweets were downloaded from Twitter using the academic version of its application programming interface (API) with the aid of the free social media analytics software Mozdeh. A set of 1000 tweets was selected for content analysis with a random number generator in Mozdeh. The results show that the number of tweets about the issue has increased over time, mainly from India, and predominantly driven by the release of films like Shikara and The Kashmir Files. The tweets show apparent universal support for the Pandits but often express strong emotions or criticize the actions of politicians, showing that the migration is an ongoing source of anguish and frustration that needs resolution. The results also show that social media analysis can give insights even into primarily offline political issues that predate the popularity of the web, and can easily incorporate international perspectives necessary to understand complex migration issues. [ABSTRACT FROM AUTHOR]

Malaria remains a major public health challenge worldwide. In order to ensure a prompt and accurate malaria diagnosis, the World Health Organization recommended the confirmatory parasitological diagnosis of malaria by microscopy and malaria rapid diagnostic test (RDT) prior to antimalarial administration and treatment. This study was designed to evaluate the performance of nested polymerase chain reaction (nested PCR), light microscopy, and Plasmodium falciparum histidine-rich protein 2 rapid diagnostic test (PfHRP2 RDT) in the detection of falciparum malaria in Akure, Nigeria. A cross-sectional and hospital-based study involving 601 febrile volunteer participants was conducted in Akure, Nigeria. Approximately 2–3 mL venous blood samples were obtained from each study participant for parasitological confirmation by microscopy and PfHRP2-based malaria RDT. Thick and thin films were prepared and viewed under the light microscope for parasite detection, parasite density quantification, and species identification, respectively. Dry blood spot samples were prepared on 3MM Whatman filter paper for nested PCR. The overall prevalence of microscopy, PfHRP2 RDT, and nested PCR were 64.89% (390/601), 65.7% (395/601), and 67.39% (405/601), respectively. The estimates of sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and Youden's j index of microscopy and RDT were 96.30, 100.00, 100.00, 92.89, 97.50, 0.963, and 95.06, 94.90, 97.47, 90.29, 95.01, and 0.899, respectively. Malaria RDT recorded higher false negativity, compared microscopy (4.94% vs. 3.70%). A near perfect agreement was reported between microscopy and nested PCR, and between PfHRP2 RDT and nested PCR with Cohen's kappa (k) values of 0.94 and 0.88, respectively. This study revealed that PfHRP2 RDT and microscopy continues to remain sensitive and specific for falciparum malaria diagnosis in the study area. [ABSTRACT FROM AUTHOR]

Several studies were conducted previously on fuel and biofuel performance of micro-turbines. The present paper combines experimental and statistical approaches to study the vibrational behavior of a gas micro-turbine supplied with different pure fuels and admixed with rapeseed oils. Experimental tests carried out at different operating conditions have allowed us to build a classification model through using discriminant analysis. The classification model can distinguish the vibrational behavior occurring when the turbine is fueled with kerosene, or pure and admixed diesel with rapeseed oil. Moreover, the methodology has even allowed us to highlight differences in vibrational behavior caused by small amounts of rapeseed oil admixed in the fuel. The model reliability, in terms of Cohen's kappa, results in optimal data classification. [ABSTRACT FROM AUTHOR]

Background: Interrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.Methods: We conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.Results: The most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.Conclusion: The authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement. [ABSTRACT FROM AUTHOR]

This article deals with the design of appropriate measures that had to be taken for the implemented measurement system. The measurement result was significantly negatively affected by several factors. Forty-five samples of shafts used in the production of surgical drills were measured. Measurements were performed by metrological appraisers with a calibrated digital micrometer. Measurement and subsequent data processing revealed low process capability (CP and PP indices). A large portion of the shafts had an observed size below the lower specific limit (LSL). Therefore, it was necessary to take corrective action. This paper focuses on the corrective measures implemented in the measurement system. The micrometer met the requirements of the standard and was metrologically capable. The shafts were measured by eight metrological appraisers, so attention was focused on the potential impact of the metrological appraiser. The measured data were evaluated by uncertainty analysis, paired t-tests, measurement systems analysis (MSA) and Cohen's kappa. The number of non-compliant shafts was shown to increase with decreasing measurement capability. The measurement system was evaluated as conditionally capable, even incapable. One possibility was to identify the optimal pair of metrological appraisers. The pair of metrological appraisers E and F appeared to be the most suitable combination for most methods. Due to the relatively high %EV index, the second option was to improve the work with the measuring instrument, that is, improve the training and supervision of metrological appraisers in the measurement process. Repeated measurements by the pair with the highest capability (metrological appraisers E and F) resulted in an increase in the value of the capability indices and a decrease in the number of shafts out of tolerance for the same shafts. As the value of these indices was lower than 1.33 during repeated measurements, corrective measures had to be taken, not only in the measurement system, but also in the production system. [ABSTRACT FROM AUTHOR]

The need to measure the degree of agreement among R raters who independently classify n subjects within K nominal categories is frequent in many scientific areas. The most popular measures are Cohen's kappa (R = 2), Fleiss' kappa, Conger's kappa and Hubert's kappa (R ≥ 2) coefficients, which have several limitations. In 2004, the delta coefficient was defined for the case of R = 2, which did not have the limitations of Cohen's kappa coefficient. This article extends the coefficient delta from R = 2 raters to R ≥ 2 (coefficient multi-rater delta), demonstrating that it can be expressed in the kappa format and has the same advantages as the coefficient delta with regard to the type kappa classic coefficients: (i) it refers to the proportion of replies that are concordant not by chance; (ii) allow to obtain a parameter that faithfully measures the degree of agreement in each category; and (iii) it is not affected by the marginal imbalance. [ABSTRACT FROM AUTHOR]

With the intention to evaluate a current h(fa)-index which considered the relative contribution of each author, Sebo P and colleagues selected senior hospital physicians from six college hospitals in Switzerland between 1 and 14 March 2020. Using Web of Science, the authors assessed the h(fa)-index of all physicians with acquired non-zero h-index. They applied Cohen's kappa statistic to calculate the inter-rater agreement between the h-index and the h(fa)-index. We argue that the agreement between the results of the two metrics is disputed by Cohen's kappa statistic. Therefore, we recommend that an additional appropriate statistical approach, the weighted Cohen's kappa statistic, should be used in this study. [ABSTRACT FROM AUTHOR]

Background: The reliability of Criteria for Assessing Prescription Quality in Chinese Hospitals (CAPQCH) has never been rigorously verified. This study was designed to verify the reliability of the CAPQCH among pharmacists in China.Methods: Fourteen pharmacists, 5 from hospitals and 9 from the communities were recruited. We randomly selected 200 prescriptions, and made the testing prescriptions including appropriate and inappropriate testing prescriptions. Pharmacists assessed these testing prescriptions according to criteria in CAPQCH. Three test sets (Set 1, Set 2, and Set 3) were evaluated at 6-month intervals. Before administration of Set 3, pharmacists were informed that achievement on Set 3 would be reflected in their performance appraisal. We also evaluated the performance based on prescription comments before and after combining several confusing criteria. Cohen's Kappa statistic, Fleiss' Kappa statistic, and accuracy were employed to evaluate reliability among pharmacists.Results: Median values of Cohen's Kappa were 0.61 in Set 1, 0.66 in Set 2, and 0.80 in Set 3; reliability is thus substantial. Our data indicate no significant differences between Set 1 and Set 2, whereas Set 3 indicates significantly improved performance. Moreover, combinations of confusing criteria contributed little to improvement of performance in prescription comments.Conclusion: Our results verified the reliability of CAPQCH application by working pharmacists. Adding performance based on prescription comments to personal appraisals was effective in improving the quality of prescription comments. These findings may be useful when future modification of the CAPQCH is considered. Moreover, this study contributes to improving the understanding of the prescription assessment situation in China. [ABSTRACT FROM AUTHOR]

Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F1 score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen’s Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen’s Kappa and the Brier score to evaluate binary classifications.

Alzheimer’s Disease (AD) is the most common cause of dementia globally. It steadily worsens from mild to severe, impairing one’s ability to complete any work without assistance. It begins to outstrip due to the population ages and diagnosis timeline. For classifying cases, existing approaches incorporate medical history, neuropsychological testing, and Magnetic Resonance Imaging (MRI), but efficient procedures remain inconsistent due to lack of sensitivity and precision. The Convolutional Neural Network (CNN) is utilized to create a framework that can be used to detect specific Alzheimer’s disease characteristics from MRI images. By considering four stages of dementia and conducting a particular diagnosis, the proposed model generates high-resolution disease probability maps from the local brain structure to a multilayer perceptron and provides accurate, intuitive visualizations of individual Alzheimer’s disease risk. To avoid the problem of class imbalance, the samples should be evenly distributed among the classes. The obtained MRI image dataset from Kaggle has a major class imbalance problem. A DEMentia NETwork (DEMNET) is proposed to detect the dementia stages from MRI. The DEMNET achieves an accuracy of 95.23%, Area Under Curve (AUC) of 97% and Cohen’s Kappa value of 0.93 from the Kaggle dataset, which is superior to existing methods. We also used the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset to predict AD classes in order to assess the efficacy of the proposed model.

Background: Vitamin A deficiency (VAD) and anemia are the most prevalent nutritional deficiency in children globally. The dried blood spot (DBS) method has been used in prevalence studies of VAD and anemia in different age groups. However, it has not yet been validated for children.Objectives: This study aimed to assess the reproducibility and validity of DBS in the diagnosis of VAD and anemia in preschoolers.Methods: Venous and capillary blood samples were collected from a representative sample of children <5 y old who attended the public health system in Rio de Janeiro. Serum retinol and hemoglobin were measured in 235 and 182 children, respectively. Serum retinol was measured with HPLC and hemoglobin was measured with spectrophotometry in samples of venous (gold standard) and capillary blood (test method, DBS). DBS reproducibility was assessed with the intraclass correlation coefficient (ICC), κ, and prevalence-adjusted and bias-adjusted κ (PABAK). DBS validity was assessed with sensitivity, specificity, accuracy index (AI), positive predictive value (PPV), and negative predictive value (NPV).Results: DBS showed very good reproducibility for serum retinol (ICC = 0.94, κ = 0.83, PABAK = 0.76) and very good/good reproducibility for hemoglobin (ICC = 0.86, κ = 0.69, PABAK = 0.69). Prevalence rates for VAD by the reference and test methods were 11.5% and 11.9%, respectively, whereas the anemia rates were 19.2% and 46.2%. The test method showed low sensitivity (33%) and PPV (32%) and high specificity (91%) and NPV (92%) for serum retinol. For hemoglobin, the test method showed fair sensitivity (71%), low PPV (30%), fair specificity (60%), and high NPV (90%). AI was 83% for VAD and 62% for anemia.Conclusions: The results suggest that DBS is adequate for the diagnosis of VAD in preschool children, but not for anemia. [ABSTRACT FROM AUTHOR]