Search

The subjective aspect of standard-setting is often criticized, yet data-driven standard-setting methods are rarely applied. Therefore, we applied a mixture Rasch model approach to setting performance standards across several testing programs of various sizes and compared the results to existing passing standards derived from traditional standard-setting methods. We found that heterogeneity of the sample is clearly necessary for the mixture Rasch model approach to standard setting to be useful. While possibly not sufficient to determine passing standards on their own, there may be value in these data-driven models for providing additional validity evidence to support decision-making bodies entrusted with establishing cut scores. They may also provide a useful tool for evaluating existing cut scores and determining if they continue to be supported or if a new study is warranted.

The COVID-19 pandemic has witnessed a renewed interest in Live Remote Proctoring (LRP), not only as a test availability measure but as a necessity to maintain business continuity. Many certification organizations have correspondingly provided LRP as an option for candidates. This study describes a retrospective, observational pilot study evaluating modality effects for three high-stakes certification examination programs administered concurrently in test center and LRP conditions. Also reported are summaries of a post-examination survey assessing drivers for selection of testing condition and candidate satisfaction, particularly with aspects of the LRP experience. Significant differences were observed in both distributions of test scores and test duration (both higher for test center candidates). Although explanatory variables are not completely understood, the authors offer insights as to factors influencing these outcomes. While satisfaction levels with LRP were reasonably positive, significant technical issues were reported by LRP candidates. The primary drivers for the selection of LRP were safety concerns related to the pandemic, simple convenience, and lack of test center availability.

The purpose of this article is to describe and analyze the psychometric properties of the Learning Orientation Questionnaire (LOQ), which have not been previously published. Psychometric validation involves the accumulation of proper empirical evidence to confirm measurement of the intended construct, and to justify the intended uses of the scales. LOQ is based upon the intentional learning theory, which is a comprehensive, holistic learning theory. Through the expertise of the LOQ's developer, educational researcher, and psychometrician, this article presents evidence of LOQ's reliability and validity according to published best practices for scale development and validation. LOQ is a reliable and valid instrument determining where learners fall along the learning orientation continuum. Education researchers can use this information to support learners to move upward on the learning orientation continuum, improving their inclination to learn, high-order thinking, and life-long learning. [ABSTRACT FROM AUTHOR]

The subjective aspect of standard‐setting is often criticized, yet data‐driven standard‐setting methods are rarely applied. Therefore, we applied a mixture Rasch model approach to setting performance standards across several testing programs of various sizes and compared the results to existing passing standards derived from traditional standard‐setting methods. We found that heterogeneity of the sample is clearly necessary for the mixture Rasch model approach to standard setting to be useful. While possibly not sufficient to determine passing standards on their own, there may be value in these data‐driven models for providing additional validity evidence to support decision‐making bodies entrusted with establishing cut scores. They may also provide a useful tool for evaluating existing cut scores and determining if they continue to be supported or if a new study is warranted. [ABSTRACT FROM AUTHOR]

It is known that the Rasch model is a special two-level hierarchical generalized linear model (HGLM). This article demonstrates that the many-faceted Rasch model (MFRM) is also a special case of the two-level HGLM, with a random intercept representing examinee ability on a test, and fixed effects for the test items, judges, and possibly other facets. This perspective suggests useful modeling extensions of the MFRM. For example, in the HGLM framework it is possible to model random effects for items and judges in order to assess their stability across examinees. The MFRM can also be extended so that item difficulty and judge severity are modeled as functions of examinee characteristics (covariates), for the purposes of detecting differential item functioning and differential rater functioning. Practical illustrations of the HGLM are presented through the analysis of simulated and real judge-mediated data sets involving ordinal responses.

The COVID-19 pandemic has witnessed a renewed interest in Live Remote Proctoring (LRP), not only as a test availability measure but as a necessity to maintain business continuity. Many certification organizations have correspondingly provided LRP as an option for candidates. This study describes a retrospective, observational pilot study evaluating modality effects for three high-stakes certification examination programs administered concurrently in test center and LRP conditions. Also reported are summaries of a post-examination survey assessing drivers for selection of testing condition and candidate satisfaction, particularly with aspects of the LRP experience. Significant differences were observed in both distributions of test scores and test duration (both higher for test center candidates). Although explanatory variables are not completely understood, the authors offer insights as to factors influencing these outcomes. While satisfaction levels with LRP were reasonably positive, significant technical issues were reported by LRP candidates. The primary drivers for the selection of LRP were safety concerns related to the pandemic, simple convenience, and lack of test center availability. [ABSTRACT FROM AUTHOR]

Simulation for education and training in health-care professions has been widely applied. However, its value as an assessment tool for competence is not fully known. Logistical barriers of simulation-based assessments have led some health-care organizations to utilize computer-based case simulations (CCSs) for assessment. This article provides a review of the literature on the identification of psychometrically sound, CCS instruments designed to measure decision-making competence in health-care professionals. CINAHL, MEDLINE, and Ovid databases identified 84 potentially relevant articles published between January 2000 and May 2017. A total of 12 articles met criteria for inclusion in this review. Findings of these 12 articles indicate that summative assessment in health care using CCSs in the form of clinical scenarios is utilized to assess higher order performance aspects of competence in the form of decision-making. Psychometric strength was validated in eight articles and supported by four replication studies. Two of the eight articles reported evidence of construct validity and support the need for evidence based on a theoretical framework. This literature review offers implications for further research on the use of CCS tools as a method for assessment of competence in health-care professionals and the need for psychometric evidence to support it. [ABSTRACT FROM AUTHOR]

The National Board of Certification and Recertification for Nurse Anesthetists (NBCRNA) references all questions on its examinations. Because examination questions should be clinically applicable, the NBCRNA is interested in knowing what reference resources clinicians rely on in their work, so that questions can be based on those or similar resources. In October 2016, the NBCRNA sent an email blast for a survey on reference usage to a weighted random selection of 14,993 student and Certified Registered Nurse Anesthetists from its database. The final response rate to the survey was 21.0% (3,155). Important findings included a greater percentage acknowledging usage of web-based references (90.7%, 95% CI = 89.6%-91.8%) than of textbooks (61.4%, 95% CI =59.7%-63.1%) or journals (62.3%, 95% CI = 60.5%-64.1%), as well as more frequent reported usage of web-based resources than textbooks. The survey also queried overall usage of journals, handbooks, and mobile apps, and of specific subcategories and titles. Characteristics of references that respondents reported as very important included easily accessible, portable, and reputable. Most findings indicated that practicing nurse anesthetists prefer web-based resources to printed textbooks or journals. With some exceptions, that preference held true across age groups and job duties. [ABSTRACT FROM AUTHOR]

The nurse anesthesia profession has been an advocate of credentialing since 1945 when the certification program and initial requirements were introduced. Substantial changes have evolved in the credentialing of health professionals over recent years to include documentation of initial and continuing competence while promoting patient safety. The National Board of Certification and Recertification for Nurse Anesthetists (NBCRNA) has conducted a 3-year investigation into continuing competence as applied to the practice of nurse anesthesia and to other professions. The purpose of this article is to present the essential concepts and current developments related to competence and continuing competence for nurse anesthetists. The NBCRNA continues to seek input from nurse anesthetists and stakeholders in its deliberations on how to define the future requirements for continuing competence. [ABSTRACT FROM AUTHOR]

The purpose of this column is to present the results of the 2007 Professional Practice Analysis (PPA) of the field of nurse anesthesia, conducted by the Council on Certification of Nurse Anesthetists. The PPA used survey and rating scale methodologies to collect data regarding the relative emphasis of various aspects of the nurse anesthesia knowledge domain and competencies. A total of 3,805 survey responses were analyzed using the Rasch rating scale model, which aggregates and transforms ordinal (rating scale) responses into linear measures of relative importance and frequency. Summaries of respondent demographics and educational and professional background are provided, as well as descriptions of how the survey results are used to develop test specifications. The results of this analysis provide evidence for the content outline and test specifications (content percentages) and thus serve as a basis of content validation for the National Certification Examination. [ABSTRACT FROM AUTHOR]

Chronic pain is a growing epidemic in America. Challenges in patients' access to care, and in reimbursement to Certified Registered Nurse Anesthetists (CRNAs) who provide pain services, have resulted in a voluntary subspecialty certification in nonsurgical pain management (NSPM) for CRNAs. An evaluation was conducted of perceptions of CRNAs toward the value of certification in NSPM. An invitation to complete the Perceived Value of Certification Tool (PVCT) was sent to 474 CRNAs who identified the subspecialty practice of NSPM upon application for recertification to the NBCRNA. Data were collected on 18 factors related to the perceived value of certification in the NSPM subspecialty. Exploratory factor analysis using principal components analysis with varimax rotation was conducted to assess the latent structure of the PVCT and to identify potential constructs of CRNAs' perceptions. Reliability was assessed using Cronbach α coefficients. Of 64 CRNAs who provided data, a 3-factor solution emerged that explained 72.25% of the overall variance: personal satisfaction, professional recognition, and competence, each with excellent to good reliability (F1: α = 0.95, F2: α = 0.94, F3: α = 0.88). Identification of the 3 constructs in this study will assist with future efforts of examination validation for the subspecialty of NSPM certification for CRNAs., Competing Interests: The authors have declared no financial relationships with any commercial entity related to the content of this article. The authors did not discuss off-label use within the article., (Copyright© by the American Association of Nurse Anesthetists.)