Search

Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example.The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review.Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories.The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Medical education researchers are often subject to challenges that include lack of funding, collaborators, study subjects, and departmental support. The construct of a research lab provides a framework that can be employed to overcome these challenges and effectively support the work of medical education researchers; however, labs are relatively uncommon in the medical education field. Using case examples, the authors describe the organization and mission of medical education research labs contrasted with those of larger research team configurations, such as research centers, collaboratives, and networks. They discuss several key elements of education research labs: the importance of lab identity, the signaling effect of a lab designation, required infrastructure, and the training mission of a lab. The need for medical education researchers to be visionary and strategic when designing their labs is emphasized, start-up considerations and the likelihood of support for medical education labs is considered, and the degree to which department leaders should support such labs is questioned.

Background Professionalism is a core competency that is difficult to assess. We examined the incidence of publication inaccuracies in Electronic Residency Application Service applications to our training program as potential indicators of unprofessional behavior. Study Design We reviewed all 2019-2020 National Resident Matching Program applicants being considered for interview. Applicant demographic characteristics recorded included standardized examination scores, gender, medical school, and medical school ranking (2019 US News & World Report). Publication verification by a medical librarian was performed for peer-reviewed journal articles/abstracts, peer-reviewed book chapters, and peer-reviewed online publications. Inaccuracies were classified as “nonserious” (eg incorrect author order without author rank promotion) or “serious” (eg miscategorization, non-peer-reviewed journal, incorrect author order with author rank promotion, nonauthorship of cited existing publication, and unverifiable publication). Multivariate logistic regression analysis was performed for demographic characteristics to identify predictors of overall inaccuracy and serious inaccuracy. Results Of 319 applicants, 48 (15%) had a total of 98 inaccuracies; after removing nonserious inaccuracies, 37 (12%) with serious inaccuracies remained. Seven publications were reported in predatory open access journals. In the regression model, none of the variables (US vs non-US medical school, gender, or medical school ranking) were significantly associated with overall inaccuracy or serious inaccuracy. Conclusions One in 8 applicants (12%) interviewing at a general surgery residency program were found to have a serious inaccuracy in publication reporting on their Electronic Residency Application Service application. These inaccuracies might represent inattention to detail or professionalism transgressions.

PURPOSE To determine the effect of patient debrief interviews on pediatric clerkship student depth of reflection and learning. METHOD The authors conducted a multi-institutional, mixed-methods, cluster randomized trial among pediatric clerkship students from May 2016 to February 2017. Intervention students completed a debrief interview with a patient-caregiver, followed by a written reflection on the experience. Control students completed a written reflection on a memorable patient encounter. Three blinded authors scored written reflections according to the 4-level REFLECT rubric to determine depth of reflection. Interrater reliability was examined using kappa. REFLECT scores were analyzed using a chi-square test; essays were analyzed using content analysis. RESULTS Eighty percent of eligible students participated. One hundred eighty-nine essays (89 control, 100 intervention) were scored. Thirty-seven percent of the control group attained reflection and critical reflection, the 2 highest levels of reflection, compared with 71% of the intervention group; 2% of the control group attained critical reflection, the highest level, compared with 31% of the intervention group (χ(3, N = 189) = 33.9, P < .001). Seven themes were seen across both groups, 3 focused on physician practice and 4 focused on patients. Patient-centered themes were more common in the intervention group, whereas physician-focused themes were more common in the control group. CONCLUSIONS Patient debrief interviews offer a unique approach to deepen self-reflection through direct dialogue and exploration of patient-caregiver experiences during hospitalization.

A key question in simulation-based education is how to maximize learning within time and resource limits, including how best to balance hands-on practice versus reflective debriefing. Several instructional design frameworks suggest setting the balance according to the type of learning objective(s); however, broad professional activities such as team-based cardiopulmonary resuscitation include several interrelated component skills. This study experimentally manipulated hands-on practice versus reflective debriefing for cardiopulmonary resuscitation skills, hypothesizing that the former best supports learning taskwork (eg, compression quality), whereas the latter best supports learning teamwork.The study was a randomized comparison trial with a pretest and posttest. Twenty-six teams of 5 to 6 first-year residents underwent either "drill" practice of key resuscitation phases, designed to maximize deliberate practice opportunities for individual and team skills, or "scrimmage" practice, designed to maximize full-scenario rehearsals and reflective debriefs. Key taskwork and teamwork behaviors were coded, and compression quality was collected and analyzed from an accelerometer.Most performance parameters improved considerably from a pretest to posttest for both taskwork (eg, percent correct compression depth 62%-81%, P = 0.01) and teamwork (eg, role leadership, 47%-70%, P = 0.00). Only 2 parameters improved differently by condition, favoring "drill" training: checking "Do Not Actively Resuscitate" wristband (odds ratio = 14.75, P = 0.03) and use of compression adjuncts (estimated marginal means = 75% versus 67%, P = 0.03).Consistent with the notion that component skills in resuscitation do not clearly and exclusively constitute "taskwork" versus "teamwork," both instructional designs led to similar improvements despite differences in the balance between hands-on practice versus reflection.

Introduction Within simulation-based mastery learning (SBML) courses, there is inconsistent inclusion of learner pretesting, which requires considerable resources and is contrary to popular instructional frameworks. However, it may have several benefits, including its direct benefit as a form of deliberate practice and its facilitation of more learner-specific subsequent deliberate practice. We consider an unexplored potential benefit of pretesting: its ability to predict variable long-term learner performance. Methods Twenty-seven residents completed an SBML course in central line insertion. Residents were tested on simulated central line insertion precourse, immediately postcourse, and after between 64 and 82 weeks. We analyzed pretest scores' prediction of delayed test scores, above and beyond prediction by program year, line insertion experiences in the interim, and immediate posttest scores. Results Pretest scores related strongly to delayed test scores (r = 0.59, P = 0.01; disattenuated ρ = 0.75). The number of independent central lines inserted also related to year-delayed test scores (r = 0.44, P = 0.02); other predictors did not discernibly relate. In a regression model jointly predicting delayed test scores, pretest was a significant predictor (β = 0.487, P = 0.011); number of independent insertions was not (β = 0.234, P = 0.198). Conclusions This study suggests that pretests can play a major role in predicting learner variance in learning gains from SBML courses, thus facilitating more targeted refresher training. It also exposes a risk in SBML courses that learners who meet immediate mastery standards may be incorrectly assumed to have equal long-term learning gains.

PURPOSE Emergency physicians (EPs) regularly manage multiple patients simultaneously, often making time-sensitive decisions around priorities for multiple patients. Few studies have explored physician cognition in multipatient scenarios. The authors sought to develop a conceptual framework to describe how EPs think in busy, multipatient environments. METHOD From July 2014 to May 2015, a qualitative study was conducted at McMaster University, using a think-aloud protocol to examine how 10 attending EPs and 10 junior residents made decisions in multipatient environments. Participants engaged in the think-aloud exercise for five different simulated multipatient scenarios. Transcripts from recorded interviews were analyzed inductively, with an iterative process involving two independent coders, and compared between attendings and residents. RESULTS The attending EPs and junior residents used similar processes to prioritize patients in these multipatient scenarios. The think-aloud processes demonstrated a similar process used by almost all participants. The cognitive task of patient prioritization consisted of three components: a brief overview of the entire cohort of patients to determine a general strategy; an individual chart review, whereby the participant created a functional patient story from information available in a file (i.e., vitals, brief clinical history); and creation of a relative priority list. Compared with residents, the attendings were better able to construct deeper and more complex patient stories. CONCLUSIONS The authors propose a conceptual framework for how EPs prioritize care for multiple patients in complex environments. This study may be useful to teachers who train physicians to function more efficiently in busy clinical environments.

Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation.Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor.Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance, not curricular time. Learners practice and retest repeatedly until they reach a designated mastery level; the final level of achievement is the same for all, although time to mastery may vary. Given the unique properties of mastery learning assessments, a thoughtful approach to establishing the performance levels and metrics that determine when a learner has demonstrated mastery is essential.Standard-setting procedures require modification when used for mastery learning settings in health care, particularly regarding the use of evidence-based performance data, the determination of appropriate benchmark or comparison groups, and consideration of patient safety consequences. Information about learner outcomes and past performance data of learners successful at the subsequent level of training can be more helpful than traditional information about test performance of past examinees. The marginally competent "borderline student" or "borderline group" referenced in traditional item-based and examinee-based procedures will generally need to be redefined in mastery settings. Patient safety considerations support conjunctive standards for key knowledge and skill subdomains and for items that have an impact on clinical outcomes. Finally, traditional psychometric indices used to evaluate the quality of standards do not necessarily reflect critical measurement properties of mastery assessments. Mastery learning and testing are essential to the achievement and assessment of entrustable professional activities and residency milestones. With careful attention, sound mastery standard-setting procedures can provide an essential step toward improving the effectiveness of health professions education, patient safety, and patient care.

Recent reviews and research agenda in simulation-based training for health care have repeatedly called for more studies comparing the effectiveness of different simulation modalities and approaches. Although a small body of such comparative research does exist, it is apparent that this line of research comes with particular methodological challenges. In this review, 20 studies comparing simulation modalities in emergency medicine were analyzed in terms of key methodological factors. Results of this literature review show that (1) past studies are largely underpowered to detect effects of the magnitude likely to be found, (2) researchers seeking to demonstrate equivalence of training approaches do not use appropriate statistical tests of equivalence, and (3) studies often use performance criterion test beds that may not support valid conclusions about trainees' future performance on the job. We discuss strengths and weaknesses of reviewed studies along these issues and recommend options for improving future comparative research in simulation-based training.