Search

Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example.The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review.Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories.The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Medical education researchers are often subject to challenges that include lack of funding, collaborators, study subjects, and departmental support. The construct of a research lab provides a framework that can be employed to overcome these challenges and effectively support the work of medical education researchers; however, labs are relatively uncommon in the medical education field. Using case examples, the authors describe the organization and mission of medical education research labs contrasted with those of larger research team configurations, such as research centers, collaboratives, and networks. They discuss several key elements of education research labs: the importance of lab identity, the signaling effect of a lab designation, required infrastructure, and the training mission of a lab. The need for medical education researchers to be visionary and strategic when designing their labs is emphasized, start-up considerations and the likelihood of support for medical education labs is considered, and the degree to which department leaders should support such labs is questioned.

Background: Research plays a large role in the professional expectations of faculty in all disciplines. In the health professions, faculty are expected to conduct research while caring for patients, educating learners, and carrying out major administrative roles. Given the difficulty in balancing these many responsibilities, it is sensible for academic institutional officers to seek to understand the factors that strongly relate to faculty research productivity, and to measure faculty members’ standing on those factors. The purpose of the study was to investigate the validity of the research productivity items from Bland and colleagues’ scale as predictors of publication counts among junior and mid-career faculty at the King Abdulaziz University Faculty of Dentistry (KAUFD). Methods: This is an observational study that was conducted at KAUFD. A questionnaire modified from the Bland model was emailed to all full time assistant and associate professors. Two main sets of statistical analyses were run: exploratory factor analysis to investigate the internal structure of the scale and regression analysis to relate scale factors to counts of publications in peer-reviewed journals. Results: Out of 106 faculty, 98 participated (92.5%). Faculty publication count in ISI-indexed journals from January 2013 to the survey time showed a Poisson distribution, with a mode of zero publications (44 participants) and a maximum of 14 publications. The exploratory factor analysis showed that factors 1-4 accounted for 46% of the variance (first factor alone explained 26% of the variance). Conclusion: Our findings suggest a different factor structure between faculty input gathered via Bland and colleagues’ (2005) questionnaire and what the authors originally proposed. Institution- or department-specific validation of this measure is warranted to make reliable conclusions of any comparisons or intervention.

IntroductionThe authors aimed to investigate how medical students interpret the observation of a more experienced clinician modeling both exemplaryandflawed behaviors as well as how that interpretation influences their subsequent clinical performance.MethodsWe presented a recorded patient sexual history to 11 medical students. The recording displayed both exemplary and flawed behaviors. Students then obtained a sexual history from a standardized patient themselves. Using an interview methodology and constructivist analytic approach, we explored the process of learning from clinical observations.ResultsStudents attended to flawed modeled behaviors, challenges specific to the task, and how areas of their own personal development were accomplished. They took a piecemeal approach to classifying modeled behaviors as done well or poorly based on previous instruction, experience, or perceived downstream effects. When applying their observations, students choose to copy, adapt, or avoid modeled behaviors based on their classification of the behavior.DiscussionTo optimize learning from observation, faculty can identify task-specific challenges and a student’s personal goals, which naturally draw the student’s attention, before observation in order to develop a shared mental model. When debriefing observed encounters, faculty may consider natural targets of learner attention, challenges specific to learning from observation, and factors likely to influence a learner’s judgement of modeled behaviors.

Background Professionalism is a core competency that is difficult to assess. We examined the incidence of publication inaccuracies in Electronic Residency Application Service applications to our training program as potential indicators of unprofessional behavior. Study Design We reviewed all 2019-2020 National Resident Matching Program applicants being considered for interview. Applicant demographic characteristics recorded included standardized examination scores, gender, medical school, and medical school ranking (2019 US News & World Report). Publication verification by a medical librarian was performed for peer-reviewed journal articles/abstracts, peer-reviewed book chapters, and peer-reviewed online publications. Inaccuracies were classified as “nonserious” (eg incorrect author order without author rank promotion) or “serious” (eg miscategorization, non-peer-reviewed journal, incorrect author order with author rank promotion, nonauthorship of cited existing publication, and unverifiable publication). Multivariate logistic regression analysis was performed for demographic characteristics to identify predictors of overall inaccuracy and serious inaccuracy. Results Of 319 applicants, 48 (15%) had a total of 98 inaccuracies; after removing nonserious inaccuracies, 37 (12%) with serious inaccuracies remained. Seven publications were reported in predatory open access journals. In the regression model, none of the variables (US vs non-US medical school, gender, or medical school ranking) were significantly associated with overall inaccuracy or serious inaccuracy. Conclusions One in 8 applicants (12%) interviewing at a general surgery residency program were found to have a serious inaccuracy in publication reporting on their Electronic Residency Application Service application. These inaccuracies might represent inattention to detail or professionalism transgressions.

There are conflicting findings in team diversity research on whether it is better for an individual on a team to be similar to or different from the rest of the team. This lab study with undergraduates completing a critical thinking and decision-making task uses optimal distinctiveness theory to examine the idea that finding a balance between these two states for team member personality will result in positive perceptions of team process. Our results supported this such that participants had the most positive perceptions of team process when optimally distinct from the rest of the team in terms of personality.

Visual diagnosis of radiographs, histology and electrocardiograms lends itself to deliberate practice, facilitated by large online banks of cases. Which cases to supply to which learners in which order is still to be worked out, with there being considerable potential for adapting the learning. Advances in statistical modeling, based on an accumulating learning curve, offer methods for more effectively pairing learners with cases of known calibrations. Using demonstration radiograph and electrocardiogram datasets, the advantages of moving from traditional regression to multilevel methods for modeling growth in ability or performance are demonstrated, with a final step of integrating case-level item-response information based on diagnostic grouping. This produces more precise individual-level estimates that can eventually support learner adaptive case selection. The progressive increase in model sophistication is not simply statistical but rather brings the models into alignment with core learning principles including the importance of taking into account individual differences in baseline skill and learning rate as well as the differential interaction with cases of varying diagnosis and difficulty. The developed approach can thus give researchers and educators a better basis on which to anticipate learners' pathways and individually adapt their future learning.

To explore how medical students completing a pediatric clerkship viewed the benefits and barriers of debrief interviews with hospitalized patients and families.In this study, focus groups were conducted with pediatric clerkship students after completion of a debrief interview. The constant comparative method was used with Mezirow's transformative learning theory as a lens to explore perceptions of the benefits and challenges of performing the interview.Focus groups revealed five benefits and two challenges. The benefits were that the debrief interviews helped students (1) humanize patients and appreciate social and environmental influences on patient health, (2) assess caregiver/patient understanding about care to correct misunderstandings, (3) actively involve caregivers/patients in treatment plan development, (4) engage patients in active expression of questions/concerns, and (5) recognize the value of their own role on the healthcare team. The challenges were that students felt (1) a lack of knowledge to answer caregivers'/patients' questions about diagnoses and (2) discomfort responding to caregiver/patient frustration, anxiety, or sadness. Student feedback on feasibility and implementation led to guidelines for selecting patients and conducting small group discussions after the debrief interviews.Debrief interviews offer a unique approach for learners to explore patient perspectives during hospitalization through direct patient engagement and dialogue, contributing to professional development, empathy, and potentially more positive patient care experiences.

The Fundamentals of Endoscopic Surgery (FES) program became required for American Board of Surgery certification as part of the Flexible Endoscopy Curriculum (FEC) for residents graduating in 2018. This study expands prior psychometric investigation of the FES skills test. We analyzed de-identified first-attempt skills test scores and self-reported demographic characteristics of 2023 general surgery residents who were required to pass FES. The overall pass rate was 83%. “Loop Reduction” was the most difficult sub-task. Subtasks related to one another only modestly (Spearman’s ρ ranging from 0.11 to 0.42; coefficient α = .55). Both upper and lower endoscopic procedural experience had modest positive association with scores (ρ = 0.14 and 0.15) and passing. Examinees who tested on the GI Mentor Express simulator had lower total scores and a lower pass rate than those tested on the GI Mentor II (pass rates = 73% vs. 85%). Removing an Express-specific scoring rule that had been applied eliminated these differences. Gender, glove size, and height were closely related. Women scored lower than men (408- vs. 489-point averages) and had a lower first-attempt pass rate (71% vs. 92%). Glove size correlated positively with score (ρ = 0.31) and pass rate. Finally, height correlated positively with score (r = 0.27) and pass rate. Statistically controlling for glove size and height did not eliminate gender differences, with men still having 3.2 times greater odds of passing. FES skills test scores show both consistencies with the assessment’s validity argument and several remarkable findings. Subtasks reflect distinct skills, so passing standards should perhaps be set for each subtask. The Express simulator-specific scoring penalty should be removed. Differences seen by gender are concerning. We argue those differences do not reflect measurement bias, but rather highlight equity concerns in surgical technology, training, and practice.

PURPOSE To determine the effect of patient debrief interviews on pediatric clerkship student depth of reflection and learning. METHOD The authors conducted a multi-institutional, mixed-methods, cluster randomized trial among pediatric clerkship students from May 2016 to February 2017. Intervention students completed a debrief interview with a patient-caregiver, followed by a written reflection on the experience. Control students completed a written reflection on a memorable patient encounter. Three blinded authors scored written reflections according to the 4-level REFLECT rubric to determine depth of reflection. Interrater reliability was examined using kappa. REFLECT scores were analyzed using a chi-square test; essays were analyzed using content analysis. RESULTS Eighty percent of eligible students participated. One hundred eighty-nine essays (89 control, 100 intervention) were scored. Thirty-seven percent of the control group attained reflection and critical reflection, the 2 highest levels of reflection, compared with 71% of the intervention group; 2% of the control group attained critical reflection, the highest level, compared with 31% of the intervention group (χ(3, N = 189) = 33.9, P < .001). Seven themes were seen across both groups, 3 focused on physician practice and 4 focused on patients. Patient-centered themes were more common in the intervention group, whereas physician-focused themes were more common in the control group. CONCLUSIONS Patient debrief interviews offer a unique approach to deepen self-reflection through direct dialogue and exploration of patient-caregiver experiences during hospitalization.

A key question in simulation-based education is how to maximize learning within time and resource limits, including how best to balance hands-on practice versus reflective debriefing. Several instructional design frameworks suggest setting the balance according to the type of learning objective(s); however, broad professional activities such as team-based cardiopulmonary resuscitation include several interrelated component skills. This study experimentally manipulated hands-on practice versus reflective debriefing for cardiopulmonary resuscitation skills, hypothesizing that the former best supports learning taskwork (eg, compression quality), whereas the latter best supports learning teamwork.The study was a randomized comparison trial with a pretest and posttest. Twenty-six teams of 5 to 6 first-year residents underwent either "drill" practice of key resuscitation phases, designed to maximize deliberate practice opportunities for individual and team skills, or "scrimmage" practice, designed to maximize full-scenario rehearsals and reflective debriefs. Key taskwork and teamwork behaviors were coded, and compression quality was collected and analyzed from an accelerometer.Most performance parameters improved considerably from a pretest to posttest for both taskwork (eg, percent correct compression depth 62%-81%, P = 0.01) and teamwork (eg, role leadership, 47%-70%, P = 0.00). Only 2 parameters improved differently by condition, favoring "drill" training: checking "Do Not Actively Resuscitate" wristband (odds ratio = 14.75, P = 0.03) and use of compression adjuncts (estimated marginal means = 75% versus 67%, P = 0.03).Consistent with the notion that component skills in resuscitation do not clearly and exclusively constitute "taskwork" versus "teamwork," both instructional designs led to similar improvements despite differences in the balance between hands-on practice versus reflection.

Background Geriatric patients account for a growing proportion of dermatology clinic visits. Although their biopsychosocial needs differ from those of younger adults, there are no geriatrics training requirements for dermatology residency programs. Objective This study explored the state of geriatrics education in dermatology programs in 2016. Methods This constructivist study employed cross-sectional, mixed-methods analysis with triangulation of semistructured interviews, surveys, and commonly used curricular materials. We used purposive sampling of 5 US academic allopathic dermatology programs of different sizes, geographic locations, and institutional resources. Participants were interviewed about informal curricula, barriers, and suggestions for improving geriatrics education, and they also completed a survey about the geriatrics topics that should be taught. The constant comparative method with grounded theory was used for qualitative analysis. We identified formal geriatrics curricular content by electronically searching and counting relevant key texts. Results Fourteen of 17 participants (82%) agreed to be interviewed, and 10 of 14 (71%) responded to the survey. Themes of what should be taught included diagnosing and managing skin diseases common in older adults, holistic treatment, cosmetic dermatology, benign skin aging, and the basic science of aging. Topics currently covered that could be expanded included communication, systems-based challenges, ethical issues, safe prescribing, quality improvement, and elder abuse. Cosmetic dermatology was the most commonly taught formal geriatrics curricular topic. Conclusions There were discrepancies among topics participants felt were important to teach about geriatric dermatology and curricular coverage of these areas. We identified challenges for expanding geriatrics curricula and potential solutions.

It has been a challenge to assess communication and professional values Milestones in emergency medicine (EM) residents using standardized methods, as mandated by the Accreditation Council for Graduate Medical Education (ACGME). This paper outlines an innovative method of assessing these Milestones using an established instructional method. EM faculty mapped the communication and professional values Milestones to an existing communication and interpersonal skills scale. We identified six communication-focused scenarios: death notification; informed consent; medical non-compliance; medical error; treatment refusal; and advanced directives. In a pilot, 18 EM residents completed these six standardized patient (SP) encounters. Our experience suggests SP encounters can support standardized direct observation of residents’ achievement of ACGME Milestones. Further effort can be made to create a tailored, behaviorally-anchored tool that uses the Milestones as the conceptual framework.

Purpose Simulation-based medical education (SBME) is gaining prominence as a tool to meet Accreditation Council for Graduate Medical Education–mandated competency-based assessment educational goals. SBME is used in radiation oncology, although the type and extent are not clear. This study reports a systematic literature review designed to clarify the type and extent of radiation oncology SBME. Methods and Materials The systematic review focused on radiation oncology SBME literature. The methods followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The inclusion criteria were identified according to the PICOS (population, intervention, comparison, outcome, and setting) framework. The population included undergraduate, graduate, and continuing medical education learners. Studies were limited to English-language studies published on or after January 1, 1990, in peer-reviewed journals. PubMed, MedEdPORTAL, and in-press articles were searched. The PubMed search was conducted using predefined search terms. References and similar articles were examined. Medical Subject Headings terms in selected articles were reviewed to ensure relevant terms were included. Results Fifty-four SBME publications met the inclusion criteria. Only 9 of 54 studies (17%) self-identified as SBME. SBME types included screen-based simulators (56%), simulated environments (13%), virtual reality and haptic systems (13%), simulated patients (11%), part-task trainers (6%), and computer-based systems with mannequins (2%). A variety of radiation oncology skill sets were addressed, including contouring (54%), treatment planning (20%), clinical decision making (17%), anatomy and/or radiology (13%), radiation biology and/or physics (13%), communication skills and/or patient education (13%), brachytherapy (13%), and immobilization (11%). A target learning population was defined in 47 studies, including residents (53%), attending physicians (36%), medical students (21%), medical physicists (11%), radiation therapists (9%), nurses (6%), administrative staff (4%), and dosimetrists (4%). Learner feedback was reported in 32 studies. Conclusions Overall, this systematic literature review provides context and guidance for future radiation oncology SBME development. Appropriately framing SBME reports in the radiation oncology literature will facilitate development, implementation, and evaluation of SBME interventions. SBME resources should be centralized to facilitate dissemination and share resources.

Analysis of the Fundamentals of Endoscopic Surgery (FES) performance exam showed higher scores for men than women. Gender differences have been reduced with task-specific practice. We assessed the effect of simulation-based mastery learning (SBML) on FES performance exam differences by gender. Forty-seven surgical trainees [29 men (m), 18 women (w)] completed a SBML curriculum and were assessed by FES. Fourteen trained on the GI Mentor 2, 18 on the Endoscopy Training System, and 15 using the Surgical Training for Endoscopic Proficiency curriculum. Performance of male and female trainees was compared. On the pre-training assessment, there were large differences between genders in FES pass rates (m 77%, w 15%, p

Introduction Within simulation-based mastery learning (SBML) courses, there is inconsistent inclusion of learner pretesting, which requires considerable resources and is contrary to popular instructional frameworks. However, it may have several benefits, including its direct benefit as a form of deliberate practice and its facilitation of more learner-specific subsequent deliberate practice. We consider an unexplored potential benefit of pretesting: its ability to predict variable long-term learner performance. Methods Twenty-seven residents completed an SBML course in central line insertion. Residents were tested on simulated central line insertion precourse, immediately postcourse, and after between 64 and 82 weeks. We analyzed pretest scores' prediction of delayed test scores, above and beyond prediction by program year, line insertion experiences in the interim, and immediate posttest scores. Results Pretest scores related strongly to delayed test scores (r = 0.59, P = 0.01; disattenuated ρ = 0.75). The number of independent central lines inserted also related to year-delayed test scores (r = 0.44, P = 0.02); other predictors did not discernibly relate. In a regression model jointly predicting delayed test scores, pretest was a significant predictor (β = 0.487, P = 0.011); number of independent insertions was not (β = 0.234, P = 0.198). Conclusions This study suggests that pretests can play a major role in predicting learner variance in learning gains from SBML courses, thus facilitating more targeted refresher training. It also exposes a risk in SBML courses that learners who meet immediate mastery standards may be incorrectly assumed to have equal long-term learning gains.

PURPOSE Emergency physicians (EPs) regularly manage multiple patients simultaneously, often making time-sensitive decisions around priorities for multiple patients. Few studies have explored physician cognition in multipatient scenarios. The authors sought to develop a conceptual framework to describe how EPs think in busy, multipatient environments. METHOD From July 2014 to May 2015, a qualitative study was conducted at McMaster University, using a think-aloud protocol to examine how 10 attending EPs and 10 junior residents made decisions in multipatient environments. Participants engaged in the think-aloud exercise for five different simulated multipatient scenarios. Transcripts from recorded interviews were analyzed inductively, with an iterative process involving two independent coders, and compared between attendings and residents. RESULTS The attending EPs and junior residents used similar processes to prioritize patients in these multipatient scenarios. The think-aloud processes demonstrated a similar process used by almost all participants. The cognitive task of patient prioritization consisted of three components: a brief overview of the entire cohort of patients to determine a general strategy; an individual chart review, whereby the participant created a functional patient story from information available in a file (i.e., vitals, brief clinical history); and creation of a relative priority list. Compared with residents, the attendings were better able to construct deeper and more complex patient stories. CONCLUSIONS The authors propose a conceptual framework for how EPs prioritize care for multiple patients in complex environments. This study may be useful to teachers who train physicians to function more efficiently in busy clinical environments.

Background Surgical skill assessment tools frequently reflect the opinions of small groups of surgeons. That raises concerns over their generalizability as well as their utilization when applied broadly. A Delphi approach could engage a broad group of experts to identify key elements for a checklist assessing coronary anastomotic skill, improving generalizability. Methods Expert surgeons in North America (10 or more years in practice, actively teaching coronary artery surgery) were contacted randomly to participate. Consenting surgeons first provided items they believed were mandatory when performing a coronary artery bypass. These were then entered into a three-round Delphi. Positive consensus was reached when 75% or more of participants ranked an item mandatory. Results Sixteen faculty consented to participate. Each participant provided 25 ± 10 items. The 407 items provided were condensed, resulting in 146 items in the final list, divided into six sections based on the conduct of the operation. Twenty-three items reached consensus in the first round, 14 in the second, and 3 in the third. These 40 items represented only 27% of the initial 146 items. Agreement within sections varied widely, from 0% for "management of assistants" to 47% for "testing and final steps." Conclusions A randomly selected group of experts using a Delphi approach can generate a checklist to assess construction of a coronary artery bypass. Considerable disagreement among experts regarding what steps are mandatory calls into question the generalizability of any locally developed checklist.

Introduction As patient volumes continue to increase, more attention must be paid to skills that foster efficiency without sacrificing patient safety. The emergency department is a fertile ground for examining leadership and management skills, especially those that concern prioritization in multi-patient environments. We sought to understand the needs of emergency physicians (EPs) and emergency medicine junior trainees with regards to teaching and learning about how best to handle busy, multi-patient environments. Method A cognitive task analysis was undertaken, using a qualitative approach to elicit knowledge of EPs and residents about handling busy emergency department situations. Ten experienced EPs and 10 junior emergency medicine residents were interviewed about their experiences in busy emergency departments. Transcripts of the interviews were analyzed inductively and iteratively by two independent coders using an interpretive description technique. Results EP teachers and junior residents differed in their perceptions of what makes an emergency department busy. Moreover, they focused on different aspects of patient care that contributed to their busyness: EP teachers tended to focus on volume of patients, junior residents tended to focus on the complexity of certain cases. The most important barrier to effective teaching and learning of managerial skills was thought to be the lack of faculty development in this skill set. Conclusions This study presents qualitative data that helps us elucidate how patient volumes affect our learning environments, and how clinical teachers and residents operate within these environments.

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance rather than curricular time. Sound assessment is the cornerstone of mastery learning systems. Inaccurate assessment of learners’ mastery or poor use of such assessments for decision-making would lead learners to be prematurely advanced or unnecessarily held back. This chapter aims to outline key issues in the validation and justification of mastery learning assessments. In mastery assessment programs, the number of items needed may be larger than in traditional settings, because many learners will take the mastery test more than once. A key important relationship to evaluate in a mastery system is whether assessment scores relate to examinees’ success in their subsequent educational unit(s), including their eventual transition to practice on patients. Comparison groups for contrasting groups methods used in mastery settings must be chosen with care. Mastery testing has potentially widespread influence on curricula and training programs.

Statistical analyses are key for deriving important insights from quantitative data in educational research. While the technical aspects of statistics can seem daunting, and expert consultation is well-advised, we do not advise thinking of analysis as a task to be “handed off” to a statistician after data is collected. Instead, analyses are best completed in close collaboration with a broad research team beginning early in the conceptualization of the research. This chapter outlines foundational concepts in statistics with the goal of familiarizing non-statisticians with key terms and concepts. We also share tips and tricks for running analyses and writing about your findings. Our hope is that readers who consider themselves statistically “challenged” will appreciate that their critical thinking about analyses can be essential to conducting sound and insightful research, provided a basic understanding of content and a team-centered approach.

Simulation-based assessment (SBA) is a promising context in which to assess learners: written examinations are easily standardized to be the same between learners but are relatively inauthentic to the context of clinical performance, while patient chart audits and direct observations of patient care are authentic yet difficult to standardize. As we will illustrate in this chapter, SBA offers many opportunities. Yet, as with other assessment strategies, SBA has its challenges. Thus it is essential that medical educators be purposeful about why and what they intend to assess, to construct SBAs that are as representative as possible of the knowledge, attitudes, and skills to be assessed while minimizing construct-irrelevant variance. In this chapter we will cover different elements to be considered when designing, implementing, and monitoring SBA.

Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation.Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor.Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

BACKGROUND: Busy environments, like the emergency department (ED), require teachers to develop instructional strategies for coaching trainees to function within these same environments. Few studies have documented the strategies used by emergency physician (EP)‐teachers within these busy, chaotic environments, instead emphasizing teaching in more predictable environments such as the outpatient clinic, hospital wards, or operating room. The authors sought to discover what strategies EP‐teachers were using and what trainees recalled experiencing when learning to handle these unpredictable, overcrowded, complex, multipatient environments. METHOD: An interpretive description study was conducted at multiple teaching hospitals affiliated with McMaster University from July 2014 to May 2015. Participants (10 EP‐teachers and 10 junior residents) were asked to recall teaching strategies related to handling ED patient flow. Participants were asked to describe techniques that they used, observed, or experienced as trainees. Two independent coders read through interview transcripts, analyzing these documents inductively and iteratively. RESULTS: Two main types of strategies to teach ED management were discovered: 1) workplace‐based methods, including both observation and in situ instruction; and 2) principle‐based advice. The most often described techniques were workplace‐based methods, which included a variety of in situ techniques ranging from conversations to managerial coaching (e.g., collaborative problem‐solving of real‐life administrative dilemmas). CONCLUSIONS: A mix of strategies are used to teach and coach trainees to handle multipatient environments. Further research is required to determine how to optimize the use of these techniques and innovate new strategies to support the learning of these crucial skills.

Context The script concordance test (SCT), designed to measure clinical reasoning in complex cases, has recently been the subject of several critical research studies. Amongst other issues, response process validity evidence remains lacking. We explored the response processes of experts on an SCT scoring panel to better understand their seemingly divergent beliefs about how new clinical data alter the suitability of proposed actions within simulated patient cases. Methods A total of 10 Argentine gastroenterologists who served as the expert panel on an existing SCT re-answered 15 cases 9 months after their original panel participation. They then answered questions probing their reasoning and reactions to other experts' perspectives. Results The experts sometimes noted they would not ordinarily consider the actions proposed for the cases at all (30/150 instances [20%]) or would collect additional data first (54/150 instances [36%]). Even when groups of experts agreed about how new clinical data in a case affected the suitability of a proposed action, there was often disagreement (118/133 instances [89%]) about the suitability of the proposed action before the new clinical data had been introduced. Experts reported confidence in their responses, but showed limited consistency with the responses they had given 9 months earlier (linear weighted kappa = 0.33). Qualitative analyses showed nuanced and complex reasons behind experts' responses, revealing, for example, that experts often considered the unique affordances and constraints of their varying local practice environments when responding. Experts generally found other experts' alternative responses moderately compelling (mean ± standard deviation 2.93 ± 0.80 on a 5-point scale, where 3 = moderately compelling). Experts switched their own preferred responses after seeing others' reasoning in 30 of 150 (20%) instances. Conclusions Expert response processes were not consistent with the classical interpretation and use of SCT scores. However, several fruitful and justifiable alternatives for the use of SCT-like methods are proposed, such as to guide assessments for learning.

An incremental (growth) theory of intelligence (mindset), compared with an entity (fixed) mindset, has been associated with improved motivation and performance. Interventions to induce incremental beliefs have improved performance on non-surgical motor tasks. We sought to evaluate the impact of 2 brief interventions to induce incremental beliefs in the context of learning a surgical task.Two randomized experiments.Secondary school students participating in medical simulation-based training activities at an academic medical center.We created 4 instructional messages intended to influence mindsets (two 60-second videos in Study 1, 2 fabricated "journal articles" in Study 2). In each study, one message emphasized that ability improves with practice (incremental); the other emphasized that ability is fixed (entity). After reviewing their randomly-assigned message, participants completed a laparoscopic cutting task as many times as they desired. Measurements included performance (product quality, self-reported task, and completion time), task persistence (repetitions), and entity beliefs.Two hundred and three students completed Study 1. Postevent entity beliefs (1 = lowest, 6 = highest) were similar between groups (incremental, 2.0vs entity, 2.0; p = 0.78). Contrary to hypothesis, the incremental video group demonstrated slower time (276vs 191 seconds; p0.0001), lower product quality (7.2vs 3.8mm deviation; p0.0001), and fewer task repetitions (1.4vs 1.8; p = 0.02). In Study 2, 113 participants provided outcomes related to mindset beliefs, but only 14 provided usable performance outcomes. Postevent entity beliefs were lower in the incremental article group (1.7vs 2.4; p0.0001). Task time (507vs 585 seconds; p = 0.40) and quality (7.1vs 7.5mm deviation; p = 0.85) were similar between groups.Brief motivational interventions can influence procedural performance and motivation. We need to better understand motivation and other affective influences on procedural skills learning. Mindset theory shows promise in this regard.

STATEMENT Despite evidence that learners vary greatly in their learning needs, practical constraints tend to favor ''one-size-fits-all'' educational approaches, in simulation-based education as elsewhere. Adaptive educational technologies - devices and/or software applications that capture and analyze relevant data about learners to select and present individually tailored learning stimuli - are a promising aid in learners' and educators' efforts to provide learning experiences that meet individual needs. In this article, we summarize and build upon the 2017 Society for Simulation in Healthcare Research Summit panel discussion on adaptive learning. First, we consider the role of adaptivity in learning broadly. We then outline the basic functions that adaptive learning technologies must implement and the unique affordances and challenges of technology-based approaches for those functions, sharing an illustrative example from healthcare simulation. Finally, we consider future directions for accelerating research, development, and deployment of effective adaptive educational technology and techniques in healthcare simulation.

Background Trainees in medical subspecialties lack validated assessment scores that can be used to prepare for their licensing examination. This paper presents the development, administration, and validity evidence of a constructed-response preparatory test (CRPT) administered to meet the needs of nephrology trainees. Methods Learning objectives from the licensing examination were used to develop a test blueprint for the preparatory test. Messick’s unified validity framework was used to gather validity evidence for content, response process, internal structure, relations to other variables, and consequences. Questionnaires were used to gather data on the trainees’ perception of examination preparedness, item clarity, and curriculum adequacy. Results There were 10 trainees and 5 faculty volunteers who took the test. The majority of trainees passed the constructed-response preparatory test. However, many scored poorly on items assessing renal pathology and physiology knowledge. We gathered the following five sources of validity evidence: (1) Content: CRPT mapped to the licensing examination blueprint, with items demonstrating clarity and range of difficulty; (2) Response process: moderate rater agreement (intraclass correlation = .58); (3) Internal structure: sufficient reliability based on generalizability theory (G-coefficient = .76 and Φ-coefficient = .53); (4) Relations to other variables: CRPT scores reflected years of exposure in nephrology and clinical practice; (5) Consequences: post-assessment survey revealed that none of the test takers felt “poorly prepared” for the upcoming summative examination and that their studying would increase in duration and be adapted in terms of content focus. Conclusions Preparatory tests using constructed response items mapped to licensure examination blueprint can be developed and used at local program settings to help prepare learners for subspecialty licensure examinations. The CRPT and questionnaire data identified shortcomings of the nephrology training program curriculum. Following the preparatory test, trainees expressed an improved sense of preparedness for their licensing examination.

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance, not curricular time. Learners practice and retest repeatedly until they reach a designated mastery level; the final level of achievement is the same for all, although time to mastery may vary. Given the unique properties of mastery learning assessments, a thoughtful approach to establishing the performance levels and metrics that determine when a learner has demonstrated mastery is essential.Standard-setting procedures require modification when used for mastery learning settings in health care, particularly regarding the use of evidence-based performance data, the determination of appropriate benchmark or comparison groups, and consideration of patient safety consequences. Information about learner outcomes and past performance data of learners successful at the subsequent level of training can be more helpful than traditional information about test performance of past examinees. The marginally competent "borderline student" or "borderline group" referenced in traditional item-based and examinee-based procedures will generally need to be redefined in mastery settings. Patient safety considerations support conjunctive standards for key knowledge and skill subdomains and for items that have an impact on clinical outcomes. Finally, traditional psychometric indices used to evaluate the quality of standards do not necessarily reflect critical measurement properties of mastery assessments. Mastery learning and testing are essential to the achievement and assessment of entrustable professional activities and residency milestones. With careful attention, sound mastery standard-setting procedures can provide an essential step toward improving the effectiveness of health professions education, patient safety, and patient care.

Context International medical graduates (IMGs) play key roles in the health systems of their host countries, but face unique challenges, which makes the provision of effective, tailored support for IMGs essential. Objectives Research on the effectiveness of educational interventions for IMGs was reviewed to characterise current knowledge and guide future research and education. Methods PubMed, Web of Science and EMBASE were searched for relevant articles published to October 2014, describing a systematic evaluation of educational interventions designed for IMGs that included at least one post-intervention outcome. Articles were coded independently by two or more researchers for content and methodology, and discussed to reach consensus. Results Twenty-two articles were identified, describing a wide variety of interventions, content and durations of intervention. Clinical topics and general principles of cross-cultural competency were the most common content areas included in curricula (13 and 12 articles, respectively). All studies deemed the interventions evaluated to be successful. However, only one study drew from theory on cross-cultural differences to guide either the curriculum or evaluation. Additionally, study designs were generally weak; no studies featured random assignment to treatment versus control groups, most studies did not use control groups at all, and no studies compared the effectiveness of different interventions. Conclusions Research into education for IMGs is critically important but currently underdeveloped. An abundance of justification studies and lack of clarification studies parallel other areas of medical education. Academic fields outside medical education, such as those of cross-cultural psychology and expatriate management, are highly relevant; researchers from these areas should be sought for collaboration. Future research should employ conceptual frameworks in order to facilitate a broader, more nuanced consideration of the diversity of individual IMGs, educational and medical contexts, interventions and outcomes. Rigorous comparative effectiveness research is lacking, but represents a promising avenue for future scholarship.

Studies have documented performance on the United States Medical Licensing Examination® (USMLE) Step 1 exam as an important factor that residency program directors consider when deciding which applicants to interview and rank. Therefore, success on this exam, though only one aspect of applicant evaluation, is important in determining future career prospects for medical students. Unfortunately, mean test scores at the University of Illinois College of Medicine at Chicago (UIC) have historically been below the national average.This retrospective and quasi-experimental mixed-methods study describes the development, evaluation, and effects of a student-initiated USMLE Step 1 preparatory program at UIC. The program provided second year students with First Aid for the USMLE Step 1 at the beginning of the academic year, as well as a six month subscription to the USMLE World question bank midyear. In addition, optional peer review sessions covering basic sciences and organ systems were taught by high-performing upperclassmen. The goals of the program were to raise mean USMLE Step 1 exam scores and increase the percentage of students passing the exam on their first time.The program premiered during the 2012-13 academic year. Data from this cohort as well as four others (N = 830; 2010-2014 examinees) were gathered. Performances between preintervention (2010-12 examinees) and postintervention (2013-14 examinees) cohorts of students were compared. Focus groups and interviews with staff and students were conducted, recorded, and analyzed to investigate the impact that the program had on student interactions and perceptions of the learning environment.There was a significant difference in exam performance pre- versus postintervention, with average USMLE Step 1 scores improving by 8.82 points following the implementation of the student-initiated program, t(5.61) = 828, p.001. The average first-attempt pass rate also increased significantly by 8%, χThis study demonstrates the efficacy of a student-initiated curriculum and provides guidance for development and implementation of examination preparatory efforts at other institutions.

Learning curves can support a competency-based approach to assessment for learning. When interpreting repeated assessment data displayed as learning curves, a key assessment question is: “How well is each learner learning?” We outline the validity argument and investigation relevant to this question, for a computer-based repeated assessment of competence in electrocardiogram (ECG) interpretation. We developed an on-line ECG learning program based on 292 anonymized ECGs collected from an electronic patient database. After diagnosing each ECG, participants received feedback including the computer interpretation, cardiologist’s annotation, and correct diagnosis. In 2015, participants from a single institution, across a range of ECG skill levels, diagnosed at least 60 ECGs. We planned, collected and evaluated validity evidence under each inference of Kane’s validity framework. For Scoring, three cardiologists’ kappa for agreement on correct diagnosis was 0.92. There was a range of ECG difficulty across and within each diagnostic category. For Generalization, appropriate sampling was reflected in the inclusion of a typical clinical base rate of 39% normal ECGs. Applying generalizability theory presented unique challenges. Under the Extrapolation inference, group learning curves demonstrated expert–novice differences, performance increased with practice and the incremental phase of the learning curve reflected ongoing, effortful learning. A minority of learners had atypical learning curves. We did not collect Implications evidence. Our results support a preliminary validity argument for a learning curve assessment approach for repeated ECG interpretation with deliberate and mixed practice. This approach holds promise for providing educators and researchers, in collaboration with their learners, with deeper insights into how well each learner is learning.

The Fundamentals of Endoscopic Surgery (FES) manual skills examination is a simulation-based assessment of five foundational skills in endoscopic surgery. With the FES skills exam becoming part of the board certification process in general surgery, continual investigation is needed to determine the validity with which the exam is supporting inferences and decision-making about examinees, as well as how it might be improved.The present study retrospectively analyzed performance and demographic details for the initial 344 examinees completing the FES skills exam.The five tasks showed distinct degrees of difficulty, with Loop Reduction being especially difficult for examinees. Tasks related to one another positively but moderately, suggesting that the exam assesses both general and task-specific skills. The number of lower-endoscopic cases completed by an examinee strongly predicted performance, while upper endoscopy experience and career level (e.g., resident vs. fellow vs. practicing) did not. Hand dominance and the type of simulator used were not found to be related to scores. However, three demographic variables that related to one another-gender, glove size, and height-were also related to performance and pass/fail status.This study's results generally support the validity argument for the FES skills exam while pointing to additional investigations to be undertaken as the exam is applied more broadly.

Pericarditis and ST-segment elevation myocardial infarction (STEMI) are often confused on electrocardiogram (EKG) interpretation and have markedly different implications. We sought to create an accurate statistical model to represent skill in distinguishing these 2 diagnoses. From a pilot study, 20

Understanding what qualities of an electrocardiogram (EKG) make it difficult for learners to interpret is crucial to developing teaching models for EKG learning. We hypothesized that EKGs could be ranked on a scale from simple pattern recognition to those requiring complex deductive analysis, and

Context Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high-stakes testing. Objectives This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats. Methods We conducted a review of SCT reports available through the Web of Science database. Additionally, we reanalysed scores from a previously published SCT administration to explore issues related to standard SCT scoring practice. Results Firstly, the predominant method for aggregate and partial credit scoring of SCTs introduces logical inconsistencies in the scoring key. Secondly, our literature review shows that SCT reliability studies have generally ignored inter-panel, inter-panellist and test–retest measurement error. Instead, studies have focused on observed levels of coefficient alpha, which is neither an informative index of internal structure nor a comprehensive index of reliability for SCT scores. As such, claims that SCT scores show acceptable reliability are premature. Finally, SCT criteria for item inclusion, in concert with a statistical artefact of the SCT format, cause anchors at the extremes of the scale to have less expected credit than anchors near or at the midpoint. Consequently, SCT scores are likely to reflect construct-irrelevant differences in examinees' response styles. This makes the test susceptible to bias against candidates who endorse extreme scale anchors more readily; it also makes two construct-irrelevant test taking strategies extremely effective. In our reanalysis, we found that examinees could drastically increase their scores by never endorsing extreme scale points. Furthermore, examinees who simply endorsed the scale midpoint for every item would still have outperformed most examinees who used the scale as it is intended. Conclusions Given the severity of these threats, we conclude that aggregate scoring of SCTs cannot be recommended. Recommendations for revisions of SCT methodology are discussed.

Recent reviews and research agenda in simulation-based training for health care have repeatedly called for more studies comparing the effectiveness of different simulation modalities and approaches. Although a small body of such comparative research does exist, it is apparent that this line of research comes with particular methodological challenges. In this review, 20 studies comparing simulation modalities in emergency medicine were analyzed in terms of key methodological factors. Results of this literature review show that (1) past studies are largely underpowered to detect effects of the magnitude likely to be found, (2) researchers seeking to demonstrate equivalence of training approaches do not use appropriate statistical tests of equivalence, and (3) studies often use performance criterion test beds that may not support valid conclusions about trainees' future performance on the job. We discuss strengths and weaknesses of reviewed studies along these issues and recommend options for improving future comparative research in simulation-based training.

PURPOSE Medical schools administer locally developed graduation competency examinations (GCEs) following the structure of the United States Medical Licensing Examination Step 2 Clinical Skills that combine standardized patient (SP)-based physical examination and the patient note (PN) to create integrated clinical encounter (ICE) scores. This study examines how different subcomponent scoring weights in a locally developed GCE affect composite score reliability and pass-fail decisions for ICE scores, contributing to internal structure and consequential validity evidence. METHOD Data from two M4 cohorts (2014: n = 177; 2015: n = 182) were used. The reliability of SP encounter (history taking and physical examination), PN, and communication and interpersonal skills scores were estimated with generalizability studies. Composite score reliability was estimated for varying weight combinations. Faculty were surveyed for preferred weights on the SP encounter and PN scores. Composite scores based on Kane's method were compared with weighted mean scores. RESULTS Faculty suggested weighting PNs higher (60%-70%) than the SP encounter scores (30%-40%). Statistically, composite score reliability was maximized when PN scores were weighted at 40% to 50%. Composite score reliability of ICE scores increased by up to 0.20 points when SP-history taking (SP-Hx) scores were included; excluding SP-Hx only increased composite score reliability by 0.09 points. Classification accuracy for pass-fail decisions between composite and weighted mean scores was 0.77; misclassification was < 5%. CONCLUSIONS Medical schools and certification agencies should consider implications of assigning weights with respect to composite score reliability and consequences on pass-fail decisions.

Recently, there has been an increased interest in technology-based training, both in academic and military settings. While many instructors have adopted computer-based training methods in their classrooms, researchers have not come to an agreement as to what type of computer-based training methods are most effective. In our study, we compared an adaptive training system to one that was non-adaptive in the context of a submarine periscope task. The adaptive system adjusted both the difficulty of scenarios and feedback given to the trainee based on their performance during training. While average learning gains between groups did not differ, trainees in the adaptive condition were more consistent in their task performance during the post-test and took less time to make their calls in both post-test and transfer scenarios. Our results generally supported the use of adaptive training over non-adaptive training, and we discuss directions for future research.