Search

Purpose: Physicians often experience moral distress from being prevented from taking what they believe to be the right course of action. Although causes and consequences of moral distress have been studied, little research offers insight into the significance of feeling morally challenged, especially in medicine. This study was undertaken to advance understanding of what physicians experience when encountering morally challenging situations and to examine how those experiences influence their interactions with the world of health care., Method: Guided by hermeneutic phenomenology, the authors conducted semistructured interviews with purposefully sampled Canadian physicians from September 2022 to January 2023. Analysis used Van Manen's 6 activities of hermeneutic research and included "story crafting" to sharpen insights. Late in that process, Van Manen's "lived existentials" was determined to provide a strong framework for physicians' experiences of moral distress., Results: Of the 12 physicians interviewed, all experienced varied and long-lasting emotional reactions to morally challenging experiences. The emotions associated with moral distress, reported as inevitable, were experienced in ways that aligned with the lived existentials of body, space, time, and relations., Conclusions: A richer understanding of physicians' lived experiences with moral distress can help mentors and educators more thoughtfully engage in conversations about it with colleagues and learners. This study highlights the core dimensions of moral distress through body, space, time, and relations, thereby providing a means to enrich discussion about emotions and their transformational power. Through such nuanced consideration, calls for emotional candor may be strengthened to enhance practice by acknowledging the scope of impact that efforts to deliver care are likely to provoke., (Copyright © 2024 the Association of American Medical Colleges.)

Purpose: This study aimed to create greater clarity about the current understanding and formulate a model of how educational comparability has been used in the literature to inform practice., Method: The authors conducted a literature search of 9 online databases, seeking articles published on comparability in distributed settings in health professions education before August 2021, with an updated search conducted in May 2023. Using a structured scoping review approach, 2 reviewers independently screened articles for eligibility with inclusion criteria and extracted key data. All authors participated in the descriptive analysis of the extracted data., Results: Twenty-four articles published between 1987 and 2021 met the inclusion criteria. Most articles were focused on medical education programs (n = 21) and located in North America (n = 18). The main rationale for discussing comparability was accreditation. These articles did not offer definitions or discussions about what comparability means. The program logic model was used as an organizing framework to synthesize the literature on practices that schools undertake to facilitate and demonstrate comparability in the design (inputs), implementation (activities), and evaluation (outcomes) of distributed education. Inputs include common learning objectives, identical assessment tools and policies, governance models that enable clear communication, and reporting structure that is supported by technological infrastructure. Activities include faculty planning meetings and faculty development training. Outcomes include student experiences and academic performances., Conclusions: This study demonstrated that a more complex understanding of the dynamics of educational processes and practices is required to better guide the practice of educational comparability within distributed education programs. In addition to highlighting the need to develop an accepted definition of educational comparability, further elucidation of the underlying dynamics among input, activities, and outcomes would help to better determine what drivers should be prioritized when considering educational change with attention to context within distributed education., (Copyright © 2024 the Association of American Medical Colleges.)

Purpose: The stakes of medical trainee selection are high, making it ironic and somewhat paradoxical that patients and the public often get little say in selection practices. The authors sought to undertake a knowledge synthesis to uncover what is known about patient engagement across the medical trainee selection continuum., Method: The authors conducted a scoping review aimed at exploring the current state of practice and research on patient engagement in medical trainee selection in 2017-2021. MeSH headings and keywords were used to capture patient, community, and standardized patient engagement in selection processes across multiple health professions. The authors employed broad inclusion criteria and iteratively refined the corpus, ultimately, limiting study selection to those reporting engagement of actual patients in selection within medicine, but maintaining a broad focus on any patient contributions across the entire selection continuum. The Cambridge Framework was adapted and used to organize the included studies., Results: In total, 2,858 abstracts were reviewed, and ultimately, 28 papers were included in the final corpus. The included studies were global but nascent. Most of the literature on this topic appears in the form of individual projects advocating for patient engagement in selection rather than cohesive programs with empirical exploration of patient engagement in selection. Job analysis methodology was particularly prominent for incorporating the patient voice into identifying competencies of relevance to selection. Direct patient engagement in early selection activities allowed the patient voice to assist candidates in determining their fit for medicine., Conclusions: Patient engagement has not been made a specific focus of study in its own right, leading the authors to encourage researchers to turn their lens more directly on patient engagement to explore how it complements the professional voice in medical trainee selection., (Copyright © 2023 the Association of American Medical Colleges.)

Purpose: The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda., Method: Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion., Results: Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process., Conclusions: This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection., (Copyright © 2023 by the Association of American Medical Colleges.)

Purpose: Safe and competent patient care depends on physicians recognizing and correcting performance deficiencies. Generating effective insight depends on feedback from credible sources. Unfortunately, physicians often have limited access to meaningful guidance. To facilitate quality improvement, many regulatory authorities have designed peer-facilitated practice enhancement programs. Their mandate to ensure practice quality, however, can create tension between formative intentions and risk (perceived or otherwise) of summative repercussions. This study explored how physicians engage with feedback when required to undergo review., Method: Between October 2018 and May 2020, 30 physicians representing various specialties and career stages were interviewed about their experiences with peer review in the context of regulatory body-mandated programs. Twenty had been reviewees and reviewers and, hence, spoke from both vantage points. Interview transcripts were analyzed using a 3-stage coding process informed by constructivist grounded theory., Results: Perceptions about the learning value of mandated peer review were mixed. Most saw value but felt anxiety about being selected due to being wary of regulatory bodies. Recognizing barriers such perceptions could create, reviewers described techniques for optimizing the value of interactions with reviewees. Their strategies aligned well with the R2C2 feedback and coaching model with which they had been trained but did not always overcome reviewees' concerns. Reasons included that most feedback was "validating," aimed at "tweaks" rather than substantial change., Conclusions: This study establishes an intriguing and challenging paradox: feedback appears often to not be recognized as feedback when it poses no threat, yet feedback that carries such threat is known to be suboptimal for inducing performance improvement. In efforts to reconcile that tension, the authors suggest that peer review for individuals with a high likelihood of strong performance may be more effective if expectations are managed through feedforward rather than feedback., (Copyright © 2023 by the Association of American Medical Colleges.)

Purpose: The medical education continuum is interrupted by several transition periods that can adversely affect performance. Most of what has been learned about such periods focuses upon movement from one stage of training to another and movement from training to practice. Established physicians, however, experience transitions throughout their careers at idiosyncratic times and with little assistance. Better understanding how physicians experience transition, where they struggle and how they adapt, would enable better support to be provided. We investigated the COVID-19-forced transition in clinical practice to virtual care, particularly its effect on physician roles and the ways that established physicians faced challenges they encountered when transitioning to virtual care., Method: Ten semistructured interviews were conducted between November 2020 and February 2021 with physicians across different specialties and practice contexts who transitioned their practice to virtual care during the COVID-19 pandemic. Interview data were analyzed iteratively using "generic qualitative methodology" with constant comparison to identify themes in relation to observations., Results: The transition to telehealth had implications that extended beyond the patient encounter, appearing to affect all aspects of the physician's practice. To reflect that, CanMEDS was chosen as a useful organizing framework. The effects, captured in the theme "changes to the physician's roles," were nuanced, illustrated a consistent need to adapt to context, and could be framed positively or negatively or both. Additionally identified themes were labeled "physicians' mental health" and "strategies to mitigate challenges." These themes highlighted that, despite the effort involved and novelty of the situation, all participants found remarkably similar ways of grappling with the challenges faced., Conclusions: While the basic roles of the physician do not appear to have changed through the transition to telehealth, our findings indicated that these roles were redefined in fundamental ways in response to changing societal needs., (Copyright © 2022 by the Association of American Medical Colleges.)

Purpose: To optimize learning, health professional training programs need to achieve the right balance between depth of practice (gaining more experience with particular skills) and breadth of practice (spreading experience across an array of activities). Better understanding how training for a particular skill set is impacted by periods of focus on a different skill set would allow improved curriculum and assessment design, thereby enhancing the efficiency of training and effectiveness of care. To this end, learning curves were used to compare performance in surgery after prolonged periods of practice to performance after gaps in surgical training., Method: Daily operative assessments from the Dalhousie obstetrics and gynecology program were analyzed retrospectively and learning curves were generated. In addition to examining the variability in learning trajectories, the impact of gaps was systematically assessed by comparing resident scores after 2 successive months in which they were not assessed operatively to those collected after 2 successive months in which they were assessed at least once., Results: Four thousand four hundred sixteen scores for 33 residents over a 10-year period were analyzed. Trajectories and peak performances were identified. Residents performed better during their third sequential month of being assessed (mean = 4.40, 95% CI = 4.33-4.46) relative to during months following a period of being away from the operating room for at least 2 months (mean = 4.21, 95% CI = 4.13-4.29; P < .01; d = 0.7). However, maximum performance achieved was more strongly related to the number of times residents experienced a gap in training (r = 0.50) than to the number of times residents experienced 3 consecutive months of training (r = 0.25)., Conclusions: Distinct patterns of development exist for individual residents. Time away from surgical practice and assessment negatively impacted short-term performance, but may improve long-term learning trajectories. This speaks to the value of spaced education and is important for the design of longitudinal skills-based training programs., (Copyright © 2021 by the Association of American Medical Colleges.)

Purpose: Physician shortages and maldistribution, particularly within family medicine, have led many medical schools worldwide to create regional medical campuses (RMCs) for clerkship training. However, Canadian medical schools have developed a number of RMCs in which all years of training (i.e., a combined model that includes both preclerkship and clinical training) are provided geographically separate from the main campus. This study addresses the question: Are combined model RMC graduates more likely to enter postgraduate training in family medicine and rural-focused programs relative to main campus graduates?, Method: The authors used a quasi-experimental research design and analyzed 2006-2016 data from the Canadian Resident Matching Service (CaRMS). Graduating students (N = 26,525) from 16 Canadian medical schools who applied for the CaRMS match in their year of medical school graduation were eligible for inclusion. The proportions of graduates who matched to postgraduate training in (1) family medicine and (2) rural-focused programs were compared for combined model RMCs and main campuses., Results: Of RMC graduates, 48.4% matched to family medicine (95% confidence interval [CI] = 46.1-50.7) compared with 37.1% of main campus graduates (95% CI = 36.5-37.7; P < .001). Of RMC graduates, 23.9% matched to rural-focused training programs (95% CI = 21.8-25.9) compared with 10.4% of main campus graduates (95% CI = 10.0-10.8; P < .001). Subanalyses ruled out a variety of potentially confounding variables., Conclusions: Combined model RMCs, in which all years of training take place away from the medical school's main campus, are associated with greater proportions of medical students entering family medicine postgraduate training and rural-focused training programs. These findings should encourage policymakers, health services agencies, and medical schools to continue seeking complements to academic medical center-based medical education., (Copyright © 2020 by the Association of American Medical Colleges.)

Purpose: Medical regulatory authorities are legally mandated to protect patients by monitoring the practice of medicine. While principally a matter of public safety, many pursue this mission by establishing quality improvement initiatives that prioritize professional development for all rather than identification of substandard performers. Engaging practitioners in directed learning opportunities, however, is rife with challenge given inherent social complexities. This study was run to explore whether relationship-centered coaching could improve physicians' perceptions of the value of engaging with College-mandated peer review., Method: A quasi-experimental analysis was performed on physician ratings of the effectiveness of peer assessor interactions and assessment processes during 3 time periods: (1) an historical control (March 2016-December 2016; n = 296); (2) a period after assessors were trained to deliver feedback using relationship-centered coaching (December 2016-March 2017; n = 96); and (3) after physicians were given more capacity to choose patient records for peer review and engage in discussion about multisource feedback results (March 2017-December 2018; n = 448)., Results: Psychometric analyses supported the aggregation of survey items into assessor interaction and assessment process subscores. Training assessors to engage in relationship-centered coaching was related with higher assessor interaction scores (4.64 vs 4.47; P < .05; d = 0.37). Assessment process scores did not increase until after additional program enhancements were made in period 3 (4.33 vs 4.17, P < .05, d = 0.29)., Conclusions: Despite peer interactions being inherently stressful for physicians when they occur in the context of regulatory authority visits, efforts to establish a quality improvement culture that prioritizes learning can improve physicians' perceptions of peer review.

Purpose: Accreditation aims to ensure all training programs meet agreed-upon standards of quality. The process is complex, resource intensive, and costly. Its benefits are difficult to assess because contextual confounds obscure comparisons between systems that do and do not include accreditation. This study explores accreditation's influence "within system" by investigating the relationship between accreditation cycle and performance on a national licensing examination., Method: Scores on the computer-based portion of the Medical Council of Canada Qualifying Examination Part I, from 1993 to 2017, were examined for all 17 Canadian medical schools. Typically completed upon graduation from medical school, results within each year were transformed for comparability across administrations and linked to timing within each school's accreditation cycle. ANOVAs were used to assess the relationship between accreditation timing and examination scores. Secondary analyses isolated 4-year from 3-year training programs and separated data generated before versus after implementation of a national midcycle informal review program., Results: Performance on the licensing exam was highest during and shortly after an accreditation site visit, falling significantly until the midpoint in the accreditation cycle (d = 0.47) before rising again. This pattern disappeared after introduction of informal interim review, but too little data have accumulated post implementation to determine if interim review is sufficient to break the influence of accreditation cycle., Conclusions: Formal, externally driven, accreditation cycles appear associated with educational processes in ways that translated into student outcomes on a national licensing examination. Whether informal, internal, interim reviews can mediate this effect remains to be seen.

Purpose: Written comments are gaining traction as robust sources of assessment data. Compared with the structure of numeric scales, what faculty choose to write is ad hoc, leading to idiosyncratic differences in what is recorded. This study offers exploration of what aspects of writing styles are determined by the faculty offering comment and what aspects are determined by the trainee being commented upon., Method: The authors compiled in-training evaluation report comment data, generated from 2012 to 2015 by 4 large North American Internal Medicine training programs. The Linguistic Index and Word Count (LIWC) was used to categorize and quantify the language contained. Generalizability theory was used to determine whether faculty could be reliably discriminated from one another based on writing style. Correlations and ANOVAs were used to determine what styles were related to faculty or trainee demographics., Results: Datasets contained 23-142 faculty who provided 549-2,666 assessments on 161-989 trainees. Faculty could easily be discriminated from one another using a variety of LIWC metrics including word count, words per sentence, and the use of "clout" words. These patterns appeared person specific and did not reflect demographic factors such as gender or rank. These metrics were similarly not consistently associated with trainee factors such as postgraduate year or gender., Conclusions: Faculty seem to have detectable writing styles that are relatively stable across the trainees they assess, which may represent an under-recognized source of construct irrelevance. If written comments are to meaningfully contribute to decision making, we need to understand and account for idiosyncratic writing styles.

Purpose: Using item analyses is an important quality-monitoring strategy for written exams. Authors urge caution as statistics may be unstable with small cohorts, making application of guidelines potentially detrimental. Given the small cohorts common in health professions education, this study's aim was to determine the impact of cohort size on outcomes arising from the application of item analysis guidelines., Method: The authors performed a Monte Carlo simulation study in fall 2015 to examine the impact of applying 2 commonly used item analysis guidelines on the proportion of items removed and overall exam reliability as a function of cohort size. Three variables were manipulated: Cohort size (6 levels), exam length (6 levels), and exam difficulty (3 levels). Study parameters were decided based on data provided by several Canadian medical schools., Results: The analyses showed an increase in proportion of items removed with decreases in exam difficulty and decreases in cohort size. There was no effect of exam length on this outcome. Exam length had a greater impact on exam reliability than did cohort size after applying item analysis guidelines. That is, exam reliability decreased more with shorter exams than with smaller cohorts., Conclusions: Although program directors and assessment creators have little control over their cohort sizes, they can control the length of their exams. Creating longer exams makes it possible to remove items without as much negative impact on the exam's reliability relative to shorter exams, thereby reducing the negative impact of small cohorts when applying item removal guidelines.

The authors describe influences associated with the incorporation of modern technologies into medical school admissions processes. Their purpose is not to critique or support specific technologies but, rather, to prompt reflection on the evolution that is afoot. Technology is now integral to the administration of multiple admissions tools, including the Medical College Admission Test, situational judgment tests, and standardized video interviews. Consequently, today's admissions landscape is transforming into an online, globally interconnected marketplace for health professions admissions tools. Academic capitalism and distance-based technologies combine to enable global marketing and dissemination of admissions tests beyond the national jurisdictions in which they are designed. As predicted by disruptive business theory, they are becoming key drivers of transformative change. The seeds of technological disruption are present now rather than something to be wary of in the future. The authors reflect on this transformation and the need for tailoring test modifications to address issues of medical student diversity and social responsibility. They comment on the online assessment of applicants' personal competencies and the potential detriments if this method were to replace admissions methods involving human contact, thanks to the ease with which institutions can implement them without cost to themselves and without adequate consideration of measurement utility or contextual appropriateness. The authors advocate for socially responsible academic capitalism within this interconnected admissions marketplace: Attending to today's transformative challenges may inform how health professions education responds to tomorrow's admissions technologies and, in turn, how tomorrow's health professionals respond to their patients' needs.

Purpose: Medical education is dependent on clinicians and other faculty who volunteer time and expertise to teaching. Unfortunately, the literature reports increasing levels of dissatisfaction, burnout, and attrition. Incentivization provides an obvious intervention, but rewards must be implemented judiciously or risk unintended consequences. With little known about the effects of incentives in medical education, the authors investigate key insights across three disciplines to explain how, why, and when incentives can be used effectively., Method: In this critical synthesis, a purposeful and iterative literature search was conducted by exploring a variety of databases to identify seminal articles, key concepts, and generative search terms. Particularly fruitful disciplines were then explored more deliberately., Results: Psychologists argue that the impact of an incentive depends on an individual's motivational drives. Organizational behaviorists draw attention to environmental incentives and disincentives that build or detract from motivation. Behavioral economists posit that size, type, and way in which an incentive is provided affect motivation differently., Conclusions: The influence of an incentive depends on how it interacts with underlying mechanisms deemed important for motivation. These mechanisms change across tasks, individuals, and contexts. Recommendations derived from the effort include being deliberate about (1) determining what is driving the individual to act, (2) considering the unique interactions between incentives and motivation types, and (3) considering barriers that may interfere with incentive effectiveness. In examining each of these, the authors argue that the field needs greater clarity regarding how, when, and why incentives operate within the many contexts in which medical educators work.

Purpose: There may be unintended consequences of broadening the competencies across which health professions trainees are assessed. This study was conducted to determine whether such broadening influences the formative guidance assessors provide to trainees and to test whether sequential collection of competency-specific assessment can overcome setbacks of simultaneous collection., Method: A randomized between-subjects experimental design, conducted in Toronto and Halifax, Canada, in 2016-2017 with paramedic educators experienced in observing/rating, in which observers' focus was manipulated. In the simultaneous condition, participants rated four unscripted (i.e., spontaneously generated) clinical performances using a six-dimension global rating scale and provided feedback. In three sequential conditions, participants were asked to rate the same performances and provide feedback but for only two of the six dimensions. Participants from these conditions were randomly merged to create a "full score" and set of feedback statements for each candidate., Results: Eighty-seven raters completed the study; 23 in the simultaneous condition and 21 or 22 for each pair of dimensions in the sequential conditions. After randomly merging participants, there were 21 "full scores" in the sequential condition. Compared with the sequential condition, participants in the simultaneous condition demonstrated reductions in the amount of unique feedback provided, increased likelihood of ignoring some dimensions of performance, lessened variety of feedback, and reduced reliability., Conclusions: Sequential or distributed assessment strategies in which raters are asked to focus on less may provide more effective assessment by overcoming the unintended consequences of asking raters to spread their attention thinly over many dimensions of competence.

The literature on multiple mini-interviews (MMIs) is replete with heterogeneous study results related to the constructs measured, correlations with other measures, and demographic relationships. Rather than view these results as contradictory, the authors ask, What if all of the results are correct? They point out that the MMI is not an assessment tool but, rather, an assessment method. Design and implementation of locally conducted MMIs in medical school admissions processes should reflect local needs. As with other local assessments, MMIs should be considered separate from nationally conducted assessments that reflect more universal competencies. With the freedom to exercise unique values in locally constructed MMIs, individual institutions, or small bands of like-minded institutions, in parallel carry the responsibility to ensure local assessment tool validity.

Purpose: The importance of confidence for learning and performance makes learners' perceptions of readiness for the next level of training valuable indicators of curricular success. The "Readiness for Clerkship" (RfC) and "Readiness for Residency" (RfR) surveys have been shown to provide reliable ratings of the relative effectiveness of various aspects of training. This study examines the generalizability of those results., Method: Surveys were administered at four medical schools approximately four months after the start of clerkship and eight months after the start of residency during 2013-2015. Collected data were anonymized. A total of 647 medical students and 483 residents participated., Results: Reliabilities of G = 0.8 could be obtained with only 6 to 12 medical students and 8 to 15 residents. Within MD programs, no meaningful differences in item ratings were observed across cohorts. Residents in each school consistently rated themselves higher than clerks on the majority of Medical Expert and Communicator competencies common to both surveys. Similar strengths and weaknesses were identified across programs, but differences were observed on five clerkship items and one residency item., Conclusions: Across four MD programs, the RfC and RfR surveys provided reliable ratings of the relative effectiveness of aspects of training with small numbers of respondents. The capacity of these surveys to efficiently identify perceived strengths and weaknesses held by cohorts of learners may, thereby, facilitate program improvement.

Purpose: Health professionals are expected to routinely assess their weaknesses, set learning goals, and monitor their achievement. Unfortunately, it is well known that these professionals often struggle with effectively integrating external data and self-perceptions. To know how best to intervene, it is critical that the health professionals community understand the cues students and practitioners use to assess their abilities. Here the authors aimed to gain insights into how and why medical students set learning goals, monitor their progress, and demonstrate their learning., Method: In 2012, the authors conducted semistructured interviews with Year 2 students (n = 20), applying an inductive approach to data analysis by iteratively developing, refining, and testing coding structures., Results: Themes were constructed through discussion and consensus: (1) Students were diverse in how they set learning goals, (2) they used a range of approaches to monitor their progress, and (3) they struggled to balance studying for exams with preparation for clinical training. Tensions observed highlight assumptions embedded in medical curricula that can be problematic., Conclusions: Educators often treat medical students as a cohesive whole, thereby creating a mismatch between assessments that are intended to be formative and information students use to monitor their progress. Despite limited exposure to clinical contexts, goal generation and monitoring often stem from a desire to prepare for clinical practice. In grappling with these tensions, it is important to be mindful that students are individualistic in how they balance their commitment to prepare for clinical work and the need to concentrate on exams.

Purpose: In-training evaluation reports (ITERs) are ubiquitous in internal medicine (IM) residency. Written comments can provide a rich data source, yet are often overlooked. This study determined the reliability of using variable amounts of commentary to discriminate between residents., Method: ITER comments from two cohorts of PGY-1s in IM at the University of Toronto (graduating 2010 and 2011; n = 46-48) were put into sets containing 15 to 16 residents. Parallel sets were created: one with comments from the full year and one with comments from only the first three assessments. Each set was rank-ordered by four internists external to the program between April 2014 and May 2015 (n = 24). Generalizability analyses and a decision study were performed., Results: For the full year of comments, reliability coefficients averaged across four rankers were G = 0.85 and G = 0.91 for the two cohorts. For a single ranker, G = 0.60 and G = 0.73. Using only the first three assessments, reliabilities remained high at G = 0.66 and G = 0.60 for a single ranker. In a decision study, if two internists ranked the first three assessments, reliability would be G = 0.80 and G = 0.75 for the two cohorts., Conclusions: Using written comments to discriminate between residents can be extremely reliable even after only several reports are collected. This suggests a way to identify residents early on who may require attention. These findings contribute evidence to support the validity argument for using qualitative data for assessment.

Purpose: Training to become a physician is an emotionally laden experience. Research in cognitive psychology indicates that emotions can influence learning and performance, but the materials used in such research (e.g., word lists) rarely reflect the complexity of material presented in medical school. The present study examined whether emotions influence learning of basic science principles., Method: Fifty-five undergraduate psychology students were randomly assigned to write about positive, negative, or neutral life events for nine minutes. Participants were then taught three physiological concepts, each in the context of a single organ system. Testing consisted of 13 clinical cases, 7 presented with the same concept/organ system pairing used during training ("near transfer") and 6 with novel pairings ("far transfer"). Testing was repeated after one week with 13 additional cases., Results: Forty-nine students provided complete data. Higher test scores were found when the concept/organ system pairing was held constant (near transfer = 51% correct vs. far = 33%; P < .001). Emotion condition influenced participants' overall performance, with individuals in the neutral condition (50.1%) performing better than those in the positive (38.2%, P < .05) and negative (37.7%, P < .001) emotion conditions., Conclusions: These data suggest that regardless of whether the emotion is positive or negative, mild affective states can impair learning of basic science concepts by novices. Demands on working memory and subsequent cognitive load provide a potential explanation. Future work will examine the extent to which these findings generalize to medical trainees.

Purpose: Interprofessional teamwork should include interprofessional feedback to optimize performance and collaboration. Social identity theory predicts that hierarchy and stereotypes may limit receptiveness to interprofessional feedback, but literature on this is sparse. This study explores perceptions among health professions students regarding interprofessional peer feedback received after a team exercise., Method: In 2012-2013, students from seven health professions schools (medicine, pharmacy, nursing, dentistry, physical therapy, dietetics, and social work) participated in a team-based interprofessional exercise early in clinical training. Afterward, they wrote anonymous feedback comments for each other. Each student subsequently completed an online survey to rate the usefulness and positivity (on five-point scales) of feedback received and guessed each comment's source. Data analysis included analysis of variance to examine interactions (on usefulness and positivity ratings) between profession of feedback recipients and providers., Results: Of 353 study participants, 242 (68.6%) accessed the feedback and 221 (62.6%) completed the survey. Overall, students perceived the feedback as useful (means across professions = 3.84-4.27) and positive (means = 4.17-4.86). There was no main effect of profession of the feedback provider, and no interactions between profession of recipient and profession of provider regardless of whether the actual or guessed provider profession was entered into the analysis., Conclusions: These findings suggest that students have positive perceptions of interprofessional feedback without systematic bias against any specific group. Whether students actually use interprofessional feedback for performance improvement and remain receptive toward such feedback as they progress in their professional education deserves further study.

Purpose: To compare the relative utility of open-book examinations (OBEs) and closed-book examinations (CBEs) given the rapid expansion and accessibility of knowledge., Method: A systematic review of peer-reviewed articles retrieved from MEDLINE, ERIC, Embase, and PsycINFO (through June 2013). In 2013-2014, articles that met inclusion criteria were reviewed by at least two investigators and coded for six outcome categories: (1) examination preparation, (2) test anxiety, (3) exam performance, (4) psychometrics and logistics, (5) testing effects, and (6) public perception., Results: From 4,192 identified studies, 37 were included. The level of learner and subject studied varied. The frequency of each outcome category was as follows: (1) exam preparation (n = 20; 54%); (2) test anxiety (n = 14; 38%); (3) exam performance (n = 30; 81%); (4) psychometrics and logistics (n = 5; 14%); (5) testing effects (n = 24; 65%); and (6) public perception (n = 5; 14%). Preexamination outcome findings were equivocal, but students may prepare more extensively for CBEs. For during-examination outcomes, examinees appear to take longer to complete OBEs. Studies addressing examination performance favored CBE, particularly when preparation for CBE was greater than for OBE. Postexamination outcomes suggest little difference in testing effects or public perception., Conclusions: Given the data available, there does not appear to be sufficient evidence for exclusively using CBE or OBE. As such, a combined approach could become a more significant part of testing protocols as licensing bodies seek ways to assess competencies other than the maintenance of medical knowledge.

Background: Health professions programs continue to search for meaningful and efficient ways to evaluate the quality of education they provide and support ongoing program improvement. Despite flaws inherent in self-assessment, recent research suggests that aggregated self-assessments reliably rank aspects of competence attained during preclerkship MD training. Given the novelty of those observations, the purpose of this study was to test their generalizability by evaluating an MD program as a whole., Method: The Readiness for Residency Survey (RfR) was developed and aligned with the published Readiness for Clerkship Survey (RfC), but focused on the competencies expected to be achieved at graduation. The RfC and RfR were administered electronically four months after the start of clerkship and six months after the start of residency, respectively. Generalizability and decision studies examined the extent to which specific competencies were achieved relative to one another., Results: The reliability of scores assigned by a single resident was G = 0.32. However, a reliability of G = 0.80 could be obtained by averaging over as few as nine residents. Whereas highly rated competencies in the RfC resided within the CanMEDS domains of professional, communicator, and collaborator, five additional medical expert competencies emerged as strengths when the program was evaluated after completion by residents., Conclusions: Aggregated resident self-assessments obtained using the RfR reliably differentiate aspects of competence attained over four years of undergraduate training. The RfR and RfC together can be used as evaluation tools to identify areas of strength and weakness in an undergraduate medical education program.

Purpose: Laboratory studies have shown that performance assessment judgments can be biased by "contrast effects." Assessors' scores become more positive, for example, when the assessed performance is preceded by relatively weak candidates. The authors queried whether this effect occurs in real, high-stakes performance assessments despite increased formality and behavioral descriptors., Method: Data were obtained for the 2011 United Kingdom Foundational Programme clinical assessment and the 2008 University of Alberta Multiple Mini Interview. Candidate scores were compared with scores for immediately preceding candidates and progressively distant candidates. In addition, average scores for the preceding three candidates were calculated. Relationships between these variables were examined using linear regression., Results: Negative relationships were observed between index scores and both immediately preceding and recent scores for all exam formats. Relationships were greater between index scores and the average of the three preceding scores. These effects persisted even when examiners had judged several performances, explaining up to 11% of observed variance on some occasions., Conclusions: These findings suggest that contrast effects do influence examiner judgments in high-stakes performance-based assessments. Although the observed effect was smaller than observed in experimentally controlled laboratory studies, this is to be expected given that real-world data lessen the strength of the intervention by virtue of less distinct differences between candidates. Although it is possible that the format of circuital exams reduces examiners' susceptibility to these influences, the finding of a persistent effect after examiners had judged several candidates suggests that the potential influence on candidate scores should not be ignored.

Purpose: Social judgment research suggests that rater unreliability in performance assessments arises from raters' differing inferences about the performer and the underlying reasons for the performance observed. These varying social judgments are not entirely idiosyncratic but, rather, tend to partition into a finite number of distinct subgroups, suggesting some "signal" in the "noise" of interrater variability. The authors investigated the proportion of variance in Mini-CEX ratings attributable to such partitions of raters' social judgments about residents., Method: In 2012 and 2013, physicians reviewed video-recorded patient encounters for seven residents, completed a Mini-CEX, and described their social judgments of the residents. Additional participants sorted these descriptions, which were analyzed using latent partition analysis (LPA). The best-fitting set of partitions for each resident served as an independent variable in a one-way ANOVA to determine the proportion of variance explained in Mini-CEX ratings., Results: Forty-eight physicians rated at least one resident (34 assessed all seven). The seven sets of social judgments were sorted by 14 participants. Across residents, 2 to 5 partitions (mode: 4) provided a good LPA fit, suggesting that subgroups of raters were making similar social judgments, while different causal explanations for each resident's performance existed across subgroups. The partitions accounted for 9% to 57% of the variance in Mini-CEX ratings across residents (mean = 32%)., Conclusions: These findings suggest that multiple "signals" do exist within the "noise" of interrater variability in performance-based assessment. It may be valuable to understand and exploit these multiple signals rather than try to eliminate them.

Purpose: Although scores on in-training evaluation reports (ITERs) are often criticized for poor reliability and validity, ITER comments may yield valuable information. The authors assessed across-rotation reliability of ITER scores in one internal medicine program, ability of ITER scores and comments to predict postgraduate year three (PGY3) performance, and reliability and incremental predictive validity of attendings' analysis of written comments., Method: Numeric and narrative data from the first two years of ITERs for one cohort of residents at the University of Toronto Faculty of Medicine (2009-2011) were assessed for reliability and predictive validity of third-year performance. Twenty-four faculty attendings rank-ordered comments (without scores) such that each resident was ranked by three faculty. Mean ITER scores and comment rankings were submitted to regression analyses; dependent variables were PGY3 ITER scores and program directors' rankings., Results: Reliabilities of ITER scores across nine rotations for 63 residents were 0.53 for both postgraduate year one (PGY1) and postgraduate year two (PGY2). Interrater reliabilities across three attendings' rankings were 0.83 for PGY1 and 0.79 for PGY2. There were strong correlations between ITER scores and comments within each year (0.72 and 0.70). Regressions revealed that PGY1 and PGY2 ITER scores collectively explained 25% of variance in PGY3 scores and 46% of variance in PGY3 rankings. Comment rankings did not improve predictions., Conclusions: ITER scores across multiple rotations showed decent reliability and predictive validity. Comment ranks did not add to the predictive ability, but correlation analyses suggest that trainee performance can be measured through these comments.

Purpose: Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought., Method: This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition., Results: A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition., Conclusions: Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.

Purpose: To examine whether or not aggregated self-assessment data of clerkship readiness can provide meaningful sources of information to evaluate the effectiveness of an educational program., Method: The 39-item Readiness for Clerkship survey was developed during academic year 2009-2010 using several key competence documents and expert review. The survey was completed by two cohorts of students (179 from the class of 2011 in February 2010, 171 from the class of 2012 in November 2010) and of clinical preceptors (384 for class of 2011 preceptors, 419 for class of 2012 preceptors). Descriptive statistics, Pearson correlations coefficients, ANOVA, and generalizability and decision studies were used to determine whether ratings could differentiate between different aspects of a training program., Results: When self-assessments were aggregated across students, their judgments aligned very well with those of faculty raters. The correlation of average scores, calculated for each item between faculty and students, was r=0.88 for 2011 and r=0.91 for 2012. This was only slightly lower than the near-perfect correlations of item averages within groups across successive years (r=0.99 for faculty; r=0.98 for students). Generalizability and decision analyses revealed that one can achieve interrater reliability in this domain with fewer students (9-21) than faculty (26-45)., Conclusions: These results provide evidence that, when aggregated, student self-assessment data from the Readiness for Clerkship Survey provide valid data for use in program evaluation that align well with an external standard.

Purpose: Medical school and residency are emotional experiences for trainees. Most research examining emotion in medicine has focused on negative moods associated with physician burnout and poor quality of life. However, positive emotional states also may have important influences on student learning and performance. The authors present a review of the literature on the influence of emotion on cognition, specifically how individuals learn complex skills and knowledge and how they transfer that information to new scenarios., Method: From September 2011 to February 2012, the authors searched Medline, PsycInfo, GoogleScholar, ERIC, and Web of Science, as well as the reference lists of relevant articles, for research on the interaction between emotion, learning, and knowledge transfer. They extracted representative themes and noted particularly relevant empirical findings., Results: The authors found articles that show that emotion influences various cognitive processes that are involved in the acquisition and transfer of knowledge and skills. More specifically, emotion influences how individuals identify and perceive information, how they interpret it, and how they act on the information available in learning and practice situations., Conclusions: There are many ways in which emotions may influence medical education. Researchers must further explore the implications of these findings to ensure that learning is not treated simply as a rational, mechanistic process but that trainees are effectively prepared to perform under a wide range of emotional conditions.

Purpose: To further the development of a substantive theory of expert diagnostic practice, the authors explored the ways in which exceptional physicians, nominated by their peers, conceptualized their own diagnostic expertise specifically and diagnostic excellence generally., Method: In this grounded theory, interview-based study, physicians at six North American research sites were nominated by their peers as exceptional diagnosticians and exceptional professionals and invited to participate in the study. A saturation sample included 34 participants, 20 exceptional diagnosticians, and 14 exceptional professionals. Using a constant comparative approach, the authors conducted one-on-one interviews with participants, transcribed the audiotapes of those interviews, and analyzed them for emergent themes. They developed a stable thematic structure and applied it to the entire data set., Results: Four interconnected themes emerged that inform the community's understanding of how physicians conceptualize expert diagnostic practice: (1) possession of extensive knowledge built through purposeful, continuous engagement in clinical practice, (2) possession of the skills to effectively gather patient stories, (3) reflective integration of that knowledge and those patient stories during diagnosis, and (4) continuous learning through clinical practice., Conclusions: Exploring these results within the context of current discourses in medical education brings to the forefront two key features of physicians' construction of diagnostic excellence: (1) the integrated nature of the medical competencies that underpin the diagnostic process and (2) the optimally adaptive nature of the diagnostic process. These findings can inform the development of practical and effective pedagogical strategies to guide trainees, clinicians, and medical educators who strive for excellence.

Purpose: Traditional medical school admissions assessment tools may be limiting diversity. This study investigates whether the Multiple Mini-Interview (MMI) is diversity-neutral and, if so, whether applying it with greater weight would dilute the anticipated negative impact of diversity-limiting admissions measures., Method: Interviewed applicants to six medical schools in 2008 and 2009 underwent MMI. Predictor variables of MMI scores, grade point average (GPA), and Medical College Admission Test (MCAT) scores were correlated with diversity measures of age, gender, size of community of origin, income level, and self-declared aboriginal status. A subset of the data was then combined with variable weight assigned to predictor variables to determine whether weighting during the applicant selection process would affect diversity among chosen applicants., Results: MMI scores were unrelated to gender, size of community of origin, and income level. They correlated positively with age and negatively with aboriginal status. GPA and MCAT correlated negatively with age and aboriginal status, GPA correlated positively with income level, and MCAT correlated positively with size of community of origin. Even extreme combinations of MMI and GPA weightings failed to increase diversity among applicants who would be selected on the basis of weighted criteria., Conclusions: MMI could not neutralize the diversity-limiting properties of academic scores as selection criteria to interview. Using academic scores in this way causes range restriction, counteracting attempts to enhance diversity using downstream admissions selection measures such as MMI. Diversity efforts should instead be focused upstream. These results lend further support for the development of pipeline programs.

Purpose: Most efforts to develop reliable evaluations of clinical competence have been oriented toward deconstructing the requisite competencies into separate scales. However, many are questioning the value of this approach on theoretical and empirical bases. This study uses "standardized narratives" to explore a different approach to assessing resident performance., Method: In 2009, based on interviews with 19 experienced clinical faculty from two institutions, 16 narrative profiles were created to represent the range of resident competence that clinical faculty might encounter during supervision. Fourteen clinicians from three institutions independently grouped the profiles into as many categories as necessary to reflect various levels of performance, described their categories, then ranked the individual profiles within each category. Then, in groups of three or four, participants negotiated a final ranking and grouping of the 16 profiles., Results: Despite interesting idiosyncracies in the factors some participants identified as guiding their rankings, there was strong consistency across the 14 clinicians regarding the rankings (single-rater intraclass correlation [ICC] = 0.86) and groupings (single-rater ICC = 0.81) of the profiles. Similarly, across institutions, the four groups were highly consistent in their final negotiated rankings (single-group ICC = 0.91) and groupings (single-group ICC = 0.87) of the profiles., Conclusions: Faculty showed more consistency in their decisions of what constitutes excellent, competent, and problematic performance in residents than implied by current assessment techniques that require deconstruction of resident competencies. This use of standardized narratives points to interesting opportunities for more authentically codifying faculty opinions of residents.

Background: To understand the modern conceptions of elite practice informing the hidden curriculum through use of peer nominations asking clinicians to identify exceptional practitioners., Method: We distributed a Web-based survey to Department of Medicine faculty at five universities in North America. Participants were asked to nominate individuals they deemed to be "outstanding practitioners" and to provide reasons. They were then asked to nominate "exceptional diagnosticians" and "exceptional professionals.", Results: Two hundred eighty-two physicians nominated 558 unique peers as "outstanding practitioners." Justifications included knowledge (45.1%), patient-related interpersonal skill (18.7%), teaching skill (10.8%), and research success (6.8%). More "exceptional diagnostician" nominees were nominated as "outstanding practitioners" (65.2%) relative to "exceptional professional" nominees (56.1%), although the effect size was small (phi = 0.09)., Conclusions: Knowledge-based competencies maintain a central role in modern conceptions of elite medical practice, although, contrary to the historical dominance of biomedical abilities, a diverse set of skills and professional aptitudes are also well represented.

Background: Measurement errors are a limitation of using rater-based assessments that are commonly attributed to rater errors. Solutions targeting rater subjectivity have been largely unsuccessful., Method: This critical review examines investigations of rater idiosyncrasy from impression formation literatures to ask new questions for the parallel problem in rater-based assessments., Results: Raters may form categorical judgments about ratees as part of impression formation. Although categorization can be idiosyncratic, raters tend to consistently construct one of a few possible interpretations of each ratee. If raters naturally form categorical judgments, an assessment system requiring ordinal or interval ratings may inadvertently introduce conversion errors due to translation techniques unique to each rater., Conclusions: Potential implications of raters forming differing categorizations of ratees combined with the use of rating scales to collect categorical judgments on measurement outcomes in rater-based assessments are explored.

Purpose: Informed self-assessment describes the set of processes through which individuals use external and internal data to generate an appraisal of their own abilities. The purpose of this project was to explore the tensions described by learners and professionals when informing their self-assessments of clinical performance., Method: This 2008 qualitative study was guided by principles of grounded theory. Eight programs in five countries across undergraduate, postgraduate, and continuing medical education were purposively sampled. Seventeen focus groups were held (134 participants). Detailed analyses were conducted iteratively to understand themes and relationships., Results: Participants experienced multiple tensions in informed self-assessment. Three categories of tensions emerged: within people (e.g., wanting feedback, yet fearing disconfirming feedback), between people (e.g., providing genuine feedback yet wanting to preserve relationships), and in the learning/practice environment (e.g., engaging in authentic self-assessment activities versus "playing the evaluation game"). Tensions were ongoing, contextual, and dynamic; they prevailed across participant groups, infusing all components of informed self-assessment. They also were present in varied contexts and at all levels of learners and practicing physicians., Conclusions: Multiple tensions, requiring ongoing negotiation and renegotiation, are inherent in informed self-assessment. Tensions are both intraindividual and interindividual and they are culturally situated, reflecting both professional and institutional influences. Social learning theories (social cognitive theory) and sociocultural theories of learning (situated learning and communities of practice) may inform our understanding and interpretation of the study findings. The findings suggest that educational interventions should be directed at individual, collective, and institutional cultural levels. Implications for practice are presented.

Background: The Multiple Mini-Interview (MMI) is useful in selecting undergraduate medical trainees. Postgraduate applicant pools have smaller numbers of more homogeneous candidates that must be actively recruited while being assessed. This paper reports on the MMI's use in assessing residency candidates., Method: Canadian and international medical graduates to three residency programs--obstetrics-gynecology and pediatrics (McMaster University) and internal medicine (University of Alberta)--underwent the MMI for residency selection (n = 484) in 2008 and 2009. Reliability was determined and candidates and interviewers completed an exit survey assessing acceptability., Results: Overall reliability of the MMI was acceptable, ranging from 0.55 to 0.72. Using 10 stations would increase reliability to 0.64-0.79. Eighty-eight percent of candidates believed they could accurately portray themselves, while 90% of interviewers believed they could reasonably judge candidates' abilities., Conclusions: The MMI provides a reliable way to assess residency candidates that is acceptable to both candidates and assessors across a variety of programs.

Background: While concern has been expressed about the validity of self-assessments, external feedback is likely filtered through self-assessment. This paper explores the relationship between self-assessments and feedback uptake., Method: During an objective structured clinical examination, students were asked to evaluate their performance and rate the quality of feedback provided by observers. Afterward, they were asked to list learning goals they generated, to indicate what activities they would undertake to fulfill those goals, and to identify which station(s) led them to generate each response. Regression analyses were used to determine which variables predicted the generation of goals/activities., Results: Students' perceptions of their own performance were more likely to result in the generation of goals/strategies than was observer feedback or student perceptions of observer feedback quality. Later stations were more likely to result in goal/strategy generation than earlier stations., Conclusions: While self-assessments may not validly indicate ability, it is still critical to determine how students perceive their ability because their opinions drive their learning goals.

Purpose: Premature closure has been identified as the single most common cause of diagnostic error. This factorial experiment explored which variables exert an unconfounded influence on physicians' diagnostic flexibility (changing their minds about the most likely diagnosis during a clinical case presentation)., Method: In 2007-2008, 256 practicing physicians viewed a clinically authentic vignette simulating a patient presenting with possible coronary heart disease (CHD) and provided their initial impression midway through the case. At the end, they answered questions about the case, indicated how they would continue their clinical investigation, and made a final diagnosis. The authors used general linear models to determine which patient factors (age, gender, socioeconomic status, race), physician factors (gender, age/experience), and process variables were related to the likelihood of physicians' changing their minds about the most likely diagnosis., Results: Physicians who had less experience, those who named a non-CHD diagnosis as their initial impression, and those who did not ask for information about the patient's prior cardiac disease history were the most likely to change their minds. Participants' certainty in their initial diagnosis, the additional information desired, the diagnostic hypotheses generated, and the follow-up intended were not related to the likelihood of change in diagnostic hypotheses., Conclusions: Although efforts encouraging physicians to avoid cognitive biases and to reason in a more analytic manner may yield some benefit, this study suggests that experience is a more important determinant of diagnostic flexibility than is the consideration of additional diagnoses or the amount of additional information collected.

Purpose: To determine how learners and physicians engaged in various structured interventions to inform self-assessment, how they perceived and used self-assessment in clinical learning and practice, and the components and processes comprising informed self-assessment and factors that influence these., Method: This was a qualitative study guided by principles of grounded theory. Using purposive sampling, eight programs were selected in Canada, the United States, the United Kingdom, the Netherlands, and Belgium, representing low, medium, and high degrees of structure/rigor in self-assessment activities. In 2008, 17 focus groups were conducted with 134 participants (53 undergraduate learners, 32 postgraduate learners, 49 physicians). Focus-group transcripts were analyzed interactively and iteratively by the research team to identify themes and compare and confirm findings., Results: Informed self-assessment appeared as a flexible, dynamic process of accessing, interpreting, and responding to varied external and internal data. It was characterized by multiple tensions arising from complex interactions among competing internal and external data and multiple influencing conditions. The complex process was evident across the continuum of medical education and practice. A conceptual model of informed self-assessment emerged., Conclusions: Central challenges to informing self-assessment are the dynamic interrelationships and underlying tensions among the components comprising self-assessment. Realizing this increases understanding of why self-assessment accuracy seems frequently unreliable. Findings suggest the need for attention to the varied influencing conditions and inherent tensions to progress in understanding self-assessment, how it is informed, and its role in self-directed learning and professional self-regulation. Informed self-assessment is a multidimensional, complex construct requiring further research.

Purpose: The drive toward competency-based education frameworks has created a tension between competing desires-for quantified, standardized measures on one hand, and for an authentic representation of what it means to be a good doctor on the other. The purpose of this study was to better understand the tensions that exist between competency frameworks and faculty's real-life experiences in evaluating residents., Method: Interviews were conducted with 19 experienced internal medicine attendings at two Canadian universities in 2007. Attendings each discussed a specific outstanding, average, and problematic resident they had supervised. Interviews were analyzed using grounded theory., Results: Eight major themes emerged reflecting how faculty conceptualize residents' performance: knowledge, professionalism, patient interactions, team interactions, systems, disposition, trust, and impact on staff. Attendings' impressions of residents did not seem to result from a linear sum of dimensions; rather, domains idiosyncratically took on variable degrees of importance depending on the resident. Relative deficiencies in outstanding residents could be overlooked, whereas strengths in problematic residents could be discounted. Some constructs (e.g., impact on staff) were not competencies at all; rather, they seem to act as explanations or evidence of attendings' opinions. Standardized evaluation forms might constrain authentic depictions of residents' performance., Conclusions: Despite concerted efforts to create standardized, objective, competency-based evaluations, the assessment of residents' clinical performance still has a strong subjective influence. Attendings' holistic impressions should not be considered invalid simply because they are subjective. Instead, assessment methods should consider novel ways of accommodating these impressions to improve evaluation.

Background: Most medical school candidates are excluded without benefit of noncognitive skills assessment. Is development of a noncognitive preinterview screening test that correlates with the well-validated Multiple Mini-Interview (MMI) possible?, Method: Study 1: 110 medical school candidates completed MMI and Computer-based Multiple Sample Evaluation of Noncognitive Skills (CMSENS)-eight 1-minute video-based scenarios and four self-descriptive questions, with short-answer-response format. Seventy-eight responses were audiotaped, 32 typewritten; all were scored by two independent raters. Study 2: 167 candidates completed CMSENS-eight videos, six self-descriptive questions, typewritten responses only, scored by two raters; 88 of 167 underwent the MMI., Results: Results for overall test generalizability, interrater reliability, and correlation with MMI, respectively, were, for Study 1, audio-responders: 0.86, 0.82, 0.15; typewritten-responders: 0.72, 0.81, 0.51; and for Study 2, 0.83, 0.95, 0.46 (correlation with disattenuation was 0.60)., Conclusions: Strong psychometric properties, including MMI correlation, of CMSENS warrant investigation into future widespread implementation as a preinterview noncognitive screening test.

Background: Because of common use of adolescent simulated patients (ASPs), clarification of the risk of negative effects associated with high-stress simulations is essential. This study evaluates the safety of suicidality simulations., Method: ASPs participated in a suicidality role or pediatric role. Explicit measures of their impact included the Suicidal Ideation Questionnaire (SIQ) and Reynolds Adolescent Depression Scale-2 (RADS-2), both of which were completed pre-/poststudy. Implicit behavioral measures of impact were also collected during training, including the Implicit Association Test., Results: Three of the 24 participants had clinical RADS-2 and/or SIQ scores. None of these adolescents' mental status deteriorated with SP participation. ASPs in the suicidality role showed behavioral effects consistent with a negative reaction, and two reported brief depression., Conclusions: ASPs participated in a suicidality simulation without evidence of suicide contagion. However, ASPs' behavioral reactions and self-reported depression suggested a transient depressive reaction.

Background: Some form of candidate-written autobiographical submission (ABS) is commonly used before interviews to screen candidates to medical school on the basis of their noncognitive characteristics. However, confidence in the validity of these measures has been questioned., Method: In 2005, applicants to McMaster University completed an off-site ABS before being interviewed and an on-site ABS at interview. Five off-site ABS questions were completed, plus eight on-site questions. On-site ABS questions were answered in variable timing conditions. ABS ratings were compared across sites and time allowed for completion., Results: Off-site ABS ratings were higher than on-site ratings, and the two sets of ratings were uncorrelated with one another. On-site ABS ratings increased with increased time allowed for completion, but the reliability of the measure was unaffected by this variable., Conclusions: Confidence that candidates independently answer preinterview ABS questions is weak. To improve ABS validity, modification of the current Web-based submission format warrants consideration.