Search

Performance on medical licensing examinations has been previously shown to be predictive of performance in practice. However, licensing examinations are closed-book and real-world medical practice increasingly requires doctors and patients to consult resources to make evidence-informed decisions. To best assess the ability of physicians and physicians-in-practice to avail themselves of point-of-care clinical resources and tools, open-book components may have an emerging role in high-stakes examinations., Competing Interests: Dr. Beth-Ann Cummings has no conflicts of interest with regard to this paper, and this work has not been funded. An earlier version of this paper was prepared by Dr. Cummings, to inform the work of the Medical Council of Canada Assessment Innovation Taskforce., (© 2022 Cummings; licensee Synergies Partners.)

Purpose: Written examinations such as multiple-choice question (MCQ) exams are a key assessment strategy in health professions education (HPE), frequently used to provide feedback, to determine competency, or for licensure decisions. However, traditional psychometric approaches for monitoring the quality of written exams, defined as items that are discriminant and contribute to increase the overall reliability and validity of the exam scores, usually warrant larger samples than are typically available in HPE contexts. The authors conducted a descriptive exploratory study to document how undergraduate medical education (UME) programs ensure the quality of their written exams, particularly MCQs., Method: Using a qualitative descriptive methodology, the authors conducted semistructured interviews with 16 key informants from 10 Canadian UME programs in 2018. Interviews were transcribed, anonymized, coded by the primary investigator, and co-coded by a second team member. Data collection and analysis were conducted iteratively. Research team members engaged in analysis across phases, and consensus was reached on the interpretation of findings via group discussion., Results: Participants focused their answers around MCQ-related practices, reporting using several indicators of quality such as alignment between items and course objectives and psychometric properties (difficulty and discrimination). The authors clustered findings around 5 main themes: processes for creating MCQ exams, processes for building quality MCQ exams, processes for monitoring the quality of MCQ exams, motivation to build quality MCQ exams, and suggestions for improving processes., Conclusions: Participants reported engaging multiple strategies to ensure the quality of MCQ exams. Assessment quality considerations were integrated throughout the development and validation phases, reflecting recent work regarding validity as a social imperative.

Purpose: Educational handover (i.e., providing information about learners' past performance) is controversial. Proponents argue handover could help tailor learning opportunities. Opponents fear it could bias subsequent assessments and lead to self-fulfilling prophecies. This study examined whether raters provided with reports describing learners' minor weaknesses would generate different assessment scores or narrative comments than those who did not receive such reports., Method: In this 2018 mixed-methods, randomized, controlled, experimental study, clinical supervisors from 5 postgraduate (residency) programs were randomized into 3 groups receiving no educational handover (control), educational handover describing weaknesses in medical expertise, and educational handover describing weaknesses in communication. All participants watched the same videos of 2 simulated resident-patient encounters and assessed performance using a shortened mini-clinical evaluation exercise form. The authors compared mean scores, percentages of negative comments, comments focusing on medical expertise, and comments focusing on communication across experimental groups using analyses of variance. They examined potential moderating effects of supervisor experience, gender, and mindsets (fixed vs growth)., Results: Seventy-two supervisors participated. There was no effect of handover report on assessment scores (F(2, 69) = 0.31, P = .74) or percentage of negative comments (F(2, 60) = 0.33, P = .72). Participants who received a report indicating weaknesses in communication generated a higher percentage of comments on communication than the control group (63% vs 50%, P = .03). Participants who received a report indicating weaknesses in medical expertise generated a similar percentage of comments on expertise compared to the controls (46% vs 47%, P = .98)., Conclusions: This study provides initial empirical data about the effects of educational handover and suggests it can-in some circumstances-lead to more targeted feedback without influencing scores. Further studies are required to examine the influence of reports for a variety of performance levels, areas of weakness, and learners., (Copyright © 2020 by the Association of American Medical Colleges.)

Introduction: Current medical education models increasingly rely on longitudinal assessments to document learner progress over time. This longitudinal focus has re-kindled discussion regarding learner handover-where assessments are shared across supervisors, rotations, and educational phases, to support learner growth and ease transitions. The authors explored the opinions of, experiences with, and recommendations for successful implementation of learner handover among clinical supervisors., Methods: Clinical supervisors from five postgraduate medical education programs at one institution completed an online questionnaire exploring their views regarding learner handover, specifically: potential benefits, risks, and suggestions for implementation. Survey items included open-ended and numerical responses. The authors used an inductive content analysis approach to analyze the open-ended questionnaire responses, and descriptive and correlational analyses for numerical data., Results: Seventy-two participants completed the questionnaire. Their perspectives varied widely. Suggested benefits of learner handover included tailored learning, improved assessments, and enhanced patient safety. The main reported risk was the potential for learner handover to bias supervisors' perceptions of learners, thereby affecting the validity of future assessments and influencing the learner's educational opportunities and well-being. Participants' suggestions for implementation focused on who should be involved, when and for whom it should occur, and the content that should be shared., Discussion: The diverse opinions of, and recommendations for, learner handover highlight the necessity for handover to maximize learning potential while minimizing potential harms. Supervisors' suggestions for handover implementation reveal tensions between assessment-of and for-learning.

Objective: To assess hospital employees' attitudes and needs regarding work commitments during disasters., Methods: A 12-item survey was distributed to employees at nine hospitals in five states. Questions addressed willingness to work during a disaster or its aftermath, support services that could encourage employees to remain for extended hours, and conflicting emergency response obligations (e.g., being a volunteer firefighter) that might prevent employees from working at the hospital. Anonymity was assured, and approval was obtained from each hospital's institutional review board., Results: Of the 2,004 surveys distributed, 1,711 (85 percent) were returned. Eighty-seven percent of respondents were willing to work after a fire/rescue/collapse mass casualty incident. Respondents were otherwise less willing to work in response to a man-made disaster (biological event: 58 percent; chemical event: 58 percent; radiation event: 57 percent) than a natural disaster (snowstorm: 83 percent; flood: 81 percent; hurricane: 78 percent; earthquake: 79 percent; tornado: 77 percent; ice storm: 75 percent; flu epidemic: 72 percent) (p < 0.001 for all comparisons by χ 2 testing). While 44 percent of respondents would come to work in response to any of the 11 disaster types listed, 19 percent were only willing to cover four or fewer types. Long-distance phone service (694, 41 percent), email access (584, 34 percent), pet care (568, 33 percent), and child care (506, 30 percent) were the most common support needs, and 365 respondents (21 percent) reported a conflicting emergency response obligation., Conclusions: The majority of hospital workers surveyed were willing to report to work in response to some types of disasters but not others, and some indicated they might not be available at all due to conflicting emergency response obligations.

Purpose: Current medical education models maintain that competencies such as professionalism and communication can be taught; however, some argue that certain attributes that make up these competencies, such as empathy, are fixed. Teachers' implicit theories, or mindsets (beliefs about the fixed versus learnable nature of human attributes) have been shown to impact their teaching and assessment practices; but little work has explored mindsets in medical education. We examined clinical supervisors' mindsets of two cognitive attributes (intelligence and clinical reasoning) and two affective attributes (moral character and empathy). Methods: Clinical supervisors ( n  = 40) from three specialities completed a survey designed to measure mindsets using two existing instruments for intelligence and moral character and 18 new items for clinical reasoning and empathy. Participants completed the survey twice for test-retest reliability ( n  = 25). Results: New items had satisfactory psychometric properties. Clinical supervisors' mindsets were mixed. Only 8% of participants saw clinical reasoning as fixed while more saw empathy (45%), intelligence (53%), and moral character (53%) as fixed - running counter to current educational models that characterize these attributes as learnable. Conclusion: This study provides evidence supporting the use of these new tools to measure mindsets that may help to better understand the impact of mindsets on medical education.

Introduction: In-training assessment reports (ITARs) summarize assessment during a clinical placement to inform decision-making and provide formal feedback to learners. Faculty development is an effective but resource-intensive means of improving the quality of completed ITARs. We examined whether the quality of completed ITARs could be improved by 'nudges' from the format of ITAR forms., Methods: Our first intervention consisted of placing the section for narrative comments at the beginning of the form, and using prompts for recommendations (Do more, Keep doing, Do less, Stop doing). In a second intervention, we provided a hyperlink to a detailed assessment rubric and shortened the checklist section. We analyzed a sample of 360 de-identified completed ITARs from six disciplines across the three academic years where the different versions of the ITAR were used. Two raters independently scored the ITARs using the Completed Clinical Evaluation Report Rating (CCERR) scale. We tested for differences between versions of the ITAR forms using a one-way ANOVA for the total CCERR score, and MANOVA for the nine CCERR item scores., Results: Changes to the form structure (nudges) improved the quality of information generated as measured by the CCERR instrument, from a total score of 18.0/45 (SD 2.6) to 18.9/45 (SD 3.1) and 18.8/45 (SD 2.6), p = 0.04. Specifically, comments were more balanced, more detailed, and more actionable compared with the original ITAR., Discussion: Nudge interventions, which are inexpensive and feasible, should be included in multipronged approaches to improve the quality of assessment reports.

Historically, students have been "consumers" of undergraduate medical education (UME) rather than stakeholders in its design and implementation. Student input has been retrospective, and although UME leaders have been open to feedback, matters most important to students have often been overlooked, leaving students feeling largely unheard. Student representation has also lacked structure and unity of feedback.A vision for effective student representation drove the creation of a partnered educational governance (PEG) model at McGill University in Montreal, Quebec, Canada, where sharing of expertise between student representatives and UME leadership has improved the UME program and the educational experience of students.The PEG model is grounded in the literature on student government, the student-as-partner framework, and theories of accountability. As part of the model, the student Medical Education Committee, an organized structure for discussion and reporting to student constituents, was established. This structure allows student representatives, entrusted by their peers and faculty, to proactively provide input to UME committees in the development of policies and curricula. The partnership between students and faculty facilitates a shared understanding of educational challenges and potential solutions.Within the first year, meaningful changes associated with the PEG model included increased student engagement in key program decisions, such as the redesign of a research course and an update to the absences and leaves policy. The PEG model enables unified student representation that is accountable and representative-and has a significant impact on outcomes-while maintaining the UME program's ownership of and responsibility for the curriculum and policies.

Context: Competency-based medical education has spurred the implementation of longitudinal workplace-based assessment (WBA) programmes to track learners' development of competencies. These hinge on the appropriate use of assessment instruments by assessors. This study aimed to validate our assessment programme and specifically to explore whether assessors' beliefs and behaviours rendered the detection of progress possible., Methods: We implemented a longitudinal WBA programme in the third year of a primarily rotation-based clerkship. The programme used the professionalism mini-evaluation exercise (P-MEX) to detect progress in generic competencies. We used mixed methods: a retrospective psychometric examination of student assessment data in one academic year, and a prospective focus group and interview study of assessors' beliefs and reported behaviours related to the assessment., Results: We analysed 1662 assessment forms for 186 students. We conducted interviews and focus groups with 21 assessors from different professions and disciplines. Scores were excellent from the outset (3.5-3.7/4), with no meaningful increase across blocks (average overall scores: 3.6 in block 1 versus 3.7 in blocks 2 and 3; F = 8.310, d.f. 2, p < 0.001). The main source of variance was the forms (47%) and only 1% of variance was attributable to students, which led to low generalisability across forms (Eρ 2  = 0.18). Assessors reported using multiple observations to produce their assessments and were reluctant to harm students by consigning anything negative to writing. They justified the use of a consistent benchmark across time by citing the basic nature of the form or a belief that the 'competencies' assessed were in fact fixed attributes that were unlikely to change., Conclusions: Assessors may purposefully deviate from instructions in order to meet their ethical standards of good assessment. Furthermore, generic competencies may be viewed as intrinsic and fixed rather than as learnable. Implementing a longitudinal WBA programme is complex and requires careful consideration of assessors' beliefs and values., (© 2018 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Objective: Rates of total knee arthroplasty (TKA) and total hip arthroplasty (THA) remain high for patients with rheumatoid arthritis (RA), who are at risk of flaring after surgery. We aimed to describe rates, characteristics, and risk factors of RA flare within 6 weeks of THA and TKA., Methods: Patients with RA were recruited prior to elective THA and TKA surgery and prospectively followed. Clinicians evaluated RA clinical characteristics 0-2 weeks before and 6 weeks after surgery. Patients answered questions regarding disease activity including self-reported joint counts and flare status weekly for 6 weeks. Per standard of care, biologics were stopped before surgery, while glucocorticoids and methotrexate (MTX) were typically continued. Multivariable logistic regression was used to identify baseline characteristics associated with postsurgical RA flares., Results: Of 120 patients, the mean age was 62 years and the median RA duration 14.8 years. Ninety-eight (82%) met 2010/1987 American College of Rheumatology/European League Against Rheumatism criteria, 53 (44%) underwent THA (and the rest TKA), and 61 (51%) were taking biologics. By 6 weeks, 75 (63%) had flared. At baseline, flarers had significantly higher disease activity (as measured by the 28-joint Disease Activity Score), erythrocyte sedimentation rate, C-reactive protein, and pain. Numerically more flarers used biologics, but stopping biologics did not predict flares, and continuing MTX was not protective. A higher baseline disease activity predicted flaring by 6 weeks (OR 2.12, p = 0.02)., Conclusion: Flares are frequent in patients with RA undergoing arthroplasty. Higher baseline disease activity significantly increases the risk. Although more patients stopping biologics flared, this did not independently predict flaring. The effect of early postsurgery flares requires further study.

Simulation allows for learner-centered health professions training by providing a safe environment to practice and make mistakes without jeopardizing patient care. It was with this goal in mind that the McGill Medical Simulation Center was officially opened on September 14, 2006, as a partnership between McGill University, the Faculty of Medicine and its affiliated hospitals. Its mandate is to provide state-of-the-art facilities to support simulation-based medical and allied health education initiatives. Since its inception, the center, recently renamed the Steinberg Center for Simulation and Interactive Learning (SCSIL), has undergone a major expansion and logged more than 130,000 learner visits. Educational activities are offered at all levels of medical and allied health care training, and include standardized patient encounters, partial task trainers, multidisciplinary courses, and high-fidelity trainers, among many others. In addition to its educational mandate, the center also supports an active research program, programs to enhance collaboration with disciplines outside of health care to spur innovation, and community outreach initiatives., (Copyright © 2017 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Introduction: Multiple-choice questions (MCQs) are a cornerstone of assessment in medical education. Monitoring item properties (difficulty and discrimination) are important means of investigating examination quality. However, most item property guidelines were developed for use on large cohorts of examinees; little empirical work has investigated the suitability of applying guidelines to item difficulty and discrimination coefficients estimated for small cohorts, such as those in medical education. We investigated the extent to which item properties vary across multiple clerkship cohorts to better understand the appropriateness of using such guidelines with small cohorts., Methods: Exam results for 32 items from an MCQ exam were used. Item discrimination and difficulty coefficients were calculated for 22 cohorts (n = 10-15 students). Discrimination coefficients were categorized according to Ebel and Frisbie (1991). Difficulty coefficients were categorized according to three guidelines by Laveault and Grégoire (2014). Descriptive analyses examined variance in item properties across cohorts., Results: A large amount of variance in item properties was found across cohorts. Discrimination coefficients for items varied greatly across cohorts, with 29/32 (91%) of items occurring in both Ebel and Frisbie's 'poor' and 'excellent' categories and 19/32 (59%) of items occurring in all five categories. For item difficulty coefficients, the application of different guidelines resulted in large variations in examination length (number of items removed ranged from 0 to 22)., Discussion: While the psychometric properties of items can provide information on item and exam quality, they vary greatly in small cohorts. The application of guidelines with small exam cohorts should be approached with caution.

Context: Over the past few decades, longitudinal integrated clerkships (LICs) have been proposed to address many perceived short-coming of traditional block clerkships. This growing interest in LICs has raised broader questions regarding the role of integration, continuity and longitudinality in medical education. A study with complementary theoretical and empirical dimensions was conducted to derive a more precise way of defining these three underlying concepts within the design of medical education curricula., Methods: The theoretical dimension involved a thematic review of the literature on integration, continuity and longitudinality in medical education. The empirical dimension surveyed all 17 Canadian medical schools on how they have operationalised integration, continuity and longitudinality in their undergraduate programmes. The two dimensions were iteratively synthesised to explore the meaning and expression of integration, continuity and longitudinality in medical education curriculum design., Results: Integration, continuity and longitudinality were expressed in many ways and forms, including: integration of clinical disciplines, combined horizontal integration and vertical integration, and programme-level integration. Types of continuity included: continuity of patients, continuity of teaching, continuity of location and peer continuity. Longitudinality focused on connected or repeating episodes of training or on connecting activities, such as encounter logging across educational episodes. Twelve of the 17 schools were running an LIC of some kind, although only one school had a mandatory LIC experience. An ordinal scale of uses of integration, continuity and longitudinality during clerkships was developed, and new definitions of these concepts in the clerkship context were generated., Conclusions: Different clerkship designs embodied different forms and levels of integration, continuity and longitudinality. A dichotomous view of LICs and rotation-based clerkships was found not to represent current practices in Canada, which instead tended to fall along a continuum of integration, continuity and longitudinality., (© 2016 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Context: Longitudinal integrated clerkships (LICs) represent a model of the structural redesign of clinical education that is growing in the USA, Canada, Australia and South Africa. By contrast with time-limited traditional block rotations, medical students in LICs provide comprehensive care of patients and populations in continuing learning relationships over time and across disciplines and venues. The evidence base for LICs reveals transformational professional and workforce outcomes derived from a number of small institution-specific studies., Objectives: This study is the first from an international collaborative formed to study the processes and outcomes of LICs across multiple institutions in different countries. It aims to establish a baseline reference typology to inform further research in this field., Methods: Data on all LIC and LIC-like programmes known to the members of the international Consortium of Longitudinal Integrated Clerkships were collected using a survey tool developed through a Delphi process and subsequently analysed. Data were collected from 54 programmes, 44 medical schools, seven countries and over 15 000 student-years of LIC-like curricula., Results: Wide variation in programme length, student numbers, health care settings and principal supervision was found. Three distinct typological programme clusters were identified and named according to programme length and discipline coverage: Comprehensive LICs; Blended LICs, and LIC-like Amalgamative Clerkships. Two major approaches emerged in terms of the sizes of communities and types of clinical supervision. These referred to programmes based in smaller communities with mainly family physicians or general practitioners as clinical supervisors, and those in more urban settings in which subspecialists were more prevalent., Conclusions: Three distinct LIC clusters are classified. These provide a foundational reference point for future studies on the processes and outcomes of LICs. The study also exemplifies a collaborative approach to medical education research that focuses on typology rather than on individual programme or context., (© 2016 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Background: Many countries have reduced resident duty hours in an effort to promote patient safety and enhance resident quality of life. There are concerns that reducing duty hours may impact residents' learning opportunities., Objectives: We (1) evaluated residents' perceptions of their current learning opportunities in a context of reduced duty hours, and (2) explored the perceived change in resident learning opportunities after call length was reduced from 24 continuous hours to 16 hours., Methods: We conducted an anonymous, cross-sectional online survey of 240 first-, second-, and third-year residents rotating through 3 McGill University-affiliated intensive care units (ICUs) in Montreal, Quebec, Canada, between July 1, 2012, and June 30, 2013. The survey investigated residents' perceptions of learning opportunities in both the 24-hour and 16-hour systems., Results: Of 240 residents, 168 (70%) completed the survey. Of these residents, 63 (38%) had been exposed to both 24-hour and 16-hour call schedules. The majority of respondents (83%) reported that didactic teaching sessions held by ICU staff physicians were useful. However, of the residents trained in both approaches to overnight call, 44% reported a reduction in learner attendance at didactic teaching sessions, 48% reported a reduction in attendance at midday hospital rounds, and 40% reported a perceived reduction in self-directed reading after the implementation of the new call schedule., Conclusions: A substantial proportion of residents perceived a reduction in the attendance of instructor-directed and self-directed reading after the implementation of a 16-hour call schedule in the ICU.