Search

In this article, the authors provide practical guidance for frontline supervisors' efforts to assess trainee performance. They focus on three areas. First, they argue the importance of promoting learner control in the assessment process, noting that providing learners agency and control can shift the stakes of assessment from high to low and promote a safe environment that facilitates learning. Second, they posit that assessment should be used to support continued development by promoting a relational partnership between trainees and supervisors. This partnership allows supervisors to reinforce desirable aspects of performance, provide real-time support for deficient areas of performance, and sequence learning with the appropriate amount of scaffolding to push trainees from competence (what they can do alone) to capability (what they are able to do with support). Finally, they advocate the importance of optimizing the use of written comments and direct observation while also recognizing that performance is interdependent in efforts to maximize assessment moments.Conclusion: Using best practices in trainee assessment can help trainees take next steps in their development in a learner-centered partnership with clinical supervisors. What is Known: • Many pediatricians are asked to assess the performance of medical students and residents they work with but few have received formal training in assessment. What is New: • This article presents evidence-based best practices for assessing trainees, including giving trainees agency in the assessment process and focusing on helping trainees take next steps in their development., (© 2021. The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.)

Processes involved in the regulation of learning have been researched for decades, because of its impact on academic and workplace performance. In fact, self-regulated learning is the focus of countless studies in health professions education and higher education in general. While we will always need competent individuals who are able to regulate their own learning, developments in healthcare require a shift from a focus on the individual to the collective: collaboration within and between healthcare teams is at the heart of high-quality patient care. Concepts of collaborative learning and collective competence challenge commonly held conceptualisations of regulatory learning and call for a focus on the social embeddedness of regulatory learning and processes regulating the learning of the collective. Therefore, this article questions the alignment of current conceptualisations of regulation of learning with demands for collaboration in current healthcare. We explore different conceptualisations of regulation of learning (self-, co-, and socially shared regulation of learning), and elaborate on how the integration of these conceptualisations adds to our understanding of regulatory learning in healthcare settings. Building on these insights, we furthermore suggest ways forward for research and educational practice., (© 2021 The Authors. Medical Education published by Association for the Study of Medical Education John Wiley & Sons Ltd.)

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence.

Objectives: With increased cross-border movement, ensuring safe and high-quality healthcare has gained primacy. The purpose of recertification is to ensure quality of care through periodically attesting doctors' professional proficiency in their field. Professional migration and facilitated cross-border recognition of qualifications, however, make us question the fitness of national policies for safeguarding patient care and the international accountability of doctors., Design and Setting: We performed document analyses and conducted 19 semistructured interviews to identify and describe key characteristics and effective components of 10 different European recertification systems, each representing one case (collective case study). We subsequently compared these systems to explore similarities and differences in terms of assessment criteria used to determine process quality., Results: Great variety existed between countries in terms and assessment formats used, targeting cognition, competence and performance (Miller's assessment pyramid). Recertification procedures and requirements also varied significantly, ranging from voluntary participation in professional development modules to the mandatory collection of multiple performance data in a competency-based portfolio. Knowledge assessment was fundamental to recertification in most countries. Another difference concerned the stakeholders involved in the recertification process: while some systems exclusively relied on doctors' self-assessment, others involved multiple stakeholders but rarely included patients in assessment of doctors' professional competence. Differences between systems partly reflected different goals and primary purposes of recertification., Conclusion: Recertification systems differ substantially internationally with regard to the criteria they apply to assess doctors' competence, their aims, requirements, assessment formats and patient involvement. In the light of professional mobility and associated demands for accountability, we recommend that competence assessment includes patients' perspectives, and recertification practices be shared internationally to enhance transparency. This can help facilitate cross-border movement, while guaranteeing high-quality patient care., Competing Interests: Competing interests: None declared., (© Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.)

Competency-based education has become central to the training and assessment of post-graduate medical trainees or residents [1]. In competency-based education, there is a strong focus on outcomes and professional performance. Typically, holistic tasks are used to train, practice and assess the defined outcomes or competencies. In residency training, these tasks are part of the day-to-day clinical practice. The performance of residents in the workplace needs to be captured and stored. A portfolio has been used as an instrument for storage and collection of workplace-based assessment and feedback in various countries, like the Netherlands and the United States. The collection of information in a portfolio can serve or be used for a variety of purposes. These are: The collection of work samples, assessment, feedback and evaluations in a portfolio enables the learner to look back, analyze and reflect. The content is used for assessment or making decisions about progress. And the portfolio is used as an instrument for quality assurance processes. In post-graduate medical education, these purposes can be combined but this is not always reported transparently. In this paper, we will discuss the different perspectives, how a portfolio can serve these three purposes and what are opportunities and challenges of combining multiple purposes.

Background: Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships., Methods: We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement., Results: The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4)., Conclusions: This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.

Purpose: Residency programs around the world use multisource feedback (MSF) to evaluate learners' performance. Studies of the reliability of MSF show mixed results. This study aimed to identify the reliability of MSF as practiced across occasions with varying numbers of assessors from different professional groups (physicians and nonphysicians) and the effect on the reliability of the assessment for different competencies when completed by both groups., Method: The authors collected data from 2008 to 2012 from electronically completed MSF questionnaires. In total, 428 residents completed 586 MSF occasions, and 5,020 assessors provided feedback. The authors used generalizability theory to analyze the reliability of MSF for multiple occasions, different competencies, and varying numbers of assessors and assessor groups across multiple occasions., Results: A reliability coefficient of 0.800 can be achieved with two MSF occasions completed by at least 10 assessors per group or with three MSF occasions completed by 5 assessors per group. Nonphysicians' scores for the "Scholar" and "Health advocate" competencies and physicians' scores for the "Health advocate" competency had a negative effect on the composite reliability., Conclusions: A feasible number of assessors per MSF occasion can reliably assess residents' performance. Scores from a single occasion should be interpreted cautiously. However, every occasion can provide valuable feedback for learning. This research confirms that the (unique) characteristics of different assessor groups should be considered when interpreting MSF results. Reliability seems to be influenced by the included assessor groups and competencies. These findings will enhance the utility of MSF during residency training.

Background: Despite acknowledgement that the Canadian Medical Educational Directives for Specialists (CanMEDS) framework covers the relevant competencies of physicians, many educators and medical professionals struggle to translate the CanMEDS roles into comprehensive training programmes for specific specialties., Aim: To gain insight into the applicability of the CanMEDS framework to guide the design of educational programmes for specific specialties by exploring stakeholders' perceptions of specialty specific competencies and examining differences between those competencies and the CanMEDS framework., Methods: This case study is a sequel to a study among ObsGyn specialists. It explores the perspectives of patients, midwives, nurses, general practitioners, and hospital boards on gynaecological competencies and compares these with the CanMEDS framework., Results: Clinical expertise, reflective practice, collaboration, a holistic view, and involvement in practice management were perceived to be important competencies for gynaecological practice. Although all the competencies were covered by the CanMEDS framework, there were some mismatches between stakeholders' perceptions of the importance of some competencies and their position in the framework., Conclusion: The CanMEDS framework appears to offer relevant building blocks for specialty specific postgraduate training, which should be combined with the results of an exploration of specialty specific competencies to arrive at a postgraduate curriculum that is in alignment with professional practice.

Background: During workplace based learning students develop professional competences and an appropriate performance. To gain insight in the learning process and to evaluate competences and performance, assessment tools are essential and need to be of good quality. We aimed to construct a competence inventory applicable as an instrument to measure the content validity of workplace based assessment tools, such as portfolio., Methods: A Delphi study was carried out based on the CanMEDS Roles Framework. In three rounds, experts (N = 25-30) were invited to score the key competences per CanMEDS role on relevance (6-point Likert-scale), and to comment on the content and formulation bearing in mind its use in workplace based assessment. A descriptive analysis of relevances and comments was performed., Results: Although all competences were scored as relevant, many comments pointed at a lack of concrete, transparent and applicable descriptions of the key competences for the purpose of assessment. Therefore, the CanMEDS roles were reformulated in this Delphi procedure as concrete learning outcomes, observable and suitable for workplace based assessment., Conclusions: A competence based inventory, ready for validating workplace based assessment tools, was constructed using a Delphi procedure and based on a clarification and concretisation of the CanMEDS roles.

Context: Mentors are increasingly involved in doctor performance assessments. Mentoring seems to be a key determinant in achieving the ultimate goal of those assessments, namely, improving doctor performance. Little is known, however, about how mentors perceive and fulfil this role., Objective: The aim of this paper is to expand understanding of the role of mentors in performance assessment., Methods: Thirty-eight mentors undertook formative performance assessments of their peers in a pilot study. A mixed-methods design was used, consisting of a postal survey (n = 28) and qualitative interviews with a subset of mentors (n = 11). Individual semi-structured interviews were completed and transcripts were analysed by two researchers using a grounded theory approach., Results: The results of the survey showed that 89% of mentors intended to continue in their mentorship role. Interviews revealed that mentors used several strategies in the assessments, including: contrasting and collating information; posing reflective questions, and goal setting. Mentors experienced difficulty in disregarding their views of the doctors evaluated. Some mentors noticed obstacles with specific interview skills such as 'paying attention to their colleagues' strengths' and 'enabling doctors to find their own solutions'. Mentors reported that they and their organisations benefited from the assessments. The perceived benefits included: improved interview skills; increased solidarity, and increased mutual respect., Conclusions: The study provides insights into what mentors can do to increase the chance that externally derived information is integrated into doctors' self-assessments. Mainly, mentors used strategies aimed at effectively delivering feedback and encouraging reflection. However, we found that mentors who took part in our study appeared to struggle with a number of obstacles related to: time investment; familiarity with the doctor assessed, and the acquiring of specific interview skills.

Background: A portfolio is used to mentor and assess students' clinical performance at the workplace. However, students and raters often perceive the portfolio as a time-consuming instrument., Purpose: In this study, we investigated whether assessment during medical internship by a portfolio can combine reliability and feasibility., Methods: The domain-oriented reliability of 61 double-rated portfolios was measured, using a generalisability analysis with portfolio tasks and raters as sources of variation in measuring the performance of a student., Results: We obtained reliability (Phi coefficient) of 0.87 with this internship portfolio containing 15 double-rated tasks. The generalisability analysis showed that an acceptable level of reliability (Phi = 0.80) was maintained when the amount of portfolio tasks was decreased to 13 or 9 using one and two raters, respectively., Conclusions: Our study shows that a portfolio can be a reliable method for the assessment of workplace learning. The possibility of reducing the amount of tasks or raters while maintaining a sufficient level of reliability suggests an increase in feasibility of portfolio use for both students and raters.

Workplace-based assessment may be further optimized by drawing upon the perspectives of multiple assessors, including those outside the trainee's discipline. Interprofessional competencies like communication and collaboration are often considered suitable for team input. We sought to characterize multidisciplinary expectations of communicator and collaborator competency roles. We adopted a constructivist grounded theory approach to explore perspectives of multidisciplinary team members on a clinical teaching unit. In semi-structured interviews, participants described expectations for competent collaboration and communication of trainees outside their own discipline. Data were analyzed to identify recurring themes, underlying concepts and their interactions using constant comparison. Three main underlying perspectives influenced interprofessional characterization of competent communication and collaboration: (1) general expectations of best practice; (2) specific expectations of supportive practice; and (3) perceived commitment to teaching practice. However, participants seemingly judged trainees outside their discipline according to how competencies were exercised to advance their own professional patient care decision-making, with minimal attention to the trainee's specific skillset demonstrated. While team members expressed commitment to supporting interprofessional competency development of trainees outside their discipline, service-oriented judgement of performance loomed large. The potential impact on the credibility of multidisciplinary sources for workplace-based assessment requires consideration. [ABSTRACT FROM AUTHOR]

Background: Despite increasing calls for patient and public involvement in health‐care quality improvement, the question of how patient evaluations can contribute to physician learning and performance assessment has received scant attention. Objective: The objective of this study was to explore, amid calls for patient involvement in quality assurance, patients' perspectives on their role in the evaluation of physician performance and to support physicians' learning and decision making on professional competence. Design: A qualitative study based on semi‐structured interviews. Setting and Participants: The study took place in a secondary care setting in the Netherlands. The authors selected 25 patients from two Dutch hospitals and through the Dutch Lung Foundation, using purposive sampling. Methods: Data were analysed according to the principles of template analysis, based on an a priori coding framework developed from the literature about patient empowerment, feedback and performance assessment. Results: The analysis unearthed three predominant patient perspectives: the proactive perspective, the restrained perspective and the outsider perspective. These perspectives differed in terms of perceived power dynamics within the doctor‐patient relationship, patients' perceived ability, and willingness to provide feedback and evaluate their physician's performance. Patients' perspectives thus affected the role patients envisaged for themselves in evaluating physician performance. Discussion and conclusion: Although not all patients are equally suitable or willing to be involved, patients can play a role in evaluating physician performance and continuing training through formative approaches. To involve patients successfully, it is imperative to distinguish between different patient perspectives and empower patients by ensuring a safe environment for feedback. [ABSTRACT FROM AUTHOR]

Purpose: National physician validation systems aim to ensure lifelong learning through periodic appraisals of physicians’ competence. Their effectiveness is determined by physicians’ acceptance of and commitment to the system. This study, therefore, sought to explore physicians’ perceptions and self-reported acceptance of validation across three different physician validation systems in Europe. Materials and methods: Using a constructivist grounded-theory approach, we conducted semi-structured interviews with 32 respiratory specialists from three countries with markedly different validation systems: Germany, which has a mandatory, credit-based system oriented to continuing professional development; Denmark, with mandatory annual dialogs and ensuing, non-compulsory activities; and the UK, with a mandatory, portfolio-based revalidation system. We analyzed interview data with a view to identifying factors influencing physicians’ perceptions and acceptance. Results: Factors that influenced acceptance were the assessment’s authenticity and alignment of its requirements with clinical practice, physicians’ beliefs about learning, perceived autonomy, and organizational support. Conclusions: Users’ acceptance levels determine any system’s effectiveness. To support lifelong learning effectively, national physician validation systems must be carefully designed and integrated into daily practice. Involving physicians in their design may render systems more authentic and improve alignment between individual ambitions and the systems’ goals, thereby promoting acceptance. [ABSTRACT FROM AUTHOR]

Introduction: Ward round skills are essential for doctors in hospital settings. Literature shows medical students’ deficiencies in these skills. Simulation has been used to train these skills. However, exposing learners to simulation at an early stage may be associated with a high cognitive load and limited learning. This study aims to determine how students experience this load and its interplay with performance and which factors promote and impair learning.Methods: Fifty-six final year medical students participated in a simulated ward round training exercise. Both students’ performance and cognitive load were measured to determine if there was any correlation and interviews were carried out to understand which factors support and impair learning.Results: Performance scores revealed deficiencies in ward round skills. Students experienced a cognitive load that weakly correlated with performance. Qualitative findings provided important insights into simulated ward-based learning. It is clear that well-designed clinical scenarios, prioritization tasks, teamwork and feedback support students’ learning process whereas distractions impair learning.Conclusions: WRS proved to be a good teaching method to improve clinical skills at this stage as the cognitive load is not too high to impair learning. Hence, including tasks in the simulation design can enhance the learning process. [ABSTRACT FROM AUTHOR]

Reforms in postgraduate medical education (PGME) exposed a gap between educational theory and clinical practice. Entrustable Professional Activities (EPAs) were introduced to assist clinicians in bridging this gap and to create better consonance between the intended and the enacted curriculum. In this viewpoint paper, we discuss the potential and the pitfalls of using EPAs in PGME. EPAs promise an effective way of teaching abstract competencies in a curriculum based on real-life professional activities that are suitable for clinical assessment. Summative judgement is used to entrust a resident step by step in a certain EPA, resulting in an increase of independent practice. However, we argue that the success of EPAs depends on (1) a balance: brief focussed descriptions against the requirements for detail and (2) a precondition: a mature and flexible workplace for learning. [ABSTRACT FROM AUTHOR]

Context The portfolio is becoming increasingly accepted as a valuable tool for learning and assessment. The validity of portfolio assessment, however, may suffer from bias due to irrelevant qualities, such as lay-out and writing style. We examined the possible effects of such qualities in a portfolio programme aimed at stimulating Year 1 medical students to reflect on their professional and personal development. In later curricular years, this portfolio is also used to judge clinical competence. Methods We developed an instrument, the Portfolio Analysis Scoring Inventory, to examine the impact of form and content aspects on portfolio assessment. The Inventory consists of 15 items derived from interviews with experienced mentors, the literature, and the criteria for reflective competence used in the regular portfolio assessment procedure. Forty portfolios, selected from 231 portfolios for which ratings from the regular assessment procedure were available, were rated by 2 researchers, independently, using the Inventory. Regression analysis was used to estimate the correlation between the ratings from the regular assessment and those resulting from the Inventory items. Results Inter-rater agreement ranged from 0.46 to 0.87. The strongest predictor of the variance in the regular ratings was ‘quality of reflection’ (R 0.80; R2 66%). No further items accounted for a significant proportion of variance. Irrelevant items, such as writing style and lay-out, had negligible effects. Conclusions The absence of an impact of irrelevant criteria appears to support the validity of the portfolio assessment procedure. Further studies should examine the portfolio's validity for the assessment of clinical competence. [ABSTRACT FROM AUTHOR]