Search

Objective: We proposed adopting billing models for secure messaging (SM) telehealth services that move beyond time-based metrics, focusing on the complexity and clinical expertise involved in patient care., Materials and Methods: We trained 8 classification machine learning (ML) models using providers' electronic health record (EHR) audit log data for patient-initiated non-urgent messages. Mixed effect modeling (MEM) analyzed significance., Results: Accuracy and area under the receiver operating characteristics curve scores generally exceeded 0.85, demonstrating robust performance. MEM showed that knowledge domains significantly influenced SM billing, explaining nearly 40% of the variance., Discussion: This study demonstrates that ML models using EHR audit log data can improve and predict billing in SM telehealth services, supporting billing models that reflect clinical complexity and expertise rather than time-based metrics., Conclusion: Our research highlights the need for SM billing models beyond time-based metrics, using EHR audit log data to capture the true value of clinical work., (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: journals.permissions@oup.com.)

Background: Although depression is one of the most common mental health disorders outpacing other diseases and conditions, poor access to care and limited resources leave many untreated. Secure messaging (SM) offers patients an online means to bridge this gap by communicating nonurgent medical questions. We focused on self-care health management behaviors and delved into SM initiation as the initial act of engagement and SM exchanges as continuous engagement patterns. This study examined whether those with depression might be using SM more than those without depression. Methods: Patient portal data were obtained from a large academic medical center's electronic health records spanning 5 years, from January 2018 to December 2022. We organized and analyzed SM initiations and exchanges using the linear mixed-effects modeling technique. Results: Our predictors correlated with SM initiations, accounting for 25.1% of variance explained. In parallel, 24.9% of SM exchanges were attributable to these predictors. Overall, our predictors demonstrate stronger associations with SM exchanges. Discussion: We examined patients with and without depression across 2,629 zip codes over five years. Our findings reveal that the predictors affecting SM initiations and exchanges are multifaceted, with certain predictors enhancing its utilization and others impeding it. Conclusions: SM telehealth service provided support to patients with mental health needs to a greater extent than those without. By increasing access, fostering better communication, and efficiently allocating resources, telehealth services not only encourage patients to begin using SM but also promote sustained interaction through ongoing SM exchanges.

Abstract: Precision education (PE) leverages longitudinal data and analytics to tailor educational interventions to improve patient, learner, and system-level outcomes. At present, few programs in medical education can accomplish this goal as they must develop new data streams transformed by analytics to drive trainee learning and program improvement. Other professions, such as Major League Baseball (MLB), have already developed extremely sophisticated approaches to gathering large volumes of precise data points to inform assessment of individual performance.In this perspective, the authors argue that medical education-whose entry into precision assessment is fairly nascent-can look to MLB to learn the possibilities and pitfalls of precision assessment strategies. They describe 3 epochs of player assessment in MLB: observation, analytics (sabermetrics), and technology (Statcast). The longest tenured approach, observation, relies on scouting and expert opinion. Sabermetrics brought new approaches to analyzing existing data in a way that better predicted which players would help the team win. Statcast created precise, granular data about highly attributable elements of player performance while helping to account for nonplayer factors that confound assessment such as weather, ballpark dimensions, and the performance of other players. Medical education is progressing through similar epochs marked by workplace-based assessment, learning analytics, and novel measurement technologies. The authors explore how medical education can leverage intersectional concepts of MLB player and medical trainee assessment to inform present and future directions of PE., (Copyright © 2023 the Association of American Medical Colleges.)

Abstract: High-quality precision education (PE) aims to enhance outcomes for learners and society by incorporating longitudinal data and analytics to shape personalized learning strategies. However, existing educational data collection methods often suffer from fragmentation, leading to gaps in understanding learner and program performance. In this article, the authors present a novel approach to PE at the University of Cincinnati, focusing on the Ambulatory Long Block, a year-long continuous ambulatory group-practice experience. Over the last 17 years, the Ambulatory Long Block has evolved into a sophisticated data collection and analysis system that integrates feedback from various stakeholders, as well as learner self-assessment, electronic health record utilization information, and clinical throughput metrics. The authors detail their approach to data prioritization, collection, analysis, visualization, and feedback, providing a practical example of PE in action. This model has been associated with improvements in both learner performance and patient care outcomes. The authors also highlight the potential for real-time data review through automation and emphasize the importance of collaboration in advancing PE. Generalizable principles include designing learning environments with continuity as a central feature, gathering both quantitative and qualitative performance data from interprofessional assessors, using this information to supplement traditional workplace-based assessments, and pairing it with self-assessments. The authors advocate for criterion referencing over normative comparisons, using user-friendly data visualizations, and employing tailored coaching strategies for individual learners. The Ambulatory Long Block model underscores the potential of PE to drive improvements in medical education and health care outcomes., (Copyright © 2023 the Association of American Medical Colleges.)

Despite the numerous calls for integrating quality improvement and patient safety (QIPS) curricula into health professions education, there are limited examples of effective implementation for early learners. Typically, pre-clinical QIPS experiences involve lectures or lessons that are disconnected from the practice of medicine. Consequently, students often prioritize other content they consider more important. As a result, they may enter clinical settings without essential QIPS skills and struggle to incorporate these concepts into their early professional identity formation. In this paper, we present twelve tips aimed at assisting educators in developing QIPS education early in the curricula of health professions students. These tips address various key issues, including aligning incentives, providing longitudinal experiences, incorporating real-world care outcomes, optimizing learning environments, communicating successes, and continually enhancing education and care delivery processes.

Abstract: Competency-based medical education (CBME) depends on effective programs of assessment to achieve the desired outcomes and goals of training. Residency programs must be able to defend clinical competency committee (CCC) group decisions about learner readiness for practice, including decisions about time-variable resident promotion and graduation. In this article, the authors describe why CCC group decision-making processes should be supported by theory and review 3 theories they used in designing their group processes: social decision scheme theory, functional theory, and wisdom of crowds. They describe how these theories were applied in a competency-based, time-variable training pilot-Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) at the University of Cincinnati internal medicine residency program in 2020-2022-to increase the defensibility of their CCC group decision-making. This work serves as an example of how use of theory can bolster validity arguments supporting group decisions about resident readiness for practice., (Copyright © 2023 the Association of American Medical Colleges.)

Background: The rapid trajectory of artificial intelligence (AI) development and advancement is quickly outpacing society's ability to determine its future role. As AI continues to transform various aspects of our lives, one critical question arises for medical education: what will be the nature of education, teaching, and learning in a future world where the acquisition, retention, and application of knowledge in the traditional sense are fundamentally altered by AI?, Objective: The purpose of this perspective is to plan for the intersection of health care and medical education in the future., Methods: We used GPT-4 and scenario-based strategic planning techniques to craft 4 hypothetical future worlds influenced by AI's integration into health care and medical education. This method, used by organizations such as Shell and the Accreditation Council for Graduate Medical Education, assesses readiness for alternative futures and effectively manages uncertainty, risk, and opportunity. The detailed scenarios provide insights into potential environments the medical profession may face and lay the foundation for hypothesis generation and idea-building regarding responsible AI implementation., Results: The following 4 worlds were created using OpenAI's GPT model: AI Harmony, AI conflict, The world of Ecological Balance, and Existential Risk. Risks include disinformation and misinformation, loss of privacy, widening inequity, erosion of human autonomy, and ethical dilemmas. Benefits involve improved efficiency, personalized interventions, enhanced collaboration, early detection, and accelerated research., Conclusions: To ensure responsible AI use, the authors suggest focusing on 3 key areas: developing a robust ethical framework, fostering interdisciplinary collaboration, and investing in education and training. A strong ethical framework emphasizes patient safety, privacy, and autonomy while promoting equity and inclusivity. Interdisciplinary collaboration encourages cooperation among various experts in developing and implementing AI technologies, ensuring that they address the complex needs and challenges in health care and medical education. Investing in education and training prepares professionals and trainees with necessary skills and knowledge to effectively use and critically evaluate AI technologies. The integration of AI in health care and medical education presents a critical juncture between transformative advancements and significant risks. By working together to address both immediate and long-term risks and consequences, we can ensure that AI integration leads to a more equitable, sustainable, and prosperous future for both health care and medical education. As we engage with AI technologies, our collective actions will ultimately determine the state of the future of health care and medical education to harness AI's power while ensuring the safety and well-being of humanity., (©Michelle I Knopp, Eric J Warm, Danielle Weber, Matthew Kelleher, Benjamin Kinnear, Daniel J Schumacher, Sally A Santen, Eneida Mendonça, Laurah Turner. Originally published in JMIR Medical Education (https://mededu.jmir.org), 25.12.2023.)

Inequity in assessment has been described as a "wicked problem"-an issue with complex roots, inherent tensions, and unclear solutions. To address inequity, health professions educators must critically examine their implicit understandings of truth and knowledge (i.e., their epistemologies) with regard to educational assessment before jumping to solutions. The authors use the analogy of a ship (program of assessment) sailing on different seas (epistemologies) to describe their journey in seeking to improve equity in assessment. Should the education community repair the ship of assessment while sailing or should the ship be scrapped and built anew? The authors share a case study of a well-developed internal medicine residency program of assessment and describe efforts to evaluate and enable equity using various epistemological lenses. They first used a postpositivist lens to evaluate if the systems and strategies aligned with best practices, but found they did not capture important nuances of what equitable assessment entails. Next, they used a constructivist approach to improve stakeholder engagement, but found they still failed to question the inequitable assumptions inherent to their systems and strategies. Finally, they describe a shift to critical epistemologies, seeking to understand who experiences inequity and harm to dismantle inequitable systems and create better ones. The authors describe how each unique sea promoted different adaptations to their ship, and challenge programs to sail through new epistemological waters as a starting point for making their own ships more equitable., (Copyright © 2023 by the Association of American Medical Colleges.)

Purpose: As competency-based medical education has become the predominant graduate medical education training model, interest in time-variable training has grown. Despite multiple competency-based time-variable training (CBTVT) pilots ongoing in the United States, little is known about how this training approach impacts learners. The authors aim to explore how their CBTVT pilot program impacted resident motivation for learning, assessment, and feedback., Method: The authors performed a qualitative educational case study on the Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) program at the University of Cincinnati from October 2020 through March 2022. Semistructured interviews were conducted with TIMELESS residents (n = 9) approximately every 6 months to capture experiences over time. The authors used inductive thematic analysis to develop themes and compared their findings with existing theories of learner motivation., Results: The authors developed 2 themes: TIMELESS had variable effects on residents' motivation for learning and TIMELESS increased resident engagement with and awareness of the program of assessment. Participants reported increased motivation to learn and seek assessment, though some felt a tension between performance (e.g., advancement through the residency program) and growth (e.g., improvement as a physician). Participants became more aware of the quality of assessments they received, in part due to TIMELESS increasing the perceived stakes of assessment, and reported being more deliberate when assessing other residents., Conclusions: Resident motivation for learning, assessment, and feedback was impacted in ways that the authors contextualize using current theories of learner motivation (i.e., goal orientation theory and attribution theory). Future research should investigate how interventions, such as coaching, guided learner reflection, or various CBTVT implementation strategies, can help keep learners oriented toward mastery learning rather than toward performance., (Copyright © 2023 by the Association of American Medical Colleges.)

Competency-based medical education (CBME) is an outcomes-based approach to education and assessment that focuses on what competencies trainees need to learn in order to provide effective patient care. Despite this goal of providing quality patient care, trainees rarely receive measures of their clinical performance. This is problematic because defining a trainee's learning progression requires measuring their clinical performance. Traditional clinical performance measures (CPMs) are often met with skepticism from trainees given their poor individual-level attribution. Resident-sensitive quality measures (RSQMs) are attributable to individuals, but lack the expeditiousness needed to deliver timely feedback and can be difficult to automate at scale across programs. In this eye opener, the authors present a conceptual framework for a new type of measure - TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs) - attuned to both automation and trainee attribution as the next evolutionary step in linking education to patient care. TRACERs have five defining characteristics: meaningful (for patient care and trainees), attributable (sufficiently to the trainee of interest), automatable (minimal human input once fully implemented), scalable (across electronic health records [EHRs] and training environments), and real-time (amenable to formative educational feedback loops). Ideally, TRACERs optimize all five characteristics to the greatest degree possible. TRACERs are uniquely focused on measures of clinical performance that are captured in the EHR, whether routinely collected or generated using sophisticated analytics, and are intended to complement (not replace) other sources of assessment data. TRACERs have the potential to contribute to a national system of high-density, trainee-attributable, patient-centered outcome measures., Competing Interests: The authors have no competing interests to declare., (Copyright: © 2023 The Author(s).)

Abstract: The information systems designed to support clinical care have evolved separately from those that support health professions education. This has resulted in a considerable digital divide between patient care and education, one that poorly serves practitioners and organizations, even as learning becomes ever more important to both. In this perspective, we advocate for the enhancement of existing health information systems so that they intentionally facilitate learning. We describe three well-regarded frameworks for learning that can point toward how health care information systems can best evolve to support learning. The Master Adaptive Learner model suggests ways that the individual practitioner can best organize their activities to ensure continual self-improvement. The PDSA cycle similarly proposes actions for improvement but at a health care organization's workflow level. Senge's Five Disciplines of the Learning Organization, a more general framework from the business literature, serves to further inform how disparate information and knowledge flows can be managed for continual improvement. Our main thesis holds that these types of learning frameworks should inform the design and integration of information systems serving the health professions. An underutilized mediator of educational improvement is the ubiquitous electronic health record. The authors list learning analytic opportunities, including potential modifications of learning management systems and the electronic health record, that would enhance health professions education and support the shared goal of delivering high-quality evidence-based health care., Competing Interests: Disclosures: Dr. Birnbaum's institution receives Grant/Research/Clinical Trial Support from Eli Lilly [JAK-STAT Immunology] for an innovation project. The remaining authors declare no conflict of interest., (Copyright © 2022 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of The Alliance for Continuing Education in the Health Professions, the Association for Hospital Medical Education, and the Society for Academic Continuing Medical Education.)

Background: Clinical competency committees (CCCs) and residency program leaders may find it difficult to interpret workplace-based assessment (WBA) ratings knowing that contextual factors and bias play a large role., Objective: We describe the development of an expected entrustment score for resident performance within the context of our well-developed Observable Practice Activity (OPA) WBA system., Design: Observational study PARTICIPANTS: Internal medicine residents MAIN MEASURE: Entrustment KEY RESULTS: Each individual resident had observed entrustment scores with a unique relationship to the expected entrustment scores. Many residents' observed scores oscillated closely around the expected scores. However, distinct performance patterns did emerge., Conclusions: We used regression modeling and leveraged large numbers of historical WBA data points to produce an expected entrustment score that served as a guidepost for performance interpretation., (© 2022. The Author(s) under exclusive licence to Society of General Internal Medicine.)

Competency-based medical education (CBME) shifts us from static assessment of learning to developmental assessment for learning. However, implementation challenges associated with CBME remain a major hurdle, especially after training and into practice. The full benefit of developmental assessment for learning over time requires collaboration, cooperation, and trust among learners, regulators, and the public that transcends each individual phase. The authors introduce the concept of an "Education Passport" that provides evidence of readiness to travel across the boundaries between undergraduate medical education, graduate medical education, and the expanse of practice. The Education Passport uses programmatic assessment, a process of collecting numerous low stakes assessments from multiple sources over time, judging these data using criterion-referencing, and enhancing this with coaching and competency committees to understand, process, and accelerate growth without end. Information in the Passport is housed on a cloud-based server controlled by the student/physician over the course of training and practice. These data are mapped to various educational frameworks such Entrustable Professional Activities or milestones for ease of longitudinal performance tracking. At each stage of education and practice the student/physician grants Passport access to all entities that can provide data on performance. Database managers use learning analytics to connect and display information over time that are then used by the student/physician, their assigned or chosen coaches, and review committees to maintain or improve performance. Global information is also collected and analyzed to improve the entire system of learning and care. Developing a true continuum that embraces performance and growth will be a long-term adaptive challenge across many organizations and jurisdictions and will require coordination from regulatory and national agencies. An Education Passport could also serve as an organizing tool and will require research and high-value communication strategies to maximize public trust in the work., Competing Interests: None reported, (© 2022 Warm, Carraccio, Kelleher, Kinnear, Schumacher, Santen; licensee Synergies Partners.)

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients., (Copyright © 2021 by the Association of American Medical Colleges.)

Introduction: Narrative assessment data are valuable in understanding struggles in resident performance. However, it remains unknown which themes in narrative data that occur early in training may indicate a higher likelihood of struggles later in training, allowing programs to intervene sooner., Methods: Using learning analytics, we identified 26 internal medicine residents in three cohorts that were below expected entrustment during training. We compiled all narrative data in the first 6 months of training for these residents as well as 13 typically performing residents for comparison. Narrative data were blinded for all 39 residents during initial phases of an inductive thematic analysis for initial coding., Results: Many similarities were identified between the two cohorts. Codes that differed between typical and lower entrusted residents were grouped into two types of themes: three explicit/manifest and three implicit/latent with six total themes. The explicit/manifest themes focused on specific aspects of resident performance with assessors describing 1) Gaps in attention to detail, 2) Communication deficits with patients, and 3) Difficulty recognizing the "big picture" in patient care. Three implicit/latent themes, focused on how narrative data were written, were also identified: 1) Feedback described as a deficiency rather than an opportunity to improve, 2) Normative comparisons to identify a resident as being behind their peers, and 3) Warning of possible risk to patient care., Discussion: Clinical competency committees (CCCs) usually rely on accumulated data and trends. Using the themes in this paper while reviewing narrative comments may help CCCs with earlier recognition and better allocation of resources to support residents' development., (© 2021. The Author(s).)

Internal medicine (IM) residents frequently perform invasive bedside procedures during residency training. Bedside procedure training in IM programs may compromise patient safety. Current evidence suggests that IM training programs rely heavily on the number of procedures completed during training as a proxy for resident competence instead of using objective postprocedure patient outcomes. The authors posit that the results of procedural training effectiveness should be reframed with outcome metrics rather than process measures alone. This article introduces the as low as reasonably achievable (ALARA) approach, which originated in the nuclear industry to increase safety margins, to help assess and reduce bedside procedural risks. Training program directors are encouraged to use ALARA calculations to define the risk trade-offs inherent in current procedural training and assess how best to reliably improve patient outcomes. The authors describe 5 options to consider: training all residents in bedside procedures, training only select residents in bedside procedures, training no residents in bedside procedures, deploying 24-hour procedure teams supervised by IM faculty, and deploying 24-hour procedure teams supervised by non-IM faculty. The authors explore how quality improvement approaches using process maps, fishbone diagrams, failure mode effects and analyses, and risk matrices can be effectively implemented to assess training resources, choices, and aims. Future research should address the drivers behind developing optimal training programs that support independent practice, correlations with patient outcomes, and methods that enable faculty to justify their supervisory decisions while adhering to ALARA risk management standards., (Copyright © 2021 by the Association of American Medical Colleges.)

The ongoing adoption of competency-based medical education (CBME) across health professions training draws focus to learner-centred educational design and the importance of fostering a growth mindset in learners, teachers, and educational programs. An emerging body of literature addresses the instructional practices and features of learning environments that foster the skills and strategies necessary for trainees to be partners in their own learning and progression to competence and to develop skills for lifelong learning. Aligned with this emerging area is an interest in Dweck's self theory and the concept of the growth mindset. The growth mindset is an implicit belief held by an individual that intelligence and abilities are changeable, rather than fixed and immutable. In this paper, we present an overview of the growth mindset and how it aligns with the goals of CBME. We describe the challenges associated with shifting away from the fixed mindset of most traditional medical education assumptions and practices and discuss potential solutions and strategies at the individual, relational, and systems levels. Finally, we present future directions for research to better understand the growth mindset in the context of CBME.

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment., (Copyright © 2021 by the Association of American Medical Colleges.)

Problem: Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system., Approach: To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time., Outcomes: The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps., Next Steps: Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments., (Copyright © 2021 by the Association of American Medical Colleges.)

Background: Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM)., Objective: To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods., Design: The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward., Participants: NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members., Main Measures: The number and description of RSQMs generated during this process., Key Results: Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission., Conclusions: Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

Introduction: Internal medicine residents perform paracentesis, but programs lack standard methods for assessing competence or maintenance of competence and instead rely on number of procedures completed. This study describes differences in resident competence in paracentesis over time., Methods: From 2016 to 2017, internal medicine residents (n = 118) underwent paracentesis simulation training. Competence was assessed using the Paracentesis Competency Assessment Tool (PCAT), which combines a checklist, global scale, and entrustment score. The PCAT also delineates two categorical cut-point scores: the Minimum Passing Standard (MPS) and the Unsupervised Practice Standard (UPS). Residents were randomized to return to the simulation lab at 3 and 6 months (group A, n = 60) or only 6 months (group B, n = 58). At each session, faculty raters assessed resident performance. Data were analyzed to compare resident performance at each session compared with initial training scores, and performance between groups at 6 months., Results: After initial training, all residents met the MPS. The number achieving UPS did not differ between groups: group A = 24 (40%), group B = 20 (34.5%), p = 0.67. When group A was retested at 3 months, performance on each PCAT component significantly declined, as did the proportion of residents meeting the MPS and UPS. At the 6-month test, residents in group A performed significantly better than residents in group B, with 52 (89.7%) and 20 (34.5%) achieving the MPS and UPS, respectively, in group A compared with 25 (46.3%) and 2 (3.70%) in group B (p < .001 for both comparison)., Discussion: Skill in paracentesis declines as early as 3 months after training. However, retraining may help interrupt skill decay. Only a small proportion of residents met the UPS 6 months after training. This suggests using the PCAT to objectively measure competence would reclassify residents from being permitted to perform paracentesis independently to needing further supervision.

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid., (Copyright © 2020 by the Association of American Medical Colleges.)

Introduction/objectives: Coaching is emerging as a form of facilitation in health professions education. Most studies focus on one-on-one coaching rather than team coaching. We assessed the experiences of interprofessional teams coached to simultaneously improve primary care residency training and interprofessional practice., Methods: This three-year exploratory mixed methods study included transformational assistance from 9 interprofessional coaches, one assigned to each of 9 interprofessional primary care teams that included family medicine, internal medicine, pediatrics, nursing, pharmacy and behavioral health. Coaches interacted with teams during 2 in-person training sessions, an in-person site visit, and then as requested by their teams. Surveys administered at 1 year and end study assessed the coaching relationship and process., Results: The majority of participants (82% at end of Year 1 and 76.6% at end study) agreed or strongly agreed that their coach developed a positive working relationship with their team. Participants indicated coaches helped them: (1) develop as teams, (2) stay on task, and (3) respond to local context issues, with between 54.3% and 69.2% agreeing or strongly agreeing that their coaches were helpful in these areas. Cronbach's alpha for the 15 coaching survey items was 0.965. Challenges included aligning the coach's expertise with the team's needs., Conclusions: While team coaching was well received by interprofessional teams of primary care professionals undertaking educational and clinical redesign, the 3 primary care disciplines have much to learn from each other regarding how to improve inter- and intra-professional collaborative practice among clinicians and staff as well as with interprofessional learners rotating through their outpatient clinics.

Background and Objectives: Much can be gained by the three primary care disciplines collaborating on efforts to transform residency training toward interprofessional collaborative practice. We describe findings from a study designed to align primary care disciplines toward implementing interprofessional education., Methods: In this mixed methods study, we included faculty, residents and other interprofessional learners in family medicine, internal medicine, and pediatrics from nine institutions across the United States. We administered a web-based survey in April/May of 2018 and used qualitative analyses of field notes to study resident exposure to team-based care during training, estimates of career choice in programs that are innovating, and supportive and challenging conditions that influence collaboration among the three disciplines. Complete data capture was attained for 96.3% of participants., Results: Among family medicine resident graduates, an estimated 87.1% chose to go into primary care compared to 12.4% of internal medicine, and 36.5% of pediatric resident graduates. Qualitative themes found to positively influence cross-disciplinary collaboration included relationship development, communication of shared goals, alignment with health system/other institutional initiatives, and professional identity as primary care physicians. Challenges included expressed concerns by participants that by working together, the disciplines would experience a loss of identity and would be indistinguishable from one another. Another qualitative finding was that overwhelming stressors plague primary care training programs in the current health care climate-a great concern. These include competing demands, disruptive transitions, and lack of resources., Conclusions: Uniting the primary care disciplines toward educational and clinical transformation toward interprofessional collaborative practice is challenging to accomplish.

Purpose: Given resource constraints, many residency programs would consider adopting an entrustment-based assessment system from another program if given the opportunity. However, it is unclear if a system developed in one context would have similar or different results in another. This study sought to determine if entrustment varied between programs (community based and university based) when a single assessment system was deployed in different contexts., Method: The Good Samaritan Hospital (GSH) internal medicine residency program adopted the observable practice activity (OPA) workplace-based assessment system from the University of Cincinnati (UC). Comparisons for OPA-mapped subcompetency entrustment progression for programs and residents were made at specific timepoints over the course of 36 months of residency. Data collection occurred from August 2012 to June 2017 for UC and from September 2013 to June 2017 for GSH., Results: GSH entrustment ratings were higher than UC for all but the 11th, 15th, and 36th months of residency (P < .0001) and were also higher for the majority of subcompetencies and competencies (P < .0001). The rate of change for average monthly entrustment was similar, with GSH having an increase of 0.041 each month versus 0.042 for UC (P = .73). Most residents progressed from lower to higher entrustment, but there was significant variation between residents in each program., Conclusions: Despite the deployment of a single entrustment-based assessment system, important outcomes may vary by context. Further research is needed to understand the contributions of tool, context, and other factors on the data these systems produce.

Purpose: To examine the reliability and attributable facets of variance within an entrustment-derived workplace-based assessment system., Method: Faculty at the University of Cincinnati Medical Center internal medicine residency program (a 3-year program) assessed residents using discrete workplace-based skills called observable practice activities (OPAs) rated on an entrustment scale. Ratings from July 2012 to December 2016 were analyzed using applications of generalizability theory (G-theory) and decision study framework. Given the limitations of G-theory applications with entrustment ratings (the assumption that mean ratings are stable over time), a series of time-specific G-theory analyses and an overall longitudinal G-theory analysis were conducted to detail the reliability of ratings and sources of variance., Results: During the study period, 166,686 OPA entrustment ratings were given by 395 faculty members to 253 different residents. Raters were the largest identified source of variance in both the time-specific and overall longitudinal G-theory analyses (37% and 23%, respectively). Residents were the second largest identified source of variation in the time-specific G-theory analyses (19%). Reliability was approximately 0.40 for a typical month of assessment (27 different OPAs, 2 raters, and 1-2 rotations) and 0.63 for the full sequence of ratings over 36 months. A decision study showed doubling the number of raters and assessments each month could improve the reliability over 36 months to 0.76., Conclusions: Ratings from the full 36 months of the examined program of assessment showed fair reliability. Increasing the number of raters and assessments per month could improve reliability, highlighting the need for multiple observations by multiple faculty raters.

Current models of quality improvement and patient safety (QIPS) education are not fully integrated with clinical care delivery, representing a major impediment toward achieving widespread QIPS competency among health professions learners and practitioners. The Royal College of Physicians and Surgeons of Canada organized a 2-day consensus conference in Niagara Falls, Ontario, Canada, called Building the Bridge to Quality, in September 2016. Its goal was to convene an international group of educational and health system leaders, educators, frontline clinicians, learners, and patients to engage in a consensus-building process and generate a list of actionable strategies that individuals and organizations can use to better integrate QIPS education with clinical care.Four strategic directions emerged: prioritize the integration of QIPS education and clinical care, build structures and implement processes to integrate QIPS education and clinical care, build capacity for QIPS education at multiple levels, and align educational and patient outcomes to improve quality and patient safety. Individuals and organizations can refer to the specific tactics associated with the 4 strategic directions to create a road map of targeted actions most relevant to their organizational starting point.To achieve widespread change, collaborative efforts and alignment of intrinsic and extrinsic motivators are needed on an international scale to shift the culture of educational and clinical environments and build bridges that connect training programs and clinical environments, align educational and health system priorities, and improve both learning and care, with the ultimate goal of achieving improved outcomes and experiences for patients, their families, and communities.

In this issue of Academic Medicine, Dewan and Norcini examine the significant variability of time-in-training between patient care "giants"-the physicians, nurse practitioners, and physician assistants who practice primary care-and they call for further studies to determine optimal training duration and eventual scope of practice. They ask, what is the minimum education and training required to practice primary care, or "how tall is the shortest giant?" In this Invited Commentary, the authors reframe the question from identifying the minimum length of training required, to identifying desired patient care outcomes. Primary care is not a uniform entity. It ranges from complex elderly chronically ill patients, to twentysomething millennials with acute problems, to pregnant women, to families, and everything in between. The authors argue that training should be fit for purpose and produce high-quality outcomes for patients. Competence should be defined by these outcomes. Drawing parallels with Major League Baseball, the authors note that time to competence development will be variable for different training programs depending on purpose, and also variable for people within those programs, even with shared purpose. While time is a tool for competence attainment, it should not be the metric by which readiness for practice is measured.

W. Edwards Deming, in his System of Profound Knowledge, asserts that leaders who wish to transform a system should understand four essential elements: appreciation for a system, theory of knowledge, knowledge about variation, and psychology. The Accreditation Council for Graduate Medical Education (ACGME) introduced the milestones program as a part of the Next Accreditation System to create developmental language for the six core competencies and facilitate programmatic assessment within graduate medical education systems. Viewed through Deming's lens, the ACGME can be seen as the steward of a large system, with everyone who provides assessment data as workers in that system. The authors use Deming's framework to illustrate the working components of the assessment system of the University of Cincinnati College of Medicine's internal medicine residency program and draw parallels to the macrocosm of graduate medical education. Successes and failures in transforming resident assessment can be understood and predicted by identifying the system and its aims, turning information into knowledge, developing an understanding of variation, and appreciating the psychology of motivation of participants. The authors offer insights from their experience for educational leaders who wish to apply Deming's elements to their own assessment systems, with questions to explore, pitfalls to avoid, and practical approaches in doing this type of work.

Medical education has shifted to a competency-based paradigm, leading to calls for improved learner assessment methods and validity evidence for how assessment data are interpreted. Clinical competency committees (CCCs) use the collective input of multiple people to improve the validity and reliability of decisions made and actions taken based on assessment data. Significant heterogeneity in CCC structure and function exists across postgraduate medical education programs and specialties, and while there is no "one-size-fits-all" approach, there are ways to maximize value for learners and programs. This paper collates available evidence and the authors' experiences to provide practical tips on CCC purpose, membership, processes, and outputs. These tips can benefit programs looking to start a CCC and those that are improving their current CCC processes.

Multiple studies have demonstrated that the information included in the Medical Student Performance Evaluation fails to reliably predict medical students' future performance. This faulty transfer of information can lead to harm when poorly prepared students fail out of residency or, worse, are shuttled through the medical education system without an honest accounting of their performance. Such poor learner handovers likely arise from two root causes: (1) the absence of agreed-on outcomes of training and/or accepted assessments of those outcomes, and (2) the lack of standardized ways to communicate the results of those assessments. To improve the current learner handover situation, an authentic, shared mental model of competency is needed; high-quality tools to assess that competency must be developed and tested; and transparent, reliable, and safe ways to communicate this information must be created.To achieve these goals, the authors propose using a learner handover process modeled after a patient handover process. The CLASS model includes a description of the learner's Competency attainment, a summary of the Learner's performance, an Action list and statement of Situational awareness, and Synthesis by the receiving program. This model also includes coaching oriented towards improvement along the continuum of education and care. Just as studies have evaluated patient handover models using metrics that matter most to patients, studies must evaluate this learner handover model using metrics that matter most to providers, patients, and learners.

Purpose: Competency-based medical education and milestone reporting have led to increased interest in work-based assessments using entrustment over time as an assessment framework. Little is known about data collected from these assessments during residency. This study describes the results of entrustment of discrete work-based skills over 36 months in the University of Cincinnati internal medicine (IM) residency program., Method: Attending physician and peer/allied health assessors provided entrustment ratings of resident performance on work-based observable practice activities (OPAs) mapped to Accreditation Council for Graduate Medicine Education/American Board of Internal Medicine reporting milestones for IM. These data were translated into milestones data and tracked longitudinally. The authors analyzed data from this new entrustment system's first 36 months (July 2012-June 2015)., Results: During the 36-month period, assessors made 364,728 milestone assessments (mapped from OPAs) of 189 residents. Residents received an annualized average of 83 assessment encounters, producing means of 3,987 milestone assessments and 4,325 words of narrative assessment. Mean entrustment ratings (range 1-5) from all assessors for all milestones rose from 2.46 for first-month residents to 3.92 for 36th-month residents (r = 0.9252, P < .001). Attending physicians' entrustment ratings were lower than peer/allied health assessors' ratings. Medical knowledge and patient care milestones were rated lower than professionalism and interpersonal and communication skills milestones., Conclusions: Entrustment of milestones appears to rise progressively over time, with differences by assessor type, competency, milestone, and resident. Further research is needed to elucidate the validity of these data in promotion, remediation, and reporting decisions.

Background: Burnout is typically viewed as an individual condition, and no link has been identified between burnout and loneliness., Objective: To investigate the association of burnout with loneliness and social network degree and centrality., Methods: A survey containing the Maslach Burnout Inventory (MBI), a 3-question loneliness scale, and a social connectivity component was sent to residents in a large urban academic medical center internal medicine residency program., Results: The response rate was 77% (95 of 124 residents). We defined significant burnout as MBI subscores of ≥ 27 for emotional exhaustion (EE), ≥ 10 for depersonalization (DP), or both. This was met by 43 (45%), 47 (49%), and 31 (33%) out of 95 respondents, respectively. Those with significant burnout had higher loneliness scores: 5.6 versus 4.5 for EE (P = .002; OR = 1.50; 95% CI 1.15-1.95); 5.4 versus 4.6 for DP (P = .024; OR = 1.33; 95% CI 1.03-1.71); and 5.8 versus 4.6 for both EE and DP (P = .001; OR = 1.54; 95% CI 1.17-2.02). Rating a larger number of coresidents as closer connections on a 5-point Likert scale was not associated with lower burnout scores. No measures of centrality were associated with burnout scores for EE and/or DP. High personal accomplishment subscores on the MBI did correlate significantly with several measures of centrality., Conclusions: Burnout was associated with loneliness in a dose-dependent fashion. Greater sense of personal accomplishment was associated with greater network centrality.

As part of the Accreditation Council for Graduate Medical Education's Next Accreditation System, residency programs must connect resident-physician education to improved patient care outcomes. Residency training programs, however, face multiple obstacles in doing so. Results from residency quality improvement (QI) curricula tend to show improvement in simple process-based measures but not in more complex outcomes of care such as diabetes or blood pressure control. In this article, the authors describe the evolution of their QI educational program for internal medicine residents at the University of Cincinnati Medical Center within the structure of a novel training model called the Ambulatory Long Block. They discuss a resident-run project that led to reduced rates of patients with uncontrolled diabetes as an example of improvement in outcome measures. Despite favorable results from that particular resident group, the successful intervention did not spread practice-wide. Using this example, they detail the phases of evolution and lessons learned from their curriculum from 2006 to 2014 within a framework of previously published general principles for successful QI education, including those of exemplary care and learning sites. Successful programs require leadership, faculty expertise and mentorship, data management, learner buy-in, and patient engagement. Their experience will hopefully be of help to others as they attempt to simultaneously improve care and education. Further research and innovation are needed in this area, including optimizing strategies for strengthening resident-driven projects through partnership with nursing, allied health, and longitudinally engaged faculty members.

Entrustable Professional Activities (EPAs) and the Next Accreditation System reporting milestones reduce general competencies into smaller evaluable parts. However, some EPAs and reporting milestones may be too broad to use as direct assessment tools. We describe our internal medicine residency curriculum and assessment system, which uses entrustment and mapping of observable practice activities (OPAs) for resident assessment. We created discrete OPAs for each resident rotation and learning experience. In combination, these serve as curricular foundation and tools for assessment. OPA performance is measured via a 5-point entrustment scale, and mapped to milestones and EPAs. Entrustment ratings of OPAs provide an opportunity for immediate structured feedback of specific clinical skills, and mapping OPAs to milestones and EPAs can be used for longitudinal assessment, promotion decisions, and reporting. Direct assessment and demonstration of progressive entrustment of trainee skill over time are important goals for all training programs. Systems that use OPAs mapped to milestones and EPAs provide the opportunity for achieving both, but require validation.