Search

The increasing use of technology in health care and health professions education is an invitation to examine how digital sources of evidence are used in making assessment claims. In this paper, we describe how four sets of terms-primary and secondary data; structured and unstructured data; development and use; and deterministic and generative-can aid in examining how data from digital sources are used in evaluating what learners know and can do. Drawing on multiple examples, this paper shows how the four sets of terms can help both developers and users of technology-based assessment systems., Competing Interests: Dr. Krumm receives research support from the Society for Improving Medical Professional Learning (SIMPL). Dr. George is the Executive Director of the Society for Improving Medical Professional Learning (SIMPL). This position is unpaid. Dr. Barone is a former employee of NBME. Dr. Smirnova received compensation from the Kern Institute for the Transformation of Medical Education, Medical College of Wisconsin, for her time helping coordinate the Next Era in Assessment meeting in Milwaukee, Wisconsin., (Copyright: © 2024 The Author(s).)

Background: Precision health is a burgeoning scientific discipline that aims to incorporate individual variability in biological, behavioral, and social factors to develop personalized health solutions. To date, emergency medicine has not deeply engaged in the precision health movement. However, rapid advances in health technology, data science, and medical informatics offer new opportunities for emergency medicine to realize the promises of precision health., Methods: In this article, we conceptualize precision emergency medicine as an emerging paradigm and identify key drivers of its implementation into current and future clinical practice. We acknowledge important obstacles to the specialty-wide adoption of precision emergency medicine and offer solutions that conceive a successful path forward., Results: Precision emergency medicine is defined as the use of information and technology to deliver acute care effectively, efficiently, and authentically to individual patients and their communities. Key drivers and opportunities include leveraging human data, capitalizing on technology and digital tools, providing deliberate access to care, advancing population health, and reimagining provider education and roles. Overcoming challenges in equity, privacy, and cost is essential for success. We close with a call to action to proactively incorporate precision health into the clinical practice of emergency medicine, the training of future emergency physicians, and the research agenda of the specialty., Conclusions: Precision emergency medicine leverages new technology and data-driven artificial intelligence to advance diagnostic testing, individualize patient care plans and therapeutics, and strategically refine the convergence of the health system and the community., (© 2024 The Author(s). Academic Emergency Medicine published by Wiley Periodicals LLC on behalf of Society for Academic Emergency Medicine.)

Purpose: Written assessment comments are needed to archive feedback and inform decisions. Regrettably, comments are often impoverished, leaving performance-relevant information undocumented. Research has focused on content and supervisor's ability and motivation to write it but has not sufficiently examined how well the undocumented information lends itself to being written as comments. Because missing information threatens the validity of assessment processes, this study examined the performance information that resists being written., Method: Two sequential data collection methods and multiple elicitation techniques were used to triangulate unwritten assessment comments. Between November 2022 and January 2023, physicians in Canada were recruited by email and social media to describe experiences with wanting to convey assessment information but feeling unable to express it in writing. Fifty supervisors shared examples via survey. From January to May 2023, a subset of 13 participants were then interviewed to further explain what information resisted being written and why it seemed impossible to express in writing and to write comments in response to a video prompt or for their own "unwritable" example. Constructivist grounded theory guided data collection and analysis., Results: Not all performance-relevant information was equally writable. Information resisted being written as assessment comments when it would require an essay to be expressed in writing, belonged in a conversation and not in writing, or was potentially irrelevant and unverifiable. In particular, disclosing sensitive information discussed in a feedback conversation required extensive recoding to protect the learner and supervisor-learner relationship., Conclusions: When documenting performance information as written comments is viewed as an act of disclosure, it becomes clear why supervisors may feel compelled to leave some comments unwritten. Although supervisors can be supported in writing better assessment comments, their failure to write invites a reexamination of expectations for documenting feedback and performance information as written comments on assessment forms., (Copyright © 2024 the Association of American Medical Colleges.)

Objective: Preliminary surgery positions are associated with a negative stigma, and this stigma may persist for those residents who later go on to obtain a categorical position. However, it is currently unknown if general surgery residents who complete a preliminary year perform differently than their categorical peers throughout training. To examine these potential differences, we compared operative performance and autonomy across all 5 years of training for those who completed a nondesignated preliminary training year with those who did not., Design: Faculty ratings of categorical general surgery residents were collected from the Society for Improving Medical and Professional Learning (SIMPL) application. Residents were categorized based on the completion of a nondesignated preliminary year ("Previous Prelim" [PP]) or not ("Traditional Categorical" [TC]). Operative performance and autonomy ratings were tracked over 5 training years and analyzed using generalized mixed effects models. Performance and autonomy outcomes were dichotomized: "not competent" or "competent" and "no meaningful autonomy" or "meaningful autonomy", respectively. Fixed effects included cohort group, academic month, case complexity, resident level (junior [PGYs 1 and 2] vs senior [PGYs 3, 4, and 5] residents), and an interaction term between cohort group and resident level, while random effects included resident, faculty, program, and procedure., Setting: Operative performance and autonomy ratings were collected from the SIMPL application between 2015 and 2023., Participants: A total of 89 general surgery residency programs, which encompassed 1,108 categorical general surgery residents., Results: A total of 42,416 evaluations were analyzed. Of these, 809 evaluations were from PP residents. Faculty ratings of PP vs TC residents revealed no significant differences in adjusted probabilities of achieving a "competent" rating as a junior (5.6%, 95% Confidence Interval [CI] 2.7%-11.1% vs 3.0%, 95% CI 2.4%-3.7%, p = 0.28) or senior (23.7%, 95% CI 12.2%-41% vs 34.1%, 95% CI 29.1%-39.5%, p = 0.59) resident. Similarly, no significant differences were identified in adjusted probabilities of achieving a "meaningful autonomy" rating between PP and TC junior (9.0%, 95% CI 5.4%-14.6% vs 8.1%, 95% CI 6.9%-9.5%, p = 0.97) or senior (42.3%, 95% CI 28.7%-57.1% vs 49.6%, 95% CI 44.9%-54.2%, p = 0.76) residents., Conclusions: PP and TC residents may have similar operative performance and autonomy ratings throughout their residency training. Although further study of more senior residents is warranted, these initial findings could help combat the negative stigma associated with a preliminary year of training., (Copyright © 2024 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Competency-based medical education (CBME) has produced large collections of data, which can provide valuable information about trainees and medical education systems. Many organizations continue to struggle with accessing, collecting, governing, analyzing, and visualizing their clinical and/or educational data. This hinders data sharing efforts within and across organizations, which are foundational in supporting system-wide improvements. Challenges to data sharing within medical education include variability in legislation, existing data policies, heterogeneity of data, inadequate data infrastructure, and various intended purposes or uses. In this eye opener, the authors describe four case studies to illustrate some of the aforementioned challenges and characterize the complexity of data sharing within medical education along two dimensions: organizational (single vs. multiple) and data type (clinical and/or educational). With the goal of better supporting data sharing initiatives, the authors introduce an action-oriented blueprint that includes a three-stage process (i.e., preparation, execution, and iteration) to highlight crucial aspects of data sharing. This evidence-informed model incorporates current best practices and aims to support data sharing initiatives within their own organizations and across multiple organizations. Finally, organizations can use this model to conceptually guide and track their progression throughout the data sharing process., Competing Interests: Dr. George is the Executive Director of the Society for Improving Medical Professional Learning (SIMPL). This position is unpaid., (Copyright: © 2024 The Author(s).)

Objectives: Emergency physicians without specialized Emergency Medical Services (EMS) training are often required to provide online medical oversight. One common ethical question faced by these physicians is the assessment for decision-making capacity in a patient who does not accept EMS transport to the hospital. We sought expert consensus for a standardized set of guiding questions and recommendations to ensure a rigorous and feasible capacity assessment., Methods: A modified Delphi method approach was used to achieve group consensus among expert individuals. Nineteen physician experts were recruited from across the country, representing populations totaling over 22 million and a variety of urban, suburban, and rural practice environments. Experts completed a Round 1 survey that included 19 questions surrounding best practices for capacity evaluation among patients refusing transport. The threshold for consensus was predefined as 80% agreement. Participants gathered virtually meeting where the results from the first round were shared with the group. Discussion generated new items and refined the language of existing items. Following the virtual meeting, a Round 2 survey was conducted, and voted on by the panel for the items that did not meet consensus in Round 1., Results: After the first round, 15 of 19 items reached consensus. Three of the items that met consensus were universally noted to require language modification for clarification. A large portion of the discussion involved the proper method of integrating patient concerns around ambulance transport (e.g., cost of transport, financial concerns, social barriers) into the capacity assessment and whether alternate care options should be discussed. After the second round of voting, one additional item was reversed to meet consensus, resulting in a total of 16 items., Conclusions: A consensus expert panel was able to agree upon 16 standardized steps to guide best practices and assist emergency physicians in real-time evaluation of patients that refuse EMS transport.

Objectives: Women remain underrepresented in the emergency medicine (EM) workforce, academic EM, and institutional leadership. In order to support women physicians in EM, we must explore factors that contribute to attrition and workplace satisfaction. For example, tensions between workplace and familial roles are important to consider as women navigate careers in EM. The logistics and stressors of workplace lactation pose a particular challenge during an already stressful time for a new mother returning to work in a busy emergency department (ED), but limited empirical data exist regarding this experience. We aimed to explore the stressors associated with workplace lactation spaces in order to better inform the creation of lactation spaces for individuals working in EDs., Methods: Our team used an exploratory qualitative design to investigate lactation-specific stressors and understand their relationship to individuals' needs when lactating in EM workplace environments. A total of 40 individuals were interviewed, highlighting post-pregnancy return-to-work (RTW) experiences of medical students, residents, advanced practice professionals, nurses, fellows, and faculty. Interviews were coded and analyzed using thematic analysis., Results: We identified both tangible and intangible characteristics of lactation spaces that contribute to stress for lactating individuals. Additionally, we discovered that participants frequently noted a desire to work simultaneously while pumping in order to feel they were self-actualizing in their dual roles of parent and clinician. Among tangible items, access to a computer within lactation space was a key driver of ability to fulfill dual roles. Among intangible characteristics, we identified three distinct, yet interrelated, subthemes, including the need for lactation spaces to be respectful of individuals' time, privacy, and general health and well-being., Conclusions: This study suggests that meeting basic lactation needs with thoughtfully designed lactation spaces can empower individuals in their roles both as a lactating parent and a clinician. EM leadership can evaluate existing lactation spaces to ensure they meet the tangible and intangible needs of lactating physicians, trainees, advanced practice professionals, and nurses., Competing Interests: The authors declare no conflicts of interest., (© 2024 The Author(s). Journal of the American College of Emergency Physicians Open published by Wiley Periodicals LLC on behalf of American College of Emergency Physicians.)

Introduction: Optimizing the performance of emergency department (ED) teams impacts patient care, but the utility of current, team-based performance assessment tools to comprehensively measure this impact is underexplored. In this study we aimed to 1) evaluate ED team performance using current team-based assessment tools during an interprofessional in situ simulation and 2) identify characteristics of effective ED teams., Methods: This mixed-methods study employed case study methodology based on a constructivist paradigm. Sixty-three eligible nurses, technicians, pharmacists, and postgraduate year 2-4 emergency medicine residents at a tertiary academic ED participated in a 10-minute in situ simulation of a critically ill patient. Participants self-rated performance using the Team Performance Observation Tool (TPOT) 2.0 and completed a brief demographic form. Two raters independently reviewed simulation videos and rated performance using the TPOT 2.0, Team Emergency Assessment Measure (TEAM), and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). Following simulations, we conducted semi-structured interviews and focus groups with in situ participants. Transcripts were analyzed using thematic analysis., Results: Eighteen team-based simulations took place between January-April 2021. Raters' scores were on the upper end of the tools for the TPOT 2.0 (R1 4.90, SD 0.17; R2 4.53, SD 0.27, IRR [inter-rater reliability] 0.47), TEAM (R1 3.89, SD 0.19; R2 3.58, SD 0.39, IRR 0.73), and Ottawa GRS (R1 6.6, SD 0.56; R2 6.2, SD 0.54, IRR 0.68). We identified six themes from our interview data: team member entrustment; interdependent energy; leadership tone; optimal communication; strategic staffing; and simulation empowering team performance., Conclusion: Current team performance assessment tools insufficiently discriminate among high performing teams in the ED. Emergency department-specific assessments that capture features of entrustability, interdependent energy, and leadership tone may offer a more comprehensive way to assess an individual's contribution to a team's performance., Competing Interests: Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. This research was generously supported by grant funding from the Emergency Medicine Foundation/Council of Residency Directors in Emergency Medicine Education Scholarship Starter Grant. There are no other conflicts of interest or sources of funding to declare.

Phenomenon : Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach : Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings : We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights : PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

Abstract: Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives. More recently, however, these clinical and educational data are also increasingly being used to train artificial intelligence (AI) models. The implications of this relatively unique secondary use of data have not been well explored. These models can support the development of sophisticated AI products that can be commercialized. While these products have the potential to support and improve the educational system, there are challenges related to validity, patient and learner consent, and biased or discriminatory outputs. The authors consider the implications of developing AI models and products using educational and clinical data from learners, discuss the uses of these products within medical education, and outline considerations that should guide the appropriate use of data for this purpose. These issues are further explored by examining how they have been navigated in an educational collaborative., (Copyright © 2023 the Association of American Medical Colleges.)

Purpose: The era of precision education is increasingly leveraging electronic health record (EHR) data to assess residents' clinical performance. But precision in what the EHR-based resident performance metrics are truly assessing is not fully understood. For instance, there is limited understanding of how EHR-based measures account for the influence of the team on an individual's performance-or conversely how an individual contributes to team performances. This study aims to elaborate on how the theoretical understandings of supportive and collaborative interdependence are captured in residents' EHR-based metrics., Method: Using a mixed methods study design, the authors conducted a secondary analysis of 5 existing quantitative and qualitative datasets used in previous EHR studies to investigate how aspects of interdependence shape the ways that team-based care is provided to patients., Results: Quantitative analyses of 16 EHR-based metrics found variability in faculty and resident performance (both between and within resident). Qualitative analyses revealed that faculty lack awareness of their own EHR-based performance metrics, which limits their ability to act interdependently with residents in an evidence-informed fashion. The lens of interdependence elucidates how resident practice patterns develop across residency training, shifting from supportive to collaborative interdependence over time. Joint displays merging the quantitative and qualitative analyses showed that residents are aware of variability in faculty's practice patterns and that viewing resident EHR-based measures without accounting for the interdependence of residents with faculty is problematic, particularly within the framework of precision education., Conclusions: To prepare for this new paradigm of precision education, educators need to develop and evaluate theoretically robust models that measure interdependence in EHR-based metrics, affording more nuanced interpretation of such metrics when assessing residents throughout training., (Copyright © 2024 the Association of American Medical Colleges.)

Purpose: Entrustable professional activities (EPAs) are a widely used framework for curriculum and assessment, yet the variability in emergency medicine (EM) training programs mandates the development of EPAs that meet the needs of the specialty as a whole. This requires eliciting and incorporating the perspectives of multiple stakeholders (i.e., faculty, residents, and patients) in the development of EPAs. Without a shared understanding of what a resident must be able to do upon graduation, we run the risk of advancing ill-prepared residents that may provide inconsistent care., Methods: In an effort to address these challenges, beginning in February 2020, the authors assembled an advisory board of 25 EM faculty to draft and reach consensus on a final list of EPAs that can be used across all training programs within the specialty of EM. Using modified Delphi methodology, the authors came to consensus on an initial list of 22 EPAs. The authors presented these EPAs to faculty supervisors, residents, and patients for refinement. The authors collated and analyzed feedback from focus groups of residents and patients using thematic analysis. The EPAs were subsequently refined based on this feedback., Results: Stakeholders in EM residency training endorsed a final revised list of 22 EPAs. Stakeholder focus groups highlighted two main thematic considerations that helped shape the finalized list of EM EPAs: attention to the meaningful nuances of EPA language and contextualizing the EPAs and viewing them developmentally., Conclusions: To foreground all key stakeholders within the EPA process for EM, the authors chose within the development process to draft; come to consensus; and refine EPAs for EM in collaboration with relevant faculty, patient, and resident stakeholders. Each stakeholder group contributed meaningfully to the content and intended implementation of the EPAs. This process may serve as a model for others in developing stakeholder-responsive EPAs., Competing Interests: The authors declare no conflicts of interest., (© 2024 Society for Academic Emergency Medicine.)

Abstract: In 1999, the National Labor Relations Board determined that residents function as employees, thereby allowing them to freely unionize. From 2020 to 2023, house staff (i.e., resident physicians and fellows) unions have significantly increased, and 8 physician training centers, representing nearly 4,000 house staff, have unionized since March 2021. While unions provide residents with an important tool in effecting change in their workplace, their introduction into the educational milieu has the potential to alter the program director (PD)-resident relationship. In this article, the authors use the educational alliance framework to detail 3 factors required to support a quality educational relationship between a resident and their PD. They also elaborate on how the introduction of unions may impact the PD-resident relationship and explore the potential unintended consequences of unionization as it pertains to this relationship. The authors then use 2 social psychology theories, naïve realism and motivated reasoning, to describe common framing dynamics that lead to conflict during collective bargaining processes. They conclude by offering strategies that PDs may use to mitigate tensions that arise in contract negotiations, even without a direct seat at the table. Ultimately, PDs should anticipate continued growth of resident unions and prepare themselves and their programs for the tensions that may arise from this action. The PD role as a neutral third party ought to be preserved, which is possible if all parties set reasonable expectations for the changes in the PD's role and responsibilities under a union. PDs should understand the 3 core aspects of the educational alliance and the importance of establishing credibility with their residents early on to build a strong foundation., (Copyright © 2024 the Association of American Medical Colleges.)