Search

BACKGROUND: Sequential testing of clinical performance is an effective strategy to reduce cost of testing. PURPOSE: To evaluate prediction accuracy and financial savings of 4 screening tests of clinical performance. METHODS: Screening tests were created from a 13-case examination taken by 434 medical students at 4 schools. Regression analysis determined prediction accuracy for 2 test outcomes. Financial savings were computed from published estimates. RESULTS: Zero false passes were obtained with the 'Total Number of Cases Passed' screening test, but it saved only 27%. Sixty-two percent savings with 5% false passes occurred with the 'Classification' screening test. The 'Scale' and 'Mini Test' screening tests would have excused 79% and 67% examinees with 5% and 1% false passes, respectively. CONCLUSIONS: Prediction accuracy varies with screening test and outcome measure. Sequential testing of clinical performance can save 40% to 60% with low false pass rates. However, programs need to consider loss of information for curriculum and individual feedback relative to financial savings. [ABSTRACT FROM AUTHOR]

Background Teaching faculty request timely feedback from residents to improve their skills. Yet even with anonymous processes, this upward feedback can be difficult to obtain as residents raise concerns about identification and repercussions. Objective To examine faculty perception of the quality and content of feedback from residents after increasing anonymity and sacrificing timeliness. Methods Between 2011 and 2017, an associate program director at a large internal medicine residency program met briefly with each resident individually to obtain feedback about their teaching faculty shortly after their rotation. To improve anonymity, residents were promised their feedback would not be released until they graduated. In 2019, all feedback was collated and released at one time to faculty. We administered 3 timed, voluntary, anonymous, 36-item closed-ended surveys to faculty asking about the content and value, and to self-identify whether the feedback was praise, constructive, or criticism. Results Exactly 189 faculty participated with 140 completing all 3 surveys (74.1% response rate). Faculty reported this feedback content to be of higher quality (81.0%, 81 of 100) and quantity (82.4%, 84 of 102) in contrast to prior feedback. More than 85.4% (88 of 103) of faculty agreed this feedback was more specific. Faculty identified less praise (median 35.0% vs median 50.0%, P <.001) and more negative constructive feedback (median 20.0% vs median 5.0%, P <.001) compared to prior feedback. More than 82% (116 of 140) of faculty reported it would change their behavior, but 3 months after receiving the feedback, only 63.6% (89 or 140) felt the same way ( P <.001). Faculty were divided on the necessity of a time delay, with 41.4% (58 of 140) believing it reduced the feedback's value. Despite the delay, 32.1% (45 of 140) felt they could identify residents. Conclusions Offering a substantial delay in feedback delivery increased anonymity and enabled residents to furnish more nuanced and constructive comments; however, faculty opinions diverged on whether this postponement was valuable.

Objectives: This trial examines the impact of the Provider Awareness and Cultural dexterity Toolkit for Surgeons (PACTS) curriculum on surgical residents' knowledge, cross-cultural care, skills, and beliefs., Background: Cross-cultural training of providers may reduce health care outcome disparities, but its effectiveness in surgical trainees is unknown., Methods: PACTS focuses on developing skills needed for building trust, working with patients with limited English proficiency, optimizing informed consent, and managing pain. The PACTS trial was a randomized crossover trial of 8 academic general surgery programs in the United States: The Early group ("Early") received PACTS between periods 1 and 2, while the Delayed group ("Delayed") received PACTS between periods 2 and 3. Residents were assessed preintervention and postintervention on Knowledge, Cross-Cultural Care, Self-Assessed Skills, and Beliefs. χ 2 and Fisher exact tests were conducted to evaluate within-intervention and between-intervention group differences., Results: Of 406 residents enrolled, 315 were exposed to the complete PACTS curriculum. Early residents' Cross-Cultural Care (79.6%-88.2%, P <0.0001), Self-Assessed Skills (74.5%--85.0%, P <0.0001), and Beliefs (89.6%-92.4%, P =0.0028) improved after PACTS; knowledge scores (71.3%-74.3%, P =0.0661) were unchanged. Delayed resident scores pre-PACTS to post-PACTS showed minimal improvements in all domains. When comparing the 2 groups in period 2, Early residents had modest improvement in all 4 assessment areas, with a statistically significant increase in Beliefs (92.4% vs 89.9%, P =0.0199)., Conclusions: The PACTS curriculum is a comprehensive tool that improved surgical residents' knowledge, preparedness, skills, and beliefs, which will help with caring for diverse patient populations., Competing Interests: The authors report no conflicts of interest., (Copyright © 2024 Wolters Kluwer Health, Inc. All rights reserved.)

Background: Patient satisfaction is critical for referrals and reimbursement of surgical faculty but remains poorly characterized for residents. We investigated whether patient evaluations of surgical trainees vary by resident gender., Methods: Surgical inpatients evaluated surgical resident care postoperatively after positively identifying trainees. Evaluations (Consumer Assessment of Healthcare Providers and Systems Surgical Care Surveys (S-CAHPS)) were scored by the "top-box" method, stratified by training level, and compared between women and men residents., Results: Ninety-one percent of patients participated (n ​= ​324/357). Patients recognized women interns less than men (75.0 ​% vs 87.2 ​%, p ​= ​0.01). S-CAHPS scores for women vs men interns were equivalent except for spending sufficient time with patients (75.6 ​% vs 88.0 ​%, p ​= ​0.02). For senior residents, there was no difference in patient recognition of women vs men (83.9 ​% vs 85.2 ​%, p ​= ​0.91) or in any S-CAHPS scores (p ​> ​0.05)., Conclusions: Gendered differences in patient evaluations of surgical trainees exist for interns but resolve by senior years. Future work should explore how patient evaluations can support trainee development while ensuring patients recognize the role of surgical residents regardless of gender., Competing Interests: Declaration of competing interest The authors declare no conflicts of interest and all financial disclosures have been made above., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Introduction: Bedside cardiac assessment (BCA) is deficient across a spectrum of noncardiology trainees. Learners not taught BCA well may become instructors who do not teach well, creating a self-perpetuating problem. To improve BCA teaching and learning, we developed a high-quality, patient-centered curriculum for medicine clerkship students that could be flexibly implemented and accessible to other health professions learners., Methods: With a constructivist perspective, we aligned learning goals, activities, and assessments. The curriculum used a "listen before you auscultate" framework, capturing patient history as context for a six-step, systematic approach. In the flipped classroom, short videos and practice questions preceded two 1-hour class activities that integrated diagnostic reasoning, pathophysiology, physical diagnosis, and reflection. Activities included case discussions, jugular venous pressure evaluation, heart sound competitions, and simulated conversations with patients. Two hundred sixty-eight students at four US and international medical schools participated. We incorporated feedback, performed thematic analysis, and assessed learners' confidence and knowledge., Results: Low posttest data capture limited quantitative results. Students reported increased confidence in BCA ability. Knowledge increased in both BCA and control groups. Thematic analysis suggested instructional design strategies were effective and peer encounters, skills practice, and encounters with educators were meaningful., Discussion: The curriculum supported active learning of day-to-day clinical competencies and promoted professional identity formation alongside BCA ability. Feedback and increased confidence on the late-clerkship posttest suggested durable learning. We recommend approaches to confirm this and other elements of knowledge, skill acquisition, or behaviors and are surveying impacts on professional identity formation-related constructs., (© 2023 Meisel et al.)

Faculty have an implicit expectation that their graduate and professional students will become leaders in their respective fields; however, there is a lack of formalized co-curricular education to prepare them to assume leadership. This article provides two examples of co-curricular leadership education programs as inspiration for others to develop and navigate the challenges of delivering programming., (© 2022 Wiley Periodicals, LLC.)

Background: Effective teamwork in interdisciplinary healthcare teams is necessary for patient safety. Psychological safety is a key component of effective teamwork. The baseline psychological safety on pediatric inpatient healthcare teams is unknown. The purpose of this study is to determine the baseline psychological safety between pediatric nurses and residents and examine the impact of an interdisciplinary nighttime simulation curriculum., Methods: A convergent, multistage mixed methods approach was used. An interprofessional simulation curriculum was implemented fall 2020 to spring 2021. Qualitative focus group data and quantitative survey data on team psychological safety were collected and compared, both pre- and post-intervention and across nurses and residents. Thematic analysis of the qualitative data was conducted, and themes integrated with survey findings., Results: Data were collected from 30 nurses and 37 residents pre-intervention and 32 and 38 post-intervention, respectively. Residents and nurses negatively rated psychological safety (pre-intervention mean = 3.40 [SD = 0.72]; post-intervention mean = 3.35 [SD = 0.81]). At both times psychological safety was rated significantly lower for residents (pre-intervention mean = 3.11 [SD = 0.76], post-intervention mean = 2.98 [SD = 0.84]) than nurses (pre-intervention mean = 3.76 [SD = 0.45], post-intervention mean = 3.79 [SD = 0.50]), all P < .001. Qualitative analysis identified six integrated themes: (1) influence of existing relationships on future interactions, (2) unsatisfactory manner and frequency of communication, (3) unsatisfactory resolution of disagreements (4) overwhelming resident workload impairs collaboration, (5) interpersonal disrespect disrupts teamwork, and (6) interprofessional simulation was useful but not sufficient for culture improvement., Conclusion: Resident-nurse team psychological safety ratings were not positive. While interprofessional simulation curriculum shows promise, additional efforts are needed to improve psychological safety among residents and nurses., (© 2022. The Author(s).)

Background: Emotional Intelligence (EI) has been linked to burnout, job satisfaction, and work performance among resident physicians. Individuals with a growth mindset believe intelligence and ability are traits that can be increased or improved upon through learning. EI and mindset have yet to be studied among plastic surgeons in the United States., Materials and Methods: An online survey was distributed to plastic surgery residents and general surgery residents at a single institution containing a validated EI survey tool (TEIQue-SF). Second, a survey was sent to all plastic surgery residents in the United States containing the TEIQue-SF and a validated survey tool to assess mindset (ITIS)., Results: The response rate for the local study was 82% (plastic surgery) and 75% (general surgery). Only 7.8% of plastic surgery residents had any prior formal EI training or education. Mean global EI scores of local plastic surgery residents were higher than the normative population sample (P <0.0001). Plastic surgery and general surgery residents had similar EI scores. Integrated residents and junior plastic surgery residents had higher Well-Being scores compared to independent (P = 0.04) and senior residents (P = 0.04). Sixty-four plastic surgery residents completed the national survey. No correlation was found between EI and ITIS scores among the national plastic surgery resident cohort., Conclusion: Despite different work profiles, general surgery and plastic surgery residents may have similar EI profiles. Mindset does not appear to correlate to EI domains. Future studies will focus on how EI and mindset may develop over the course of residency training and their relationship to overall resident wellness., (Copyright © 2021. Published by Elsevier Inc.)

Background: Limited exposure to surgeons early on in medical school may adversely impact students' clerkship experiences and professional development. This explanatory sequential mixed methods study investigates a perceived discrepancy between surgical and nonsurgical instructors in our institution's preclinical curriculum., Methods: The demographics of preclinical faculty were assessed before and after a curricular reform. Semi-structured interviews with 13 surgical faculty explored barriers and facilitators to surgeon involvement. Responses were inductively coded and thematically analyzed., Results: Surgeons' contributions to preclinical instruction fell from 10% to 5% across the curriculum reform. Barriers both leading to and reinforced by surgeons' limited involvement relate to surgeon, medical school, and student factors. Participants proposed three solutions to barriers in each domain., Conclusions: Surgeons provide a minority of our preclinical instruction and may be disproportionately impacted by reform efforts. Deliberate efforts are necessary to increase opportunities for surgeons to engage with preclinical medical students., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Background: This study characterizes prevalence, frequency, and forms of patient-derived gender-based discrimination (GBD) experienced by resident physicians, as well as their experiences witnessing and reporting patient-derived GBD., Methods: A web-based survey was sent to residents from 12 programs at three academic institutions., Results: Response rate was 47.9% (309/645) with 55.0% of respondents identifying as women. Women were more likely than men to experience patient-derived GBD during residency (100% vs 68.8%, p < 0.001), including inappropriate physical contact, receiving less trust from patients, and being mistaken for a nurse (p < 0.001). While 85.9% of residents personally experienced and 95.0% of residents witnessed patient-derived GBD, only 3.4% of residents formally reported patient-derived GBD. Women were more likely to report negative personal and professional consequences of patient-derived GBD., Conclusions: Patient-derived GBD is pervasive and disproportionately affects women residents. Current reporting mechanisms are not adequately capturing nor addressing patient-derived GBD., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Objective: Minimally invasive surgery (MIS) is an integral component of General Surgery training and practice. Yet, little is known about how much autonomy General Surgery residents achieve in MIS procedures, and whether that amount is sufficient. This study aims to establish a contemporary benchmark for trainee autonomy in MIS procedures. We hypothesize that trainees achieve progressive autonomy, but fail to achieve meaningful autonomy in a substantial percentage of MIS procedures prior to graduation., Setting/participants: Fifty General Surgery residency programs in the United States, from September 1, 2015 to March 19, 2020. All Categorical General Surgery Residents and Attending Surgeons within these programs were eligible., Design: Data were collected prospectively from attending surgeons and categorical General Surgery residents. Trainee autonomy was assessed using the 4-level Zwisch scale (Show and Tell, Active Help, Passive Help, and Supervision Only) on a smartphone application (SIMPL). MIS procedures included all laparoscopic, thoracoscopic, endoscopic, and endovascular/percutaneous procedures performed by residents during the study. Primary outcomes of interest were "meaningful autonomy" rates (i.e., scores in the top 2 categories of the Zwisch scale) by postgraduate year (PGY), and "progressive autonomy" (i.e., differences in autonomy between PGYs) in MIS procedures, as rated by attending surgeons. Primary outcomes were determined with descriptive statistics, one-way analysis of variance (ANOVA) and Z-tests. Secondary analyses compared (i) progressive autonomy between common MIS procedures, and (ii) progressive autonomy in MIS vs. non-MIS procedures., Results: A total of 106,054 evaluations were performed across 50 General Surgery residency programs, of which 38,985 (37%) were for MIS procedures. Attendings performed 44,842 (42%) of all evaluations, including 16,840 (43%) of MIS evaluations, while residents performed the rest. Overall, meaningful autonomy in MIS procedures increased from 14.1% (PGY1s) to 75.9% (PGY5s), with significant (p < 0.001) increases between each PGY level. Meaningful autonomy rates were higher in the MIS vs. non-MIS group [57.2% vs. 48.0%, p < 0.001], and progressed more rapidly in MIS vs. non-MIS, (p < 0.05). The 7 most common MIS procedures accounted for 83.5% (n = 14,058) of all MIS evaluations. Among PGY5s performing these procedures, meaningful autonomy rates (%) were: laparoscopic appendectomy (95%); laparoscopic cholecystectomy (93%); diagnostic laparoscopy (87%); upper/lower endoscopy (85%); laparoscopic hernia repair (72%); laparoscopic partial colectomy (58%); and laparoscopic sleeve gastrectomy (45%)., Conclusions: US General Surgery residents receive progressive autonomy in MIS procedures, and appear to progress more rapidly in MIS versus non-MIS procedures. However, residents fail to achieve meaningful autonomy in nearly 25% of MIS cases in their final year of residency, with higher rates of meaningful autonomy only achieved in a small subset of basic MIS procedures., (Copyright © 2020. Published by Elsevier Inc.)

Background: Valid methods for providing detailed formative feedback on direct laryngoscopy and endotracheal intubation (ETI) performance do not exist. We are developing an observation-based assessment tool for measuring performance and providing feedback during ETI., Methods: Based on the literature and interviews of experts, we proposed an initial ETI metric with 22 items. Six anesthesiology experts used it to assess the quality of ETI performance in videotaped intubations. Following metric revisions, 2 expert groups assessed 2 collections of videos (27 total) using the revised metric. Two reference standards for comparison with metric scores were created with a third and fourth group of experts; (1) an average global rating (1-100) of each ETI performance and (2) average rank-ordered performance from best to worst. Rater agreement and correlations between the 2 methods were calculated. Regression analysis determined items that optimally discriminated quality. When calculating a score based on all clinically important terms, multiple weightings were evaluated., Results: Metric items had high average rater agreement (80%) with intraclass correlation coefficients averaging 0.83. Correlations of the reference rank and score were high for both video collections (-0.96, P < .05, and -0.95, P < .05). Regression coefficients for different item weighting methods indicated strong relationships with global ratings (averaging r = 0.89, P < .05) and rankings averaging -0.85, P < .05). Prediction of global ratings using regression achieved high accuracy ( R 2 = 0.8218)., Conclusions: High observer agreement and strong correlations between metric and rank data support the validity of using this metric to assess ETI performance. Different weighting models yielded scores that correlated strongly with the ratings and ranks from global assessment. When using the metric to predict competency, a 3-item regression model is most accurate in predicting a global score., Competing Interests: Funding: National Institutes of Health 5R01HL119248-04, (© 2020 Society for Education in Anesthesia.)

Background: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment., Methods: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail., Results: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room., Conclusions: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Objective: Optimal methods for medical student assessment in surgery remain elusive. Faculty- and housestaff-written evaluations constitute the chief means of student assessment in medical education. However, numerous studies show that this approach has poor specificity and a high degree of subjectivity. We hypothesized that an objective structured clinical examination (OSCE) in the surgery clerkship would provide additional data on student performance that would confirm or augment other measures of assessment., Design: We retrospectively reviewed data from OSCEs, National Board of Medical Examiners shelf examinations, oral presentations, and written evaluations for 51 third-year Harvard Medical School students rotating in surgery at Massachusetts General Hospital from 2014 to 2015. We expressed correlations between numeric variables in Pearson coefficients, stratified differences between rater groups by one-way analysis of variance, and compared percentages with 2-sample t-tests. We examined commentary from both OSCE and clinical written evaluations through textual analysis and summarized these results in percentages., Results: OSCE scores and clinical evaluation scores correlated poorly with each other, as well as with shelf examination scores and oral presentation grades. Textual analysis of clinical evaluation comments revealed a heavy emphasis on motivational factors and praise, whereas OSCE written comments focused on cognitive processes, patient management, and methods to improve performance., Conclusions: In this single-center study, an OSCE provided clinical skills data that were not captured elsewhere in the surgery clerkship. Textual analysis of faculty evaluations reflected an emphasis on interpersonal skills, rather than appraisal of clinical acumen. These findings suggest complementary roles of faculty evaluations and OSCEs in medical student assessment., (Copyright © 2017 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

It is acknowledged that interprofessional communication and teamwork are foundational for high-quality, safe medical practice. The theory of distributed knowledge posits that each person has unique knowledge and experiences that can contribute to a broader group perspective. Patient care can be positively influenced by a robust and interprofessionally shared understanding of the complexities of health and illness. One would expect a variety of perspectives to be evident in all healthcare providers, including incoming health science students from different professional fields. To examine this notion, medical, nursing, and pharmacy students (n = 24) at the start of an interprofessional training experience were asked to write an essay on factors that contribute to health and/or illness. Their essays were thematically coded to generate a list of factors considered key to health/illness and compare responses across fields to better understand how knowledge is distributed across entering learners. Results show that students from different professional backgrounds emphasise some shared and some divergent factors. This distributed knowledge can be a rich source of understanding and shared thinking across professions. It is also a critical source of individual power to contribute to the construction of an ongoing, collaborative discourse about health.

Purpose: Some General Surgery Milestones can be difficult to assess in traditional clinical settings and especially difficult to assess in junior residents. The purpose of this pilot study was to METHODS: A total of 9 categorical interns participated in a comprehensive, 4-module, SP scenario designed to evaluate and manage right upper quadrant pain. SP checklist scores (SP%) were converted to Milestone-equivalent scores for direct comparison (SP-C). Milestone scores were analyzed from 3 different sources: SP, faculty (FAC), and CCC. Interns completed course evaluations at the end of each session. Spearman's rho was used to determine correlations. Wilcoxon signed rank tests were used to test for differences between scores from different sources., Results: Individual intern Milestone scores from the 3 sources (SP-C, FAC, and CCC) did not correlate. All 7 mean Milestone scores from SPs were significantly higher than from FAC and CCC. FAC and CCC scores were statistically equivalent except for Systems-Based Practice 1 (SBP1) and Patient Care 3 (PC3) where CCC scores were significantly higher than FAC. Mean SP% scores for PC1 were significantly lower than for PROF1, MK1, MK2, and ICS1 (p < 0.05). Interns felt the modules were moderately to very useful., Conclusions: Developing an SP scenario for Milestones evaluation is feasible. SPs, faculty observers, and CCC each use different data to provide a unique source of Milestone assessment. SP scenarios may be ideally suited to assess specific resident strengths and weaknesses and provide individualized feedback, thus augmenting traditional evaluations. Additional SP scenarios, assessing a broader range of skills and Milestones, are advisable for more reliable estimates of resident performance., (Copyright © 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Background: Physiologic and psychological stress are commonly experienced by operating room (OR) personnel, yet there is little research about the stress levels in OR teams and their impact on performance. Previously published procedures to measure physiologic activation are invasive and impractical for the OR. The purpose of this study was to determine the practicality of a new watch-sized device to measure galvanic skin response (GSR) in OR team members during high-fidelity surgical simulations., Methods: Interprofessional OR teams wore sensors on the wrist (all) and ankle (surgeons and scrub nurses/technicians) during the orientation, case, and debriefing phases for 17 simulations of a surgical airway case. Data were compared across all simulation phases, collectively and for each professional group., Results: Forty anesthesiology residents, 35 surgery residents, 27 OR nurses, 12 surgical technicians, and 7 CRNAs participated. Collectively, mean wrist GSR levels significantly increased from orientation phase to the case (0.40-0.62 μS; P < .001) and remained elevated even after the simulation was over (0.40-0.67 μS; P < .001). Surgery residents were the only group that demonstrated continued increases in wrist GSR levels throughout the entire simulation (change in GSR = 0.21 to 0.32 to 0.11 μS; P < .01). Large intraindividual differences (≤ 200 times) were found in both wrist and ankle GSR. There was no correlation between wrist and ankle data., Conclusion: Continuous GSR monitoring of all professionals during OR simulations is feasible, but would be difficult to implement in an actual OR environment. Large variation in individual levels of physiologic activation suggests complementary qualitative research is needed to better understand how people respond to stressful OR situations., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Background: Although emotional intelligence (EI) may have a role in the development of Accreditation Council for Graduate Medical Education core competencies, few studies have measured resident EI across specialties. This study aimed to describe the EI of resident physicians across multiple specialties., Methods: Three hundred twenty five surgery, pediatric, and pathology residents at 3 large academic institutions were invited to complete the psychometrically validated Trait Emotional Intelligence Questionnaire., Results: The response rate was 42.8% (n = 139). Global EI of all residents (101.0 ± 8.1) was comparable with, but less variable than, the general population sample and was not statistically different between specialties. Compared with the norm sample, residents in the 3 specialty groups demonstrated unique combinations of areas of relative high and low development., Conclusions: There exist distinct strengths and opportunities for the development for surgery, pediatrics, and pathology residents. Future investigations could use EI profiling to create educational interventions to develop specific areas of EI and assess correlation with resident performance., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Background: E-learning is increasingly common in undergraduate medical education. Internet-based multimedia materials should be designed with millennial learner utilization preferences in mind for maximal impact., Methods: Medical students used all 20 Web Initiative for Surgical Education of Medical Doctors modules from July 1, 2013 to October 1, 2013. Data were analyzed for topic frequency, time and week day, and access to questions., Results: Three thousand five hundred eighty-seven students completed 35,848 modules. Students accessed modules for average of 51 minutes. Most frequent use occurred on Sunday (23.1%), Saturday (15.4%), and Monday (14.3%). Friday had the least use (8.2%). A predominance of students accessed the modules between 7 and 10 PM (34.4%). About 80.4% of students accessed questions for at least one module. They completed an average of 40 ± 30 of the questions. Only 827 students (2.3%) repeated the questions., Conclusions: Web Initiative for Surgical Education of Medical Doctors has peak usage during the weekend and evenings. Most frequently used modules reflect core surgical problems. Multiple factors influence the manner module questions are accessed., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Background: Because academic literature indicates that emotional intelligence (EI) is tied to work performance, job satisfaction, burnout, and client satisfaction, there is great interest in understanding physician EI., Objective: To determine whether gender differences in resident EI profiles mirror EI gender differences in the general population., Study Design (include Participants and Setting): A total of 325 residents in 3 types of residency programs (pathology, pediatrics, and general surgery) at 3 large academic institutions were invited electronically to complete the validated Trait Emotional Intelligence Questionnaire (TEIQue), a tool consisting of 153 items that cluster to 15 independent facets, 4 composite factors, and 1 global EI score., Results: The response rate was 42.8% (n = 139, women = 84). Global EI was not significantly different between men and women resident physicians (p = 0.74). Women scored higher than men in the TEIQue facets impulse control (p = 0.004) and relationships (p = 0.004). Men scored higher than women in 2 facets, stress management (p = 0.008) and emotion management (p = 0.023). Within surgery (n = 85, women = 46), women scored higher than men in impulse control (p = 0.006), whereas men scored higher in stress management (p = 0.008)., Conclusions: Men and women residents across 3 specialties demonstrated near-identical global EI scores. However, gender differences in specific TEIQue facets suggest that similar to the general population, men and women residents may benefit from specific training of different EI domains to enhance well-rounded development. The lack of significant gender differences within surgery may indicate that surgery attracts individuals with particular EI profiles regardless of gender. Future research should focus on the functional relationship between educational interventions that promote targeted EI development and enhanced clinical performance., (Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Purpose: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy., Methods: Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated., Results: Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high., Conclusions: The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines., (Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Background: Problems with communication and team coordination are frequently linked to adverse events in medicine. However, there is little experimental evidence to support a relationship between observer ratings of teamwork skills and objective measures of clinical performance., Aim: Our main objective was to test the hypothesis that observer ratings of team skill will correlate with objective measures of clinical performance., Methods: Nine teams of medical students were videotaped performing two types of teamwork tasks: (1) low fidelity classroom-based patient assessment and (2) high fidelity simulated emergent care. Observers used a behaviourally anchored rating scale to rate each individual on skills representative of assertiveness, decision-making, situation assessment, leadership, and communication. A checklist-based measure was used to assess clinical team performance., Results: Moderate to high inter-observer correlations and moderate correlations between cases established the validity of a behaviourally anchored team skill rating tool for simulated emergent care. There was moderate to high correlation between observer ratings of team skill and checklist-based measures of team performance for the simulated emergent care cases (r = 0.65, p = 0.06 and r = 0.97, p < 0.0001)., Conclusions: These results provide prospective evidence of a positive relationship between observer ratings of team skills and clinical team performance in a simulated dynamic health care task.

Objectives: To assess the individual and team skills acquired from an interactive training program to prepare emergency personnel to respond to terrorist acts., Methods: We developed a 16-hour, two-day, multimedia- and simulation-enhanced course that places learners in realistic situations using the equipment required to respond to various chemical, biologic, radiologic, and explosive acts of terrorism. Small-group sessions and drills were conducted. Errors in skill performance were corrected immediately, and then skills were repeated to achieve mastery. Participants included emergency medical technicians, paramedics, nurses, and physicians. Team performance was assessed over four successive scenarios using a 100-mm visual analog scale. Individual learner skill acquisition was assessed with precourse and postcourse evaluation of selected skills in a randomized sampling of consenting learners., Results: Nearly all teams achieved mastery of the required skills by the second assessment rotation. Individual learners demonstrated significant gains in the ability to emergently don personal protective equipment and administer a nerve agent antidote kit., Conclusions: An interactive, simulation-enhanced curriculum of terrorism response training for emergency responders can produce significant, quantifiable individual and team skill gain. Future studies should further address performance benchmarks for these newly acquired skills.

Objectives: This report synthesises a subset of 31 journal articles on high-fidelity simulation-based medical education containing 32 research studies drawn from a larger qualitative review published previously. These studies were selected because they present adequate data to allow for quantitative synthesis. We hypothesised an association between hours of practice in simulation-based medical education and standardised learning outcomes measured as weighted effect sizes., Methods: Journal articles were screened using 5 exclusion and inclusion criteria. Response data were extracted and 3 judges independently coded each study. Learning outcomes were standardised using a common metric, the average weighted effect size (AWES), due to the heterogeneity of response measures in individual studies. anova was used to evaluate AWES differences due to hours of practice on a high-fidelity medical simulator cast in 5 categories. The eta squared (eta2) statistic was used to assess the association between AWES and simulator practice hours., Results: There is a strong association (eta2=0.46) between hours of practice on high-fidelity medical simulators and standardised learning outcomes. The association approximates a dose-response relationship., Conclusions: Hours of high-fidelity simulator practice have a positive, functional relationship with standardised learning outcomes in medical education. More rigorous research methods and more stringent journal editorial policies are needed to advance this field of medical education research.

Introduction: Responding to acts of terrorism requires the effective use of public-safety and medical-response resources. The knowledge, skills and attitudes necessary to respond to future threats is unfamiliar to most emergency responders., Objectives: The purpose of this report to describe the development, implementation and evaluation of a multidisciplinary, interactive and simulation-enhanced course to prepare responders to acts of terrorism., Methods: We used a 5-step systematic process to develop a blended-learning, simulation-enhanced training program. Learners completed a self-confidence questionnaire and written examination prior to the course and a self-confidence questionnaire, written examination and course evaluation when they finished the course., Results: From July 7, 2003 to March 8, 2005, 497 consenting learners completed the course. After course completion, learners demonstrated significant increases in their knowledge of terrorism response (t = -64.3, df = 496, p < 0.05) and their confidence in responding to terrorist events (t = -45.5, df = 496, p < 0.05). Learner feedback about the course was highly positive., Conclusions: We successfully implemented a two-day course for professionals likely to respond to terrorist acts that included scenario-based performance training and assessment. Course participants increased their knowledge and were more confident in their ability to respond to acts of terrorism after participating in this course.

Background: Using computer-based simulation to assess clinical skill-a key competence for medical trainees-enables standardization and exposure to a broad sample of physical findings. The purpose of this study is to provide evidence of construct validity for a computer-based outcome measure of neurology clinical skills., Method: A total of 128 medical students and neurology residents at four institutions volunteered to take a 34-question computer-based test designed to measure neurology clinical skills. Subjects were classified into three groups based on level of training: novice, intermediate, and experienced., Results: Overall performance increased with level of training. Question difficulty discriminated between groups as predicted. Twenty-six of 34 individual items discriminated between novices and more advanced learners. This test separated learners at different levels of training with a consistency of .92., Conclusion: This study provided evidence of construct validity for a computer-based outcome measure of neurology clinical skills.

Since appropriate treatment of patients in the first few hours of ischemic stroke may decrease the risk of long-term disability, prehospital providers should recognize, assess, manage and communicate about stroke patients in an effective and time-efficient manner. This requires the instruction and evaluation of a wide range of competencies including clinical skills, patient investigation and management and communication skills. The authors developed and assessed the effectiveness of a simulation-enhanced stroke course that incorporates several different learning strategies to evaluate competencies in the care of acute stroke patients. The one-day, interactive, emergency stroke course features a simulation-enhanced, blended-learning approach that includes didactic lectures, tabletop exercises, and focused-examination training and small-group sessions led by paramedic instructors as standardized patients portraying five key neurological syndromes. From January to October 2000, 345 learners were assessed using multiple-choice tests as were randomly selected group of 73 learners using skills' checklists during two pre- and two post-course simulated patient encounters. Among all learners there was a significant gain in knowledge (pre: 53.9%+/-13.9 and post: 85.4%+/-8.5; p<0.001), and for the 73 learners a significant improvement in their clinical and communication skills (p<0.0001 for all). By using a simulation-enhanced, blended-learning approach, pre-hospital paraprofessionals were successfully trained and evaluated in a wide range of competences that will lead to the more improved recognition and management of acute stroke patients.

Review Date: 1969 to 2003, 34 years., Background and Context: Simulations are now in widespread use in medical education and medical personnel evaluation. Outcomes research on the use and effectiveness of simulation technology in medical education is scattered, inconsistent and varies widely in methodological rigor and substantive focus., Objectives: Review and synthesize existing evidence in educational science that addresses the question, 'What are the features and uses of high-fidelity medical simulations that lead to most effective learning?'., Search Strategy: The search covered five literature databases (ERIC, MEDLINE, PsycINFO, Web of Science and Timelit) and employed 91 single search terms and concepts and their Boolean combinations. Hand searching, Internet searches and attention to the 'grey literature' were also used. The aim was to perform the most thorough literature search possible of peer-reviewed publications and reports in the unpublished literature that have been judged for academic quality., Inclusion and Exclusion Criteria: Four screening criteria were used to reduce the initial pool of 670 journal articles to a focused set of 109 studies: (a) elimination of review articles in favor of empirical studies; (b) use of a simulator as an educational assessment or intervention with learner outcomes measured quantitatively; (c) comparative research, either experimental or quasi-experimental; and (d) research that involves simulation as an educational intervention., Data Extraction: Data were extracted systematically from the 109 eligible journal articles by independent coders. Each coder used a standardized data extraction protocol., Data Synthesis: Qualitative data synthesis and tabular presentation of research methods and outcomes were used. Heterogeneity of research designs, educational interventions, outcome measures and timeframe precluded data synthesis using meta-analysis., Headline Results: Coding accuracy for features of the journal articles is high. The extant quality of the published research is generally weak. The weight of the best available evidence suggests that high-fidelity medical simulations facilitate learning under the right conditions. These include the following: providing feedback--51 (47%) journal articles reported that educational feedback is the most important feature of simulation-based medical education; repetitive practice--43 (39%) journal articles identified repetitive practice as a key feature involving the use of high-fidelity simulations in medical education; curriculum integration--27 (25%) journal articles cited integration of simulation-based exercises into the standard medical school or postgraduate educational curriculum as an essential feature of their effective use; range of difficulty level--15 (14%) journal articles address the importance of the range of task difficulty level as an important variable in simulation-based medical education; multiple learning strategies--11 (10%) journal articles identified the adaptability of high-fidelity simulations to multiple learning strategies as an important factor in their educational effectiveness; capture clinical variation--11 (10%) journal articles cited simulators that capture a wide variety of clinical conditions as more useful than those with a narrow range; controlled environment--10 (9%) journal articles emphasized the importance of using high-fidelity simulations in a controlled environment where learners can make, detect and correct errors without adverse consequences; individualized learning--10 (9%) journal articles highlighted the importance of having reproducible, standardized educational experiences where learners are active participants, not passive bystanders; defined outcomes--seven (6%) journal articles cited the importance of having clearly stated goals with tangible outcome measures that will more likely lead to learners mastering skills; simulator validity--four (3%) journal articles provided evidence for the direct correlation of simulation validity with effective learning., Conclusions: While research in this field needs improvement in terms of rigor and quality, high-fidelity medical simulations are educationally effective and simulation-based education complements medical education in patient care settings.

Purpose: The purpose of this article is to summarize this author's view on "where we are" with standardized patient-based assessments of clinical performance and to offer three directions for further research and development., Summary: The push for more objective outcome data has fueled proliferation of the most researched innovation in the history of medical education. Near-random clinical experiences of students do not provide consistent, repeated practice with important clinical cases to achieve minimally adequate performance on these objective performance examinations, leading to scoring "psychogymnastics" to titrate fail rates. The second area is to modify these examinations to reflect features at higher levels of professional development such as situational awareness. Theories of professional development should guide changes. The third area incorporates multiperson scenarios; a clinician with a family or a team in the operating room. Simulation of complex situations, especially those requiring rapid, accurate communication and action can reduce medical errors and improve patient safety., Conclusions: Standardized patient-based examinations provide objective outcome data but require artificial adjustments in scoring due to inconsistent learning opportunities. Theoretical research on professional development, acquisition of expertise and team functioning provides fertile, new directions to take standardized patient-based examinations to the next level.

Purpose: This study reports the development of reliable multimedia, computer-based measures of bedside neurology skills., Method: A consortium of neurologists and medical educators (1) identified bedside skills and (2) created a computer-based test. Test-item stems were multimedia clips of standardized patients. Options appeared as text. Sixty-one junior medical students responded to these items., Results: The 77-item test yielded a reliability coefficient of 0.85. Subgroup item analysis resulted in reliability coefficients of 0.73 for the 20-question "pretest" (Test A) and 0.75 for the 20-question "posttest" (Test B)., Conclusion: Two sets of test items were developed that can be used as outcome measures in studies that assess the effectiveness of educational interventions in bedside neurology.

Background: Objective evaluations of residents' clinical skills reveal serious deficits., Purpose: To develop, implement, and evaluate outcomes from a review course in cardiology bedside skills for internal medicine residents., Methods: We used a 1-group pretest-posttest design with historical comparisons. The study was conducted at the University of Miami School of Medicine as part of the internal medicine residency program from July 1999 to June 2000. A total of 67 2nd- and 3rd-year medicine residents received an educational intervention involving deliberate practice using simulation technology. A total of 155 4th-year medical students in one intervention and one comparison group (n = 53) served as historical comparisons. Outcome measures were a reliable computer-delivered pretest and posttest that evaluate cardiology bedside skills., Results: Residents who received the review course and medical students who received a comparable educational intervention showed large and statistically significant pretest-to-posttest improvement in bedside skills. These 2 groups are also significantly and substantially different at posttest from a comparison group of 4th-year medical students that did not receive a specific educational intervention., Conclusion: Educational interventions using simulation technology that engage learners in deliberate practice of clinical skills produce large improvements in a relatively short time, with little faculty involvement.

Background: Because of the content of the American Board of Surgery (ABS) certifying (oral) examination, there is a perception that those in some subspecialty surgical training programs at the time of the examination may have a lower pass rate. In addition, the format of the oral examination has prompted the use of specialized preparation such as "mock orals" and commercial courses. The purpose of this study was to correlate the pass rates on the ABS certifying examination with the practice status and methods of specialized preparation., Materials and Methods: A survey covering demographic information, type of surgical practice at the time of the examination, methods of preparation, and results of the examination was distributed to 1997 and 1998 graduates via a request to surgical residency program directors., Results: One hundred one of 268 program directors supplied 717 names. There were 465 responses. Surveys distributed by the other 167 program directors resulted in an additional 81 responses. Four hundred ten (75%) of the respondents had taken the certifying examination. The total pass rate was 91%. There were no significant differences in the pass rate between those in private practice general surgery; those in academic general surgery; Thoracic, Vascular, or Plastic Surgery Fellows; those in other surgical fellowships; and those in the military or research. No significant differences in the pass rates were noted between those who prepared with formal mock orals, with informal mock orals, with a commercial course, with combinations of the three, and with no specialized preparation., Conclusion: Performance on the ABS certifying examination was not influenced by the candidate's practice status at the time of the examination. A substantial percentage of examinees either are exposed to or perceive the need to pursue specialized preparation for the examination, a behavior that in general produces good results.

Medical students must learn to recognize occupational and environmental-related illness. An occupational and environmental medicine curriculum can achieve this goal. The curriculum must be evaluated to ensure that medical students are learning to recognize exposure-related health conditions and to evaluate if this ability correlates with medical interviewing skills. A case, formatted for an Objective Structured Clinical Examination (OSCE), was developed to evaluate student performance on an exposure-related clinical problem. The OSCE results were analyzed to identify the areas that differentiated the students who recognized an exposure-related medical condition from those who did not. We conclude that an OSCE is an effective curriculum evaluation tool to assess whether a core occupational and environmental-related curriculum is contributing to student learning in exposure history-taking and associated clinical reasoning skills.

Changes in medical practice that limit instruction time and patient availability, the expanding options for diagnosis and management, and advances in technology are contributing to greater use of simulation technology in medical education. Four areas of high-technology simulations currently being used are laparoscopic techniques, which provide surgeons with an opportunity to enhance their motor skills without risk to patients; a cardiovascular disease simulator, which can be used to simulate cardiac conditions; multimedia computer systems, which includes patient-centered, case-based programs that constitute a generalist curriculum in cardiology; and anesthesia simulators, which have controlled responses that vary according to numerous possible scenarios. Some benefits of simulation technology include improvements in certain surgical technical skills, in cardiovascular examination skills, and in acquisition and retention of knowledge compared with traditional lectures. These systems help to address the problem of poor skills training and proficiency and may provide a method for physicians to become self-directed lifelong learners.

The pressures of a changing health care system are making inroads on the commitment and effort that both basic science and clinical faculty can give to medical education. A tool that has the potential to compensate for decreased faculty time and thereby to improve medical education is multimedia computer instruction that is applicable at all levels of medical education, developed according to instructional design principles, and supported by evidence of effectiveness. The authors describe the experiences of six medical schools in implementing a comprehensive computer-based four-year curriculum in bedside cardiology developed by a consortium of university cardiologists and educational professionals. The curriculum consisted of ten interactive, patient-centered, case-based modules focused on the history, physical examination, laboratory data, diagnosis, and treatment. While an optimal implementation plan was recommended, each institution determined its own strategy. Major goals of the project, which took place from July 1996 to June 1997, were to identify and solve problems of implementation and to assess learners' and instructors' acceptance of the system and their views of its value. A total of 1,586 students used individual modules of the curriculum 6,131 times. Over 80% of students rated all aspects of the system highly, especially its clarity and educational value compared with traditional lectures. The authors discuss the aspects of the curriculum that worked, problems that occurred (such as difficulties in scheduling use of the modules in the third year), barriers to change and ways to overcome them (such as the type of team needed to win acceptance for and oversee implementation of this type of curriculum), and the need in succeeding years to formally assess the educational effectiveness of this and similar kinds of computer-based curricula.

Purpose: To determine whether medical students were prepared to assess risk and counsel patients about prevention of HIV infection, and whether HIV-related experience produced better knowledge and counseling skills., Method: In 1995, students at four North Carolina medical schools interviewed a standardized patient portraying a young woman concerned about HIV infection. The standardized patient recorded whether students asked risk-behavior questions and provided risk-reduction advice. A 21-item questionnaire assessed the students' knowledge of HIV testing and prevention. Students indicated whether they had had experience in educational settings related to HIV or STDs., Results: 415 students completed both the patient interview and the questionnaire. Many failed to ask the patient about several HIV-risk behaviors. Although nearly all (98%) inquired about condom use, fewer than two thirds asked about the patient's history of STDs, number of sexual partners, or specific sexual practices. Most students advised the patient to use condoms. The average score on the knowledge test was 79%; 70% of students confused anonymous with confidential testing, more than half overestimated the risk of HIV transmission from a needle stick, and nearly one in ten did not know how to use a condom. Educational exposures did not produce significantly better risk assessment, counseling information, or knowledge scores., Conclusion: A majority of experienced medical students did not assess several important risk factors of a patient concerned about HIV infection, and many would have provided incorrect information related to HIV testing and prevention of infection. Patient contact in traditional clinical settings did not influence prevention knowledge or behavior. More innovative methods are needed to train students in HIV-infection prevention and counseling.

Purpose: This research involved the development, and in particular the evaluation of the reliability, of scales to measure medical faculty attitudes toward clinical evaluation (ACE) of medical students. The intent was to create measures that yield reliable data and have practical utility in medical education research and faculty development., Method: A systematic, eight-step scale development protocol was used to create the instrument. In early 1993 factor analysis was used on data from 217 clinical faculty at four medical schools to refine the measures. Internal consistency and test-retest reliability analyses were performed. Analyses were also done to determine whether the attitude scores were influenced by such faculty demographic attributes as employing medical school, gender, age, tenure track status, academic rank, or academic department., Results: An initial pool of 52 items was reduced to 30 items based on iterative reliability studies. Factor analysis on the 30 items yielded two scales: (1) Quality of Evaluation Procedures, 12 items, alpha = .81; and (2) Content of Departmental Evaluations, eight items, alpha = .85. Test-retest reliabilities (12 weeks) for the scales were .67 and .74, respectively. Faculty demographics did not influence attitudes about the quality of evaluation procedures. However, family physicians showed a slightly more positive attitude toward the content of departmental evaluations than did physicians in five other medical specialties., Conclusion: The goal of developing reliable measures of faculty attitudes toward clinical evaluation of medical students has been achieved. With baseline reliabilities established, future research should assess the validity and utility of the scales, especially in the context of clinical practice examinations.

The primary goal of medical education is to foster development of clinical competence in trainees at all levels. Variable clinical experience, inconsistent methods of instruction, and ambiguous evaluation criteria undermine this goal. Standardized patients, trained to consistently portray a wide variety of clinical cases, can help overcome many of these educational problems. This article describes the development and application of standardized patients throughout medical training at The University of Texas Medical Branch, Galveston, in the freshman interviewing course, the second-year physical diagnosis course, third-year clerkships, a fourth-year final exercise, and residency training. Development of this program is discussed in the context of a broader literature in medical education, and investigation of variables affecting standardized patient and student performance is reported. Future directions for use of standardized patients in monitoring and promoting the development of clinical competence are discussed.