Search

Purpose: Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method., Method: In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment., Results: Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74)., Conclusions: EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.

Purpose: To determine the diagnostic justification proficiency of senior medical students across a broad spectrum of cases with common chief complaints and diagnoses., Method: The authors gathered diagnostic justification exercise data from the Senior Clinical Comprehensive Examination taken by Southern Illinois University School of Medicine's students from the classes of 2011 (n = 67), 2012 (n = 66), and 2013 (n = 79). After interviewing and examining standardized patients, students listed their key findings and diagnostic possibilities considered, and provided a written explanation of how they used key findings to move from their initial differential diagnoses to their final diagnosis. Two physician judges blindly rated responses., Results: Student diagnostic justification performance was highly variable from case to case and often rated below expectations. Of the students in the classes of 2011, 2012, and 2013, 57% (38/67), 23% (15/66), and 33% (26/79) were judged borderline or poor on diagnostic justification performance for more than 50% of the cases on the examination., Conclusions: Student diagnostic justification performance was inconsistent across the range of cases, common chief complaints, and underlying diagnoses used in this study. More than 20% of students exhibited borderline or poor diagnostic justification performance on more than 50% of the cases. If these results are confirmed in other medical schools, attention needs to be directed to investigating new curricular methods that ensure deliberate practice of these competencies across the spectrum of common chief complaints and diagnoses and do not depend on the available mix of patients.

Purpose: Fostering ability to organize and use medical knowledge to guide data collection, make diagnostic decisions, and defend those decisions is at the heart of medical training. However, these abilities are not systematically examined prior to graduation. This study examined diagnostic justification (DXJ) ability of medical students shortly before graduation., Method: All senior medical students in the Classes of 2011 (n = 67) and 2012 (n = 70) at Southern Illinois University were required to take and pass a 14-case, standardized patient examination prior to graduation. For nine cases, students were required to write a free-text response indicating how they used patient data to move from their differential to their final diagnosis. Two physicians graded each DXJ response. DXJ scores were compared with traditional standardized patient examination (SCCX) scores., Results: The average intraclass correlation between raters' rankings of DXJ responses was 0.75 and 0.64 for the Classes of 2011 and 2012, respectively. Student DXJ scores were consistent across the nine cases. Using SCCX and DXJ scores led to the same pass-fail decision in a majority of cases. However, there were many cases where discrepancies occurred. In a majority of those cases, students would fail using the DXJ score but pass using the SCCX score. Common DXJ errors are described., Conclusions: Commonly used standardized patient examination component scores (history/physical examination checklist score, findings, differential diagnosis, diagnosis) are not direct, comprehensive measures of DXJ ability. Critical deficiencies in DXJ abilities may thus go undiscovered.

Purpose: Little is known about the acquisition of clinical reasoning skills in medical school, the development of clinical reasoning over the medical curriculum as a whole, and the impact of various curricular methodologies on these skills. This study investigated (1) whether there are differences in clinical reasoning skills between learners at different years of medical school, and (2) whether there are differences in performance between students at schools with various curricular methodologies., Method: Students (n = 2,394) who had completed zero to three years of medical school at five U.S. medical schools participated in a cross-sectional study in 2008. Students took the same diagnostic pattern recognition (DPR) and clinical data interpretation (CDI) tests. Percent correct scores were used to determine performance differences. Data from all schools and students at all levels were aggregated for further analysis., Results: Student performance increased substantially as a result of each year of training. Gains in DPR and CDI performance during the third year of medical school were not as great as in previous years across the five schools. CDI performance and performance gains were lower than DPR performance and gains. Performance gains attributable to training at each of the participating medical schools were more similar than different., Conclusions: Years of training accounted for most of the variation in DPR and CDI performance. As a rule, students at higher training levels performed better on both tests, though the expected larger gains during the third year of medical school did not materialize.

Purpose: During the transition from medical school to internship, trainees experience high levels of stress related to pages on the inpatient wards. The steep learning curve during this period may also affect patient safety. The authors piloted the use of simulated pages to improve medical student preparedness, decrease stress related to pages, and familiarize medical students with common patient problems., Method: A multidisciplinary team at Southern Illinois University School of Medicine developed simulated pages that were tested among senior medical students. Sixteen medical students were presented with 11 common patient scenarios. Data on assessment, management, and global performance were collected. Mean confidence levels were evaluated pre- and postintervention. Students were also surveyed on how the simulated pages program influenced their perceived comfort in managing patient care needs and the usefulness of the exercise in preparing them to handle inpatient pages., Results: Mean scores on the assessment and management portions of the scenarios varied widely depending on the scenario (range -15.6 ± 41.6 to 95.7 ± 9.5). Pass rates based on global performance ranged from 12% to 93%. Interrater agreement was high (mean kappa = 0.88). Students' confidence ratings on a six-point scale increased from 1.87 preintervention to 3.53 postintervention (P < .0001)., Conclusions: Simulated pages engage medical students and may foster medical student preparedness for internship. Students valued the opportunity to simulate "on call" responsibilities, and exposure to simulated pages significantly increased their confidence levels. Further studies are needed to determine effects on patient safety outcomes.

Background: This project sought to study the effectiveness of a curriculum to enhance the intraoperative clinical judgment and procedural skill of surgical residents., Method: A multiinstitutional, prospective, randomized study was performed. A cognitive task analysis of laparoscopic cholecystectomy (LC) was conducted on which instructional activities and measurement instruments were designed. Residents were randomly assigned to a control or intervention group. Subjects took written pre- and posttests examining procedure-related judgment and knowledge. The intervention group participated in a three-session curriculum emphasizing LC critical decisions and error prevention. All subjects were evaluated performing the procedure on a cadaveric model. Scores from written and practical exams were compared using independent-sample and paired Student t tests., Results: Written examination scores increased for both groups. The intervention group scored significantly higher (P < .05) on the written posttest than the control group. There were no differences between groups on the practical examination. Reliability coefficients for the written examination ranged from .65 to .75. Reliability coefficients for the oral exam, technical skill, and error items on the porcine practical exam were .83, .90, and .53., Conclusions: The curriculum resulted in enhanced performance on a written exam designed to assess intraoperative judgment, but no differences in technical skills, showing important implications for future skills lab curriculum formats.

Background: Skill acquisition and maintenance requires spaced deliberate practice. Assessing medical students' physical examination performance ability is resource intensive. The authors assessed the nature and size of physical examination performance samples necessary to accurately estimate total physical examination skill., Method: Physical examination assessment data were analyzed from second year students at the University of Illinois College of Medicine at Chicago in 2002, 2003, and 2004 (N = 548). Scores on subgroups of physical exam maneuvers were compared with scores on the total physical exam, to identify sound predictors of total test performance., Results: Five exam subcomponents were sufficiently correlated to overall test performance and provided adequate sensitivity and specificity to serve as a means to prompt continued student review and rehearsal of physical examination technical skills., Conclusions: Selection and administration of samples of the total physical exam provide a resource-saving approach for promoting and estimating overall physical examination skills retention.

Purpose: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments., Method: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments., Results: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%)., Conclusions: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

The authors offer a practical guide for improving the appraisal of a resident's performance. They identify six major factors that compromise the process of observing, measuring, and characterizing a resident's current performance, forecasting future performance, and making decisions about the resident's progress. Factors that compromise any of these steps lead to individual and collective uncertainty and decrease faculty confidence when making decisions on a resident's progress. The six factors, addressed in order of importance, are inaccuracies due to (1) incomplete sampling of performance, (2) rater memory constraints, (3) hidden performance deficits of the resident, (4) lack of performance benchmarks, (5) faculty members' hesitancy to act on negative performance information, and (6) systematic rater error. The description of each factor is followed by a number of specific suggestions on what residency programs can do to eliminate or minimize the impact of these factors. While this article is couched in the context of the performance evaluation of residents, everything included pertains to measuring and appraising medical students' and practicing physicians' clinical performance as well.

Purpose: To determine whether attending physicians' post-rotation performance ratings and written comments detect surgery residents' clinical performance deficits., Method: Residents' performance records from 1997-2002 in the Department of Surgery, Southern Illinois University School of Medicine, were reviewed to determine the percentage of times end-of-rotation performance ratings and/or comments detected deficiencies leading to negative end-of-year progress decisions., Results: Thirteen of 1,986 individual post-rotation ratings (0.7%) nominally noted a deficit. Post-rotation ratings of "good" or below were predictive of negative end-of-year progress decisions. Eighteen percent of residents determined to have some deficiency requiring remediation received no post-rotation performance ratings indicating that deficiency. Written comments on post-rotation evaluation forms detected deficits more accurately than did numeric ratings. Physicians detected technical skills performance deficits more frequently than applied knowledge and professional behavior deficits. More physicians' post-rotation numeric ratings contradicted performance deficits than supported them. More written comments supported deficits than contradicted them in the technical skills area. In the applied knowledge and professional behavior areas, more written comments contradicted deficits than supported them., Conclusions: A large percentage of performance deficiencies only became apparent when the attending physicians discussed performance at the annual evaluation meetings. Annual evaluation meetings may (1) make patterns of residents' behavior apparent that were not previously apparent to individual physicians, (2) provide evidence that strengthens the individual attending's preexisting convictions about residents' performance deficiencies, or (3) lead to erroneous conclusions. The authors believe deficiencies were real and that their findings can be explained by a combination of reasons one and two.

Purpose: To describe an academic performance incentive system (APIS) and faculty perception of it; explore the impacts of incentive level, faculty rank, clinical practice volume, and administrative responsibility on academic productivity; and describe the APIS's use in maintaining congruence between department mission and activities., Method: A list of teaching, research, and academic service activities was developed, which full-time faculty (n = 33) used to report activities. Clinical faculty members received incentive income based on credits earned. APIS initially distributed 1% of practice plan receipts (subsequently increased to 3% and then 5%). Productivity was measured by differences in APIS points achieved. Satisfaction of all faculty participants was measured by survey., Results: Faculty members (n = 20) who participated throughout averaged 22 credits per month (nine to 42 credits), and quarterly incentive bonuses ranged from 145 US dollars to 6,128 US dollars. Average credits earned per month were 24 for the 1% incentive, 23 for the 3% incentive, and 20 for the 5% incentive. Faculty members with administrative responsibilities were as productive academically as were their non-administrative counterparts. Senior faculty members were as productive as junior faculty. Faculty members who were more productive clinically were more productive academically. Seventy percent of respondents reported they were either very satisfied or somewhat satisfied with the APIS. Seventy-eight percent felt that the APIS accurately reflected their academic productivity. Most respondents (81%) felt that the amount of money allocated to the incentive system was appropriate (15% felt it should be increased and one respondent recommended reduction)., Conclusions: The APIS system has been well accepted by faculty and allows for data-driven discussion of the department's mission and activities.

In July 2001 the Accreditation Council for Graduate Medical Education (ACGME) charged U.S. residency training programs to implement a curriculum and evaluation plan covering six competencies. The authors describe the curriculum and evaluation strategy of the first surgical training program developed to meet the competencies, and list each competency and the teaching method and measurement instruments used. Implementation began July 1, 2001, and the program was fully operational on July 1, 2002. Meeting the curriculum challenges required modification of the existing curriculum and the addition of new instructional units. Nine additional evaluation instruments were needed. The largest investment was in planning and implementation, a one-time development cost. Staff workload increased by 252 hours; this is expected to be a continuing annual requirement. Faculty workload increased by two hours per resident and each resident's workload increased by 112 hours per year (2.3 hours per week). The transition was smoother than expected. Faculty and residents' buy-in was crucial. Faculty and residents were alerted to upcoming changes at the beginning of the year in a grand rounds presentation on the ACGME competencies and the approach to meeting requirements. Updates were presented periodically. The authors recommend that residency programs engaged in similar efforts make effective use of instruments developed elsewhere and collaborate with other programs rather than develop everything locally. The program's benefits include time savings and the availability of validity data and norms to inform decision making on residents' and program progress.

Purpose: To describe the Patient Findings Questionnaire (PFQ) and compare its scores and pass/fail decisions with those obtained from standardized patient (SP) examination checklists., Method: Checklists and PFQs were used to assess data acquisition by 790 second-year medical students. PFQs were composed of multiple-choice items designed to determine whether examines had acquired key historical patient information., Results: At the item level, the two measurement methods yielded the same decisions about data acquisition on 88% of observed occasions. Most discrepancies (74%) involved SPs rating examinees as having elicited information when the examinee was unable to answer the associated PFQ item. At the test level, the two instruments yielded the same pass/fail decision on a large majority of occasions., Conclusions: The PFQ and checklist yielded similar data acquisition scores and decisions at the item and test levels. Replacement of the checklist with the PFQ should result in examinees' behaving in a way more consistent with recommended interviewing practices.

Purpose: To determine whether medical students' interpersonal and communication skills, as measured by satisfaction ratings given by standardized patients (SPs), improves or deteriorates with additional training and experience., Method: A longitudinal study was made of a single cohort of 133 medical students scheduled to graduate in 1994 from the University of Illinois College of Medicine at Chicago. Clinical performance examinations employing clinical encounters with SPs were given at the end of the students' second year at the end of the first quarter of their fourth year. The second-year examination consisted of four student-SP encounters for each student; the fourth-year examination consisted of 16 such encounters. For each examination, SP perceptions of interpersonal and communication skills were measured using six items from the American Board of Internal Medicine Patient Satisfaction Questionnaire; the items employed five ordinal-scale descriptors (from 1 = poor to 5 = excellent). The unit of analysis was the student-SP encounter. For each year, ratings on each item were averaged across encounters to produce a mean rating for each item; in addition, ratings across all the encounters and items were averaged to produce a mean total patient satisfaction rating. The percentage of fail or poor ratings given for each item was calculated, as was the average of these percentages across the items. All rating calculations were done between years and between genders. Two-tailed, paired t-tests and F-ratios were used for comparisons between years and between genders, respectively. Finally, ratings for a comparison group of first-, second-, and third-year internal medicine residents were gathered under similar testing conditions., Results: Mean total patient satisfaction with the students' clinical performances improved from the first examination (3.51, SD, .84) to the second (3.60, SD, .86). The percentage of student encounters receiving fair or poor ratings decreased from 14.5% in the second year to 12% in the fourth year; resident encounters received a fair or poor rating 7.8% of the time. The female students received fewer fair or poor ratings than did the male students both in their second year (11.7% vs 16%) and in their fourth year (9.3% vs 12.8%). Of the six patient-satisfaction items, both the students and the residents consistently were ranked lowest in encouraging patients to ask questions and answering patient questions., Conclusion: Contrary to the stated beliefs of some physician-educators that students' skills deteriorate during the course of training, these medical students' interpersonal and communication skills improved with additional training and experience.

Purpose: To conduct the first of a series of pilot projects of the clinical competence assessment (CCA) of the Educational Commission for Foreign Medical Graduates (ECFMG) in order to provide profiles of clinical competencies of graduates of foreign medical schools for residency directors in the United States and for governments and institutions in other countries., Method and Results: In September 1992 the first pilot project of the ECFMG CCA was conducted for a program director who wanted to evaluate ten first-year residents in a midwestern U.S. program. The CCA consists of integrated clinical encounters with ten standardized patients, 60 laser videodisc pictorials, and analysis of test items of previously completed ECFMG certification examinations. Profiles of the following clinical competencies were provided to the program director: data gathering (history and physical examination), interviewing and interpersonal skills, diagnosis and management skills, interpretation of diagnostic and laboratory procedures, written communication of information to the health care team, and spoken-English proficiency. The profiles were provided as individual scores compared with mean scores of a reference group of 525 first-year residents who took the CCA at four U.S. assessment centers, and as percentile scores with a range of one standard error of measurement., Conclusion: The individual performance data in this first pilot project were valuable to the program director, who used them to supplement scores on a written examination during the first residency year. The pilot project has shown the ECFMG CCA to be a useful tool for program directors to evaluate applicants and residents who are graduates of foreign medical schools.

Criteria are presented for determining whether licensure and/or achievement test results should be used as a basis for making various types of comparisons and judgments about medical education programs for program evaluation purposes. United States Medical Licensing Examination (USMLE) examinations and National Board of Medical Examiners (NBME) subject tests are then considered as sources of evaluation data for medical education programs by evaluating these examinations in the light of the proposed criteria. It is concluded that NBME examination and USMLE results may properly be considered one of several tools to use in an evaluation of medical education programs but are not recommended as the sole basis for a judgment of program effectiveness. However, care must be taken to ensure that test score differences are large enough to be real and to have practical significance. Furthermore, changes in curriculum should not be considered or implemented on the basis of NBME/USMLE results unless the changes in test scores are maintained for at least two years. The focus should generally be on inordinately high failure rates rather than on changes in mean class scores. Efforts to develop objective, standardized, and controlled measures of other clinical competencies for national use should be encouraged to minimize the temptation to use USMLE Step 1 and Step 2 scores as the sole basis for comparisons of medical education programs.

Eighteen questions are posed that the authors believe address the major technical issues involved in the application of standardized patients (SPs). Following each question, selected empirical evidence and commentary are provided in response to the question and as background for further consideration of the issues. The authors conclude that the evidence from this large and systematic knowledge base about SP examinations is encouraging and should be cause for optimism. Moreover, it is noted that far less is known about the measurement properties of conventional procedures for clinical assessment in medical school courses and clerkships. The collective evidence from this extensive SP research base suggests that medical educators should move ahead to realize the benefits of the high-fidelity standardized approach provided by standardized-patient examinations.

Performance of senior medical students on an objectively scored examination of clinical competence based on standardized-patient cases was used to assess the predictive validity of the two most commonly used admission measures, the Medical College Admissions Test and the undergraduate grade point average. The students were in the classes of 1986 and 1987 at Southern Illinois University School of Medicine. The correlations of the admissions measures with clinical performances were quite weak, and none of the admissions measures consistently showed a clear advantage as a predictor of clinical performance. Correlations of the admissions measures with scores on National Board of Medical Examiners (NBME) Part I and Part II examinations were small to moderate, although somewhat larger than the correlations with clinical performances. Correlations were corrected for attenuation due to differential unreliabilities of the clinical examination results and the scores on NBME examinations, and for restriction of range due to the stringent medical school selection process. Corrected correlations were small to moderate and showed the same pattern as the uncorrected ones. The study documents that traditional admissions measures are useful for selecting students who will perform effectively in clinical as well as basic science settings.