Search

Objective: The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees., Summary Background Data: Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved., Methods: The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties., Results: Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items., Conclusions: A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust., Competing Interests: The authors report no conflicts of interest., (Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.)

Purpose: Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method., Method: In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment., Results: Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74)., Conclusions: EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.

Objective: To establish the number of operative performance observations needed for reproducible assessments of operative competency., Background: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown., Methods: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement., Results: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings., Conclusion: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.

Background: We investigated attending surgeon decisions regarding resident operative autonomy, including situations where operative autonomy was discordant with performance quality., Methods: Attending surgeons assessed operative performance and documented operative autonomy granted to residents from 14 general surgery residency programs. Concordance between performance and autonomy was defined as "practice ready performance/meaningfully autonomous" or "not practice ready/not meaningfully autonomous." Discordant circumstances were practice ready/not meaningfully autonomous or not practice ready/meaningfully autonomous. Resident training level, patient-related case complexity, procedure complexity, and procedure commonality were investigated to determine impact on autonomy., Results: A total of 8,798 assessments were collected from 429 unique surgeons assessing 496 unique residents. Practice-ready and exceptional performances were 20 times more likely to be performed under meaningfully autonomous conditions than were other performances. Meaningful autonomy occurred most often with high-volume, easy and common cases, and less complex procedures. Eighty percent of assessments were concordant (38% practice ready/meaningfully autonomous and 42% not practice ready/not meaningfully autonomous). Most discordant assessments (13.8%) were not practice ready/meaningfully autonomous. For fifth-year residents, practice ready/not meaningfully autonomous ratings (9.7%) were more frequent than not practice ready/meaningfully autonomous ratings (7.5%). Ten surgeons (2.3%) failed to afford residents meaningful autonomy on any occasion., Conclusion: Resident operative performance quality is the most important determinant in attending surgeon decisions regarding resident autonomy., (Copyright © 2018 Elsevier Inc. All rights reserved.)

Background: Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements., Methods: Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.S. general surgery resident training programs between September 1, 2015 and December 31, 2016. With 5 of the most commonly performed core general surgery procedures, we correlated the level of autonomy achieved by each resident with the number of procedures they had performed before the evaluation period, with the intent of identifying specific target numbers that would correlate with the achievement of meaningful autonomy for each procedure with most residents., Results: Whereas a definitive target number was identified for laparoscopic appendectomy (i.e. 25), for the other 4 procedures studied (i.e. laparoscopic cholecystectomy, 52; open inguinal hernia repair, 42; ventral hernia repair, 35; and partial colectomy, 60), target numbers identified were less definitive and/or were higher than many residents will experience during their surgical residency training., Conclusions: We conclude that procedural target numbers are generally not effective in predicting procedural competence and should not be used as the basis for determining residents' readiness for independent practice., (Copyright © 2017. Published by Elsevier Inc.)

Background: Educating residents in the operating room requires balancing patient safety, operating room efficiency demands, and resident learning needs. This study explores 4 factors that influence the amount of autonomy supervising surgeons afford to residents., Methods: We evaluated 7,297 operations performed by 487 general surgery residents and evaluated by 424 supervising surgeons from 14 training programs. The primary outcome measure was supervising surgeon autonomy granted to the resident during the operative procedure. Predictor variables included resident performance on that case, supervising surgeon history with granting autonomy, resident training level, and case difficulty., Results: Resident performance was the strongest predictor of autonomy granted. Typical autonomy by supervising surgeon was the second most important predictor. Each additional factor led to a smaller but still significant improvement in ability to predict the supervising surgeon's autonomy decision. The 4 factors together accounted for 54% of decision variance (r = 0.74)., Conclusion: Residents' operative performance in each case was the strongest predictor of how much autonomy was allowed in that case. Typical autonomy granted by the supervising surgeon, the second most important predictor, is unrelated to resident proficiency and warrants efforts to ensure that residents perform each procedure with many different supervisors., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Objective: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy., Background: The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role., Methods: Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation., Results: A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at "Practice Ready" or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%-94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy ("Passive Help" or "Supervision Only") increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence ("Supervision Only") was 33.3%., Conclusions: US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

Objective: To provide recommended practice guidelines for assessing single operative performances and for combining results of operative performance assessments into estimates of overall operative performance ability., Summary Background Data: Operative performance is one defining characteristic of surgeons. Assessment of operative performance is needed to provide feedback with learning benefits to surgical residents in training and to assist in making progress decisions for residents. Operative performance assessment has been a focus of investigation over the past 20 years. This review is designed to integrate findings of this research into a set of recommended operative performance practices., Methods: Literature from surgery and from other pertinent research areas (psychology, education, business) was reviewed looking for evidence to inform practice guideline development. Guidelines were created along with a conceptual and scientific foundation for each guideline., Results: Ten guidelines are provided for assessing individual operative performances and 10 are provided for combing data from individual operative performances into overall judgments of operative performance ability., Conclusions: The practice guidelines organize available information to be immediately useful to program directors, to support surgical training, and to provide a conceptual framework upon which to build as the base of pertinent knowledge expands through future research and development efforts.

Purpose: Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs., Methods: Between September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail., Results: A total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures., Conclusions: SIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed., (Copyright © 2016. Published by Elsevier Inc.)

Background: Global, end-of-rotation evaluations are often difficult to interpret due to their high level of abstraction (eg, excellent, good, poor) and the bias toward high ratings. This study documents the utility of and measurement characteristics of serious problem items, an alternative item format., Methods: This report is based on 4,234 faculty performance ratings for 105 general surgery residents. Faculty members reported whether each resident had a serious problem for each of 8 areas of clinical performance and 6 areas of professional behavior., Results: A total of 263 serious problems were reported. The performance category with the most total serious problems noted was knowledge and that with the least problems noted was relations with patients and family members. Seven residents accounted for 86.9% of all serious problem reports. Each resident had serious problems in multiple performance areas. Problems were reported most frequently in knowledge, management, technical/procedural skills, ability to assume responsibility within level of competence, and problem identification. Citations of these serious problems were most common in year 3. Traditional ratings of global performance were not an adequate means for identifying residents with serious performance problems., Conclusion: Serious problem ratings can communicate faculty concerns about residents more directly and can be used as a complement to conventional global rating scales without substantially increasing faculty workload., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Background: The idea of competency-based education sounds great on paper. Who wouldn't argue for a standardized set of performance-based assessments to assure competency in graduating students and residents? Even so, conceptual concerns have already been raised about this new system and there is yet no evidence to refute their veracity., Aims: We argue that practical concerns deserve equal consideration, and present evidence strongly suggesting these concerns should be taken seriously., Method: Specifically, we share two historical examples that illustrate what happened in two disparate contexts (K-12 education and the Department of Defense [DOD]) when competency (or outcomes-based) assessment frameworks were implemented. We then examine how observation and assessment of clinical performance stands currently in medical schools and residencies, since these methodologies will be challenged to a greater degree by expansive lists of competencies and milestones., Results/conclusions: We conclude with suggestions as to a way forward, because clearly the assessment of competency and the ability to guarantee that graduates are ready for medical careers is of utmost importance. Hopefully the headlong rush to competencies, milestones, and core entrustable professional activities can be tempered before even more time, effort, frustration and resources are invested in an endeavor which history suggests will collapse under its own weight.

Objective: A valid measure of resident operative performance ability requires direct observation and accurate rating of multiple resident performances under the normal range of operating conditions. The challenge is to create an operative performance rating (OPR) system that: is easy to use, encourages completion of many ratings immediately after performances and minimally disrupts supervising surgeons' work days. The purpose of this study was to determine whether a score based on a single-item overall OPR provides a valid and stable appraisal of resident operative performances., Design: A retrospective comparison of a single-item OPR with a gold-standard rating based on multiple procedure-specific and general OPR items., Setting: Data were collected in the general surgery residency program at Southern Illinois University from 2001 through 2012., Participants: Assessments of 1033 operative performances (3 common procedures, 2 laparoscopic, and 1 open) by general surgery residents were collected. OPRs based on single-item overall performance scale scores were compared with gold-standard ratings for the same performances., Results: Differences in performance scores using the 2 scales averaged 0.02 points (5-point scale). Correlations of the single-item and gold-standard scale scores averaged 0.95. Based on generalizability analyses of laparoscopic cholecystectomy ratings, each instrument required 5 observations to achieve reliabilities of 0.80 and 11 observations to achieve reliabilities of 0.90. Only 4.4% of single-item ratings misclassified the performance when compared with the gold-standard rating and all misclassifications were near misses. For 80% of misclassified ratings, single-item ratings were lower., Conclusions: Single-item operative performance measures produced ratings that were virtually identical to gold-standard scale ratings. Misclassifications occurred infrequently and were minor in magnitude. Ratings using the single-item scale: take less time to complete, should increase the sample of procedures rated, and encourage attending surgeons to complete ratings immediately after observing performances. Face-to-face and written comments and suggestions should continue to be used to provide the granular feedback residents need to improve subsequent performances., (Copyright © 2015 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Objectives: Clinical Assessment and Management Examination--Outpatient (CAMEO) is a metric for evaluating the clinical performance of surgery residents. The aim of this study was to investigate the measurement characteristics of CAMEO and propose how it might be used as an evaluation tool within the general surgery milestones project., Design: A total of 117 CAMEO evaluations were gathered and used for analysis. Internal consistency reliability was estimated, and item characteristics were explored. A Kruskal-Wallis procedure was performed to discern how well the instrument discriminated between training levels. An exploratory factor analysis was also conducted to understand the dimensionality of the evaluation., Setting: CAMEO evaluations were collected from 2 departments of surgery geographically located in the Midwestern United States. Combined, the participating academic institutions graduate approximately 18 general surgery residents per year., Participants: In this retrospective data analysis, the number of evaluations per resident ranged from 1 to 7, and evaluations were collected from 2006 to 2013. For the purpose of data analysis, residents were classified as interns (postgraduate year 1 [PGY1]), juniors (PGY2-3), or seniors (PGY4-5)., Results: CAMEO scores were found to have high internal consistency (Cronbach's α = 0.96), and all items were highly correlated (≥ 0.86) to composite CAMEO scores. Scores discriminated between senior residents (PGY4-5) and lower level residents (PGY1-3). Per an exploratory factor analysis, CAMEO was revealed to measure a single dimension of "clinical competence.", Conclusions: The findings of this research aligned with related literature and verified that CAMEO scores have desirable measurement properties, making CAMEO an attractive resource for evaluating the clinical performance of surgery residents., (Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Purpose: Operative performance ratings (OPRs) need adequate clarity and detail to support self-directed learning and valid progress decisions. This study was designed to determine (1) the elapsed time between observing operative performances and completing performance ratings under field conditions and (2) the effect of increased elapsed time on rating clarity and detail., Methods: Overall, 895 OPRs by 19 faculty members for 37 general surgery residents were the focus of this study. The elapsed time between observing the performance and completing the evaluation was recorded. No-delay comparison data included 45 additional ratings of 8 performances collected under controlled conditions immediately following the performance by 17 surgeons whose sole responsibility was to observe and rate the performances. Item-to-item OPR variation and the presence and nature of comments were indicators of evaluation clarity, detail, and quality., Results: Elapsed time between observing and evaluating performances under field conditions were as follows: 1 day or less, 116 performances (13%); 2 to 3 days, 178 performances (20%); 4 to 14 days, 377 performances (42%); and more than 14 days, 224 performances (25%). Overall, 87% of performances rated more than 14 days after observation had no item-to-item ratings variation compared with 62% rated with a delay of 4 to 14 days, 41% rated with a delay of 2 to 3 days, 42% rated within 1 day, and 2% rated immediately. In addition, 70% of ratings completed more than 14 days after observation had no written comments, compared with 49% for those completed with a delay of 4 to 14 days, 45% for those completed in 2 to 3 days, and 46% for those completed within 1 day. Moreover, 47% of comments submitted after more than 14 days were exclusively global comments (less instructionally useful) compared with 7% for those completed with a delay of 4 to 14 days and 5% for those completed in 1 to 3 days., Conclusions: The elapsed time between observation and rating of operative performances should be recorded. Immediate ratings should be encouraged. Ratings completed more than 3 days after observation should be discouraged and discounted, as they lack clarity and detail about the performance., (Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Background: Communication breakdowns and care coordination problems often cause preventable adverse patient care events, which can be especially acute in the trauma setting, in which ad hoc teams have little time for advanced planning. Existing teamwork curricula do not address the particular issues associated with ad hoc emergency teams providing trauma care., Methods: Ad hoc trauma teams completed a preinstruction simulated trauma encounter and were provided with instruction on appropriate team behaviors and team communication. Teams completed a postinstruction simulated trauma encounter immediately afterward and 3 weeks later, then completed a questionnaire. Blinded raters rated videotapes of the simulations., Results: Participants expressed high levels of satisfaction and intent to change practice after the intervention. Participants changed teamwork and communication behavior on the posttest, and changes were sustained after a 3-week interval, though there was some loss of retention., Conclusions: Brief training exercises can change teamwork and communication behaviors on ad hoc trauma teams., (Copyright © 2014 Elsevier Inc. All rights reserved.)

Purpose: Fostering ability to organize and use medical knowledge to guide data collection, make diagnostic decisions, and defend those decisions is at the heart of medical training. However, these abilities are not systematically examined prior to graduation. This study examined diagnostic justification (DXJ) ability of medical students shortly before graduation., Method: All senior medical students in the Classes of 2011 (n = 67) and 2012 (n = 70) at Southern Illinois University were required to take and pass a 14-case, standardized patient examination prior to graduation. For nine cases, students were required to write a free-text response indicating how they used patient data to move from their differential to their final diagnosis. Two physicians graded each DXJ response. DXJ scores were compared with traditional standardized patient examination (SCCX) scores., Results: The average intraclass correlation between raters' rankings of DXJ responses was 0.75 and 0.64 for the Classes of 2011 and 2012, respectively. Student DXJ scores were consistent across the nine cases. Using SCCX and DXJ scores led to the same pass-fail decision in a majority of cases. However, there were many cases where discrepancies occurred. In a majority of those cases, students would fail using the DXJ score but pass using the SCCX score. Common DXJ errors are described., Conclusions: Commonly used standardized patient examination component scores (history/physical examination checklist score, findings, differential diagnosis, diagnosis) are not direct, comprehensive measures of DXJ ability. Critical deficiencies in DXJ abilities may thus go undiscovered.

Objective: This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment., Background: Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training., Methods: Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation., Results: A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances., Conclusions: OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.

Background: This study explored the amount of guidance provided to residents in the operating room (OR) and the relationship of OR guidance with postgraduate year (PGY) and operative performance rating (OPR)., Methods: We used OPR instruments to collect data from supervising surgeons after each performance. External expert raters blindly rated the amount of guidance for 5 videotaped performances., Results: Three hundred sixty-eight performances were analyzed for 5 procedures performed by 26 residents with 16 supervising surgeons over 6 months. Guidance ratings varied with procedure, individual supervising surgeons varied in the amount of guidance reported, the amount of guidance decreased as residents' PGY level increased, and the correlation between guidance rating and overall performance was .62. In comparison cases, most supervising surgeons underestimated the amount of guidance provided., Conclusions: Controlling for the amount of supervising surgeon guidance has important implications for training and evaluation as we strive to prepare residents to practice independently., (Copyright © 2012 Elsevier Inc. All rights reserved.)

Purpose: Little is known about the acquisition of clinical reasoning skills in medical school, the development of clinical reasoning over the medical curriculum as a whole, and the impact of various curricular methodologies on these skills. This study investigated (1) whether there are differences in clinical reasoning skills between learners at different years of medical school, and (2) whether there are differences in performance between students at schools with various curricular methodologies., Method: Students (n = 2,394) who had completed zero to three years of medical school at five U.S. medical schools participated in a cross-sectional study in 2008. Students took the same diagnostic pattern recognition (DPR) and clinical data interpretation (CDI) tests. Percent correct scores were used to determine performance differences. Data from all schools and students at all levels were aggregated for further analysis., Results: Student performance increased substantially as a result of each year of training. Gains in DPR and CDI performance during the third year of medical school were not as great as in previous years across the five schools. CDI performance and performance gains were lower than DPR performance and gains. Performance gains attributable to training at each of the participating medical schools were more similar than different., Conclusions: Years of training accounted for most of the variation in DPR and CDI performance. As a rule, students at higher training levels performed better on both tests, though the expected larger gains during the third year of medical school did not materialize.

Background: Current remediation strategies for students failing standardized patient examinations represent poorly targeted approaches since the specific nature of clinical performance weaknesses has not been defined., Purpose: The purpose is to determine the impact of a specifically targeted clinical performance course required of students who failed a clinical performance examination., Methods: A month-long clinical performance course, targeted to treat specific types of clinical performance deficiencies, was designed to remediate students failing standardized patient examinations in 2007 (n=8) and 2008 (n=5). Participating students were assessed on pre- and postperformance measures, including multiple-choice tests that measured diagnostic pattern recognition and clinical data interpretation and clinical performance measures using standardized clinical encounters. Comparisons between average pre- and postintervention performance scores were computed using paired sample t tests. Results were adjusted for regression toward the mean., Results: In both 2007 and 2008, the mean preintervention clinical data interpretation and standardized patient examination scores were below the criterion referenced passing standard set for the clinical competency exam. In both years the mean postintervention scores for the participants were above the passing standard for these two examinations. Pre- and postintervention differences were statistically significant in both cases., Conclusions: This study provides insight into the reasons that students fail clinical performance examinations and elucidates one method by which such students may be successfully remediated.

Purpose: During the transition from medical school to internship, trainees experience high levels of stress related to pages on the inpatient wards. The steep learning curve during this period may also affect patient safety. The authors piloted the use of simulated pages to improve medical student preparedness, decrease stress related to pages, and familiarize medical students with common patient problems., Method: A multidisciplinary team at Southern Illinois University School of Medicine developed simulated pages that were tested among senior medical students. Sixteen medical students were presented with 11 common patient scenarios. Data on assessment, management, and global performance were collected. Mean confidence levels were evaluated pre- and postintervention. Students were also surveyed on how the simulated pages program influenced their perceived comfort in managing patient care needs and the usefulness of the exercise in preparing them to handle inpatient pages., Results: Mean scores on the assessment and management portions of the scenarios varied widely depending on the scenario (range -15.6 ± 41.6 to 95.7 ± 9.5). Pass rates based on global performance ranged from 12% to 93%. Interrater agreement was high (mean kappa = 0.88). Students' confidence ratings on a six-point scale increased from 1.87 preintervention to 3.53 postintervention (P < .0001)., Conclusions: Simulated pages engage medical students and may foster medical student preparedness for internship. Students valued the opportunity to simulate "on call" responsibilities, and exposure to simulated pages significantly increased their confidence levels. Further studies are needed to determine effects on patient safety outcomes.

Background: The aim of this study was to compare the laboratory teaching of a basic technical skill by a nonphysician skills coach and a faculty surgeon., Methods: Medical students were randomized to instruction of skin suturing in the skills laboratory by a faculty surgeon or by a nonphysician skills coach. Testing of performance occurred at 3 time points. Other faculty surgeons, blinded to identities and training groups, rated performance., Results: Forty-nine students participated. Baseline fourth-year student mean scores showed no significant difference between training groups. Third-year and fourth-year student performance showed no difference between training groups on postintervention testing. Delayed testing also showed no difference in third-year student scores., Conclusions: Training by either a nonsurgeon skills coach or a faculty surgeon resulted in no difference in performance on a basic surgical skill. This was true for students with and without prior experience and was also true after subsequent clinical experiences. Nonphysician coaches may ease the teaching burden of surgical faculty members while providing similar quality of instruction for trainees.

Purpose: An accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues the examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) by studying additional factors that may influence reliability, accuracy, and interpretability of results., Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2001 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) the patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance., Results: In all, 566 evaluations were analyzed, which consisted of open colectomy (n = 125), open inguinal hernia (n = 103), laparoscopic cholecystectomy (n = 199), and excisional biopsy (n = 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. The median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment. Most were completed within 4.5 to 22 days. Procedure-specific and global scale scores resulted in similar rank-ordering of performances (single-measure intraclass correlation using the consistency model = 0.88; 95% confidence interval [CI] = 0.87-0.90) and similar absolute OPRS scores (single-measure intraclass correlation using the consistency model = 0.89; 95% CI, 0.87-0.90). Evaluating surgeons differed in stringency of ratings across procedures (average difference = 1.4 points of 5 possible points). Resident performance improved with increasing PGY level for all 4 procedures., Conclusions: Substantial time elapses between performance in the operating room and the completion of the evaluation. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performance observations by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial.

Background: This project sought to study the effectiveness of a curriculum to enhance the intraoperative clinical judgment and procedural skill of surgical residents., Method: A multiinstitutional, prospective, randomized study was performed. A cognitive task analysis of laparoscopic cholecystectomy (LC) was conducted on which instructional activities and measurement instruments were designed. Residents were randomly assigned to a control or intervention group. Subjects took written pre- and posttests examining procedure-related judgment and knowledge. The intervention group participated in a three-session curriculum emphasizing LC critical decisions and error prevention. All subjects were evaluated performing the procedure on a cadaveric model. Scores from written and practical exams were compared using independent-sample and paired Student t tests., Results: Written examination scores increased for both groups. The intervention group scored significantly higher (P < .05) on the written posttest than the control group. There were no differences between groups on the practical examination. Reliability coefficients for the written examination ranged from .65 to .75. Reliability coefficients for the oral exam, technical skill, and error items on the porcine practical exam were .83, .90, and .53., Conclusions: The curriculum resulted in enhanced performance on a written exam designed to assess intraoperative judgment, but no differences in technical skills, showing important implications for future skills lab curriculum formats.

Background: Working knowledge of physicians manifests as a combination of diagnostic pattern recognition and clinical data interpretation (analytic fact checking)., Purpose: The purpose was to study medical student acquisition of these abilities as a function of years of medical training/experience., Methods: A cross-sectional study involving students who had completed 0, 1, 2, and 3 years of medical school. All students at all levels of training took the same tests of diagnostic pattern recognition and clinical data interpretation. Percent correct scores were calculated and used to estimate learning curves. A cohort of family physicians also took the test to provide a benchmark., Results: Student diagnostic pattern recognition and clinical data interpretation ability demonstrated a steady upward growth curve but leveled off in Year 3. Diagnostic pattern recognition performance was consistently higher than clinical data interpretation performance. The rate of diagnostic performance gain with training and experience was also higher., Conclusions: Medical students acquired diagnostic pattern recognition ability and all years of medical training contributed. The rate of clinical data interpretation performance improvement was slower, and the absolute performance level was lower. What was surprising was the lower rate of improvement in diagnostic pattern recognition and clinical data interpretation performance for students during their 1st year of clinical training. Students' understanding of findings and their relationships to disease processes may be affected by their limited patient experience.

Background: Skill acquisition and maintenance requires spaced deliberate practice. Assessing medical students' physical examination performance ability is resource intensive. The authors assessed the nature and size of physical examination performance samples necessary to accurately estimate total physical examination skill., Method: Physical examination assessment data were analyzed from second year students at the University of Illinois College of Medicine at Chicago in 2002, 2003, and 2004 (N = 548). Scores on subgroups of physical exam maneuvers were compared with scores on the total physical exam, to identify sound predictors of total test performance., Results: Five exam subcomponents were sufficiently correlated to overall test performance and provided adequate sensitivity and specificity to serve as a means to prompt continued student review and rehearsal of physical examination technical skills., Conclusions: Selection and administration of samples of the total physical exam provide a resource-saving approach for promoting and estimating overall physical examination skills retention.

Background: Curricula for surgical technical skills laboratories have traditionally been designed to accommodate the clinical activities of residents, so they typically consist of individual, episodic training sessions. We believe that the skills laboratory offers an opportunity to design a surgical skills curriculum based on the fundamental elements known to be important for motor skill instruction. We hypothesized that training novices with such a curriculum for a 1-month period would yield skills performance levels equivalent to those of second year surgery residents who had trained in a traditional program., Study Design: Fourth-year medical students served as study subjects (novice group) during a 4-week senior elective. They were taught each skill during a 1-week period. Subjects received instruction by a content expert followed by a 1-week period of deliberate practice with feedback. The novice performances were videotaped both before and after the intervention, and each videotape was evaluated in a blinded fashion by experts using a validated evaluation instrument. These results were compared with skill performance ratings of first- and second-year surgery residents that had been accumulated over the previous 3 years., Results: Average performance ratings for the novices substantially improved for all four skills after training. There was no marked difference between average performance ratings of postintervention novice scores when compared with the average scores in the resident group. Inter-rater agreement in scoring for the videotaped novice performances exceeded 0.87 (intraclass correlation) for all ratings of pre- and posttraining., Conclusions: These results demonstrate the effectiveness of a laboratory-based training program that includes fundamentals of motor skills acquisition.

Background: Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format., Methods: Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations., Results: Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable., Conclusions: Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.

Purpose: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments., Method: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments., Results: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%)., Conclusions: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

The authors offer a practical guide for improving the appraisal of a resident's performance. They identify six major factors that compromise the process of observing, measuring, and characterizing a resident's current performance, forecasting future performance, and making decisions about the resident's progress. Factors that compromise any of these steps lead to individual and collective uncertainty and decrease faculty confidence when making decisions on a resident's progress. The six factors, addressed in order of importance, are inaccuracies due to (1) incomplete sampling of performance, (2) rater memory constraints, (3) hidden performance deficits of the resident, (4) lack of performance benchmarks, (5) faculty members' hesitancy to act on negative performance information, and (6) systematic rater error. The description of each factor is followed by a number of specific suggestions on what residency programs can do to eliminate or minimize the impact of these factors. While this article is couched in the context of the performance evaluation of residents, everything included pertains to measuring and appraising medical students' and practicing physicians' clinical performance as well.

Background: This study investigated the concurrent and predictive validity of end-of-rotation (EOR) clinical performance ratings., Study Design: Surgeon EOR ratings of residents were collected and compared with end-of-year (EOY) progress decisions and to EOR and EOY confidential judgments of resident ability to provide patient care without direct supervision., Results: Eighty percent to 85% of EOR ratings were Excellent or Very Good. Five percent or fewer were Fair or Poor. Almost all residents receiving Excellent or Very Good EOR ratings also received positive EOR judgments about ability to provide patient care without direct supervision. Residents rated Fair or Poor received negative EOR judgments about ability to provide patient care without direct supervision. As the cumulative percent of Good, Fair, and Poor EOR ratings increased, the number of residents promoted without stipulations at the end of the year decreased and the percentage of faculty members who judged the residents capable of providing effective patient care without direct supervision at the end of the year declined. All residents receiving 40% or more EOR ratings below Very Good had stipulations associated with their promotion., Conclusions: Despite use of descriptive anchors on the scale, clinical performance ratings have no direct meaning. Their meaning needs to be established in the same manner as is done in setting normal values for diagnostic tests, ie, by establishing the relationship between EOR ratings and practice outcomes.

Purpose: To determine whether attending physicians' post-rotation performance ratings and written comments detect surgery residents' clinical performance deficits., Method: Residents' performance records from 1997-2002 in the Department of Surgery, Southern Illinois University School of Medicine, were reviewed to determine the percentage of times end-of-rotation performance ratings and/or comments detected deficiencies leading to negative end-of-year progress decisions., Results: Thirteen of 1,986 individual post-rotation ratings (0.7%) nominally noted a deficit. Post-rotation ratings of "good" or below were predictive of negative end-of-year progress decisions. Eighteen percent of residents determined to have some deficiency requiring remediation received no post-rotation performance ratings indicating that deficiency. Written comments on post-rotation evaluation forms detected deficits more accurately than did numeric ratings. Physicians detected technical skills performance deficits more frequently than applied knowledge and professional behavior deficits. More physicians' post-rotation numeric ratings contradicted performance deficits than supported them. More written comments supported deficits than contradicted them in the technical skills area. In the applied knowledge and professional behavior areas, more written comments contradicted deficits than supported them., Conclusions: A large percentage of performance deficiencies only became apparent when the attending physicians discussed performance at the annual evaluation meetings. Annual evaluation meetings may (1) make patterns of residents' behavior apparent that were not previously apparent to individual physicians, (2) provide evidence that strengthens the individual attending's preexisting convictions about residents' performance deficiencies, or (3) lead to erroneous conclusions. The authors believe deficiencies were real and that their findings can be explained by a combination of reasons one and two.

Background: Global ratings based on observing convenience samples of clinical performance form the primary basis for appraising the clinical competence of medical students, residents, and practicing physicians. This review explores cognitive, social, and environmental factors that contribute unwanted sources of score variation (bias) to clinical performance evaluations., Summary: Raters have a 1 or 2-dimensional conception of clinical performance and do not recall details. Good news is reported more quickly and fully than bad news, leading to overly generous performance evaluations. Training has little impact on accuracy and reproducibility of clinical performance ratings., Conclusions: Clinical performance evaluation systems should assure broad, systematic sampling of clinical situations; keep rating instruments short; encourage immediate feedback for teaching and learning purposes; encourage maintenance of written performance notes to support delayed clinical performance ratings; give raters feedback about their ratings; supplement formal with unobtrusive observation; make promotion decisions via group review; supplement traditional observation with other clinical skills measures (e.g., Objective Structured Clinical Examination); encourage rating of specific performances rather than global ratings; and establish the meaning of ratings in the manner used to set normal limits for clinical diagnostic investigations.

The purpose of this study was to establish construct validation of a flexible sigmoidoscopy simulator by comparing training-level grouped subjects. These included clerical staff (n = 10), residents (n = 19), and experts (n = 5). Each participant performed 3 scopes. The ANOVA group-based results for trainer-measured variables are shown in Table 1. These results demonstrate that the flexible sigmoidoscopy simulator distinguished the trained from the untrained and the resident from the expert. Although there was no statistically significant differences between the senior residents and the experts, the expert commonly outperformed the residents. Establishing the transferability of simulator training to real life is next. If the transfer of skill can be established, it may give rise to a new skills training approach.

Background: Simulation-based training provides minimal feedback and relies heavily on self-assessment. Research has shown medical trainees are poor self-assessors. The purpose of this study was to examine trainees' ability to self-assess technical skills using a simulation-trainer., Methods: Twenty-one medical students performed 10 repetitions of a simulated task. After each repetition they estimated their time and errors made. These were compared with the simulator data., Results: Task time (P < 0.0001) and errors made (P < 0.0001) improved with repetition. Both self-assessment curves reflected their actual performance curves (P < 0.0001). Self-assessment of time did not improve in accuracy (P = 0.26) but error estimation did (P = 0.01) when compared with actual performance., Conclusions: Novices demonstrated improved skill acquisition using simulation. Their estimates of performance and accuracy of error estimation improved with repetition. Clearly, practice enhances technical skill self-assessment. These results support the notion of self-directed skills training and could have significant implications for residency training programs.

In July 2001 the Accreditation Council for Graduate Medical Education (ACGME) charged U.S. residency training programs to implement a curriculum and evaluation plan covering six competencies. The authors describe the curriculum and evaluation strategy of the first surgical training program developed to meet the competencies, and list each competency and the teaching method and measurement instruments used. Implementation began July 1, 2001, and the program was fully operational on July 1, 2002. Meeting the curriculum challenges required modification of the existing curriculum and the addition of new instructional units. Nine additional evaluation instruments were needed. The largest investment was in planning and implementation, a one-time development cost. Staff workload increased by 252 hours; this is expected to be a continuing annual requirement. Faculty workload increased by two hours per resident and each resident's workload increased by 112 hours per year (2.3 hours per week). The transition was smoother than expected. Faculty and residents' buy-in was crucial. Faculty and residents were alerted to upcoming changes at the beginning of the year in a grand rounds presentation on the ACGME competencies and the approach to meeting requirements. Updates were presented periodically. The authors recommend that residency programs engaged in similar efforts make effective use of instruments developed elsewhere and collaborate with other programs rather than develop everything locally. The program's benefits include time savings and the availability of validity data and norms to inform decision making on residents' and program progress.

Purpose: To conduct the first of a series of pilot projects of the clinical competence assessment (CCA) of the Educational Commission for Foreign Medical Graduates (ECFMG) in order to provide profiles of clinical competencies of graduates of foreign medical schools for residency directors in the United States and for governments and institutions in other countries., Method and Results: In September 1992 the first pilot project of the ECFMG CCA was conducted for a program director who wanted to evaluate ten first-year residents in a midwestern U.S. program. The CCA consists of integrated clinical encounters with ten standardized patients, 60 laser videodisc pictorials, and analysis of test items of previously completed ECFMG certification examinations. Profiles of the following clinical competencies were provided to the program director: data gathering (history and physical examination), interviewing and interpersonal skills, diagnosis and management skills, interpretation of diagnostic and laboratory procedures, written communication of information to the health care team, and spoken-English proficiency. The profiles were provided as individual scores compared with mean scores of a reference group of 525 first-year residents who took the CCA at four U.S. assessment centers, and as percentile scores with a range of one standard error of measurement., Conclusion: The individual performance data in this first pilot project were valuable to the program director, who used them to supplement scores on a written examination during the first residency year. The pilot project has shown the ECFMG CCA to be a useful tool for program directors to evaluate applicants and residents who are graduates of foreign medical schools.

Objective: To develop an assessment of clinical competence of graduates of foreign medical schools and to determine the reliability and validity of the assessment and the feasibility of large-scale administration., Design: The Educational Commission for Foreign Medical Graduates (ECFMG) clinical competence study included (1) clinical encounters with standardized patients to assess history taking, physical examination, and communication skills; (2) laser videodisk pictorials to assess identification and interpretation of diagnostic procedures; (3) written clinical vignettes to assess diagnosis and management skills; and (4) assessment of spoken English. A uniform method of operating the test centers and of training the standardized patients was developed., Setting: Medical schools and their primary teaching hospitals and affiliated hospitals., Participants: Six hundred twenty-four first-year residents, of whom 525 are graduates of foreign medical schools., Main Outcome Measures: Scores, reliability coefficients, validity measures, feasibility of multisite administration, trends of scores over time, and acceptability by examinees., Results: The ECFMG clinical competence assessment was conducted at four geographically separate test centers. Reliability coefficients were high (.85) for the integrated clinical encounter and were in a reasonable range (.71 to .82) for all test components. The assessment adds to the predictability of the residents' performance in the hospital over that of current ECFMG certification examinations. Test security was addressed by demonstrating no consistent pattern of change in scores over testing dates. Virtually all examinees thought the assessment was appropriate. Standardized patients were able to assess spoken English accurately., Conclusion: The feasibility of conducting a reliable and valid test of clinical competence for graduates of foreign medical schools was demonstrated for this test population.

Doctor ratings of clerkship performance are often discounted as not accurately reflecting clinical competence. Such ratings are influenced by the following uncontrolled variables: case difficulty; differing rater focus and standards; lack of agreement on what constitutes acceptable performance; and collective patient care responsibility masks individual contributions. Standardized direct measures of clinical competence were developed to control these factors and allow direct comparisons of student performance. Students saw 18 patients representing frequently occurring and important patient problems. Student actions and decisions were recorded and subsequent responses to questions revealed knowledge of pathophysiology, basis for actions, use and interpretation of laboratory investigations, and management. Actions and responses were graded using a pre-set key. The examination covered 73% of designated clinical competencies. Examinations scores corresponded with independent measures of clinical competence. Reliability studies indicated that new cases can be substituted in subsequent years with confidence that scores will maintain similar meaning. Costs are +6.95 per student per case, which is modest considering the quality and quantity of information acquired. Methods described are practical for evaluation of clerks and residents and for licensing and specialty certification examinations.

Written examinations are widely used for assessment in clinical clerkships and for licensure and specialty board certification, as opposed to assessment based on actual performance with patients. This reliance on written examinations is due to their ease of use and perceived objectivity and occurs despite the fact that the examinations assess few components of clinical competence. Simulated patients can standardize the presentation of a patient problem; and, if the patients are employed in an assessment in a manner parallel to the design of written test items, the assessment can have an objectivity similar to that enjoyed by written tests. Such an assessment allows the major components of clinical competence to be tested. The results and feasibility of using simulated patients in a multiple-station assessment of an entire senior class in January 1986 are described. A second assessment was administered to a different senior class in December 1986. This latter assessment was designed in collaboration with another medical school that administered the same assessment to its senior students in March 1987.

Performance of senior medical students on an objectively scored examination of clinical competence based on standardized-patient cases was used to assess the predictive validity of the two most commonly used admission measures, the Medical College Admissions Test and the undergraduate grade point average. The students were in the classes of 1986 and 1987 at Southern Illinois University School of Medicine. The correlations of the admissions measures with clinical performances were quite weak, and none of the admissions measures consistently showed a clear advantage as a predictor of clinical performance. Correlations of the admissions measures with scores on National Board of Medical Examiners (NBME) Part I and Part II examinations were small to moderate, although somewhat larger than the correlations with clinical performances. Correlations were corrected for attenuation due to differential unreliabilities of the clinical examination results and the scores on NBME examinations, and for restriction of range due to the stringent medical school selection process. Corrected correlations were small to moderate and showed the same pattern as the uncorrected ones. The study documents that traditional admissions measures are useful for selecting students who will perform effectively in clinical as well as basic science settings.