Search

The American College of Rheumatology (ACR) Adult Rheumatology In-Training Examination (ITE) is a feedback tool designed to identify strengths and weaknesses in the content knowledge of individual fellows-in-training and the training program curricula. We determined whether scores on the ACR ITE, as well as scores on other major standardized medical examinations and competency-based ratings, could be used to predict performance on the American Board of Internal Medicine (ABIM) Rheumatology Certification Examination.Between 2008 and 2012, 629 second-year fellows took the ACR ITE. Bivariate correlation analyses of assessment scores and multiple linear regression analyses were used to determine whether ABIM Rheumatology Certification Examination scores could be predicted on the basis of ACR ITE scores, United States Medical Licensing Examination scores, ABIM Internal Medicine Certification Examination scores, fellowship directors' ratings of overall clinical competency, and demographic variables. Logistic regression was used to evaluate whether these assessments were predictive of a passing outcome on the Rheumatology Certification Examination.In the initial linear model, the strongest predictors of the Rheumatology Certification Examination score were the second-year fellows' ACR ITE scores (β = 0.438) and ABIM Internal Medicine Certification Examination scores (β = 0.273). Using a stepwise model, the strongest predictors of higher scores on the Rheumatology Certification Examination were second-year fellows' ACR ITE scores (β = 0.449) and ABIM Internal Medicine Certification Examination scores (β = 0.276). Based on the findings of logistic regression analysis, ACR ITE performance was predictive of a pass/fail outcome on the Rheumatology Certification Examination (odds ratio 1.016 [95% confidence interval 1.011-1.021]).The predictive value of the ACR ITE score with regard to predicting performance on the Rheumatology Certification Examination supports use of the Adult Rheumatology ITE as a valid feedback tool during fellowship training.

Importance In 1990, the American Board of Internal Medicine (ABIM) ended lifelong certification by initiating a 10-year Maintenance of Certification (MOC) program that first took effect in 2000. Despite the importance of this change, there has been limited research examining associations between the MOC requirement and patient outcomes. Objective To measure associations between the original ABIM MOC requirement and outcomes of care. Design, Setting, and Participants Quasi-experimental comparison between outcomes for Medicare beneficiaries treated in 2001 by 2 groups of ABIM-certified internal medicine physicians (general internists). One group (n = 956), initially certified in 1991, was required to fulfill the MOC program in 2001 (MOC-required) and treated 84 215 beneficiaries in the sample; the other group (n = 974), initially certified in 1989, was grandfathered out of the MOC requirement (MOC-grandfathered) and treated 69 830 similar beneficiaries in the sample. We compared differences in outcomes for the beneficiary cohort treated by the MOC-required general internists before (1999-2000) and after (2002-2005) they were required to complete MOC, using the beneficiary cohort treated by the MOC-grandfathered general internists as the control. Main Outcomes and Measures Quality measures were ambulatory care–sensitive hospitalizations (ACSHs), measured using prevention quality indicators. Ambulatory care–sensitive hospitalizations are hospitalizations triggered by conditions thought to be potentially preventable through better access to and quality of outpatient care. Other outcomes included health care cost measures (adjusted to 2013 dollars). Results Annual incidence of ACSHs (per 1000 beneficiaries) increased from the pre-MOC period (37.9 for MOC-required beneficiaries vs 37.0 for MOC-grandfathered beneficiaries) to the post-MOC period (61.8 for MOC-required beneficiaries vs 61.4 for MOC-grandfathered beneficiaries) for both cohorts, as did annual per-beneficiary health care costs (pre-MOC period, $5157 for MOC-required beneficiaries vs $5133 for MOC-grandfathered beneficiaries; post-MOC period, $7633 for MOC-required beneficiaries vs $7793 for MOC-grandfathered beneficiaries). The MOC requirement was not statistically associated with cohort differences in the growth of the annual ACSH rate (per 1000 beneficiaries, 0.1 [95% CI, −1.7 to 1.9];P = .92), but was associated with a cohort difference in the annual, per-beneficiary cost growth of −$167 (95% CI, −$270.5 to −$63.5;P = .002; 2.5% of overall mean cost). Conclusion and Relevance Imposition of the MOC requirement was not associated with a difference in the increase in ACSHs but was associated with a small reduction in the growth differences of costs for a cohort of Medicare beneficiaries.

Background Electronic resources are increasingly used in medical practice. Their use during high-stakes certification examinations has been advocated by many experts, but whether doing so would affect the capacity to differentiate between high and low abilities is unknown. Objective To determine the effect of electronic resources on examination performance characteristics. Design Randomized controlled trial. Setting Medical certification program. Participants 825 physicians initially certified by the American Board of Internal Medicine (ABIM) who passed the Internal Medicine Certification examination or sat for the Internal Medicine Maintenance of Certification (IM-MOC) examination in 2012 to 2015. Intervention Participants were randomly assigned to 1 of 4 conditions: closed book using typical or additional time, or open book (that is, UpToDate [Wolters Kluwer]) using typical or additional time. All participants took the same modified version of the IM-MOC examination. Measurements Primary outcomes included item difficulty (how easy or difficult the question was), item discrimination (how well the question differentiated between high and low abilities), and average question response time. Secondary outcomes included examination dimensionality (that is, the number of factors measured) and test-taking strategy. Item response theory was used to calculate question characteristics. Analysis of variance compared differences among conditions. Results Closed-book conditions took significantly less time than open-book conditions (mean, 79.2 seconds [95% CI, 78.5 to 79.9 seconds] vs. 110.3 seconds [CI, 109.2 to 111.4 seconds] per question). Mean discrimination was statistically significantly higher for open-book conditions (0.34 [CI, 0.32 to 0.35] vs. 0.39 [CI, 0.37 to 0.41] per question). A strong single dimension showed that the examination measured the same factor with or without the resource. Limitation Only 1 electronic resource was evaluated. Conclusion Inclusion of an electronic resource with time constraints did not adversely affect test performance and did not change the specific skill or factor targeted by the examination. Further study on the effect of resource inclusion on other examinations is warranted. Primary Funding Source ABIM Foundation.

PURPOSE Experienced clinicians derive many diagnoses intuitively, because most new problems they see closely resemble problems they've seen before. The majority of these diagnoses, but not all, will be correct. This study determined whether further reflection regarding initial diagnoses improves diagnostic accuracy during a high-stakes board exam, a model for studying clinical decision making. METHOD Keystroke response data were used from 500 residents who took the 2010 American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. Data included time to initial response on each question, whether the answer was correct, and whether or not the resident changed her or his initial response. The focus was on 80 diagnosis questions that comprised realistic clinical vignettes with multiple-choice single-best answers. Cognitive skill (ability) was measured using overall exam scores. Case complexity was determined using item difficulty (proportion of examinees that correctly answered the question). A hierarchical generalized linear model was used to assess the relationship between time spent on initial responses and the probability of correctly answering the questions. RESULTS On average, residents changed their responses on 12% of all diagnosis questions (or 9.6 questions out of 80). Changing an answer from incorrect to correct was almost twice as likely as changing an answer from correct to incorrect. The relationship between response time and accuracy was complex. CONCLUSIONS Further reflection appears to be beneficial to diagnostic accuracy, especially for more complex cases.

Growing concern about rising costs and potential harms of medical care has stimulated interest in assessing physicians' ability to minimize the provision of unnecessary care.To assess whether graduates of residency programs characterized by low-intensity practice patterns are more capable of managing patients' care conservatively, when appropriate, and whether graduates of these programs are less capable of providing appropriately aggressive care.Cross-sectional comparison of 6639 first-time takers of the 2007 American Board of Internal Medicine certifying examination, aggregated by residency program (n = 357).Intensity of practice, measured using the End-of-Life Visit Index, which is the mean number of physician visits within the last 6 months of life among Medicare beneficiaries 65 years and older in the residency program's hospital referral region.The mean score by program on the Appropriately Conservative Management (ACM) (and Appropriately Aggressive Management [AAM]) subscales, comprising all American Board of Internal Medicine certifying examination questions for which the correct response represented the least (or most, respectively) aggressive management strategy. Mean scores on the remainder of the examination were used to stratify programs into 4 knowledge tiers. Data were analyzed by linear regression of ACM (or AAM) scores on the End-of-Life Visit Index, stratified by knowledge tier.Within each knowledge tier, the lower the intensity of health care practice in the hospital referral region, the better residency program graduates scored on the ACM subscale (P .001 for the linear trend in each tier). In knowledge tier 4 (poorest), for example, graduates of programs in the lowest-intensity regions had a mean ACM score in the 38th percentile compared with the 22nd percentile for programs in the highest-intensity regions; in tier 2, ACM scores ranged from the 75th to the 48th percentile in regions from lowest to highest intensity. Graduates of programs in low-intensity regions tended, more weakly, to score better on the AAM subscale (in 3 of 4 knowledge tiers).Regardless of overall medical knowledge, internists trained at programs in hospital referral regions with lower-intensity medical practice are more likely to recognize when conservative management is appropriate. These internists remain capable of choosing an aggressive approach when indicated.

PURPOSE One way to ensure quality of health care in the United States is through maintenance of certification (MOC). In this study, the authors explored whether participation in the internal medicine MOC program varies by physician-level characteristics, professional activities, and the size and location of the practice. They also sought to determine which component of MOC was incomplete for physicians who participated but did not complete the program. METHOD The authors used a theoretical realist approach to understand whether participation in the American Board of Internal Medicine MOC program varies according to physician and practice characteristics. The data came from a study sample that consisted of all physicians whose original certification was granted in internal medicine from 1990 through 1999; the study was conducted in 2013. Chi-square tests of independence and a multinomial logistic regression were conducted to determine which physician-level characteristics, professional activities, and practice characteristics were significantly associated with MOC participation. RESULTS Results showed that physicians who completed MOC tended to have higher certification exam scores; were younger; were U.S. medical graduates; practiced as subspecialists and in the Midwest; spent more time in patient care, teaching, or administration; worked in nonsolo practices; or were employed in counties with less than 20% of persons in poverty. CONCLUSIONS As certifying boards evaluate their programs, they need to continuously improve their features to assure the public that physicians maintaining certification are providing high-quality patient care.

Supplemental Digital Content is available in the text., Purpose To determine whether the 2003 Accreditation Council for Graduate Medical Education (ACGME) duty hours reform affected medical knowledge as reflected by written board scores for internal medicine (IM) residents. Method The authors conducted a retrospective cohort analysis of postgraduate year 1 (PGY-1) Internal Medicine residents who started training before and after the 2003 duty hour reform using a merged data set of American Board of Internal Medicine (ABIM) Board examination and the National Board of Medical Examiners (NMBE) United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge test scores. Specifically, using four regression models, the authors compared IM residents beginning PGY-1 training in 2000 and completing training unexposed to the 2003 duty hours reform (PGY-1 2000 cohort, n = 5,475) to PGY-1 cohorts starting in 2001 through 2005 (n = 28,008), all with some exposure to the reform. Results The mean ABIM board score for the unexposed PGY-1 2000 cohort (n = 5,475) was 491, SD = 85. Adjusting for demographics, program, and USMLE Step 2 exam score, the mean differences (95% CI) in ABIM board scores between the PGY-1 2001, 2002, 2003, 2004 and 2005 cohorts minus the PGY-1 2000 cohort were −5.43 (−7.63, −3.23), −3.44 (−5.65, −1.24), 2.58 (0.36, 4.79), 11.10 (8.88, 13.33) and 11.28 (8.98, 13.58) points respectively. None of these differences exceeded one-fifth of an SD in ABIM board scores. Conclusions The duty hours reforms of 2003 did not meaningfully affect medical knowledge as measured by scores on the ABIM board examinations.

Importance US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed. Objective To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings. Design, Setting, and Participants Cross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3). Exposures Program director ratings on the RAES and milestone ratings. Main Outcomes and Measures Correlations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems. Results Corresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values P values P P P P Conclusions and Relevance Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.

There is limited understanding of how clinical reasoning is developed, maintained, and objectively assessed. Using the theoretical lens of deliberate practice, we explored physicians' views on clinical reasoning. We compared responses from internists (faculty) and internal medicine interns, to identify potential qualitative and/or quantitative differences in how clinical reasoning is developed and maintained.Participants' free-text comments regarding how clinical reasoning is developed, maintained, and objectively assessed were analyzed. Three investigators coded responses using a constant-comparative, grounded theory approach. We also compared the frequencies of each theme between the 2 groups.Twenty-two faculty and 17 interns participated in this study. Faculty and intern themes for how clinical reasoning is developed, maintained, and objectively assessed were similar, but quantitative and qualitative differences emerged. Interrater reliability of themes was high (overall kappa: 0.92; range: 0.88-0.98). Only experts (faculty) mentioned the value of teaching for development and maintenance of clinical reasoning. Interns focused on knowledge acquisition activities and use of online resources. Experts and intern participants both struggled with how to best measure clinical reasoning; direct observation was rarely mentioned as a strategy.Consistent with our theoretical expectations, we found quantitative and qualitative differences in participants' responses, which have implications for teaching and assessment of clinical reasoning. By capturing the types of activities and their relative frequencies within and between these groups, this work adds to the deliberate practice literature.

To learn whether there are differences among certified and self-designated cardiologists, internists, and family practitioners in terms of the mortality of their patients with acute myocardial infarction (AMI).Data on all patients admitted with AMI were collected for calendar year 1993 by the Pennsylvania Health Care Cost Containment Council and analyzed. Certified and self-designated family practitioners, internists, and cardiologists (n = 4,546) were compared with respect to the characteristics of their patients' illnesses. In addition, a regression model was fitted in which mortality was the dependent measure and the independent variables were the probability of death, hospital characteristics (location and the availability of advanced cardiac care), and physician characteristics (patient volume, years since graduation from medical school, specialty, and certification status).On average, cardiologists treated more patients than did generalists, and their patients were less severely ill. In the regression analysis, all variables were statistically significant except the availability of advanced cardiac care. Holding all other variables constant, treatment by a certified physician was associated with a 15% reduction in mortality among patients with AMI.Less patient mortality was associated with treatment by physicians who were cardiologists, cared for larger numbers of AMI patients, were closer to their graduation from medical school, and were certified.

Objective: To determine the impact of variability in answer key construction (i.e., option weighting) on total errors of measurement and to compare several designs for reducing this effect. Methods: A video-based format that assesses ability to interpret arteriograms is studied because it reproduces with high fidelity an important task faced by cardiologists. Responses given by highly qualified examinees are used to develop answer keys by the aggregate scoring method, and generalizability theory is applied to estimate the error variance in answer key construction. Sample: Two hundred and two examinees who volunteered to participate in the study after taking their certifying examination in cardiovascular diseases. Results: Results show less scorer variability than case variability; however, several scorers make a sizeable reduction in measurement error. Conclusion: Although there is some error in the answer key construction process, its size is relatively small and it should not be a major impediment to the use of performance-based item formats.

Introduction: Board certification has evolved from a “point-in-time” event to a process of periodic learning and reevaluation of medical competence through maintenance of certification (MOC). To better understand MOC participation, the transtheoretical model (TTM) was used to describe physicians' perceptions of MOC as a sequence of attitudinal changes. Method: Data were from a survey of internal medicine (IM) physicians' attitudes toward periodic reevaluation through MOC. An overall importance or decisional balance score was computed for each physician by summing his or her ratings across the 10 quality measures. The decisional balance score was used to classify physicians according to their acceptance of MOC, aligned with the 3 early TTM stages-of-change groups—precontemplation (PC), contemplation (C), and preparation (P)—where PC was least accepting and P was most accepting. Effect sizes assessed whether differences in attitudes toward reevaluation via MOC were of sufficient magnitude to support the TTM principles. Results: The difference in degree of acceptance of MOC between the P group and the PC and C groups was significant (p < 0.001), but the effect size was lower than predicted by the “strong” principle. Resistance to MOC for the PC and C groups was significantly greater than the P group (p < 0.001) and supported the “weak” principle. Physicians' beliefs about how often they should demonstrate performance on quality measures aligned well with the American Board of Internal Medicine's MOC requirements, with the P group believing in more frequent assessments than the PC and C groups (p < 0.001). Conclusions: Results show that physicians in the Preparation stage had overcome resistance to MOC as predicted by the “weak” principle of the TTM, but their attitude scores about the benefits of MOC were below what was expected by theory. This suggests that the structure of MOC may have made it easier for physicians to overcome barriers to MOC participation but may have lacked adequate resources to promote the benefits of participating in the process. More effort is needed to understand the specific benefits of MOC for reevaluating competencies, how to engage physicians and other stakeholders in the design of MOC, and how to communicate the rationale and evidence to those who are less accepting of MOC.

Contains fulltext : 125473.pdf (Publisher’s version ) (Open Access) Burnout is prevalent in residency training and practice and is linked to medical error and suboptimal patient care. However, little is known about how burnout affects clinical reasoning, which is essential to safe and effective care. The aim of this study was to examine how burnout modulates brain activity during clinical reasoning in physicians. Using functional Magnetic Resonance Imaging (fMRI), brain activity was assessed in internal medicine residents (n = 10) and board-certified internists (faculty, n = 17) from the Uniformed Services University (USUHS) while they answered and reflected upon United States Medical Licensing Examination and American Board of Internal Medicine multiple-choice questions. Participants also completed a validated two-item burnout scale, which includes an item assessing emotional exhaustion and an item assessing depersonalization. Whole brain covariate analysis was used to examine blood-oxygen-level-dependent (BOLD) signal during answering and reflecting upon clinical problems with respect to burnout scores. Higher depersonalization scores were associated with less BOLD signal in the right dorsolateral prefrontal cortex and middle frontal gyrus during reflecting on clinical problems and less BOLD signal in the bilateral precuneus while answering clinical problems in residents. Higher emotional exhaustion scores were associated with more right posterior cingulate cortex and middle frontal gyrus BOLD signal in residents. Examination of faculty revealed no significant influence of burnout on brain activity. Residents appear to be more susceptible to burnout effects on clinical reasoning, which may indicate that residents may need both cognitive and emotional support to improve quality of life and to optimize performance and learning. These results inform our understanding of mental stress, cognitive control as well as cognitive load theory.

To study relationships between clinical skill measures assessed at the beginning of general internists' careers and their career outcomes and practice characteristics.General Internist Community Tracking Study Physician Survey respondents (2000-2001, 2004-2005) linked with residency program evaluations and American Board of Internal Medicine board certification examination score records; n = 2,331.Cross-sectional regressions of career outcome and practice characteristic measures on board examination scores/success, residency evaluations interacted with residency type, and potential confounding variables.Failure to achieve board certification was associated with $27,206 (18 percent, p.05) less income and 14.9 percent more minority patients relative to physicians scoring in the bottom quartile on their initial examination who eventually became certified (p.01). Other skill measures were not associated with income. Scoring in the top rather than bottom quartile on the board certification examination was associated with 9 percent increased likelihood of reporting high career satisfaction (p.05). Among physicians trained in community hospital residency programs, lower evaluations were associated with 14.5 percent higher share of minority patients (p.05). Both skill measures were associated with practice type.There are associations between early career skill measures and career outcomes. In addition, minority patients are more likely to be treated by physicians with lower early career clinical skills measures than nonminority patients.

Changes in certification requirements and examinee characteristics are likely to influence the validity of the evidence associated with interpretations made based on test data. We examined whether changes in Educational Commission for Foreign Medical Graduates (ECFMG) certification requirements over time were associated with changes in internal medicine (IM) residency program director ratings and certification examination scores. Comparisons were made between physicians who were ECFMG-certified before and after the Clinical Skills Assessment (CSA) requirement. A multivariate analysis of covariance was conducted to examine the differences in program director ratings based on CSA cohort and whether the examinees emigrated for undergraduate medical education (national vs. international students). A univariate analysis of covariance was conducted to examine differences in scores from the American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. For both analyses, United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores were used as covariates. Results indicate that, of those certified by ECFMG between 1993 and 1997, 17 % (n = 1,775) left their country of citizenship for undergraduate medical education. In contrast, 38 % (n = 1,874) of those certified between 1999 and 2003 were international students. After adjustment by covariates, the main effect of cohort membership on the program director ratings was statistically significant (Wilks’ λ = 0.99, F5, 15391 = 19.9, P < 0.001). However, the strength of the relationship between cohort group and the ratings was weak (η = 0.01). The main effect of migration status was statistically significant and weak (Wilks’ λ = 0.98, F5,15391 = 45.3, P < 0.01; η = 0.02). Differences in ABIM Internal Medicine Certification Examination scores based on whether or not CSA were required was statistically significant, although the magnitude of the association between these variables was very small. The findings suggest that the implementation of an additional evaluation of skills (e.g., history-taking, physical examination) as a prerequisite to postgraduate medical education (residency) provides some additional, relevant data to those who select ECFMG-certified residents.

Managing diabetes and preventing its associated morbidities require active partnerships between physicians and patients. Studies to date lack the level of detail to quantify the degree to which interventions that are more controlled by physicians influence outcomes versus those that are more controlled by patients. Using the Archimedes model, we simulated a thirty-year clinical trial and compared the effects of three sets of interventions over which physicians have progressively less control: compliance with process-of-care standards, such as conducting foot and retinal exams and screening for signs of early kidney disease; control of biomarkers, such as hemoglobin A1c and blood pressure; and lifestyle modifications, such as patients' switching to healthier diets and losing weight. We found that if all US adults diagnosed with type 2 diabetes met quality targets in all of these areas, they would experience a nearly 16 percent increase in quality-adjusted life-years and a nearly 23 percent reduction in fifteen-year mortality over the thirty-year simulation period. Meeting aggressive biomarker targets yielded the most benefit. Meeting conservative biomarker targets came next, followed closely by meeting process-of-care standards. The incremental benefits of complying fully with diet and smoking cessation yielded the least benefit. Thus, through measures more readily within their control, and through collaboration with their patients, physicians have a substantial opportunity to improve outcomes. These findings can inform policy makers' rational resource allocation decisions and the design of programs to improve diabetes care.

In competency-testing programs, score equating is often used to ensure equivalent pass/fail decisions over different forms of the same examination. When a judgmental standard-setting method (e.g., Angoff, 1971) is used to pick the pass/fail point, the designs and statistical techniques of score equating can be applied, with favorable results, to the experts' judgments rather than examinees' performances (Norcini, 1990). The purpose of this study was to determine whether anchor item sets varying in difficulty and discrimination affect the precision of cutting score equivalents generated through judge rescaling as much as equivalents produced by score equating. Analyses were based on four groups of experts (ns = 9 and 11) and two groups of examinees (ns = 250 and 1,000). Tucker equating (Gulliksen, 1950) was applied to obtain cutting score equivalents, and the expert rescaling produced similar results regardless of the characteristics of the anchor sets; rescaling was also more precise than equating. For sc...

PURPOSE: To examine the association between physicians' cognitive skills and their performance on a composite measure of diabetes care that included process, outcome, and patient experience measures. METHOD: The sample was 676 physicians from the United States with time-limited certification in general internal medicine between 2005 and 2009. Scores from the American Board of Internal Medicine (ABIM) internal medicine maintenance of certification (MOC) examination were used to measure practicing physicians' cognitive skills (scores reflect fund of medical knowledge, diagnostic acumen, and clinical judgment). Practice performance was assessed using a diabetes composite measure aggregated from clinical and patient experience measures obtained from the ABIM Diabetes Practice Improvement Module. RESULTS: Using multiple regression analyses and controlling for physician and patient characteristics, MOC examination scores were significantly associated with the diabetes composite scores (β = .22, P < .001). The association was particularly stronger with intermediate outcomes than with process and patient experience measures. Performance in the endocrine disease content domain of the examination was more strongly associated with the diabetes composite scores (β = .19, P < .001) than the performance in other medical content domains (β = .06-.14). CONCLUSIONS: Physicians' cognitive skills significantly relate to their performance on a comprehensive composite measure for diabetes care. Although significant, the modest association suggests that there are unique aspects of physician competence captured by each assessment alone and that both must be considered when assessing a physician's ability to provide high-quality care.

Expanded abstract Citation Prasad M, Holmboe ES, Lipner RS, Hess BJ, Christie JD, Bellamy SL, Rubenfeld GD, Kahn JM. Clinical Protocols and Trainee Knowledge About Mechanical Ventilation. JAMA. 2011; 306(9):935-941. PubMed PMID: 21900133 This is available on http://www.pubmed.gov Background Clinical protocols are associated with improved patient outcomes; however, they may negatively affect medical education by removing trainees from clinical decision making. Methods Objective: To study the relationship between critical care training with mechanical ventilation protocols and subsequent knowledge about ventilator management. Design: A retrospective cohort equivalence study linking a national survey of mechanical ventilation protocol availability with knowledge about mechanical ventilation. Exposure to protocols was defined as high intensity if an intensive care unit had 2 or more protocols for at least 3 years and as low intensity if 0 or 1 protocol. Setting: Accredited US pulmonary and critical care fellowship programs. Subjects: First-time examinees of the American Board of Internal Medicine (ABIM) Critical Care Medicine Certification Examination in 2008 and 2009. Intervention: N/A Outcomes: Knowledge, measured by performance on examination questions specific to mechanical ventilation management, calculated as a mechanical ventilation score using item response theory. The score is standardized to a mean (SD) of 500 (100), and a clinically important difference is defined as 25. Variables included in adjusted analyses were birth country, residency training country, and overall first-attempt score on the ABIM Internal Medicine Certification Examination. Results The 90 of 129 programs (70%) responded to the survey. Seventy seven programs (86%) had protocols for ventilation liberation, 66 (73%) for sedation management, and 54 (60%) for lung-protective ventilation at the time of the survey. Eighty eight (98%) of these programs had trainees who completed the ABIM Critical Care Medicine Certification Examination, totaling 553 examinees. Of these 88 programs, 27 (31%) had 0 protocols, 19 (22%) had 1 protocol, 24 (27%) had 2 protocols, and 18 (20%) had 3 protocols for at least 3 years. 42 programs (48%) were classified as high intensity and 46 (52%) as low intensity, with 304 trainees (55%) and 249 trainees (45%), respectively. In bi-variable analysis, no difference in mean scores was observed in high-intensity (497; 95% CI, 486-507) vs low-intensity programs (497; 95% CI, 485-509). Mean difference was 0 (95% CI, -16 to 16), with a positive value indicating a higher score in the high-intensity group. In multivariable analyses, no association of training was observed in a high-intensity program with mechanical ventilation score (adjusted mean difference, -5.36; 95% CI, -20.7 to 10.0). Conclusions Among first-time ABIM Critical Care Medicine Certification Examination examinees, training in a high-intensity ventilator protocol environment compared with a low-intensity environment was not associated with worse performance on examination questions about mechanical ventilation management.

Objective Little is known about the best approaches and format for measuring physicians’ communication skills in an online environment. This study examines the reliability and validity of scores from two Web-based communication skill assessment formats. Methods We created two online communication skill assessment formats: (a) MCQ (multiple-choice questions) consisting of video-based multiple-choice questions; (b) multi-format including video-based multiple-choice questions with rationales, Likert-type scales, and free text responses of what physicians would say to a patient. We randomized 100 general internists to each test format. Peer and patient ratings collected via the American Board of Internal Medicine (ABIM) served as validity sources. Results Seventy-seven internists completed the tests (MCQ: 38; multi-format: 39). The adjusted reliability was 0.74 for both formats. Excellent communicators, as based on their peer and patient ratings, performed slightly better on both tests than adequate communicators, though this difference was not statistically significant. Physicians in both groups rated test format innovative (4.2 out of 5.0). Conclusion The acceptable reliability and participants’ overall positive experiences point to the value of ongoing research into rigorous Web-based communication skills assessment. Practice implications With efficient and reliable scoring, the Web offers an important way to measure and potentially enhance physicians’ communication skills.

Much research has been devoted to addressing challenges in achieving reliable assessments of physicians’ clinical performance but less work has focused on whether valid and accurate classification decisions are feasible. This study used 957 physicians certified in internal medicine (IM) or a subspecialty, who completed the American Board of Internal Medicine (ABIM) Diabetes Practice Improvement Module (PIM). Ten clinical and two patient-experience measures were aggregated into a composite measure. The composite measure score was highly reliable (r = .91) and classification accuracy was high across the entire score scale (>0.90), which indicated that it is possible to differentiate high-performing and low-performing physicians. Physicians certified in endocrinology and those who scored higher on their IM certification examination had higher composite scores, providing some validity evidence. In summary, it is feasible to create a psychometrically robust composite measure of physicians’ clinical performance, specifically for the quality of care they provide to patients with diabetes.

Interventional cardiology, with large numbers of complex procedures and potentially serious complications, stands out as an obvious discipline in which to apply simulation to help prevent medical errors. The objective of the study was to determine whether it is feasible to develop a valid and reliable evaluation approach using medical simulation to assess technical and cognitive skills of physicians performing coronary interventions.Clinical case scenarios were developed by a committee of subject matter experts, who defined key decision nodes, such as stent positioning, and introduced unanticipated complications, such as coronary perforation. Subjects were 115 physicians from 10 U.S. healthcare institutions at three levels of expertise: novice, skilled, or expert. Subjects completed a questionnaire, one practice case and six test cases on a SimSuite simulator (Medical Simulation Corporation, Denver, CO), and an opinion survey. Clinical specialists rated subjects' procedural skills.A technical and cognitive skills evaluation of performance in interventional cardiology procedures using medical simulation yielded results that distinguished between a novice group and skilled or expert groups (P0.001) and scores correlated moderately with clinical specialist ratings of subjects' procedural skills and with number and complexity of procedures performed in practice during the previous year. Approximately 90% of subjects generally thought that the cases were well simulated and presented situations encountered in practice.This study suggests that an evaluation approach using high-fidelity medical simulation to assess technical and cognitive skills of physicians performing interventional cardiology procedures can be used to identify physicians who are extremely poor performers and not likely to be providing appropriate patient care. We believe that use of a high-fidelity simulator incorporating situations with multiple events, immediate feedback, and high sensory load would complement the results of traditional written examinations of medical knowledge to provide a more comprehensive assessment of physician ability in interventional cardiology.

Health reform legislation grants authority for patient-centered medical home pilot projects to test changes in the way primary care is provided. There is concern that using a measurement tool to qualify medical homes that is solely based on the presence or absence of "system elements" may miss the point conceptually and lead physicians astray in attempts to transform their entire practices. To find out whether and how practice characteristics explain health care quality, we examined risk-adjusted composite measures of quality for common chronic and acute care conditions and preventive care from 202 general internists working primarily in small primary care office settings. We found that current conceptions and measures of what constitutes "successful" practice systems and care are incomplete, and have limited associations with measures of health care quality. Future research should explore more fully the issues around physician competence, including competence in systems and quality improvement; the interactive nature of clinical practice; and other important system elements not captured by current tools.

A shortage of primary care physicians is expected, due in part to decreasing numbers of physicians entering general internal medicine (GIM). Practicing general internists may contribute to the shortage by leaving internal medicine (IM) for other careers in and out of medicine. To better understand mid-career attrition in IM. Mail survey to a national sample of internists originally certified by the American Board of Internal Medicine in GIM or an IM subspecialty during the years 1990 to 1995. Self-reported current status as working in IM, working in another medical or non-medical field, not currently working but plan to return, or retired; and career satisfaction. Nine percent of all internists in the 1990–1995 certification cohorts and a significantly larger proportion of general internists (17%) than IM subspecialists [(4%) P

Background Effective care coordination requires good physician-to-physician communication. The authors evaluated a new tool called the Communication with Referring Physicians Practice Improvement Module (CRP-PIM), which assesses and encourages improved communication among physician consultants and referring physicians. Method Eight-hundred three consultants (internists and subspecialists) completed a practice system survey and were rated by 12,212 referring physicians on 13 communication processes using a six-point scale. Consultants received an interactive performance report and selected targets for improvement. Data were analyzed using descriptive statistics, correlations, t tests, and factor analysis. Results Mean overall rating was high, at 5.53 (SD 0.23, range 2.46-5.95); consultants still identified areas for improvement. The generalizability coefficient for overall ratings was 0.78. Factor analysis supported two categories of ratings associated with consultants' gender, subspecialty, residency performance ratings, and specific practice system features. Conclusions The CRP-PIM provides a psychometrically viable measure and encourages consultants to improve communication with referring physicians.

Background The relationship between physicians' cognitive skill and the delivery of evidence-based processes of care is not well characterized.Therefore, we set out to determine associations between general internists' performance on the American Board of Internal Medicine maintenance of certification examination and the receipt of important processes of care by Medicare patients. Methods Physicians were grouped into quartiles based on their performance on the American Board of Internal Medicine examination. Hierarchical generalized linear models examined associations between examination scores and the receipt of processes of care by Medicare patients. The main outcome measures were the associations between diabetes care, using a composite measure of hemoglobin A 1c , and lipid testing and retinal screening, mammography, and lipid testing in patients with cardiovascular disease and the physician's performance on the American Board of Internal Medicine examination, adjusted for the number of Medicare patients with diabetes and cardiovascular disease in a physician's practice panel; frequency of visits; patient comorbidity, age, and ethnicity; and physician training history and type of practice. Results Physicians scoring in the top quartile were more likely to perform processes of care for diabetes (composite measure odds ratio [OR], 1.17; 95% confidence interval [CI], 1.07-1.27) and mammography screening (OR, 1.14; 95% CI, 1.08-1.21) than physicians in the lowest physician quartile, even after adjustment for multiple factors. There was no significant difference among the groups in lipid testing of patients with cardiovascular disease (OR, 1.00; 95% CI, 0.91-1.10). Conclusion Our findings suggest that physician cognitive skills, as measured by a maintenance of certification examination, are associated with higher rates of processes of care for Medicare patients.

Few studies have systematically and rigorously examined the quality of care provided in educational practice sites.The objectives of this study were to (1) describe the patient population cared for by trainees in internal medicine residency clinics; (2) assess the quality of preventive cardiology care provided to these patients; (3) characterize the practice-based systems that currently exist in internal medicine residency clinics; and (4) examine the relationships between quality, practice-based systems, and features of the program: size, type of program, and presence of an electronic medical record.This is a cross-sectional observational study.This study was conducted in 15 Internal Medicine residency programs (23 sites) throughout the USA.The participants included site champions at residency programs and 709 residents.Abstracted charts provided data about patient demographics, coronary heart disease risk factors, processes of care, and clinical outcomes. Patients completed surveys regarding satisfaction. Site teams completed a practice systems survey.Chart abstraction of 4,783 patients showed substantial variability across sites. On average, patients had between 3 and 4 of the 9 potential risk factors for coronary heart disease, and approximately 21% had at least 1 important barrier of care. Patients received an average of 57% (range, 30-77%) of the appropriate interventions. Reported satisfaction with care was high. Sites with an electronic medical record showed better overall information management (81% vs 27%) and better modes of communication (79% vs 43%).This study has provided insight into the current state of practice in residency sites including aspects of the practice environment and quality of preventive cardiology care delivered. Substantial heterogeneity among the training sites exists. Continuous measurement of the quality of care provided and a better understanding of the training environment in which this care is delivered are important goals for delivering high quality patient care.

SIGNIFICANT QUALITY-OF-CARE GAPS ARE WELL DOCUmented in the United States. These reports have focused mostly on underuse of performance measures of important processes of care, and some outcomes of care. Others have argued that the cause of underuse of these evidence-based processes of care is usually not deficient physician knowledge about whether to perform the examination or order the test, but rather poorly designed, dysfunctional microsystems of care unable to deliver effective, efficient, and reliable care. Consequently, much of the recent work in quality improvement has focused on changing microsystems of care “to deliver the right care for the right patient at the right time, all the time.” What is often overlooked in quality improvement, but equally important, is that effective microsystems must have highly competent clinicians, who possess sufficient knowledge and clinical skills to make and execute evidence-based decisions, exercise informed clinical judgment, and deal effectively with uncertainty. Clinical judgment and the ability to deal with uncertainty are especially critical with respect to misuse and overuse of processes of care. Misuse and overuse of processes of care (eg, overprescribing antibiotics and unnecessary imaging and procedures) put patients at greater risk for unnecessary complications. Physician knowledge and clinical judgment also are central to making correct diagnoses. The majority of current performance measures assume a correct diagnosis, but more than that current measures cover only a fraction of the myriad health problems seen by physicians on a daily basis and likely will never address unusual or less common but no less important or serious conditions. Furthermore, many symptoms and signs that prompt patients to see physicians are often not well-defined and a diagnosis often remains uncertain after the initial visit. Clinical judgment is crucial in determining when further intervention is necessary or when watchful waiting may be the best approach. Even when an accurate diagnosis is made, prudent clinical judgment is necessary to determine appropriate care, including the correct diagnostic tests, critical to the efficiency and effectiveness aspects of quality. Our objectives in this Commentary are to discuss the relationship between medical knowledge and quality and how the secure examination component of specialty board certification—with its primary focus on assessing physician knowledge, diagnostic acumen, and clinical judgment—is an important complement to current performance measures. Recognizing this importance, in 2006 the American Board of Internal Medicine instituted a new requirement for all physicians with time-limited certificates to evaluate their performance in practice to address physician competence in practice-based learning and improvement and systemsbased practice. We hope this discussion will stimulate dialogue about the need for more comprehensive physician performance measurement in the era of public reporting.

BACKGROUND To assess the psychometric properties of the three components of the Diabetes Practice Improvement Module, to compare reliabilities of composites to individual measures, and to identify associations among practice-based and patient-based measures. METHOD Data include practice systems surveys of 626 physicians, 13,965 chart audits, and 12,927 patient surveys. Quality composites were identified using factor analysis. Means with reliabilities (intraclass correlation coefficient [ICC] and Cronbach's alpha) are reported. Associations among patient-based quality measures and practice measures with case-mix adjustments were estimated via hierarchical models. RESULTS Composite ICCs range from 0.11 to 0.54, and single items range from 0.05 to 0.49. Staff communication, efficiency, care access, and patient knowledge correlate with patient satisfaction (P < .001). Clinical outcomes are associated with clinical processes (e.g., annual foot exam) and appropriate treatment (P < .001). Patient adjusters (e.g., overall health or factors limiting self-care) are important for the models; physician characteristics used (e.g., age, practice size) seem less important. CONCLUSIONS Composites require smaller patient sample sizes and result in more reliable measures than do individual items. Additionally, the data show meaningful relationships between composites; physician-directed components (i.e., clinical processes and treatments) are related to clinical outcomes, and patients are clearly more satisfied with care if it is easily accessible and if communication about care is good.

The Institute of Medicine's reports and discussions on quality of medical care have focused on a systems-based approach to quality improvement. Our objective is to summarize evidence and theory about the role of a physician's current board certification status in quality improvement. The first body of evidence includes the validity of board certification demonstrated by the testing process, the relationship of examination scores with other measures of physician competence, and the relationship between certification status and clinical outcomes. The second body of evidence involves the adaptation of error prevention theory to medical care. Patient safety is enhanced when problem-solving uses readily accessed habits of behavior, the same behavior necessary to achieve board certification. The third body of evidence, obtained through a Gallup poll, demonstrates that certification and maintenance of certification are highly valued by the public. The majority of respondents thought it important for physicians to be reevaluated on their qualifications every few years and that physicians should do more to demonstrate ongoing competence than is currently required by the profession. We conclude that a physician's current certification status should be among the evidence-based measures used in the quality movement.

To establish whether successful certifying examination performances of doctors are associated with their patients' mortality and length of stay following acute myocardial infarction.Risk adjusted mortality and survivors' length of stay were compared for doctors who had satisfactorily completed training in internal medicine or cardiology and attempted the relevant examination. Specifically, the study investigated the joint effects of hospital location, availability of advanced cardiac care, doctors' specializations, certifying examination performances, year certification was first attempted and patient volume.Data on all acute myocardial infarctions in Pennsylvania for the calendar year 1993 were collected by the Pennsylvania Health Care Cost Containment Council. These data were combined with physician information from the database of the American Board of Internal Medicine.Holding all variables constant, successful examination performance (i.e. certification in internal medicine or cardiology) was associated with a 19% reduction in mortality. Decreased mortality was also correlated with treatment in hospitals located outwith either rural or urban settings and with management by a cardiologist. Shorter stays were not related to examination performance but were associated with treatment by high volume cardiologists who had recently finished training and who cared for their patients in hospitals located outwith rural or urban settings.The results of the study add to the evidence supporting the validity of the certifying examination and lend support to the concept that fund of knowledge is related to quality of practice.

RECENT CONTROVERSY ABOUT THE AMERICAN Board of Internal Medicine (ABIM) Maintenance of Certification (MOC) process has centered on the role of the secure examination. Some physicians object to taking any examination, whereas others assert that the internal medicine and subspecialty examinations are not sufficiently focused to reflect physician practices. Others have questioned why the examination is not “open-book” to mirror a real-world practice environment. Even though all 24 certifying boards of the American Board of Medical Specialties require secure examinations for MOC, and many have required recertification for decades, the questions raised regarding ABIM examinations have relevance to all medical specialties. This Commentary intends to clarify the science and philosophy behind examination development and the current psychometric principles that guide the process. Certifying boards strive to provide the public with useful information about a physician’s competence while meeting expectations both of physicians and the public for a relevant, valid, and reliable assessment process. The ABIM’s MOC is a multidimensional program that meets these goals through tools designed to evaluate several competencies. Physicians choose assessments of knowledge, practice improvement, and communication skills based on personal evaluation of their learning needs and their practice profile. The secure examination component of MOC assesses the broad-based knowledge that the public expects of a physician who identifies himself or herself as a general internist or subspecialist. Its purpose is to define a discipline as well as test an individual’s clinical knowledge and judgment. The ABIM follows the Standards for Educational and Psychological Testing established through a collaboration of the American Psychological Association, the American Educational Research Association, and the National Council of Measurement and Evaluation. These standards represent a wellestablished professional consensus concerning appropriate and fair test use and are based on psychometrics, the science of testing that governs examination administration, production, content, scoring, generalization, and extrapolation to real-world practice. Developing an examination that meets these standards is a labor-intensive undertaking. Clinical content and case scenarios in the ABIM’s MOC examinations are developed by carefully selected committee members to ensure adequate content coverage and to include the perspectives of physicians both in practice and in academia. Topics in which physician intervention has significant effects on patient health outcomes are identified. Each topic proposed for the test is rated by practicing physicians, program directors, and trainees for importance and frequency in practice. Low-rated topics are typically excluded from the MOC examination. Question stems provide mini case simulations or patient vignettes that present a realistic description of the patient, symptoms, and laboratory values, requiring the integration of information, prioritization of alternatives, and use of clinical judgment in reaching an appropriate conclusion and deciding on a course of action. Vignette questions are more relevant to real-world patient care than nonvignettes. Through committee review and pretesting, questions are evaluated to ensure that they are evidence-based, technically sound, and fair. The performance of each question is reviewed for content, difficulty level, and how well it discriminates between physicians who clearly demonstrate competency in the domains expected of internists and those who do not. If a question is too difficult or does not discriminate well among physicians, it may be revised or discarded. The ABIM MOC examinations as currently constructed maintain high reproducibility of test scores (a Cronbach 0.90 [1.0 is the maximum]). To date, case-based multiple-choice questions portrayed as patient vignettes are the most efficient, cost-effective, and feasible method for assessing a broad domain area and extrapolating to real-world patient care. The ABIM is currently studying ways to incorporate computer clinical simulation into the examination to better assess clinical skills and increase realism. To set a credible pass-fail standard, the ABIM relies on a committee of knowledgeable internists from private practice and academic medicine to set an absolute stan

Physicians who are disciplined by state licensing boards are more likely to have demonstrated unprofessional behavior in medical school. Information is limited on whether similar performance measures taken during residency can predict performance as practicing physicians.To determine whether performance measures during residency predict the likelihood of future disciplinary actions against practicing internists.Retrospective cohort study.State licensing board disciplinary actions against physicians from 1990 to 2006.66,171 physicians who entered internal medicine residency training in the United States from 1990 to 2000 and became diplomates.Predictor variables included components of the Residents' Annual Evaluation Summary ratings and American Board of Internal Medicine (ABIM) certification examination scores.2 performance measures independently predicted disciplinary action. A low professionalism rating on the Residents' Annual Evaluation Summary predicted increased risk for disciplinary action (hazard ratio, 1.7 [95% CI, 1.3 to 2.2]), and high performance on the ABIM certification examination predicted decreased risk for disciplinary action (hazard ratio, 0.7 [CI, 0.60 to 0.70] for American or Canadian medical school graduates and 0.9 [CI, 0.80 to 1.0] for international medical school graduates). Progressively better professionalism ratings and ABIM certification examination scores were associated with less risk for subsequent disciplinary actions; the risk ranged from 4.0% for the lowest professionalism rating to 0.5% for the highest and from 2.5% for the lowest examination scores to 0.0% for the highest.The study was retrospective. Some diplomates may have practiced outside of the United States. Nondiplomates were excluded.Poor performance on behavioral and cognitive measures during residency are associated with greater risk for state licensing board actions against practicing physicians at every point on a performance continuum. These findings support the Accreditation Council for Graduate Medical Education standards for professionalism and cognitive performance and the development of best practices to remediate these deficiencies.

The American Board of Medical Specialties (ABMS) adopted a framework, called Maintenance of Certification (MOC), for all certifying boards to evaluate physicians' competence throughout their careers, with the goal of improving the quality of health care. The MOC participation rates of the American Board of Internal Medicine (ABIM) show that 23% of general internists and 14% of subspecialists choose not to renew their respective certificates.To study U.S. internists' perceptions about the forces driving them to maintain certification.Mail survey.A nationally representative sample of certified internists in the United States.Physicians originally certified in internal medicine, a subspecialty, or an area of added qualifications in 1990, 1991, or 1992.The overall rate of response to the survey was 51%. Although 91% of all participants are still working in internal medicine or its subspecialties, this percentage is notably lower among general internists (79%). Of those still working in the field of internal medicine or its subspecialties, approximately half report being required to maintain their specialty certificate by at least 1 employer, but only approximately one third of those who completed or enrolled in MOC report this requirement as a reason for participating. Those who completed or enrolled in MOC do so more for positive professional reasons than for monetary benefits or professional advancement. The most common reasons for not participating are the perceptions that it takes too much time, is too expensive, and is not required for employment.Respondents were volunteers from an early cohort of diplomates entering the program, and those with less positive attitudes may have responded at higher rates. Results are based on self-reported data, and misconceptions about program requirements may have led to some inaccurate responses.The relatively large percentage of general internists who left internal medicine mostly to work in another medical field explains why rates of MOC participation for general internists seem lower than those for subspecialists (77% vs. 86%). Although positive professional reasons clearly have a compelling internal influence on program participation, it is less clear whether employers' requirements are an equally compelling external influence. Although half of all respondents report that MOC is required by 1 of their employers, only one third of those who participate in the program describe it as a reason for participating.

The purpose of this study was to determine whether two variations on the typical Angoff group standard-setting process would produce sufficiently consistent results to recommend their use. Judgments obtained from a group of experts during a meeting were compared with judgments gathered from the same group before and after the meeting. The results indicate that differences between passing scores obtained with the three variations are relatively small, but those gathered before the meeting were less consistent than ratings gathered during and after the meeting. These results imply that judgments gathered after an initial traditional group-process session can provide an efficient alternative mechanism for setting cutting scores using the Angoff method. This research was supported by The American Board of Internal Medicine, but does not necessarily reflect its opinions or policies.

The performance of candidates on the geriatric medicine items on the American Board of Internal Medicine's 1980, 1981, and 1982 Certifying Examinations was analyzed. For the 1981 and 1982 examinations, candidates who trained in residency programs having a special emphasis on geriatric medicine performed significantly better on the geriatric items than did candidates who attended other residency programs, regardless of overall medical knowledge. This finding suggests the efficacy of training in geriatric medicine as measured by a written examination.

The performance of practicing internists on the American Board of Internal Medicine's 1980 Recertification Examination was examined in two studies. In the first study, a psychometric common-item equating technique was used to compare the performance of 1980 recertification candidates with that of 1979 certification candidates. Results showed that the knowledge base of practicing internists was similar to that of residents completing training. The second study analyzed the performance of 1980 recertification candidates to determine whether being certified or having an interest in a subspecialty affects a physician's performance on items in that area. The results showed that subspecialists do significantly better than general internists on items pertaining to their area of specialization. Similar outcomes were found for internists with a special interest in a subspecialty area. These findings establish the importance of continued periodic evaluation and support the development of an evaluation tool tailored to the physician's area of concentration.