Search

Retrieval practice improves long-term retention. Use of interactive retrieval practice in large group, in-person and online live classes, in combination with outside resources, is unreported for medical physiology classes. The primary study purpose was to compare student cohorts' performance with or without retrieval practice in renal physiology classes, relative to the national average on customized national examinations in renal physiology, nonphysiology, and all questions. The secondary purpose was to examine the students' educational experience. For the primary purpose, we used a nonequivalent group, posttest-only design. For the secondary purpose, we used cross-sectional and qualitative designs. We analyzed examination results of 684 students in four academic years. For renal physiology questions, students performed significantly better in years with retrieval practice compared with years without it (P < 0.001). There was no change in nonphysiology scores over the four years. Performance in all questions, too, significantly improved (P < 0.001). A large majority (86%) of students indicated retrieval practice helped them learn renal physiology. Student ratings of quality in online classes, which featured interactive retrieval practice, were higher than that of in-person classes (P < 0.001). Qualitative analysis revealed students found interactive retrieval practice, scaffolding, outside resources, and the instructor's teaching style helpful. Educators in medical physiology classes can use our findings to implement interactive retrieval practice.

This article presents a new approach to the analysis of how students answer tests and how they allocate resources in terms of time on task and revisiting previously answered questions. Previous research has shown that in high-stakes assessments, most test takers do not end the testing session early, but rather spend all of the time they were assigned to take the test. Rather than being an indication of speededness, this was found to be caused by test takers' tendency to revisit previous items even if they already provided answers to all questions. In accordance with this information, the proposed approach models revisit patterns simultaneously with responses and response times to gain a better understanding of the relationship between speed, ability, and revisit tendency. The empirical data analysis revealed that examinees' tendency to revisit items was strongly related to their speed and subgroups of examinees displayed different test-taking behaviors.

Previous research on rapid responding tends to implicitly consider examinees as either engaging in solution behavior or purely guessing. However, particularly in a high-stakes testing context, examinees perceiving that they are running out of time may consider the remaining items for less time than necessary to provide a fully informed response, but longer than a truly rapid guess. This partial consideration results in a response that misrepresents true ability, but with accuracy above the level of pure chance. To address this limitation of existing methodology, we propose an empirical approach that attempts to disentangle fully and partially informed responses to be used as a preliminary measure of the extent to which speededness may be distorting test score validity. We first illustrate and validate the approach using an experimental dataset in which the amount of time per item was manipulated. Next, applications of this approach are demonstrated using observational data in a more realistic context through four operational exams in which speededness is unknown.

Test scoring models vary in their generality, some even adjust for examinees answering multiple-choice items correctly by accident (guessing), but no models, that we are aware of, automatically adjust an examinee's score when there is internal evidence of cheating. In this study, we use a combination of jackknife technology with an adaptive robust estimator to reduce the bias in examinee scores due to contamination through events such as having access to some of the test items in advance of the test administration. We illustrate our methodology with a data set of test items we knew to have been divulged to a subset of the examinees.

Problem: Practitioners working with multiple-choice tests have long utilized Item Response Theory (IRT) models to evaluate the performance of test items for quality assurance. The use of similar applications for performance tests, however, is often encumbered due to the challenges encountered in working with complicated data sets in which local calibrations alone provide a poor model fit. Purpose: The purpose of this study was to investigate whether the item calibration process for a performance test, computer-based case simulations (CCS), taken from the United States Medical Licensing Examination® (USMLE®) Step 3® examination may be improved through explanatory IRT models. It was hypothesized that explanatory IRT may help improve data modeling for performance assessment tests by allowing important predictors to be added to a conventional IRT model, which are limited to item predictors alone. Methods: The responses of 767 examinees from a six-item CCS test were modeled using the Partial Credit Model (PCM) and four explanatory model extensions, each incorporating one predictor variable of interest. Predictor variables were the examinees' gender, the order in which examinees encountered an individual item (item sequence), the time it took each examinee to respond to each item (response time), and examinees' ability score on the multiple-choice part of the examination. Results: Results demonstrate a superior model fit for the explanatory PCM with examinee ability score from the multiple-choice portion of Step 3. Explanatory IRT model extensions might prove useful in complex performance assessment test settings where item calibrations are often problematic due to short tests and small samples. Recommendations: Findings of this study have great value in practice and implications for researchers working with small or complicated response data. Explanatory IRT methodology not only provides a way to improve data modeling for performance assessment tests but also enhances the inferences made by allowing important person predictors to be incorporated into a conventional IRT model.

Phipps, Osborne, Dyer, and Ball, (2008) posited that, "Agricultural education in secondary schools has played an important role in enhancing student achievement in the core subject areas…" (p. 4), while Enderlin and Osborne (1992) reported that agricultural students received higher test scores in biology than students in other science classes. However, further evaluation of this academic integration is warranted to determine which practices are most beneficial to students. This study sought to determine if there was a relationship between the number of agricultural education classes that students took and the subsequent outcomes on the Alabama High School Graduation Exam. To address the research question, binary logistic regression was employed. Results indicated that the model did predict the outcomes on the language and math portion of the exam while the model failed to predict outcomes on the social studies, biology, and reading portions of the exam. This study should be replicated using standardized tests in other states. Comparable data should be collected for students not enrolled in agriculture classes so that the groups mean scores could be compared.

The purpose of this study was to evaluate whether models based on pre-admission testing, including performance on the Medical College Admission Test (MCAT), performance on required courses in the medical school curriculum, or a combination of both could accurately predict performance of medical students on the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Models were produced using stepwise linear regression and feed forward neural networks. Notable accuracy in predicting Step 1 and Step 2 scores were achieved from models integrating pre-admission variables with medical school coursework grades. Of interest, the coursework grades contributed far greater to these models than the pre-admission variables except the MCAT. (Contains 2 tables.)

This descriptive article describes the use of clinical case-based portfolios in histopathology teaching laboratories in conjunction with virtual microscopy not only to integrate histology and pathology disciplines for first and second year medical students but also to stimulate student engagement, promote self-directed and group-based learning and enhance student-to-student interaction in a structured manner. Portfolios consisted of PowerPoint files encompassing four to five clinical case studies relevant to the topics covered that week. Portfolios integrated study materials provided in the module-specific lectures, clinical skill lectures, and online interactive content. Two sets of portfolios, Individual and Group, were used. Individual Portfolios were completed by each student and uploaded prior to the laboratory session. Group Portfolios were completed by students working together in small groups during the laboratory session with minimal faculty assistance. The functional utility and acceptance of Individual and Group Portfolios among first- and second-year medical students was evaluated using electronic surveys and examination performances. Both first- and second-year students agreed that the use of portfolios in conjunction with virtual microscopy promoted understanding and encouraged discussion of the topics covered during the week and that group members worked well together and contributed to the completion of the portfolios. Performances on the Histology and Cell Biology and Pathology sections on the United States Medical Licensing Examination® (USMLE®) remained consistent and in line with national averages. Overall, use of portfolios promoted peer teaching and contributed towards successful transition to the new system-based integrated curriculum with continued strong performance on the USMLE.

Open Educational Resources (OER) can offer educators the necessary flexibility for tailoring educational resources to better fit their educational goals. Although the number of OER repositories is growing fast, few studies have been conducted to empirically test the effectiveness of OER integration in the classroom. Furthermore, very little is known about the benefits of utilizing OER in high school mathematics classrooms for students with learning disabilities. The purpose of this study was to examine the effects of Multimedia Open Resources for Education (MORE) on the mathematics performance and motivation of high school students diagnosed with learning disabilities. The independent variable was the use of three MORE to supplement mathematics instruction. The outcome measures included students' achievement and motivation. At the end of the four-week study, a posttest was conducted, followed by the Course Interest Survey (CIS) based on Keller's ARCS motivational theory. The results showed that the achievement test scores between the experimental group and the control group were not significantly different. However, the results of the CIS showed a statistically significant difference in motivation between the two groups (p < 0.05) in all of the four motivational components - attention, relevance, confidence, and satisfaction.

Much of the content delivered during medical students' preclinical years is assessed nationally by such testing as the United States Medical Licensing Examination® (USMLE®) Step 1 and Comprehensive Osteopathic Medical Licensing Examination® (COMPLEX-USA®) Step 1. Improvement of student study/learning strategies skills is associated with academic success in internal and external (USMLE Step 1) examinations. This research explores the strength of association between the Learning and Study Strategies Inventory (LASSI) scores and student performance in the anatomical sciences and USMLE Step 1 examinations. The LASSI inventory assesses learning and study strategies based on ten subscale measures. These subscales include three components of strategic learning: skill (Information processing, Selecting main ideas, and Test strategies), will (Anxiety, Attitude, and Motivation) and self-regulation (Concentration, Time management, Self-testing, and Study aid). During second year (M2) orientation, 180 students (Classes of 2016, 2017, and 2018) were administered the LASSI survey instrument. Pearson Product-Moment correlation analyses identified significant associations between five of the ten LASSI subscales (Anxiety, Information processing, Motivation, Selecting main idea, and Test strategies) and students' performance in the anatomical sciences and USMLE Step 1 examinations. Identification of students lacking these skills within the anatomical sciences curriculum allows targeted interventions, which not only maximize academic achievement in an aspect of an institution's internal examinations, but in the external measure of success represented by USMLE Step 1 scores.

The Angoff standard setting method relies on content experts to review exam items and make judgments about the performance of the minimally proficient examinee. Unfortunately, at times content experts may have gaps in their understanding of specific exam content. These gaps are particularly likely to occur when the content domain is broad and/or highly technical, or when non-expert stakeholders are included in a standard setting panel (e.g., parents, administrators, or union representatives). When judges lack expertise regarding specific exam content, the ratings associated with those items may be bias. This study attempts to illustrate the impact of rating unfamiliar items on Angoff passing scores. The study presents a comparison of Angoff ratings for typical items with those identified by judges as containing unfamiliar content. The results indicate that judges tend to perceive unfamiliar items as being artificially difficult resulting in systematically lower Angoff ratings. The results suggest that when judges are forced to rate unfamiliar items, the validity of the resulting classification decision may be jeopardized.

The relative impact of admissions factors and curricular measures on the first medical licensing exam (United States Medical Licensing Exam [USMLE] Step 1) scores is examined. The inclusion of first-year and second-year curricular measures nearly doubled the variance explained in Step 1 scores from the amount explained by the combination of preadmission demographic characteristics and admissions factors. In addition, the relationship between the Medical College Admissions Test (MCAT) and Step 1 scores becomes counterintuitive in models that include curricular measures, where students with the lowest combined admissions metrics (MCAT, grade-point average) score "higher", on average, than those with some of the highest admissions metrics. Overreliance on traditional metrics in admissions decisions can exclude students from medical school who have the ability to succeed.

Changes in medical school curricula often require educators to develop teaching strategies that decrease contact hours while maintaining effective pedagogical methods. When faced with this challenge, faculty at the University of Cincinnati College of Medicine converted the majority of in-person histology laboratory sessions to self-study modules that utilize multiple audiovisual modalities and a virtual microscope platform. Outcomes related to this shift were investigated through performance on in-house examinations, results of the United States Medical Licensing Examination® (USMLE®) Step 1 Examination, and student feedback. Medical School College Admissions Test® (MCAT®) scores were used as a covariate when comparing in-house examinations. Results revealed no significant change in performance on in-house examinations when the content being assessed was controlled (F(2, 506) = 0.676, P = 0.51). A significant improvement in overall practical examination grade averages was associated with the self-study modules (F(6, 1164) = 10.213, P < 0.01), but gradual changes in examination content may explain this finding. The histology and cell biology portion of USMLE Step 1 Examination remained consistent throughout the time period that was investigated. Student feedback regarding the self-study modules was positive and suggested that features such as instructor narrated videos were an important component of the self-study modules because they helped recreate the experience of in-person laboratory sessions. Positive outcomes from the student perspective and no drop in examination performance suggests that utilizing self-study modules for histology laboratory content may be an option for educators faced with the challenge of reducing contact hours without eliminating content.

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was 0.79 (Documentation = 0.63, Differential Diagnosis = 0.90, Justification = 0.48, and Workup = 0.54). The majority of score variance was due to case specificity (13%) and case-task specificity (31%), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5%). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.

Success in residency matching is largely contingent upon standardized exam scores. Identifying predictors of standardized exam performance could promote primary intervention and lead to design insights for preclinical courses. We hypothesized that clinically relevant courses with an emphasis on higher-order cognitive understanding are most strongly associated with performance on United States Medical Licensing Examination Step exams and National Board of Medical Examiners clinical subject exams. Academic data from students between 2007 and 2012 were collected. Preclinical course scores and standardized exam scores were used for statistical modeling with multiple linear regression. Preclinical courses were categorized as having either a basic science or a clinical knowledge focus. Medical College Admissions Test scores were included as an additional predictive variable. The study sample comprised 795 graduating medical students. Median score on Step 1 was 234 (interquartile range 219-245.5), and 10.2% (81/795) scored lower than one standard deviation below the national average (205). Pathology course score was the strongest predictor of performance on all clinical subject exams and Step exams, outperforming the Medical College Admissions Test in strength of association. Using Pathology score <75 as a screening metric for Step 1 score <205 results in sensitivity and specificity of 37 and 97%, respectively, and a likelihood ratio of 11.9. Performance in Pathology, a clinically relevant course with case-based learning, is significantly related to subsequent performance on standardized exams. Multiple linear regression is useful for identifying courses that have potential as risk stratifiers.

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60%), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep[superscript 2] = 0.68, ? = 0.60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < 0.01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.

Correlates of achievement in the basic science years in medical school and on the Step 1 of the United States Medical Licensing Examination® (USMLE®), (Step 1) in relation to preadmission variables have been the subject of considerable study. Preadmissions variables such as the undergraduate grade point average (uGPA) and Medical College Admission Test® (MCAT®) scores, solely or in combination, have previously been found to be predictors of achievement in the basic science years and/or on the Step 1. The purposes of this retrospective study were to: (1) determine if our statistical analysis confirmed previously published relationships between preadmission variables (MCAT, uGPA, and applicant pool size), and (2) study correlates of the number of failures in five M1 courses with those preadmission variables and failures on Step 1. Statistical analysis confirmed previously published relationships between all preadmission variables. Only one course, Microscopic Anatomy, demonstrated significant correlations with all variables studied including the Step 1 failures. Physiology correlated with three of the four variables studied, but not with the Step 1 failures. Analyses such as these provide a tool by which administrators will be able to identify what courses are or are not responding in appropriate ways to changes in the preadmissions variables that signal student performance on the Step 1.

The University of Alabama School of Medicine (UASOM) instituted a fully integrated, organ system-based preclinical curriculum in 2007. Gross anatomy and embryology were integrated with other basic science disciplines throughout the first two years of undergraduate medical education. Here we describe the methods of instruction and integration of gross anatomy and embryology in this curriculum as well as challenges faced along the way. Gross anatomy and embryology are taught through a combination of didactic lectures, team-based learning activities, and cadaveric dissection laboratories. Vertical integration occurs through third- and fourth-year anatomy and embryology elective courses. Radiology is integrated with anatomy instruction through self-study modules and hands-on ultrasound sessions. Our model of anatomy instruction is time efficient, clinically relevant, and effective as demonstrated by student performance on the United States Medical Licensing Examination® (USMLE®) Step 1 examination. We recommend that medical schools considering full integration of gross anatomy and embryology (1) carefully consider the sequencing of organ system modules, (2) be willing to sacrifice anatomical detail for clinical application, (3) provide additional electives to third- and fourth-year students, and (4) integrate radiology with anatomical education.

While the first year of medical school is challenging for all students, there may be specific issues for students from rural areas, economically disadvantaged backgrounds, ethnic minorities, or nontraditional age groups. A Summer Prematriculation Program (SPP) was created to prepare entering at-risk students for the demands of medical school. For the past 2 yr, an emphasis was placed on the development of appropriate study plans and skills. On presurveys, students predicted an increase in their number of study hours per lecture hour, from 7.6 h in undergraduate coursework to 9.1 h in medical school coursework (n = 35). These study plans were infeasible given the rigorous didactic lecture schedule in medical school. Interventions were made through lectures on study plans and modeling of appropriate study habits using engaging lectures in the SPP physiology course. At the end of the program, a postsurvey was given, and students reported a reduction in the planned hours of study to a more realistic 3.9 h of study time per hour of lecture. Furthermore, students planned to decrease their use of textbooks while increasing their use of concept mapping, videos, and peer teaching. The majority of students completing the SPP program with a study skills emphasis performed well in the Medical Physiology course, with 4 students honoring in the course, 27 students passing, and 2 students remediating the course after an initial failure. These results indicate that at-risk medical students may have inappropriate study plans that can be improved through participation in a program that emphasizes study skills development.

Previous research on the impact of text and formatting changes on test-item performance has produced mixed results. This matter is important because it is generally acknowledged that "any" change to an item requires that it be recalibrated. The present study investigated the effects of seven classes of stylistic changes on item difficulty, discrimination, and response time for a subset of 65 items that make up a standardized test for physician licensure completed by 31,918 examinees in 2012. One of two versions of each item (original or revised) was randomly assigned to examinees such that each examinee saw only two experimental items, with each item being administered to approximately 480 examinees. The stylistic changes had little or no effect on item difficulty or discrimination; however, one class of edits--changing an item from an open lead-in (incomplete statement) to a closed lead-in (direct question)--did result in slightly longer response times. Data for nonnative speakers of English were analyzed separately with nearly identical results. These findings have implications for the conventional practice of repretesting (or recalibrating) items that have been subjected to minor editorial changes.

The purpose of the present study was to extend past work with the Angoff method for setting standards by examining judgments at the judge level rather than the panel level. The focus was on investigating the relationship between observed Angoff standard setting judgments and empirical conditional probabilities. This relationship has been used as a measure of internal consistency by previous researchers. Results indicated that judges varied in the degree to which they were able to produce internally consistent ratings; some judges produced ratings that were highly correlated with empirical conditional probabilities and other judges' ratings had essentially no correlation with the conditional probabilities. The results also showed that weighting procedures applied to individual judgments both increased panel-level internal consistency and produced convergence across panels.

Growing reliance on complex constructed response items has generated considerable interest in automated scoring solutions. Many of these solutions are described in the literature; however, relatively few studies have been published that "compare" automated scoring strategies. Here, comparisons are made among five strategies for machine-scoring examinee performances of computer-based case simulations, a complex item format used to assess physicians' patient-management skills as part of the Step 3 United States Medical Licensing Examination. These strategies utilize expert judgments to obtain various (a) case-specific or (b) generic scoring algorithms. The various compromises between efficiency, validity, and reliability that characterize each scoring approach are described and compared.

This study sought to determine the academic and professional outcomes of medical school graduates who failed the United States Licensing Examination Step 1 on the first attempt. This retrospective cohort study was based on pooled data from 2,003 graduates of six Midwestern medical schools in the classes of 1997-2002. Demographic, academic, and career characteristics of graduates who failed Step 1 on the first attempt were compared to graduates who initially passed. Fifty medical school graduates (2.5%) initially failed Step 1. Compared to graduates who initially passed Step 1, a higher proportion of graduates who initially failed Step 1 became primary care physicians (26/49 [53%] vs. 766/1,870 [40.9%]), were more likely at graduation to report intent to practice in underserved areas (28/50 [56%] vs. 419/1,939 [21.6%]), and more likely to take 5 or more years to graduate (11/50 [22.0%] vs. 79/1,953 [4.0%]). The relative risk of first attempt Step 1 failure for medical school graduates was 13.4 for African Americans, 7.4 for Latinos, 3.6 for matriculants greater than 22 years of age, 3.2 for women, and 2.3 for first generation college graduates. The relative risk of not being specialty board certified for those graduates who initially failed Step 1 was 2.2. Our observations regarding characteristics of graduates in our study cohort who initially failed Step 1 can inform efforts by medical schools to identify and assist students who are at particular risk of failing Step 1.

Anatomy instruction has evolved over the past two decades as many medical schools have undergone various types of curricular reform. To provide empirical evidence about whether or not curricular changes impact the acquisition and retention of anatomy knowledge, this study investigated the effect of variation in gross anatomy course hours, curricular approach (stand-alone versus integrated), and laboratory experience (dissection versus dissection and prosection) on USMLE Steps 1 and 2 Clinical Knowledge (CK) scores. Gross anatomy course directors at 54 United States schools provided information about their gross anatomy courses via an online survey (response rate of 42%). Survey responses were matched with USMLE scores for 6,411 examinees entering LCME-accredited schools in 2007 and taking Step 1 for the first time in 2009. Regression analyses were conducted to examine relationships between gross anatomy instructional characteristics and USMLE performance. Step 1 total scores, Step 1 gross anatomy sub-scores, and Step 2 CK scores were unrelated to instructional hours, controlling for MCAT scores. Examinees from schools with integrated curricula scored slightly lower on Steps 1 and 2 CK than those from stand-alone courses (effect sizes of 2.1 and 1.9 on score scales with SDs of 22 and 20, respectively). Examinees with dissection and prosection experience performed slightly better on Step 2 CK than examinees in courses with dissection only laboratories (effect size of 1.2). Results suggest variation in course hours is unrelated to performance on Steps 1 and 2 CK. Although differences were observed in relation to curricular approach and laboratory experience, effect sizes were small. (Contains 4 tables and 3 figures.)

This study investigated the extent to which the performance data format impacted data use in Angoff standard setting exercises. Judges from two standard settings (a total of five panels) were randomly assigned to one of two groups. The full-data group received two types of data: (1) the proportion of examinees selecting each option and (2) plots showing the proportion of examinees selecting the correct answer by deciles defined by total test score. The options-only group received only the option data. Results indicated that judgments in the full-data group were in substantially closer alignment with the empirical data than those in the options-only group. This suggests that either the decile data alone or the combination of both pieces of data leads to a greater reliance on the data. The results are discussed from the perspective of the validity/credibility of the resulting cut scores. (Contains 6 figures and 6 tables.)

Although a few studies report sizable score gains for examinees who repeat performance-based assessments, research has not yet addressed the reliability and validity of inferences based on ratings of repeat examinees on such tests. This study analyzed scores for 8,457 single-take examinees and 4,030 repeat examinees who completed a 6-hour clinical skills assessment required for physician licensure. Each examinee was rated in four skill domains: data gathering, communication-interpersonal skills, spoken English proficiency, and documentation proficiency. Conditional standard errors of measurement computed for single-take and multiple-take examinees indicated that ratings were of comparable precision for the two groups within each of the four skill domains; however, conditional errors were larger for low-scoring examinees regardless of retest status. In addition, on their first attempt multiple-take examinees exhibited less score consistency across the skill domains but on their second attempt their scores became more consistent. Further, the median correlation between scores on the four clinical skill domains and three external measures was 0.15 for multiple-take examinees on their first attempt but increased to 0.27 for their second attempt, a value, which was comparable to the median correlation of 0.26 for single-take examinees. The findings support the validity of inferences based on scores from the second attempt. (Contains 3 figures, 2 tables and 2 notes.)