Search

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was 0.79 (Documentation = 0.63, Differential Diagnosis = 0.90, Justification = 0.48, and Workup = 0.54). The majority of score variance was due to case specificity (13%) and case-task specificity (31%), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5%). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.

Background: SNAPPS is a learner-centered approach to case presentations that was shown, in American studies, to facilitate the expression of clinical reasoning and uncertainties in the outpatient setting. Aim: To evaluate the SNAPPS technique in an Asian setting. Methods: We conducted a quasi-experimental trial comparing the SNAPPS technique to the usual-and-customary method of case presentations for fifth-year medical students in an ambulatory internal medicine clerkship rotation at Khon Kaen University, Thailand. We created four experimental groups to test main and maturation effects. We measured 12 outcomes at the end of the rotations: total, summary, and discussion presentation times, number of basic clinical findings, summary thoroughness, number of diagnoses in the differential, number of justified diagnoses, number of basic attributes supporting the differential, number of student-initiated questions or discussions about uncertainties, diagnosis, management, and reading selections. Results: SNAPPS users (90 case presentations), compared with the usual group (93 presentations), had more diagnoses in their differentials (1.81 vs. 1.42), more basic attributes to support the differential (2.39 vs. 1.22), more expression of uncertainties (6.67% vs. 1.08%), and more student-initiated reading selections (6.67% vs. 0%). Presentation times did not differ between groups (12 vs. 11.2 min). There were no maturation effects detected. Conclusions: The use of the SNAPPS technique among Thai medical students during their internal medicine ambulatory care clerkship rotation did facilitate the expression of their clinical reasoning and uncertainties. More intense student-preceptor training is needed to better foster the expression of uncertainties. [ABSTRACT FROM AUTHOR]

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists. [ABSTRACT FROM AUTHOR]