Search

Background: Validation of examinations is usually based on classical test theory. In this study, we analysed a key feature examination according to item response theory and compared the results with those of a classical test theory approach., Methods: Over the course of five years, 805 fourth-year undergraduate students took a key feature examination on general medicine consisting of 30 items. Analyses were run according to a classical test theory approach as well as using item response theory. Classical test theory analyses are reported as item difficulty, discriminatory power, and Cronbach's alpha while item response theory analyses are presented as item characteristics curves, item information curves and a test information function., Results: According to classical test theory findings, the examination was labelled as easy. Analyses according to item response theory more specifically indicated that the examination was most suited to identify struggling students. Furthermore, the analysis allowed for adapting the examination to specific ability ranges by removing items, as well as comparing multiple samples with varying ability ranges., Conclusions: Item response theory analyses revealed results not yielded by classical test theory. Thus, both approaches should be routinely combined to increase the information yield of examination data.

Background: Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome., Methods: Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities., Results: Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values., Conclusions: Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

Background: Evaluation is an integral part of curriculum development in medical education. Given the peculiarities of bedside teaching, specific evaluation tools for this instructional format are needed. Development of these tools should be informed by appropriate frameworks. The purpose of this study was to develop a specific evaluation tool for bedside teaching based on the Stanford Faculty Development Program's clinical teaching framework., Methods: Based on a literature review yielding 47 evaluation items, an 18-item questionnaire was compiled and subsequently completed by undergraduate medical students at two German universities. Reliability and validity were assessed in an exploratory full information item factor analysis (study one) and a confirmatory factor analysis as well as a measurement invariance analysis (study two)., Results: The exploratory analysis involving 824 students revealed a three-factor structure. Reliability estimates of the subscales were satisfactory (α = 0.71-0.84). The model yielded satisfactory fit indices in the confirmatory factor analysis involving 1043 students., Discussion: The new questionnaire is short and yet based on a widely-used framework for clinical teaching. The analyses presented here indicate good reliability and validity of the instrument. Future research needs to investigate whether feedback generated from this tool helps to improve teaching quality and student learning outcome.

Background: Financial incentives are effective in moderating physician and patient behaviour, but they have not been studied in the context of medical education., Aim: This study assessed whether financial incentives can motivate students to acquire electrocardiogram (ECG) interpretation skills., Methods: Students enrolled for a cardio-respiratory teaching module (n = 121) were randomised to an intervention (financial incentive) or a control (book voucher raffle) condition. All students took three validated exams of ECG interpretation skills (at module entry, module exit and seven weeks later). Only the exit exam was financially incentivised in the intervention group. The primary outcome was the proportion of students who correctly identified ≥60% of clinically important diagnoses in the exit exam., Results: Financial incentives more than doubled the odds of correctly identifying ≥60% of diagnoses in the exit exam (adjusted odds ratio 2.44, 95% confidence interval 1.05-5.67) and significantly increased student learning time. However, there was no significant effect on performance levels in the retention exam., Conclusions: Financial incentives increase reported learning time and examination results in the short-term. The lack of a sustained effect on performance suggests that financial incentives may foster a superficial or strategic rather than a deep approach to learning.