Search

Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.

Developed a method to estimate the cutscore precisions for empirical standard-setting methods by using resampling. Illustrated the method with two actual datasets consisting of 86 Dutch medical residents and 155 Canadian medical students taking objective structured clinical examinations. Results show the applicability of the method. (SLD)

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA. [ABSTRACT FROM AUTHOR]

Objective To determine the effectiveness of single-point benchmarking and longitudinal benchmarking for inter-school educational evaluation. Methods We carried out a mixed, longitudinal, cross-sectional study using data from 24 annual measurement moments (4 tests × 6 year groups) over 4 years for 4 annual progress tests assessing the graduation-level knowledge of all students from 3 co-operating medical schools. Participants included undergraduate medical students (about 5000) from 3 medical schools. The main outcome measures involved between-school comparisons of progress test results based on different benchmarking methods. Results Variations in relative school performance across different tests and year groups indicate instability and low reliability of single-point benchmarking, which is subject to distortions as a result of school-test and year group-test interaction effects. Deviations of school means from the overall mean follow an irregular, noisy pattern obscuring systematic between-school differences. The longitudinal benchmarking method results in suppression of noise and revelation of systematic differences. The pattern of a school’s cumulative deviations per year group gives a credible reflection of the relative performance of year groups. Conclusions Even with highly comparable curricula, single-point benchmarking can result in distortion of the results of comparisons. If longitudinal data are available, the information contained in a school’s cumulative deviations from the overall mean can be used. In such a case, the mean test score across schools is a useful benchmark for cross-institutional comparison. [ABSTRACT FROM AUTHOR]

Objective To determine whether items of progress tests used for inter-curriculum comparison favour students from the medical school where the items were produced (i.e. whether the origin bias of test items is a potential confounder in comparisons between curricula). Methods We investigated scores of students from different schools on subtests consisting of progress test items constructed by authors from the different schools. In a cross-institutional collaboration between 3 medical schools, progress tests are jointly constructed and simultaneously administered to all students at the 3 schools. Test score data for 6 consecutive progress tests were investigated. Participants consisted of approximately 5000 undergraduate medical students from 3 medical schools. The main outcome measure was the difference between the scores on subtests of items constructed by authors from 2 of the collaborating schools (subtest difference score). Results The subtest difference scores showed that students obtained better results on items produced at their own schools. This effect was more pronounced in Years 2–5 of the curriculum than in Year 1, and diminished in Year 6. Conclusions Progress test items were subject to origin bias. As a consequence, all participating schools should contribute equal numbers of test items if tests are to be used for valid and fair inter-curriculum comparisons. [ABSTRACT FROM AUTHOR]

Purpose To ascertain whether the grade point average (GPA) of school-leaving examinations is related to study success, career development and scientific performance. The problem of restriction of range was expected to be partially reduced due to the use of a national lottery system weighted in favour of students with higher GPAs. Method We studied the students ( n = 398) admitted to the Faculty of Medicine, University of Groningen, the Netherlands in 1982 and 1983. Data concerning drop-out and study progress were derived from the student administration. Data about career development were obtained from annual interviews with graduates ( n = 318) between 1993 and 2000. Literature searches yielded data concerning scientific performance. Multiple linear regression and logistic regression were used to analyse the data. The variables ‘gender’ and ‘cohort’ and their interaction were included in the analyses to account for variation in the general level of the dependent variable and the effect of GPA on the dependent variable. Results GPA scores had no effect on drop-out rate. High GPA scores were associated with significantly less time to graduation, more chance of a preferred placement for specialist training and higher scientific output. GPA was not related to income. Gender differences were found for study duration and scientific output. Women graduated earlier and published less. Conclusion The GPA of school-leaving examinations was found to be related to study success, career development and scientific performance. In this study the usual problem of restriction of range was shown to be absent. The weighted lottery procedure even resulted in an over-dispersion of candidates relative to the applicants. The resulting effect sizes were in agreement with those reported in other studies. [ABSTRACT FROM AUTHOR]

Context The portfolio is becoming increasingly accepted as a valuable tool for learning and assessment. The validity of portfolio assessment, however, may suffer from bias due to irrelevant qualities, such as lay-out and writing style. We examined the possible effects of such qualities in a portfolio programme aimed at stimulating Year 1 medical students to reflect on their professional and personal development. In later curricular years, this portfolio is also used to judge clinical competence. Methods We developed an instrument, the Portfolio Analysis Scoring Inventory, to examine the impact of form and content aspects on portfolio assessment. The Inventory consists of 15 items derived from interviews with experienced mentors, the literature, and the criteria for reflective competence used in the regular portfolio assessment procedure. Forty portfolios, selected from 231 portfolios for which ratings from the regular assessment procedure were available, were rated by 2 researchers, independently, using the Inventory. Regression analysis was used to estimate the correlation between the ratings from the regular assessment and those resulting from the Inventory items. Results Inter-rater agreement ranged from 0.46 to 0.87. The strongest predictor of the variance in the regular ratings was ‘quality of reflection’ (R 0.80; R2 66%). No further items accounted for a significant proportion of variance. Irrelevant items, such as writing style and lay-out, had negligible effects. Conclusions The absence of an impact of irrelevant criteria appears to support the validity of the portfolio assessment procedure. Further studies should examine the portfolio's validity for the assessment of clinical competence. [ABSTRACT FROM AUTHOR]

For more than two decades the Medical School in Maastricht, the Netherlands, has used simulated patients (SPs) to provide students with opportunities to practise their skills in communication and physical examination. In this educational setting a student meets a SP in a videotaped session. Feedback by the SP to the student at the end of the session is considered an important educational feature. We found no instruments to assess individual SP performance during those sessions. To develop a valid, reliable and feasible instrument to evaluate the performance of SPs. The content of the instrument was validated through interviews with students, teachers and experts who are involved with SPs. They were asked to indicate key features of good SP performance. Based on the interviews, a written checklist was developed to measure individual SP performance. The instrument was evaluated in a regular SP session at the medical school, involving 152 students and their teachers. All interviewees considered the scale to be satisfactory and the instrument to be valid. The feasibility and reliability of the checklists were investigated using the data of 398 returned checklists. Cronbach's alpha was found to be 0.73. Generalizability analysis showed that 12 completed checklists were required to obtain a reliable assessment of one SP. The Maastricht Assessment of Simulated Patients (MaSP) appears to be a valid, reliable and feasible tool to assess the performance of SPs in an educational setting. [ABSTRACT FROM AUTHOR]

Objectives: This study explores the optimal focus for negotiation skills development training by investigating how often medical residents negotiate in practice, and how they perceive the effectiveness of their negotiation capabilities., Methods: An exploratory study was performed using a questionnaire regarding the medical residents' working environment, negotiation frequency, knowledge and skills using a 5-point Likert scale, multiple choice questions and open questions. Exploratory factor analysis with principal component analysis, varimax rotation, reliability analysis, and content analysis were used to reduce the number of variables. Descriptive and interferential statistics and multiple regression analysis were used to analyze the data., Results: We analyzed the responses of 60 medical residents. The findings showed that the perceived development of their negotiation knowledge (M=3.06, SD=0.83) was less than their negotiation skills (M=3.69, SD=0.47). Their attitude during negotiations, especially females, differed substantially in the interactions with nurses than with their supervisors. Medical residents with more working experience, better negotiation skills or who worked in hierarchical environments negotiated more frequently with their supervisors. Medical residents with better collaboration skills and negotiation knowledge demonstrated better negotiation skills., Conclusions: This study underlines medical residents' need for negotiation training. In addition to the basic negotiation knowledge and skills, training programs in negotiation should focus on the medical residents' awareness of their attitudes during negotiations, combining the assertiveness shown in interactions with supervisors with the empathy and emotional engagement present in interactions with nurses.  Furthermore, attention should be paid to the influence of the environmental hierarchy on negotiation skill development.

Background: Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program., Methods: We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students., Results: Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs., Conclusions: The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

Background: Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships., Methods: We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement., Results: The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4)., Conclusions: This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.

General practitioners (GPs) are increasingly called upon to identify patients at risk for hereditary cancers, and their genetic competencies need to be enhanced. This article gives an overview of a research project on how to build effective educational modules on genetics, assessed by randomized controlled trials (RCTs), reflecting the prioritized educational needs of primary care physicians. It also reports on an ongoing study to investigate long-term increase in genetic consultation skills (1-year follow-up) and interest in and satisfaction with a supportive website on genetics among GPs. Three oncogenetics modules were developed: an online Continuing Professional Development (G-eCPD) module, a live genetic CPD module, and a "GP and genetics" website (huisartsengenetica.nl) providing further genetics information applicable in daily practice. Three assessments to evaluate the effectiveness (1-year follow-up) of the oncogenetic modules were designed: 1.An online questionnaire on self-reported genetic competencies and changes in referral behaviour, 2.Referral rates from GPs to clinical genetics centres and 3.Satisfaction questionnaire and visitor count analytics of supportive genetics website. The setting was Primary care in the Netherlands and three groups of study participants were included in the reported studies:. Assessment 1. 168 GPs responded to an email invitation and were randomly assigned to an intervention or control group, evaluating the G-eCPD module (n = 80) or the live module (n = 88). Assessment 2. Referral rates by GPs were requested from the clinical genetics centres, in the northern and southern parts of the Netherlands (Amsterdam and Maastricht), for the two years before (2010 [n = 2510] and 2011 [n = 2940]) and the year after (2012 [n = 2875]) launch of the oncogenetics CPD modules and the website. Assessment 3. Participants of the website evaluation were all recruited online. When they visited the website during the month of February 2013, a pop-up invitation came up. Of the 1350 unique visitors that month, only 38 completed the online questionnaire. Main outcomes measure showed long-term (self-reported) genetic consultation skills (i.e. increased genetics awareness and referrals to clinical genetics centres) among GPs who participated in the oncogenetic training course, and interest in and satisfaction with the supportive website. 42 GPs (52%) who previously participated in the G-eCPD evaluation study and 50 GPs (57%) who participated in the live training programme responded to the online questionnaire on long-term effects of educational outcome. Previous RCTs showed that the genetics CPD modules achieved sustained improvement of oncogenetic knowledge and consultation skills (3-months follow-up). Participants of these RCTs reported being more aware of genetic problems long term; this was reported by 29 GPs (69%) and 46 GPs (92%) participating in the G-eCPD and live module evaluation studies, respectively (Chisquare test, p<0.005). One year later, 68% of the respondents attending the live training reported that they more frequently referred patients to the clinical genetics centres, compared to 29% of those who attended the online oncogenetics training (Chisquare test, p<0.0005). However, the clinical genetics centres reported no significant change in referral numbers one year after the training. Website visitor numbers increased, as did satisfaction, reflected in a 7.7 and 8.1 (out of 10) global rating of the website (by G-eCPD and live module participants, respectively). The page most often consulted was "family tree drawing". Self-perceived genetic consultation skills increased long-term and GPs were interested in and satisfied with the supportive website. Further studies are necessary to see whether the oncogenetics CPD modules result in more efficient referral. The results presented suggest we have provided a flexible and effective framework to meet the need for effective educational programmes for non-geneticist healthcare providers, enabling improvement of genetic medical care.

Background: Teachers' conceptions of learning and teaching are partly unconscious. However, they are critical for the delivery of education and affect students' learning outcomes. Lasting changes in teaching behaviour can only be realized if conceptions of teachers have been changed accordingly. Previously we constructed a questionnaire named COLT to measure conceptions. In the present study, we investigated if different teacher profiles could be assessed which are based on the teachers' conceptions. These teacher profiles might have implications for individual teachers, for faculty development activities and for institutes. Our research questions were: (1) Can we identify teacher profiles based on the COLT? (2) If so, how are these teacher profiles associated with other teacher characteristics?, Methods: The COLT questionnaire was sent electronically to all teachers in the first three years of the undergraduate curriculum of Medicine in two medical schools in the Netherlands with student-centred education. The COLT (18 items, 5 point Likert scales) comprises three scales: 'teacher centredness', 'appreciation of active learning' and 'orientation to professional practice'. We also collected personal information about the participants and their occupational characteristics. Teacher profiles were studied using a K-means cluster analysis and calculating Chi squares., Results: The response rate was 49.4% (N = 319/646). A five-cluster solution fitted the data best, resulting in five teacher profiles based on their conceptions as measured by the COLT. We named the teacher profiles: Transmitters (most traditional), Organizers, Intermediates, Facilitators and Conceptual Change Agents (most modern). The teacher profiles differed from each other in personal and occupational characteristics., Conclusions: Based on teachers' conceptions of learning and teaching, five teacher profiles were found in student-centred education. We offered suggestions how insight into these teacher profiles might be useful for individual teachers, for faculty development activities and for institutes and departments, especially if involved in a curriculum reform towards student-centred education.

Purpose: Studies that compare the effects of problem-based and conventional medical curricula on student performance generally show no significant differences. However, curriculum comparison studies are at best quasi-experimental, so they are prone to forms of selection bias. The purpose of this study was to reanalyze data from such comparisons by controlling for two potential biases: differential student attrition and differential exposure., Method: The authors reanalyzed 104 previously published comparisons involving a single, problem-based medical school in the Netherlands (Maastricht University's medical school), using student attrition and study duration data from this school and the schools with which it was compared. The authors removed bias by reequalizing the comparison groups in terms of attrition and study duration., Results: The uncorrected data showed no differences between problem-based and conventional curricula: Mean effect sizes as expressed by Cohen d were 0.02 for medical knowledge and 0.07 for diagnostic reasoning. However, the reanalysis demonstrated medium-level effect sizes favoring the problem-based curriculum. After corrections for attrition and study duration, the mean effect size for knowledge acquisition was 0.31 and for diagnostic reasoning was 0.51., Conclusions: Effects of the Maastricht problem-based curriculum were masked by differential attrition and differential exposure in the original studies. Because this school has been involved in many studies included in influential literature reviews published in the past 20 years, the authors' findings have implications for the assessment of the value of problem-based learning put forward by these reviews.

Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself). Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category 'Dealing with oneself' consisted of descriptions of a student's attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase.

Purpose: Clinical teaching's importance in the medical curriculum has led to increased interest in its evaluation. Instruments for evaluating clinical teaching must be theory based, reliable, and valid. The Maastricht Clinical Teaching Questionnaire (MCTQ), based on the theoretical constructs of cognitive apprenticeship, elicits evaluations of individual clinical teachers' performance at the workplace. The authors investigated its construct validity and reliability, and they used the underlying factors to test a causal model representing effective clinical teaching., Method: Between March 2007 and December 2008, the authors asked students who had completed clerkship rotations in different departments of two teaching hospitals to use the MCTQ to evaluate their clinical teachers. To establish construct validity, the authors performed a confirmatory factor analysis of the evaluation data, and they estimated reliability by calculating the generalizability coefficient and standard error measurement. Finally, to test a model of the factors, they fitted a structural linear model to the data., Results: Confirmatory factor analysis yielded a five-factor model which fit the data well. Generalizability studies indicated that 7 to 10 student ratings can produce reliable ratings of individual teachers. The hypothesized structural linear model underlined the central roles played by modeling and coaching (mediated by articulation)., Conclusions: The MCTQ is a valid and reliable evaluation instrument, thereby demonstrating the usefulness of the cognitive apprenticeship concept for clinical teaching during clerkships. Furthermore, a valuable model of clinical teaching emerged, highlighting modeling, coaching, and stimulating students' articulation and exploration as crucial to effective teaching at the clinical workplace.

Introduction: Models for training urological procedures without burdening patients are available at varying costs. We examined the value of training on a low-fidelity model in addition to training on a high-fidelity simulator in a cystoscopy training program., Methods: Thirty-two medical students were randomized to an intervention and a control group. The former started by performing cystoscopy on a low-cost, low-fidelity, glass globe model before moving on to training on the URO Mentor (UM), a computerized simulator. The control group took part in the same UM training program but not in the low-fidelity training. Performance on UM was assessed by a global rating score, percentage of correctly inspected areas of the bladder (% inspected areas), time, and number of traumas caused., Results: The intervention group had generally higher scores. Its global rating score on task 1 was significantly higher than that of the control group (Mann-Whitney U test, P = 0.046, effect size 0.6) and the group also scored higher, albeit not significantly, on time and % inspected areas. All students said they valued training with UM, but the appreciation of the intervention group was stronger (mean 8.9 vs. 8.1 on a scale from 1 to 10, P = 0.017, effect size 1.8)., Conclusion: A low-fidelity glass globe model seemed to be an inexpensive educational tool to practice the first steps of cystoscopy. It may reduce training time on the UM simulator. The combined use of a low- and high-fidelity training model may provide an optimal learning effect.

The potential richness of the feedback for learners and teachers is one of the educational advantages of progress tests (PTs). Every test administration yields information on a student's knowledge level in each sub-domain of the test (cross-sectional information), and it adds a next point to the corresponding knowledge growth curve (longitudinal information). Traditional paper-based feedback has severe limitations and requires considerable effort from the learners to give meaning to the data. We reasoned that the PT data should be flexibly accessible in all pathways and with any available comparison data, according to the personal interest of the learner. For that purpose, a web-based tool (Progress test Feedback, the ProF system) was developed. This article presents the principles and features of the generated feedback and shows how it can be used. In addition to enhancement of the feedback, the ProF database of longitudinal PT-data also provides new opportunities for research on knowledge growth, and these are currently being explored.

Unlabelled: The learning environment of undergraduate research internships has received little attention, compared to postgraduate research training. This study investigates students' experiences with research internships, particularly the quality of supervision, development of research skills, the intellectual and social climate, infrastructure support, and the clarity of goals and the relationship between the experiences and the quality of students' research reports and their overall satisfaction with internships., Method: A questionnaire (23 items, a 5-point Likert scale) was administered to 101 Year five veterinary students after completion of a research internship. Multiple linear regression analyses were conducted with quality of supervision, development of research skills, climate, infrastructure and clarity of goals as independent variables and the quality of students' research reports and students' overall satisfaction as dependent variables., Results: The response rate was 79.2%. Students' experiences are generally positive. Students' experiences with the intellectual and social climate are significantly correlated with the quality of research reports whilst the quality of supervision is significantly correlated with both the quality of research reports and students' overall satisfaction with the internship., Conclusion: Both the quality of supervision and the climate are found to be crucial factors in students' research learning and satisfaction with the internship.

Objectives: This study set out to examine how much time students and teachers devote to different learning-oriented interactions during seminar sessions and students' and teachers' perceptions about the occurrence and desirability of these interactions., Methods: Students and teachers participating in eight seminar group sessions in Year 4 of an undergraduate veterinary curriculum completed an 11-item questionnaire which asked them to rate, on a 5-point Likert scale, the frequency of occurrence and level of desirability of three learning-oriented types of interaction: exploratory questioning; cumulative reasoning, and handling of conflict about knowledge. The questionnaire also invited positive and negative responses to aspects of group interactions and an overall mark (1-10) for the seminars and group interactions. Four group sessions were video-recorded and analysed using a coding scheme. The amount of time devoted to the different interactions was calculated., Results: Both students and teachers gave scores of 3.0-3.5 for frequency of occurrence of exploratory questioning and cumulative reasoning and < 3.0 for occurrence of handling of conflict about knowledge. The desired occurrences of all interaction types were significantly higher than the actual occurrences according to students and teachers. Teachers were responsible for the majority of the interactions (93%). The percentages of session time devoted to teacher-centred cumulative reasoning, exploratory questioning and handling of conflict about knowledge were 65.8%, 15.6% and 3.1%, respectively., Conclusions: Group interactions in seminar groups are dominated by the posing of questions by teachers to students. The moderate occurrence of group interactions as perceived by students and teachers may be explained by the inadequate preparation of teachers and students to stimulate group interactions.

Background: Research indicates that the quality of supervision strongly influences the learning of medical students in clinical practice. Clinical teachers need feedback to improve their supervisory skills. The available instruments either lack a clear theoretical framework or are not suitable for providing feedback to individual teachers. We developed an evaluation instrument based on the 'cognitive apprenticeship model'., Aim: The aim was to estimate the content validity of the developed instrument., Method: Item relevance was rated on a five-point scale (1 = highly irrelevant, 5 = highly relevant) by three groups of stakeholders in undergraduate clinical teaching: educationalists (N = 12), doctors (N = 16) and students (N = 12). Additionally, stakeholders commented on content, wording and omission of items., Results: The items were generally rated as very relevant (Mean = 4.3, SD = 0.38, response = 95%) and any differences between the stakeholder groups were small. The results led to elimination of 4 items, rewording of 13 items and addition of 1 item., Discussion: The cognitive apprenticeship model appears to offer a useful framework for the development of an evaluation instrument aimed at providing feedback to individual clinical teachers on the quality of student supervision. Further studies in larger populations will have to establish the instrument's statistical validity and generalizability.

Context: We addressed the assessment of professional behaviour in tutorial groups by investigating students' perceptions of the frequency and impact of critical incidents that impede this assessment and 5 factors underlying these critical incidents., Methods: A questionnaire asking students to rate the frequency and impact of 40 critical incidents relating to effective assessment of professional behaviour on a 5-point Likert scale was developed and sent to all undergraduate medical students in Years 2-4 of a 6-year undergraduate curriculum., Results: The response rate was 70% (n = 393). Important factors underlying critical incidents are: lack of effective interaction; lack of thoroughness; tutors' failure to confront students with unprofessional behaviour; lack of effort to find solutions, and lack of student motivation. Confirmatory factor analysis showed a good model fit. Because the relationship between frequency of occurrence and degree of impediment varies, the best information about the true impact of critical incidents and the underlying factors is provided by the product of frequency and degree of impediment. Frequency of occurrence remains stable and degree of impediment increases in Years 2-4., Conclusions: The results of this study can be used to design and improve faculty development programmes aimed at improving assessment of professional behaviour. Training programmes should motivate tutors by providing background information as to why and how sound assessment of professional behaviour is to be performed and encourage tutors to confront students with and discuss all aspects of professional behaviour, as well as provide appropriate feedback.

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.

Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE., Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making., Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment., Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.

An academic department of education serving the entire university and a strategic choice by the Faculty of Medicine to support educational innovation through education research are the historical cornerstones of the education research program of the University of Maastricht. Over the years, the department's initial exclusive research focus on the evaluation of problem-based learning has widened to include theory-based applied research covering the broad domain of education. The program focuses on themes: the learning of students and teachers, characteristics of powerful learning environments, and assessment and evaluation of learning and teaching. Although modest in terms of resources, the program is firmly anchored within the Faculty's organizational structure. Educational relevance and professional alignment are the most prominent determinants of the success of the program. These features sustain the institutional mission of educational excellence as well as the high ranking of the Faculty of Medicine's medical training program among the training programs of the Netherlands' medical schools. A break in this self-perpetuating mechanism--due either to internal politics or to staffing problems--forms the main risk factor for the continuation of the department.

Context: For more than two decades the Medical School in Maastricht, the Netherlands, has used simulated patients (SPs) to provide students with opportunities to practise their skills in communication and physical examination. In this educational setting a student meets a SP in a videotaped session. Feedback by the SP to the student at the end of the session is considered an important educational feature. We found no instruments to assess individual SP performance during those sessions., Objective: To develop a valid, reliable and feasible instrument to evaluate the performance of SPs., Methods: The content of the instrument was validated through interviews with students, teachers and experts who are involved with SPs. They were asked to indicate key features of good SP performance. Based on the interviews, a written checklist was developed to measure individual SP performance. The instrument was evaluated in a regular SP session at the medical school, involving 152 students and their teachers., Main Outcomes: All interviewees considered the scale to be satisfactory and the instrument to be valid. The feasibility and reliability of the checklists were investigated using the data of 398 returned checklists. Cronbach's alpha was found to be 0.73. Generalizability analysis showed that 12 completed checklists were required to obtain a reliable assessment of one SP., Conclusions: The Maastricht Assessment of Simulated Patients (MaSP) appears to be a valid, reliable and feasible tool to assess the performance of SPs in an educational setting.