Search

The assessment of workplace learning by educators at the workplace is a complex and inherently social process, as the workplace is a participatory learning environment. We therefore propose seeing assessment as a process of judgment embedded in a community of practice and to this purpose use the philosophy of inferentialism to unravel the judgment process of workplace educators by seeing it as an interrelated system of judgments, actions and reasons. Focussing on the unfolding of a process, we applied a longitudinal holistic case study design. Results show that educators are engaged in a constant judgment process during which they use multiple and adaptive frames of reference when forming their judgment about students. They construct an overarching image of students that develops throughout the placement, and their judgments about students go hand in hand with their actions relating to fostering independent practice. [ABSTRACT FROM AUTHOR]

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory. [ABSTRACT FROM AUTHOR]

Introduction The Clinical Teaching Effectiveness Instrument (CTEI) was developed to evaluate the quality of the clinical teaching of educators. Its authors reported evidence supporting content and criterion validity and found favourable reliability findings. We tested the validity and reliability of this instrument in a European context and investigated its reliability as an instrument to evaluate the quality of clinical teaching at group level rather than at the level of the individual teacher. Methods Students participating in a surgical clerkship were asked to fill in a questionnaire reflecting a student−teacher encounter with a staff member or a resident. We calculated variance components using the ur genova program. For individual score interpretation of the quality of clinical teaching the standard error of estimate was calculated. For group interpretation we calculated the root mean square error. Results The results did not differ statistically between staff and residents. The average score was 3.42. The largest variance component was associated with rater variance. For individual score interpretation a reliability of > 0.80 was reached with 7 ratings or more. To reach reliable outcomes at group level, 15 educators or more were needed with a single rater per educator. Discussion The required sample size for appraisal of individual teaching is easily achievable. Reliable findings can also be obtained at group level with a feasible sample size. The results provide additional evidence of the reliability of the CTEI in undergraduate medical education in a European setting. The results also showed that the instrument can be used to measure the quality of teaching at group level. [ABSTRACT FROM AUTHOR]

The practice of assessment is governed by an interesting paradox. On the one hand good assessment requires substantial resources which may exceed the capacity of a single institution and we have reason to doubt the quality of our in-house examinations. On the other hand, our parsimonity with regard to our resources makes us reluctant to pool efforts and share our test material. This paper reports on an initiative to share test material across different medical schools. Three medical schools in The Netherlands have successfully set up a partnership for a specific testing method: progress testing. At present, these three schools collaboratively produce high-quality test items. The jointly produced progress tests are administered concurrently by these three schools and one other school, which buys the test. The steps taken in establishing this partnership are described and results are presented to illustrate the unique sort of information that is obtained by cross-institutional assessment. In addition, plans to improve test content and procedure and to expand the partnership are outlined. Eventually, the collaboration may even extend to other test formats. This article is intended to give evidence of the feasibility and exciting potential of between school collaboration in test development and test administration. Our experiences have demonstrated that such collaboration has excellent potential to combine economic benefit with educational advantages, which exceed what is achievable by individual schools. [ABSTRACT FROM AUTHOR]

Simulation-based testing methods have been developed to meet the need for assessment procedures that are both authentic and well-structured. It is widely acknowledged that, although the authenticity of a procedure may be a contributing factor to its validity, authenticity alone never is a sufficient factor. In this paper we describe the mainstream development of various simulation-based approaches, with their strengths and weaknesses. The purpose is not to provide a review based on an extensive meta-analysis but to present crucial factors in the development of these methods and their implications for current and future developments. The description of these simulation-based instruments uses a subdivision according to the layers of Miller's pyramid. Written and computer-based simulations are aimed at measuring the ‘knows how’ layer, observation-based techniques such as standardised patient-based examinations and objective structured clinical examinations target the ‘shows how’ layer and performance practice measures assess performance at the ‘does’ layer. In all simulations, case specificity was found to pose the most prominent threat to reliability, while too much structure threatened to trivialise the assessment. The conclusion is that authentic and reliable assessment is predicated on a wise balance between efficiency and adequate content sampling. [ABSTRACT FROM AUTHOR]

The article discusses various reports published within the issue, including one on the symposium that addressed the utility of European licensing examination by Mereke Gorsira, another by Ronald Harden on the five myths around the benefits of licensure examination, and one by Peter McCrorie and Katherine Boursicot exploring the graduating assessment systems across Great Britain in terms of testing time, content, format and standard setting procedures.

Introduction: Programmatic assessment (PA) is an approach to assessment aimed at optimizing learning which continues to gain educational momentum. However, the theoretical underpinnings of PA have not been clearly described. An explanation of the theoretical underpinnings of PA will allow educators to gain a better understanding of this approach and, perhaps, facilitate its use and effective implementation. The purpose of this article is twofold: first, to describe salient theoretical perspectives on PA; second to examine how theory may help educators to develop effective PA programs, helping to overcome challenges around PA. Results: We outline a number of learning theories that underpin key educational principles of PA: constructivist and social constructivist theory supporting meaning making, and longitudinality; cognitivist and cognitive development orientation scaffolding the practice of a continuous feedback process; theory of instructional design underpinning assessment as learning; self-determination theory (SDT), self-regulation learning theory (SRL), and principles of deliberate practice providing theoretical tenets for student agency and accountability. Conclusion: The construction of a plausible and coherent link between key educational principles of PA and learning theories should enable educators to pose new and important inquiries, reflect on their assessment practices and help overcome future challenges in the development and implementation of PA in their programs. [ABSTRACT FROM AUTHOR]

Vocational education aims at preparing students for their future as professionals; thus, students are often required to undergo a form of on-the-job learning. Assessment of professional performance during on-the-job learning is not based on single observations, but on prolonged participation at the workplace culminating in judgement by a workplace educator. However, little is known about how educators reach a judgement. This paper, therefore, aims at exploring how educators inform their judgements about students' professional performance. We consider the workplace to be a participatory learning environment, which entails that judgement about professional performance should be made in relation to the community students participated in and that the educator from that community has a central role. Our study takes an explorative approach based on interviews and uses template analysis to generate insight. Findings revealed that educators use multiple strategies to gather and process information about students to inform their judgement. Findings also showed that assessment starts at the first meeting, when the educator forms a first impression of the student and judges whether or not the student is suitable for the job. Educators at the workplace also showed that they are aware of the context in which they form their judgement. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the authors discusses the papers within the issue on topics including a model for an assessment program, the assessment program of the Cleveland Clinic Lerner College of Medicine, and the implementation of the assessment program model in a workplace setting.

Introduction: For health professionals, the development of insight into their performance is vital for safe practice, professional development and self-regulation. This study investigates whether the development of dental trainees' insight, when provided with external feedback on performance, can be assessed using a single criterion on a simple global ratings form such as the Longitudinal Evaluation of Performance or Mini Clinical Evaluation Exercise. Methods: Postgraduate dental trainees (N = 139) were assessed using this tool on a weekly basis for 6 months. Regression analysis of the data was carried out using SPSS, and a short trainer questionnaire was implemented to investigate feasibility. Results: Ratings for insight were shown to increase with time in a similar manner to the growth observed in other essential skills. The gradient of the slope for growth of insight was slightly less than that of the other observed skills. Trainers were mostly positive about the new criterion assessing trainees' insight, although the importance of training for trainers in this process was highlighted. Discussion: Our data suggest that practitioners' insight into their performance can be developed with experience and regular feedback. However, this is most likely a complex skill dependent on a number of intrinsic and external factors. Conclusion: The development of trainees' insight into their performance can be assessed using a single criterion on a simple global ratings form. The process involves no additional burden on evaluators in terms of their time or cost, and promotes best practice in the provision of feedback for trainees. [ABSTRACT FROM AUTHOR]

The article presents an abstract of the article "Medical Education: Challenges for Educationalists," by L.W.T. Schulwirth and C.P.M. van der Vleuten.

Objectives The clinical learning environment is an influential factor in work-based learning. Evaluation of this environment gives insight into the educational functioning of clinical departments. The Postgraduate Hospital Educational Environment Measure (PHEEM) is an evaluation tool consisting of a validated questionnaire with 3 subscales. In this paper we further investigate the psychometric properties of the PHEEM. We set out to validate the 3 subscales and test the reliability of the PHEEM for both clerks (clinical medical students) and registrars (specialists in training). Methods Clerks and registrars from different hospitals and specialties filled out the PHEEM. To investigate the construct validity of the 3 subscales, we used an exploratory factor analysis followed by varimax rotation, and a cluster analysis known as Mokken scale analysis. We estimated the reliability of the questionnaire by means of variance components according to generalisability theory. Results A total of 256 clerks and 339 registrars filled out the questionnaire. The exploratory factor analysis plus varimax rotation suggested a 1-dimensional scale. The Mokken scale analysis confirmed this result. The reliability analysis showed a reliable outcome for 1 department with 14 clerks or 11 registrars. For multiple departments 3 respondents combined with 10 departments provide a reliable outcome for both groups. Discussion The PHEEM is a questionnaire measuring 1 dimension instead of the hypothesised 3 dimensions. The sample size required to achieve a reliable outcome is feasible. The instrument can be used to evaluate both single and multiple departments for both clerks and registrars. [ABSTRACT FROM AUTHOR]

Introduction: A framework for the definition of generic objectives for community-based education (CBE) was developed for undergraduate medical programmes, particularly for developing countries. To probe the validity of the set of CBE objectives generated by this approach, opinions from a wider audience were sought . Method: Questionnaires were sent to 72 medical schools in 36 developing countries. Half of the addressees were randomly drawn from the list of institutional members of The Network: Toward Unity For Health (TUFH) and stratified according to developing countries. Another 36 medical schools were randomly drawn from non-Network: TUFH schools from the same country where the selected Network: TUFH addressee was located, or from a neighbouring country . Results: A total of 43 medical schools responded to the questionnaire (60% response rate), 31 out of the 36 addressed Network: TUFH members (86%) and 12 out of 36 addressed non-Network: TUFH schools (33%). Out of all 43 respondents 39 (91%) had implemented CBE in their curricula. Opinions of Network: TUFH and non-Network: TUFH schools on the framework and the generic objectives were not significantly different. Out of the 21 proposed objectives, 17 were scored as relevant by 75% or over of all responders and one out of the four objectives considered to be less relevant by the responders was deleted . Conclusion: A framework to develop generic CBE objectives and a derived set of 21 objectives were modified based on input by 43 medical schools residing in developing countries distributed all over the globe. The outcome is a validated set of 20 generic objectives for CBE programmes in developing countries . [ABSTRACT FROM AUTHOR]

Context: There has been an increase in the number of medical schools implementing community-based educational (CBE) programmes. However, there are doubts whether CBE programmes are appropriately implemented. As a case study, the CBE programme of the Medical Faculty of Diponegoro University (MFDU) in Semarang, Indonesia was evaluated . Objectives: To acquire MFDU's students' opinions on their CBE programme as part of a comprehensive evaluation of that programme, and to generate recommendations for improvement of MFDU's CBE programme . Methods: Coles and Grant's model for curriculum evaluation was applied. This model recommends triangulation of data generated from comparison of the programme as designed “on paper”, as implemented “in action” by the faculty, and “as experienced” by the students. To assess the curriculum as experienced by the students, direct participatory observation was performed and focus group discussions were conducted to collect students' opinions . Results: Students specifically signalled: (1) frequent overlap of lectures given during the CBE clerkship and in the previous part of the curriculum, (2) mismatch between their activities in the community and the community's felt health needs, (3) greater benefits of the CBE programme for Primary Health Care (PHC) centres and students than for target communities, and (4) incidental defective co-operation between students and health providers or community health workers . Conclusion: Students' opinions yielded more in-depth information on the CBE programme evaluated and facilitated formulation of recommendations for improvement of that CBE programme . [ABSTRACT FROM AUTHOR]

Context: A growing number of health professions schools have implemented programmes for community-based education (CBE) for their students. There are indications, however, that particularly in developing countries, CBE programmes are not always optimally implemented or sustained. Objective: To test the suitability of an established method for curriculum evaluation, combined with a set of generic objectives for CBE programmes, for evaluation of CBE programmes. Methods: As a case study, Coles and Grant's model for curriculum evaluation was applied to the CBE programme of the Medical Faculty of Diponegoro University (MFDU) in Semarang, Indonesia. Document analysis yielded information on the programme on paper; participatory observation and staff interviews on the programme in action. In addition, MFDU's CBE programme was evaluated against a set of generic objectives for CBE programmes recently designed by us. Results: MFDU has created great opportunities for its CBE programme in which, however, also significant weaknesses were revealed. (1) In the community, much time was spent on formal teaching; (2) Students' work in the community was not jointly identified with community members regarding the community's felt health needs; (3) There was rarely continuity, and evaluation or follow-up of the students' work in the community; and (4) No systematic programme evaluations are carried out. Discussion: This evaluation study showed shortcomings in the implementation of MFDU's CBE programme. The major weaknesses identified point at an underutilization of the opportunities and potential jeopardization of the facilities in the community. On the other hand, more time is needed in the CBE programme to establish the health needs to be addressed jointly with the community and to assess the impact of activities undertaken. A thorough review of the CBE programme, perhaps taking the outcomes of this study into account, could turn MFDU's CBE programme into a fine example for other medical schools in Indonesia and beyond. Conclusion: Coles and Grant's method for curriculum evaluation proved suitable for evaluation of a CBE programme in a developing country. After additional comparison with a reference list of objectives for CBE programmes, reasoned suggestions for programme can be made. [ABSTRACT FROM AUTHOR]

Various measures have been introduced to enhance learning experiences in clerkships, generally with limited success. This study evaluated the impact of a multifaceted approach on the effectiveness of learning in a surgical clerkship. In accordance with results obtained in continuing medical education, several interventions were introduced simultaneously. We compared students' evaluations of the traditional surgical clerkship with those of the restructured clerkship. Two consecutive cohorts of students were asked to complete a questionnaire about the quality and quantity of their learning experiences. Cohort 1 ( n = 28) undertook the traditional clerkship and cohort 2 ( n = 72) the restructured clerkship. A Mann–Whitney test was used to compare outcomes between the 2 cohorts. There were few statistically significant differences between cohorts 1 and 2. Overall, quality indicators did not differ between the 2 cohorts. A short-term multifaceted intervention led to a slight increase in the performance of clinical skills and a slight decrease in time spent on activities of limited educational value. The intervention may have been too brief to produce substantial effects. Future interventions should also target teachers, including trainees, in order to assess their opinions and address their educational needs. [ABSTRACT FROM AUTHOR]

The availability of a framework for the definition of generic objectives for community-based education (CBE) programmes may assist in the rational design of objectives for specific CBE programmes. Factors impacting on community health from the perspective of a developing country were collected. Potential assistance from medical students to communities to improve their health status was determined. Competencies required in students to execute tasks in the community were defined and eventually educational objectives to develop these competencies in the students were established. Factors impacting on community health and activities of medical students in CBE programmes were identified by review of literature and Internet resources. Competencies desired for execution of tasks by students and educational objectives to develop these competencies were defined by us and checked against pertinent literature. A draft table representing the 4 elements of the framework was discussed by an international group of experts for external validation. A total of 26 factors impacting on community health were identified and clustered in 5 domains. Twenty-one generic objectives for CBE programmes were defined to develop the required competencies in students. Analogues of each of these 21 objectives were found in at least 1 publication specifying objectives for specific CBE programmes but none of these publications stated any objective not covered by our list of generic objectives. It proved possible to develop a framework to define generic objectives for CBE programmes. An example was elaborated from the perspective of a medical school in a developing country. [ABSTRACT FROM AUTHOR]

PurposeTo assess whether case-based questions elicit different thinking processes from factual knowledge-based questions. Method20 general practitioners (GPs) and 20 students solved case-based questions and matched factual knowledge-based questions while thinking aloud. Verbatim protocols were analysed. Five indicators were defined: extent of protocols; immediate responses; re-reading of information given in the stem or case after the question had been read; order of re-reading information, and type of consideration, i.e. ‘true–false’ type or ‘vector’, that is, a deliberation which has a magnitude and a direction. ResultsCases elicited longer protocols than factual knowledge questions. Students re-read more given information than GPs. GPs gave an immediate response on twice as many occasions as students. GPs re-ordered the case information, whereas students re-read the information in the order it was presented. This ordering difference was not found in the factual knowledge questions. Factual knowledge questions mainly led to ‘true–false’ considerations, whereas cases elicited mainly ‘vector’ considerations. ConclusionShort case-based questions lead to thinking processes which represent problem-solving ability better than those elicited by factual knowledge questions. [ABSTRACT FROM AUTHOR]

Background: Scores on an outcome measurement instrument depend on the type and settings of the instrument used, how instructions are given to patients, how professionals administer and score the instrument, etc. The impact of all these sources of variation on scores can be assessed in studies on reliability and measurement error, if properly designed and analyzed. The aim of this study was to develop standards to assess the quality of studies on reliability and measurement error of clinician-reported outcome measurement instruments, performance-based outcome measurement instrument, and laboratory values.Methods: We conducted a 3-round Delphi study involving 52 panelists.Results: Consensus was reached on how a comprehensive research question can be deduced from the design of a reliability study to determine how the results of a study inform us about the quality of the outcome measurement instrument at issue. Consensus was reached on components of outcome measurement instruments, i.e. the potential sources of variation. Next, we reached consensus on standards on design requirements (n = 5), standards on preferred statistical methods for reliability (n = 3) and measurement error (n = 2), and their ratings on a four-point scale. There was one term for a component and one rating of one standard on which no consensus was reached, and therefore required a decision by the steering committee.Conclusion: We developed a tool that enables researchers with and without thorough knowledge on measurement properties to assess the quality of a study on reliability and measurement error of outcome measurement instruments. [ABSTRACT FROM AUTHOR]

Abstract: Assessments are widely used in dental education to record the academic progress of students and ultimately determine whether they are ready to begin independent dental practice. Whilst some would consider this a “rite‐of‐passage” of learning, the concept of assessments in education is being challenged to allow the evolution of “assessment for learning.” This serves as an economical use of learning resources whilst allowing our learners to prove their knowledge and skills and demonstrating competence. The Association for Dental Education in Europe and the American Dental Education Association held a joint international meeting in London in May 2017 allowing experts in dental education to come together for the purposes of Shaping the Future of Dental Education. Assessment in a Global Context was one topic in which international leaders could discuss different methods of assessment, identifying the positives, the pitfalls and critiquing the method of implementation to determine the optimum assessment for a learner studying to be a healthcare professional. A post‐workshop survey identified that educators were thinking differently about assessment, instead of working as individuals providing isolated assessments; the general consensus was that a longitudinally orientated systematic and programmatic approach to assessment provide greater reliability and improved the ability to demonstrate learning. [ABSTRACT FROM AUTHOR]

Background:In clerkships, students are expected to self-regulate their learning. How clinical departments and their routine approach on clerkships influences students’ self-regulated learning (SRL) is unknown. Aim:This study explores how characteristic routines of clinical departments influence medical students’ SRL. Methods:Six focus groups including 39 purposively sampled participants from one Dutch university were organized to study how characteristic routines of clinical departments influenced medical students’ SRL from a constructivist paradigm, using grounded theory methodology. The focus groups were audio recorded, transcribed verbatim and were analyzed iteratively using constant comparison and open, axial and interpretive coding. Results:Students described that clinical departments influenced their SRL through routines which affected the professional relationships they could engage in and affected their perception of a department’s invested effort in them. Students’ SRL in a clerkship can be supported by enabling them to engage others in their SRL and by having them feel that effort is invested in their learning. Conclusions:Our study gives a practical insight in how clinical departments influenced students’ SRL. Clinical departments can affect students’ motivation to engage in SRL, influence the variety of SRL strategies that students can use and how meaningful students perceive their SRL experiences to be. [ABSTRACT FROM PUBLISHER]

A letter to the editor is presented in response to the progress testing material within a School of Medicine and between schools by Doctor Van der Vleuten and his colleagues.

Background: The development of reflective learning skills is a continuous process that needs scaffolding. It can be described as a continuum, with the focus of reflection differing in granularity from recent, concrete activities to global competency development. Aim: To explore learners' perceptions regarding the effects of two reflective writing activities designed to stimulate reflection at different degrees of granularity during clinical training. Methods: Totally 142 respondents (students and recent graduates) completed a questionnaire. Quantitative and qualitative data were triangulated. Results: Immediate reflection-on-action was perceived to be more valuable than delayed reflection-on-competency-development because it facilitated day-to-day improvement. Delayed reflection was perceived to facilitate overall self-assessment, self-confidence and continuous improvement, but this perception was mainly found among graduates. Detailed reflection immediately after a challenging learning experience and broad reflection on progress appeared to serve different learning goals and consequently require different arrangements regarding feedback and timing. Conclusions: Granularity of focus has consequences for scaffolding reflective learning, with immediate reflection on concrete events and reflection on long-term progress requiring different approaches. Learners appeared to prefer immediate reflection-on-action. [ABSTRACT FROM AUTHOR]

It has been shown that medical students have a higher rate of depressive symptoms than the general population and age- and sex-matched peers. This study aimed to estimate the prevalence of depressive symptoms among the medical students of a large school following a traditional curriculum and its relation to personal background variables. A descriptive-analytic, cross-sectional study was conducted in a medical school in Riyadh, Saudi Arabia. The medical students of King Saud University in Riyadh, Saudi Arabia, were screened for depressive symptoms using the 21-item Beck Depression Inventory. A high prevalence of depressive symptoms (48.2%%) was found, it was either mild (21%%), moderate (17%%), or severe (11%%). The presence and severity of depressive symptoms had a statistically significant association with early academic years ( p < 0.000) and female gender ( p < 0.002). The high prevalence of depressive symptoms is an alarming sign and calls for remedial action, particularly for the junior and female students. [ABSTRACT FROM AUTHOR]

Context Graduate medical education is currently facing major educational reforms. There is a lack of empirical evidence in the literature about the learning processes of residents in the clinical workplace. This qualitative study uses a ‘grounded theory’ approach to continue the development of a theoretical framework of learning in the clinical workplace by adding the perspective of attending doctors. Methods A total of 21 Dutch attending doctors involved in the training of residents in obstetrics and gynaecology participated in 1 of 3 focus group sessions. They discussed their perceptions of how residents learn and what factors influence residents' learning. A grounded theory approach was used to analyse the transcribed discussions. Results Three related themes emerged. The first concerned the central role of participation in work-related activities: according to attending doctors, residents learn by tackling the everyday challenges of clinical work. The second involved the ways in which attending doctors influence what residents learn from work-related activities. The final theme focused on attending doctors' views of the essential characteristics of residents and their development during residency. Conclusions Attending doctors' perspectives complement current insights derived from similar research among residents and from related literature. As part of an ongoing effort to further develop understanding of how residents learn, this study adds several ways in which attending doctors strive to combine guidance in both patient care and resident training. Furthermore, attending doctors' perspectives draw attention to other aspects of learning in the clinical workplace, such as the role of confidence and the balance between supervision and independence. [ABSTRACT FROM AUTHOR]

Objectives Medical councils worldwide have outlined new standards for postgraduate medical education. This means that residency programmes will have to integrate modern educational views into the clinical workplace. Postgraduate medical education is often characterised as a process of learning from experience. However, empirical evidence regarding the learning processes of residents in the clinical workplace is lacking. This qualitative study sought insight into the intricate process of how residents learn in the clinical workplace. Methods We carried out a qualitative study using focus groups. A grounded theory approach was used to analyse the transcribed tape recordings. A total of 51 obstetrics and gynaecology residents from teaching hospitals and affiliated general hospitals participated in 7 focus group discussions. Participants discussed how they learn and what factors influence their learning. Results An underlying theoretical framework emerged from the data, which clarified what happens when residents learn by doing in the clinical workplace. This framework shows that work-related activities are the starting point for learning. The subsequent processes of ‘interpretation’ and ‘construction of meaning’ lead to refinement and expansion of residents' knowledge and skills. Interaction plays an important role in the learning process. This is in line with both cognitivist and sociocultural views on learning. Conclusions The presented theoretical framework of residents' learning provides much needed empirical evidence for the actual learning processes of residents in the clinical workplace. The insights it offers can be used to exploit the full educational potential of the clinical workplace. [ABSTRACT FROM AUTHOR]

Sharing and collaboration relating to progress testing already takes place on a national level and allows for quality control and comparisons of the participating institutions. This study explores the possibilities of international sharing of the progress test after correction for cultural bias and translation problems. Three progress tests were reviewed and administered to 3043 Pretoria and 3001 Maastricht medical students. In total, 16% of the items were potentially biased and removed from the test items administered to the Pretoria students (9% due to translation problems; 7% due to cultural differences). Of the three clusters (basic, clinical and social sciences) the social sciences contained most bias (32%), basic sciences least (11%). The differences that were found, comparing the student results of both schools, seem a reflection of the deliberate accentuations that both curricula pursue. The results suggest that the progress test methodology provides a versatile instrument that can be used to assess medical schools across the world. Sharing of test material is a viable strategy and test outcomes are interesting and can be used in international quality control. [ABSTRACT FROM AUTHOR]

Assessment drives the educational behaviour of students and supervisors. Therefore, an assessment programme targeted at specific competencies may be expected to motivate supervisors and students to pay more attention to those competencies. In-training assessment (ITA) is regarded as a feasible method for assessing a broad range of competencies. Before and after the implementation of an ITA programme in an undergraduate Internal Medicine clerkship we surveyed students on the frequency of unobserved and observed supervision, and the quality of feedback as inferred from the seniority of the person providing it. After the implementation of the ITA programme supervision increased, but the difference was not statistically significant. The quality of feedback showed no significant change either. Inter-student variation in supervision and feedback remained invariably high after the implementation of the ITA programme. Whether these results are attributable to the way the programme was implemented or to the way the results were assessed remains to be clarified. [ABSTRACT FROM AUTHOR]

The study was designed to illustrate how residents perceive the Mini Clinical Examination Exercise as an assessment tool and its influence on their approach to learning and studying. A phenomenographic approach was applied. All 16 residents from a cardiology training program in Buenos Aires were included. Results show that in all cases residents demonstrate an intrinsic interest in the subject matter. They show self-regulating strategies when required to select, relate and make critical appraisals of their own. They consistently demonstrate an aim to build a relationship between individual experience and their chosen topic. The residents feel comfortable because it melds with their routine. Residents find the Mini Clinical Examination Exercise to be a useful assessment tool with a favorable influence towards a constructive approach to study and learning. [ABSTRACT FROM AUTHOR]

INTRODUCTION Structured assessment, embedded in a training programme, with systematic observation, feedback and appropriate documentation may improve the reliability of clinical assessment. This type of assessment format is referred to as in-training assessment (ITA). The feasibility and reliability of an ITA programme in an internal medicine clerkship were evaluated. The programme comprised 4 ward-based test formats and 1 outpatient clinic-based test format. Of the 4 ward-based test formats, 3 were single-sample tests, consisting of 1 student-patient encounter, 1 critical appraisal session and 1 case presentation. The other ward-based test and the outpatient-based test were multiple sample tests, consisting of 12 ward-based case write-ups and 4 long cases in the outpatient clinic. In all the ITA programme consisted of 19 assessments. METHODS During 41 months, data were collected from 119 clerks. Feasibility was defined as over two thirds of the students obtaining 19 assessments. Reliability was estimated by performing generalisability analyses with 19 assessments as items and 5 test formats as items. RESULTS A total of 73 students (69%) completed 19 assessments. Reliability expressed by the generalisability coefficients was 0.81 for 19 assessments and 0.55 for 5 test formats. CONCLUSIONS The ITA programme proved to be feasible. Feasibility may be improved by scheduling protected time for assessment for both students and staff. Reliability may be improved by more frequent use of some of the test formats. [ABSTRACT FROM AUTHOR]

CONTEXT Continuing professional development (CPD) of general practitioners. OBJECTIVE Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated. METHODS To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A border-line regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs. RESULTS Both methods are applicable and reliable; the obtained standards are credible according to the GPs. CONCLUSIONS Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used — i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given — is important because methods differ in practical aspects. [ABSTRACT FROM AUTHOR]

To investigate the experiences and opinions of programme directors, clinical supervisors and trainees on an in-training assessment (ITA) programme on a broad spectrum of competence for first year training in anaesthesiology. How does the programme work in practice and what are the benefits and barriers? What are the users' experiences and thoughts about its effect on training, teaching and learning? What are their attitudes towards this concept of assessment? Semistructured interviews were conducted with programme directors, supervisors and trainees from 3 departments. Interviews were audiotaped and transcribed. The content of the interviews was analysed in a consensus process among the authors. The programme was of benefit in making goals and objectives clear, in structuring training, teaching and learning, and in monitoring progress and managing problem trainees. There was a generally positive attitude towards assessment. Trainees especially appreciated the coupling of theory with practice and, in general, the programme inspired an academic dialogue. Issues of uncertainty regarding standards of performance and conflict with service declined over time and experience with the programme, and departments tended to resolve practical problems through structured planning. Three interrelated factors appeared to influence the perceived value of assessment in postgraduate education: (1) the link between patient safety and individual practice when assessment is used as a licence to practise without supervision rather than as an end-of-training examination; (2) its benefits to educators and learners as an educational process rather than as merely a method of documenting competence, and (3) the attitude and rigour of assessment practice. [ABSTRACT FROM AUTHOR]

The evidence suggests that a longitudinal training of communication skills embedded in a rich clinical context is most effective. In this study we evaluated the acquisition of communication skills under such conditions. In a longitudinal design the communication skills of a randomly selected sample of 25 trainees of a three-year postgraduate training programme for general practice were assessed at the start and at the end of training. Eight videotaped real life consultations were rated per measurement and per trainee, using the MAAS-Global scoring list. The results were compared with each other and with those of a reference group of 94 experienced GPs. The mean score of the MAAS-Global was slightly increased at the end of training (2.4) compared with the start (2.2). No significant difference was found between the final results of the trainees and the reference group. According to the criteria of the rating scale the performance of both trainees and GPs was unsatisfactory. The results of this study indicate that communication skills do not improve in a three-year postgraduate training comprising both a rich clinical context and a longitudinal training of communication skills, and that an unsatisfactory level still exists at the end of training. Moreover, GPs do not acquire communication skills during independent practice as they perform comparably to the trainees. Further research into the measurement of communication skills, the teaching procedures, the role of the GP-trainer as a model and the influence of rotations through hospitals and the like, is required. [ABSTRACT FROM AUTHOR]

In many student-centred curricula, teachers work in small groups with students. Guiding small groups is rather complex. Teachers need to aim to stimulate students in these groups towards constructive, self-directed, contextual and collaborative learning. It is also important that teachers demonstrate adequate interpersonal behaviour towards students. In this study, an instrument was developed to provide teachers with feedback about their performance to improve their behaviour. The aim of this study is to report on the development, reliability and validity of an instrument to evaluate teachers' performance in guiding small groups. The items of the instrument were based on five theoretical dimensions: constructive, self-directed, contextual and collaborative learning and the teacher's interpersonal behaviour. Students were asked to fill out this instrument and evaluate the performance of the teacher guiding the small group at the end of several courses. The instrument was tested in a pilot-study. Confirmatory factor analysis and generalizability studies indicated that the instrument is fairly valid and reliable when six student responses are available per teacher. Teachers receive detailed feedback about their performance indicating strong and weak aspects. Further research is needed to determine whether this feedback indeed leads to improvement of teachers' behaviour. [ABSTRACT FROM AUTHOR]

Background: A new reform on postgraduate education in Denmark requires a formal in-training assessment in all specialties. The aim of this study was to survey the implementation and acceptability of the first example of a nation-wide in-training assessment programme for first-year trainees in anaesthesiology developed by a working group under the Danish Society of Anaesthesiology and Intensive Care Medicine.Methods: A questionnaire about the implementation of the programme in practice and the characteristics of the trainees was sent to the educational responsible consultant (ERC) in each of the 26 anaesthetic departments in the country with first-year trainees in anaesthesiology. Standard evaluations of the assessment programme were regularly collected from trainees.Results: Twenty-five (96%) departments returned the questionnaire. In total the departments reported on 100 trainees and 83 of these had been enrolled in the programme. Thirteen departments reported in total on 27 trainees who had completed their first year of training and these departments had applied a median 21 (range 17-21) of the 21 tests included in the entire programme. Time constraints and resistance among senior clinicians were the most frequently cited barriers to implementation. Evaluations from trainees showed a generally positive attitude towards most of the programme. They especially praised the programme's effect on structuring training and having a positive effect on learning.Conclusion: The in-training assessment programme has been widely implemented across the country. The majority of the programme was acceptable to trainees and had a positive effect on structuring training and on fostering learning. [ABSTRACT FROM AUTHOR]

Purpose To examine the validity of a written knowledge test of skills for performance on an OSCE in postgraduate training for general practice. Methods A randomly-selected sample of 47 trainees in general practice took a knowledge test of skills, a general knowledge test and an OSCE. The OSCE included technical stations and stations including complete patient encounters. Each station was checklist rated and global rated. Results The knowledge test of skills was better correlated to the OSCE than the general knowledge test. Technical stations were better correlated to the knowledge test of skills than stations including complete patient encounters. For the technical stations the rating system had no influence on the correlation. For the stations including complete patient encounters the checklist rating correlated better to the knowledge test of skills than the global rating. Conclusion The results of this study support the predictive validity of the knowledge test of skills. In postgraduate training for general practice a written knowledge test of skills can be used as an instrument to estimate the level of clinical skills, especially for group evaluation, such as in studies examining the efficacy of a training programme or as a screening instrument for deciding about courses to be offered. This estimation is more accurate when the content of the test matches the skills under study. However, written testing of skills cannot replace direct observation of performance of skills. [ABSTRACT FROM AUTHOR]

Introduction An earlier study showed that an Angoff procedure with ≥ 10 recently graduated students as judges can be used to estimate the passing score of a progress test. As the acceptability and feasibility of this approach are questionable, we conducted an Angoff procedure with test item writers as judges. This paper reports on thereliability and credibility of this procedure and compares the standards set by the two different panels. Methods Fourteen item writers judged 146 test items. Recently graduated students had assessed these items in a previous study. Generalizability was investigated as a function of the number of items and judges. Credibility was judged by comparing the pass/fail rates associated with the Angoff standard, a relative standard and a fixed standard. The Angoff standards obtained by item writers and graduates were compared. Results The variance associated with consistent variability of item writers across items was 1·5% and for graduate students it was 0·4%. An acceptable error score required 39 judges. Item-Angoff estimates of the two panels and item P -values correlated highly. Failure rates of 57%, 55% and 7% were associated with the item writers' standard, the fixed standard and the graduates' standard, respectively. Conclusion The graduates' and the item writers' standards differed substantially, as did the associated failure rates. A panel of 39 item writers is not feasible. The item writers' passing score appears to be less credible. The credibility of the graduates' standard needs further evaluation. The acceptability and feasibility of a panel consisting of both students and item writers may be worth investigating. [ABSTRACT FROM AUTHOR]

Background Knowledge is an essential component of medical competence and a major objective of medical education. Thus, the degree of acquisition of knowledge by students is one of the measures of the effectiveness of a medical curriculum. We studied the growth in student knowledge over the course of Maastricht Medical School's 6-year problem-based curriculum. Methods We analysed 60 491 progress test (PT) scores of 3226 undergraduate students at Maastricht Medical School. During the 6-year curriculum a student sits 24 PTs (i.e. four PTs in each year), intended to assess knowledge at graduation level. On each test occasion all students are given the same PT, which means that in year 1 a student is expected to score considerably lower than in year 6. The PT is therefore a longitudinal, objective assessment instrument. Mean scores for overall knowledge and for clinical, basic, and behavioural/social sciences knowledge were calculated and used to estimate growth curves. Findings Overall medical knowledge and clinical sciences knowledge demonstrated a steady upward growth curve. However, the curves for behavioural/social sciences and basic sciences started to level off in years 4 and 5, respectively. The increase in knowledge was greatest for clinical sciences (43%), whereas it was32% and 25% for basic and behavioural/social sciences, respectively. Interpretation Maastricht Medical School claims to offer a problem-based, student-centred, horizontally and vertically integrated curriculum in the first 4 years, followed by clerkships in years 5 and 6. Students learn by analysing patient problems and exploring pathophysiological explanations. Originally, it was intended that students’ knowledge of behavioural/social sciences would continue to increase during their clerkships. However, the results for years 5 and 6 show diminishing growth in basic and behavioural/social sciences knowledge compared to overall and clinical sciences knowledge, which appears to suggest... [ABSTRACT FROM AUTHOR]

Background Medical education uses the cognitive apprenticeship model of student learning extensively. Students rotate among different hospitals and out- patient clinics where they are exposed to a range of professionally relevant contexts. Here they learn to think and act in different domains under the supervision of experts. Previous research has shown that these learning situations involve little teaching. Students see a narrow range of patient problems and feedback is limited. The aim of this study is to investigate relationships among some educational variables in the out-patient clinic. Method This paper provides a theoretical model that specifies the factors influencing the effectiveness of student rotations at out-patient clinics. The model makes distinctions between input variables, such as organizational quality, number of students contemporaneously involved and available space, and process variables, such as patient mix and supervision, and the output variable of the effectiveness of rotations in out-patient clinics. Results The model was tested against empirical data from evaluative surveys and showed a reasonable fit. The model offers suggestions for improving the learning environment of clinical rotations. Discussion The strength of this study lies in its process evaluation perspective which investigates interactions between intervening variables rather than the influence of particular variables in isolation from other variables. [ABSTRACT FROM AUTHOR]

Aim This study was conducted to investigate the value of a written knowledge test of communication skills for predicting scores on a performance test of communication skills. Method A paper-and-pencil test of knowledge about communication skills and a performance test of communication skills, consisting of four stations with standardised patients, were administered to students of two classes of the medical schools of Maastricht and Leiden, the Netherlands. The results on these tests were compared. Results From the results of both instruments, the classes of the participating students could be recognised equally well: 60% correct qualifications of the classes by the knowledge test and 64% by the multiple station examination. Between the two tests an overall, disattenuated correlation of 0·60 was found (N = 133, P < 0·01), suggesting moderate predictive value of the knowledge test for the performance test of communication skills. The correlation is stronger for students from Maastricht medical school than for their colleagues in Leiden. Correlation between the knowledge of communication skills test and other available test results of the participating Maastricht students is close to zero, suggesting that the test measures a distinct quality of students' competence. Discussion The paper-and-pencil test of knowledge of communication skills has predictive value for the performance of these skills, but this value seems to be less pronounced than similar findings for clinical procedural skills. The stronger relationship between 'knowing how' and 'showing' in the Maastricht student group might be indicative of an effect of the training format. [ABSTRACT FROM AUTHOR]

In order to share the issues involved in setting up a communication skills training programme in a medical school, the development of such a programme at the Skillslab of Maastricht Medical School, the Netherlands, is described and the problems encountered are addressed. A multidisciplinary working group developed teaching goals for communication skills, focused on observable behaviour to be displayed by students. These teaching goals were incorporated in a generic model for doctor-patient communication. A longitudinal training programme was created, throughout the four years of the preclinical curriculum. Students meet in small groups of 10, once every 2 weeks. In between group sessions they practise consultation skills in simulated patient contacts. Communication skills are assessed in the annual multiple station examination. In the development of this programme the following consecutive actions were taken: teaching material was produced, and an assessment tool was developed, as were instruments for programme evaluation. The programme evaluation allowed student feedback to teachers, the teachers' departments, and the administration of the medical school. Finally, teacher training was professionalized. [ABSTRACT FROM AUTHOR]

Introduction General practice trainers hold a key position in general practice training, especially through their provision of a role model. Their own competence in general practice care is important in this regard. The purpose of the study was to evaluate whether a quality assessment programme could identify the strengths and weaknesses of GP trainers in four main domains of general practice care. Methods The quality assessment programme comprised validated tests on four domains of general practice: general medical knowledge, knowledge of medical-technical skills, consultation skills and practice management. The criterion for the identification of relative strengths and weaknesses of GP trainers was a variation in the scores of trainers indicating higher and lower scores (strengths and weaknesses) within each domain. Results GP trainers (n=105) were invited to participate in the study and 90% (n=94) did so. The variation in scores allowed the indication of strengths and weaknesses. Main strengths were: general medical knowledge of the digestive system; knowledge of medical skills relating to the skin; consultation skills concerning empathy; practice management with regard to accessibility. Main weaknesses were: general medical knowledge of the neurological system; knowledge of the medical/technical skills relating to the endocrine metabolic and nutritional system; consultation skills regarding shared decision making; practice management involving cooperation with staff and other care providers. Discussion This first systematic evaluation of GP trainers identified their strengths and weaknesses. The weaknesses identified will be used in the improvement process as topics for collective improvement in the GP trainers’ general curriculum and in individual learning plans. [ABSTRACT FROM AUTHOR]

Background: Medical students increasingly participate in rotations in primary care settings, such as general practices. Although general practices can differ markedly from hospital settings, research on the instructional quality of general practice rotations is relatively scarce. Aim: We conducted a study to identify which aspects make a significant contribution to the instructional quality of general practice clerkships, as perceived by students. Method: After completing their general practice attachment, 155 fifth-year medical students filled out an evaluation questionnaire. Exploratory factor analysis and reliability analysis identified clusters of correlated independent variables. Subsequent regression analysis revealed the relations between the reduced set of independent variables and the dependent variable ‘Instructional quality’. Results: Both the quality of supervision and the size and diversity of the patient mix substantially affected students’ experienced instructional quality. Opportunities and facilities to perform independently were correlated with instructional quality, but did not affect the instructiveness directly. Conclusions: Supervision, patient mix and independence are crucial factors for learning in general practice. This is consistent with findings in hospital settings. The perceived quality of instruction hinges on supervision, which is not only the variable most strongly related to instructional quality, but which also affects both the patient mix and students’ independence. [ABSTRACT FROM AUTHOR]