Search

The way quality of assessment has been perceived and assured has changed considerably in the recent 5 decades. Originally, assessment was mainly seen as a measurement problem with the aim to tell people apart, the competent from the not competent. Logically, reproducibility or reliability and construct validity were seen as necessary and sufficient for assessment quality and the role of human judgement was minimised. Later, assessment moved back into the authentic workplace with various workplace-based assessment (WBA) methods. Although originally approached from the same measurement framework, WBA and other assessments gradually became assessment processes that included or embraced human judgement but based on good support and assessment expertise. Currently, assessment is treated as a whole system problem in which competence is evaluated from an integrated rather than a reductionist perspective. Current research therefore focuses on how to support and improve human judgement, how to triangulate assessment information meaningfully and how to construct fairness, credibility and defensibility from a systems perspective. But, given the rapid changes in society, education and healthcare, yet another evolution in our thinking about good assessment is likely to lurk around the corner.

Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption--just like PBL--will take many years ahead of us.

Programmatic assessment is both a philosophy and a method for assessment. It has been developed in medical education as a response to the limitation of the dominant testing or measurement approaches and to better align with changes in how medical competence was conceptualised. It is based on continual collection of assessment and feedback information which is collected in a portfolio and periodically discussed with a mentor or coach. Students produce reflections in which they analyse the assessment information triangulated across methods to meaningfully--rather than purely numerically-analyse their strengths and weakness in the different competency domains. Decisions about students' progress are made proportionally, in that low-stakes decisions are made when only limited assessment information is available but high-stakes decisions will and can be made on rich information. Quality of the process and credibility of the outcomes is determined by a combination of psychometric methods, qualitative rigour, and through organisational due processes. The method is still young, preliminary results are promising but more empirical justification research is needed.

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments--rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically "the locus of regulation of learning" (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use--and, consequently, the effectiveness--of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting "idiosyncratic rater variance" is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.

Clinical workplaces offer postgraduate trainees a wealth of opportunities to learn from experience. To promote deliberate and meaningful learning self-regulated learning skills are foundational. We explored trainees' learning activities related to patient encounters to better understand what aspects of self-regulated learning contribute to trainees' development, and to explore supervisor's role herein. We conducted a qualitative non-participant observational study in seven general practices. During two days we observed trainee's patient encounters, daily debriefing sessions and educational meetings between trainee and supervisor and interviewed them separately afterwards. Data collection and analysis were iterative and inspired by a phenomenological approach. To organise data we used networks, time-ordered matrices and codebooks. Self-regulated learning supported trainees to increasingly perform independently. They engaged in self-regulated learning before, during and after encounters. Trainees' activities depended on the type of medical problem presented and on patient, trainee and supervisor characteristics. Trainees used their sense of confidence to decide if they could manage the encounter alone or if they should consult their supervisor. They deliberately used feedback on their performance and engaged in reflection. Supervisors appeared vital in trainees' learning by reassuring trainees, discussing experience, knowledge and professional issues, identifying possible unawareness of incompetence, assessing performance and securing patient safety. Self-confidence, reflection and feedback, and support from the supervisor are important aspects of self-regulated learning in practice. The results reflect how self-regulated learning and self-entrustment promote trainees' increased participation in the workplace. Securing organized moments of interaction with supervisors is beneficial to trainees' self-regulated learning.

Every year, thousands of students go abroad for part of their study programme. Supervision from the home institution is then crucial for good study progress. Providing supervision and feedback at a distance is challenging. This project aims to identify bottlenecks for supervision and hypothesises that online supervisory group meetings with videoconferencing contribute to good supervision. Study 1 showed that students who were abroad perceived lower-quality supervision and feedback on all measured aspects, compared with students staying at the home institution. Within a quasi-experimental design, Study 2 showed that students participating in online meetings experienced better supervision than those who were abroad without online meetings, and they were equally positive about supervision as students who stayed at the home institution. These findings stress the need for extra support for students during their stay abroad and the large potential of videoconferencing in optimising supervision at a distance.

A tool used in student-centered education is discussion among students in small learning groups. The Western origin of student-centered education, coupled with cross-cultural differences in communication styles, may detract from its cross-cultural applicability. This study investigates how in student-centered education, students' cultural backgrounds are expressed in discussions and shape students' discussion behaviors and skills. A comparative case study using problem-based learning as a student-centered model was conducted in three medical schools located in East Asia, Western Europe and the Middle East. Four cultural factors were found to potentially cause students, especially those in the non-Western schools, to refrain from speaking up, asking questions, and challenging others in discussions. Six contextual factors mediated the influence of the cultural factors. The findings were incorporated in a conceptual model. The conclusion seems justified that student-centered education is feasible in different cultural contexts, but across these contexts, processes and outcomes are likely to differ.

Contrary to classical problem-based learning, in guided problem-based learning, the learning goals are predetermined by the instructor--on the basis of a detailed decomposition of the subject matter to be studied--to activate prior knowledge and to structure self-study and subsequent group discussion. This study investigated the effects of classical problem-based learning and guided problem-based learning, for different prior knowledge levels, on perceived value and usefulness of the learning activity and on conceptual understanding of statistics. Participants randomly assigned 110 students to 10 problem-based learning groups, and subsequently, the 10 groups were allocated randomly to classical problem-based learning or guided problem-based learning. The results indicate that guided problem-based learning tends to enhance conceptual understanding of statistics more than break; classical problem-based learning once students have some prior knowledge of the subject. Furthermore, guided problem-based learning tends to increase students' awareness of the value and usefulness of the learning activity.

Although elaboration has been investigated frequently, there is little evidence for the beneficial effect of elaboration in problem-based learning. A controlled experiment tested the effect of elaboration during problem-based discussion on recall. Sixty-seven students observed a video-recorded, problem-based discussion. In one experimental condition, a tutor in the video encouraged participants to elaborate by asking elaborative questions. In a second condition, the tutor asked superficial questions. After the discussion, all participants studied a text with relevant new information. Elaborative questions had no significant effect on recall of idea units from the text, p = 0.39, ?[superscript 2] = 0.01. High-ability students outperformed low-ability students, p = 0.04, ?[superscript 2] = 0.07, but this effect did not interact with the experimental treatment, p = 0.22, ?[superscript 2] = 0.02. Suggestions for further research are presented.

This study set out to test whether relevant prior knowledge would moderate a positive effect on academic achievement of elaboration during small-group discussion. In a 2 × 2 experimental design, 66 undergraduate students observed a video showing a small-group problem-based discussion about thunder and lightning. In the video, a teacher asked questions to the observing participants. Participants either elaborated by responding to these questions, or did not elaborate, but completed a distraction task after each question. They received either relevant or irrelevant prior knowledge before the discussion. After the discussion, all participants studied a text about thunder and lighting and completed immediate and delayed-recall tests for this text. Elaboration had no main effect on recall, but there was a significant interaction effect between relevant prior knowledge and elaboration. The results suggest that elaboration is helpful for students with more prior knowledge, but harmful for students with less prior knowledge.

Anatomy education often consists of a combination of lectures and laboratory sessions, the latter frequently including surface anatomy. Studying surface anatomy enables students to elaborate on their knowledge of the cadaver's static anatomy by enabling the visualization of structures, especially those of the musculoskeletal system, move and function in a living human being. A recent development in "teaching methods" for surface anatomy is body painting, which several studies suggest increases both student motivation and knowledge acquisition. This article focuses on a "teaching approach" and is a translational contribution to existing literature. In line with best evidence medical education, the aim of this article is twofold: to briefly inform teachers about constructivist learning theory and elaborate on the principles of constructive, collaborative, contextual, and self-directed learning; and to provide teachers with an example of how to implement these learning principles to change the approach to teaching surface anatomy. Student evaluations of this new approach demonstrate that the application of these learning principles leads to higher student satisfaction. However, research suggests that even better results could be achieved by further adjustments in the application of contextual and self-directed learning principles. Successful implementation and guidance of peer physical examination is crucial for the described approach, but research shows that other options, like using life models, seem to work equally well. Future research on surface anatomy should focus on increasing the students' ability to apply anatomical knowledge and defining the setting in which certain teaching methods and approaches have a positive effect. (Contains 1 table and 3 figures.)

Traditionally, an anatomy practical examination is conducted using a free response format (FRF). However, this format is resource-intensive, as it requires a relatively large time investment from anatomy course faculty in preparation and grading. Thus, several interventions have been reported where the response format was changed to a selected response format (SRF). However, validity evidence from those interventions has not proved entirely adequate for the practical anatomy examination, and thus, further investigation was required. In this study, the validity evidence of SRF was examined using multiple choice questions (MCQs) constructed according to different levels of Bloom's taxonomy in comparison with the traditional free response format. A group of 100 medical students registered in a gross anatomy course volunteered to be enrolled in this study. The experimental MCQ examinations were part of graded midterm and final steeplechase practical examination. Volunteer students were instructed to complete the practical examinations twice, once in each of two separate examination rooms. The two separate examinations consisted of a traditional free response format and MCQ format. Scores from the two examinations (FRF and MCQ) displayed a strong correlation, even with higher level Bloom's taxonomy questions. In conclusion, the results of this study provide empirical evidence that the SRF (MCQ) response format is a valid method and can be used as an alternative to the traditional FRF steeplechase examination. (Contains 4 tables.)

Purpose: This study aims to investigate quality of feedback as offered by supervisor-assessors with varying levels of assessor expertise following assessment of performance in residency training in a health care setting. It furthermore investigates if and how different levels of assessor expertise influence feedback characteristics. Design/methodology/approach: Experienced (n = 18) and non-experienced (n = 16) supervisor-assessors with different levels of assessor expertise in general practice (GP) watched two videotapes, each presenting a trainee in a "real-life" patient encounter. After watching each videotape, participants documented performance ratings, wrote down narrative feedback comments and verbalized their feedback. Deductive content analysis of feedback protocols was used to explore quality of feedback. Between-group differences were assessed using qualitative-based quantitative analysis of feedback data. Findings: Overall, specificity and usefulness of both written and verbal feedback was limited. Differences in assessor expertise did not seem to affect feedback quality. Research limitations/implications: Results of the study are limited to a specific setting (GP) and assessment context. Further study in other settings and larger sample sizes may contribute to better understanding of the relation between assessor characteristics and feedback quality. Practical implications: Findings suggest that even with supervisor-assessors with varying levels of assessor expertise who are trained in performance assessment and the provision of performance feedback, high-quality feedback is not self-evident; coaching "on the job" of feedback providers and continuous evaluation of feedback processes in performance management systems is crucial. Instruments should facilitate provision of meaningful feedback in writing. Originality/value: The paper investigates quality of feedback immediately following assessment of performance, and links feedback quality to assessor expertise. Findings can contribute to improvement of performance management systems and assessments for developmental purposes. (Contains 2 tables and 2 figures.)

Little is known about the influence of language background in problem-based learning (PBL) tutorial groups on group processes and students' academic achievement. This study investigated the relationship between language background, secondary school score, tutorial group processes, and students' academic achievement in PBL. A validated tutorial group effectiveness questionnaire was administered to undergraduate medical students in a PBL curriculum at the Nelson R. Mandela School of Medicine (NRMSM) in South Africa. Although 58 percent of the students did not speak English as their first language, the tutorials were in English. Furthermore, secondary school scores differed strongly due to inadequate resources between secondary schools. A path analysis was conducted to test a causal model in which the two independent variables were English as the First Language (EFL) and secondary school scores. These variables were assumed to influence the process variables (cognitive, motivational, and demotivational group processes). Input and process variables were assumed to influence the two output variables, being overall group productivity and students' academic achievement. All data were analyzed at the individual student level (N = 387). A very good model fit was found (CMIN/DF = 0.68, GFI = 1.00, TLI = 1.02, CFI = 1.00, RMSEA = 0.00). EFL and secondary school scores positively affected students' academic achievement (respectively beta = 0.24 and beta = 0.16). EFL negatively affected motivational group processes (beta = -0.22). Cognitive group processes positively influenced overall group productivity (beta = 0.31) and so did motivational group processes (beta = 0.27). Demotivational group processes negatively predicted academic achievement and overall group productivity (beta = -0.15, and beta = -0.25). The model resulted in an R-square of 0.15 and 0.45 for academic achievement and overall group productivity, respectively. EFL and secondary school scores had a positive effect on academic achievement. Cognitive and motivational group processes had a positive effect on overall group productivity, while EFL negatively impacted motivational group processes. We recommend English language development courses to be formally included into curricula to enhance student learning.

Collaborative approaches such as Problem Based Learning (PBL) may provide the opportunity to bring together diverse students but their efficacy in practice and the complications that arise due to the mixed ethnicity needs further investigation. This study explores the key advantages and problems of heterogeneous PBL groups from the students' and teachers' opinions. Focus groups were conducted with a stratified sample of second year medical students and their PBL teachers. We found that students working in heterogeneous groupings interact with students with whom they don't normally interact with, learn a lot more from each other because of their differences in language and academic preparedness and become better prepared for their future professions in multicultural societies. On the other hand we found students segregating in the tutorials along racial lines and that status factors disempowered students and subsequently their productivity. Among the challenges was also that academic and language diversity hindered student learning. In light of these the recommendations were that teachers need special diversity training to deal with heterogeneous groups and the tensions that arise. Attention should be given to create "the right mix" for group learning in diverse student populations. The findings demonstrate that collaborative heterogeneous learning has two sides that need to be balanced. On the positive end we have the "ideology" behind mixing diverse students and on the negative the "practice" behind mixing students. More research is needed to explore these variations and their efficacy in more detail. (Contains 1 table.)

Seventy students participated in an experiment to measure the effects of either providing explanations or listening during small group discussions on recall of related subject-matter studied after the discussion. They watched a video of a small group discussing a problem. In the first experimental condition, the video was stopped at various points in time, enabling the participants to verbally respond to the discussion. In the second condition, they listened to the same discussion, without contributing. In the control condition, they listened to a discussion that was not related to the subject-matter subsequently studied. After the discussion, all participants studied a text and answered questions that tested their recall of information from this text. No immediate differences in recall were found. One month later, participants who had actively engaged in explaining remembered more from the text. The conclusion appears justified that actively providing explanations during a discussion positively affects long-term memory. (Contains 2 figures and 1 table.)

In the complex practice of today's blended learning, educators need to be able to evaluate both online and face-to-face communication in order to get the full picture of what is going on in blended learning scenarios. The aim of this study was to investigate the reliability and feasibility of a practical instrument for analysing face-to-face communication that is in alignment with tools for analysing online communication. We examined whether an existing coding scheme for online communication could be modified to evaluate face-to-face communication. We investigated the reliability, validity and usability of a modified version of the Transcript Analysis Tool (TAT) in analysing communication among students during small-group sessions comprising presentations by students and discussion. The modified TAT proved to be effective in obtaining a detailed description of students' participation in the communication and of the focus of the communication. Evaluation was more efficient when direct video analysis was used than when a transcript of the same video was analysed. Direct analysis took about 2 h for 20 min of video recording and had an inter-rater reliability of [kappa] = 0.56. The modified TAT appears to be an effective and efficient instrument to analyse face-to-face communication in small group sessions. Its combined use with an evaluation instrument for online learning may create a powerful evaluation instrument for blended learning. Further studies are needed to establish the value of such an instrument. (Contains 6 tables and 3 figures.)

Innovations in undergraduate medical education, such as integration of disciplines and problem based learning, have given rise to concerns about students' knowledge of anatomy. This article originated from several studies investigating the knowledge of anatomy of students at the eight Dutch medical schools. The studies showed that undergraduate students uniformly perceived deficiencies in their anatomical knowledge when they started clinical training regardless of their school's didactic approach. A study assessing students' actual knowledge of clinical anatomy revealed no relationship between students' knowledge and the school's didactic approach. Test failure rates based on absolute standards set by different groups of experts were indicative of unsatisfactory levels of anatomical knowledge, although standards differed markedly between the groups of experts. Good test performance by students seems to be related to total teaching time for anatomy, teaching in clinical context, and revisiting anatomy topics in the course of the curriculum. These factors appeared to outweigh the effects of disciplinary integration or whether the curriculum was problem-based or traditional.

Quality control policies towards Dutch vocational schools have changed dramatically because the government questioned examination quality. Schools must now demonstrate assessment quality to a new Examination Quality Center. Since teachers often design assessments, they must be involved in quality issues. This study therefore explores teachers' opinions on assessment quality evaluation criteria. Pre-vocational and vocational teachers (N=211) responded to a questionnaire. Contrary to expectations, results show that teachers deem classical and competency-based quality criteria equally important. Vocational teachers gave higher importance scores than pre-vocational teachers, possibly due to the pressure they experience to improve the quality of their assessments.

Context: In-training assessment (ITA), defined as multiple assessments of performance in the setting of day-to-day practice, is an invaluable tool in assessment programmes which aim to assess professional competence in a comprehensive and valid way. Research on clinical performance ratings, however, consistently shows weaknesses concerning accuracy, reliability and validity. Attempts to improve the psychometric characteristics of ITA focusing on standardisation and objectivity of measurement thus far result in limited improvement of ITA-practices. Purpose: The aim of the paper is to demonstrate that the psychometric framework may limit more meaningful educational approaches to performance assessment, because it does not take into account key issues in the mechanics of the assessment process. Based on insights from other disciplines, we propose an approach to ITA that takes a constructivist, social-psychological perspective and integrates elements of theories of cognition, motivation and decision making. A central assumption in the proposed framework is that performance assessment is a judgment and decision making process, in which rating outcomes are influenced by interactions between individuals and the social context in which assessment occurs. Discussion: The issues raised in the article and the proposed assessment framework bring forward a number of implications for current performance assessment practice. It is argued that focusing on the context of performance assessment may be more effective in improving ITA practices than focusing strictly on raters and rating instruments. Furthermore, the constructivist approach towards assessment has important implications for assessment procedures as well as the evaluation of assessment quality. Finally, it is argued that further research into performance assessment should contribute towards a better understanding of the factors that influence rating outcomes, such as rater motivation, assessment procedures and other contextual variables.

As assessment methods are changing, the way to determine their quality needs to be changed accordingly. This article argues for the use Competence Assessment Programs (CAPs), combinations of traditional tests and new assessment methods which involve both formative and summative assessments. To assist schools in evaluating their CAPs, a self-evaluation procedure was developed, based on 12 quality criteria for CAPs developed in earlier studies. A self-evaluation was chosen as it is increasingly used as an alternative to external evaluation. The CAP self-evaluation is carried out by a group of functionaries from the same school and comprises individual self-evaluations and a group interview. The CAP is rated on the 12 quality criteria and a piece of evidence is asked for to support these ratings. In this study, three functionaries from eight schools (N=24) evaluated their CAP using the self-evaluation procedure. Results show that the group interview was very important as different perspectives on the CAP are assembled here into an overall picture of the CAP's quality. Schools seem to use mainly personal experiences to support their ratings and need to be supported in the process of carrying out a self-evaluation.

Introduction: Many of the residents who supervise medical students in clinical practice are unfamiliar with the principles of effective supervision. Training in teaching skills is therefore seen as an effective strategy to improve the quality of clinical supervision. Method: Twenty seven medical residents were matched and assigned to an experimental group (n = 14) and a control group (n = 13). The experimental group participated in a two-day workshop on teaching skills. Using standardized questionnaires, the teaching abilities of all participants were assessed anonymously by medical students, before and after the workshop, to determine for any effect of the intervention. Results: A significant improvement in the teaching abilities of the medical residents in the experimental group was observed following the workshop (t = -2.68, p = 0.02). The effect size within the experimental group was large (d = 1.17), indicating that the workshop led to a measurable positive change in the medical residents' teaching abilities. The effect size estimated from the post intervention scores on teaching ability of the two groups showed a moderate improvement (d = 0.57) in the experimental group compared with the control group. Discussion: Medical students rated the teaching abilities of the workshop participants after the training more highly than those of the residents in the control group. The ability to adjust teaching to the needs of the students and teach effective communication and diagnostic clinical skills were among the features that characterized effective teaching. Properly designed, teacher-training workshops could be effective and feasible methods to improve the quality of teaching by medical residents.

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.