Search

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments--rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically "the locus of regulation of learning" (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use--and, consequently, the effectiveness--of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself). Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category "Dealing with oneself" consisted of descriptions of a student's attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase. (Contains 5 tables and 1 figure.)

Little is known about the influence of language background in problem-based learning (PBL) tutorial groups on group processes and students' academic achievement. This study investigated the relationship between language background, secondary school score, tutorial group processes, and students' academic achievement in PBL. A validated tutorial group effectiveness questionnaire was administered to undergraduate medical students in a PBL curriculum at the Nelson R. Mandela School of Medicine (NRMSM) in South Africa. Although 58 percent of the students did not speak English as their first language, the tutorials were in English. Furthermore, secondary school scores differed strongly due to inadequate resources between secondary schools. A path analysis was conducted to test a causal model in which the two independent variables were English as the First Language (EFL) and secondary school scores. These variables were assumed to influence the process variables (cognitive, motivational, and demotivational group processes). Input and process variables were assumed to influence the two output variables, being overall group productivity and students' academic achievement. All data were analyzed at the individual student level (N = 387). A very good model fit was found (CMIN/DF = 0.68, GFI = 1.00, TLI = 1.02, CFI = 1.00, RMSEA = 0.00). EFL and secondary school scores positively affected students' academic achievement (respectively beta = 0.24 and beta = 0.16). EFL negatively affected motivational group processes (beta = -0.22). Cognitive group processes positively influenced overall group productivity (beta = 0.31) and so did motivational group processes (beta = 0.27). Demotivational group processes negatively predicted academic achievement and overall group productivity (beta = -0.15, and beta = -0.25). The model resulted in an R-square of 0.15 and 0.45 for academic achievement and overall group productivity, respectively. EFL and secondary school scores had a positive effect on academic achievement. Cognitive and motivational group processes had a positive effect on overall group productivity, while EFL negatively impacted motivational group processes. We recommend English language development courses to be formally included into curricula to enhance student learning.

In the complex practice of today's blended learning, educators need to be able to evaluate both online and face-to-face communication in order to get the full picture of what is going on in blended learning scenarios. The aim of this study was to investigate the reliability and feasibility of a practical instrument for analysing face-to-face communication that is in alignment with tools for analysing online communication. We examined whether an existing coding scheme for online communication could be modified to evaluate face-to-face communication. We investigated the reliability, validity and usability of a modified version of the Transcript Analysis Tool (TAT) in analysing communication among students during small-group sessions comprising presentations by students and discussion. The modified TAT proved to be effective in obtaining a detailed description of students' participation in the communication and of the focus of the communication. Evaluation was more efficient when direct video analysis was used than when a transcript of the same video was analysed. Direct analysis took about 2 h for 20 min of video recording and had an inter-rater reliability of [kappa] = 0.56. The modified TAT appears to be an effective and efficient instrument to analyse face-to-face communication in small group sessions. Its combined use with an evaluation instrument for online learning may create a powerful evaluation instrument for blended learning. Further studies are needed to establish the value of such an instrument. (Contains 6 tables and 3 figures.)

Progress testing provides data on the growth of students' knowledge over the course of the curriculum obtained from the results of all students in the curriculum on periodical similar tests pitched at end-of-curriculum level. Since 2001, three medical schools have jointly constructed and administered four progress tests annually. All students in the 6-year undergraduate curricula of these schools take the same tests resulting in 24 distinct measurements per academic year (four tests for six student year groups), which may be used to compare performance between and within schools. Because single point measurements had proven unreliable, we devised a method to use cumulative information to compare schools' test performance. This cumulative deviation method involves calculation of the deviations of schools' scores from the cross-institutional average score for 24 measurement moments in 1 year. The current study shows that it appears to be feasible to use a combination of the cumulative deviation method and trend analysis for subdomains of medical knowledge to detect strengths and weaknesses in knowledge development in medical curricula. We illustrate the method by applying it to data from 16 consecutive progress tests administered to all students (4,300) of three medical schools in the academic years 2001/2002 through 2004/2005.

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.

Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.

Developed a method to estimate the cutscore precisions for empirical standard-setting methods by using resampling. Illustrated the method with two actual datasets consisting of 86 Dutch medical residents and 155 Canadian medical students taking objective structured clinical examinations. Results show the applicability of the method. (SLD)

Compared four standard-setting procedures for an objective structure clinical examination (OSCE) in medical education. Applied Angoff, borderline, relative, and holistic procedures to the data used to establish a cutoff score for a pass/fail decision. The Angoff and borderline procedures gave similar results; however, the relative and holistic methods gave widely divergent results. (Author/DB)

Sequential, standardized patient-based tests were given to medical students at the University of Maastricht (Netherlands); the first test was a screening test that evaluated the efficiency/validity of cutoff scores. Findings indicated that stringent pass/fail cutoff scores on the screening test produced optimal results. Fewer than 0.2 percent of students who passed the screening test failed the complete test. (DB)

Objective: To compare postpartum hemorrhage (PPH) patient outcomes before and after an in situ instructional design based PPH simulation attended by obstetrics and gynecology (OBGYN) residents.Methods: This uncontrolled before-and-after study was conducted in Recife, Brazil including all 1388 women delivering from June to August 2012 and all 1357 delivering from June to August 2013. The 36 OBGYN residents were divided into13 teams of two or three participants and were trained through ID based PPH simulation training with the following eight steps: (1) prior knowledge activation, (2) video demonstration, (3) dual-coding PPH protocol discussion-an image association during the training, (4) training scenario # 1, (5) debriefing, (6) training scenario # 2 with immediate feedback, (7) training scenario # 3, and (8) debriefing with self-assessment. The training scenarios had an increasing level of complexity. The main goal of the training was the adequate management of PPH and situational awareness improvement-the ability to anticipate, recognize, and intercept unfolding error chains. The primary patient outcomes rates used for the before and after comparison were therapeutic uterotonics use within 24 h of birth and blood transfusion. Secondary outcomes were therapeutic oxytocin mean dosage IU within 24 h of birth, postpartum Hb < 6 g/dL, among others. Chi-square test was used for categorical variables comparison and independent t-test for continuous variables.Results: PPH rates were 100 (7.2% of 2012 deliveries) and 80 cases (5.9% of 2013 deliveries), respectively. Comparison of primary post- and pre-simulation outcomes revealed no significant differences. However, in the comparison for therapeutic oxytocin mean dosage IU within 24 h of birth, there was an increase found after the simulation (15.98 ± 7.4 versus 25.1 ± 12.3; p < .001). For all other outcome measures, there were no statistical differences.Conclusions:In situ ID based PPH simulation leads to an increase in the mean dosage of oxytocin after training, in selected cases. This may indicate better situational awareness when managing women with PPH. [ABSTRACT FROM AUTHOR]

Objectives: This study explores the optimal focus for negotiation skills development training by investigating how often medical residents negotiate in practice, and how they perceive the effectiveness of their negotiation capabilities. Methods: An exploratory study was performed using a questionnaire regarding the medical residents' working environment, negotiation frequency, knowledge and skills using a 5- point Likert scale, multiple choice questions and open questions. Exploratory factor analysis with principal component analysis, varimax rotation, reliability analysis, and content analysis were used to reduce the number of variables. Descriptive and interferential statistics and multiple regression analysis were used to analyze the data. Results: We analyzed the responses of 60 medical residents. The findings showed that the perceived development of their negotiation knowledge (M=3.06, SD=0.83) was less than their negotiation skills (M=3.69, SD=0.47). Their attitude during negotiations, especially females, differed substantially in the interactions with nurses than with their supervisors. Medical residents with more working experience, better negotiation skills or who worked in hierarchical environments negotiated more frequently with their supervisors. Medical residents with better collaboration skills and negotiation knowledge demonstrated better negotiation skills. Conclusions: This study underlines medical residents' need for negotiation training. In addition to the basic negotiation knowledge and skills, training programs in negotiation should focus on the medical residents' awareness of their attitudes during negotiations, combining the assertiveness shown in interactions with supervisors with the empathy and emotional engagement present in interactions with nurses. Furthermore, attention should be paid to the influence of the environmental hierarchy on negotiation skill development. [ABSTRACT FROM AUTHOR]

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA. [ABSTRACT FROM AUTHOR]