Search

Context: One central consideration in health professions education (HPE) is to ensure we are making sound and justifiable decisions based on the assessment instruments we use on health professionals. To achieve this goal, HPE assessment researchers have drawn on Kane's argument-based framework to ascertain the validity of their assessment tools. However, the original four-inference model proposed by Kane - frequently used in HPE validation research - has its limitations in terms of what each inference entails and what claims and sources of backing are housed in each inference. The under-specification in the four-inference model has led to inconsistent practices in HPE validation research, posing challenges for (i) researchers who want to evaluate the validity of different HPE assessment tools and/or (ii) researchers who are new to test validation and need to establish a coherent understanding of argument-based validation., Methods: To address these identified concerns, this article introduces the expanded seven-inference argument-based validation framework that is established practice in the field of language testing and assessment (LTA). We explicate (i) why LTA researchers experienced the need to further specify the original four Kanean inferences; (ii) how LTA validation research defines each of their seven inferences and (iii) what claims, assumptions and sources of backing are associated with each inference. Sampling six representative validation studies in HPE, we demonstrate why an expanded model and a shared disciplinary validation framework can facilitate the examination of the validity evidence in diverse HPE validation contexts., Conclusions: We invite HPE validation researchers to experiment with the seven-inference argument-based framework from LTA to evaluate its usefulness to HPE. We also call for greater interdisciplinary dialogue between HPE and LTA since both disciplines share many fundamental concerns about language use, communication skills, assessment practices and validity in assessment instruments., (© 2024 The Author(s). Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Objective: This study explored how the Syrian crisis, training conditions, and relocation influenced the National Medical Examination (NME) scores of final-year medical students., Methods: Results of the NME were used to denote the performance of final-year medical students between 2014 and 2021. The NME is a mandatory standardised test that measures the knowledge and competence of students in various clinical subjects. We categorised the data into two periods: period-I (2014-2018) and period-II (2019-2021). Period-I represents students who trained under hostile circumstances, which refer to the devastating effects of a decade-long Syrian crisis. Period-II represents post-hostilities phase, which is marked by a deepening economic crisis., Results: Collected data included test scores for a total of 18 312 final-year medical students from nine medical schools (from six public and three private universities). NME scores improved significantly in period-II compared with period-I tests (p < 0.0001). Campus location or relocation during the crisis affected the results significantly, with higher scores from students of medical schools located in lower-risk regions compared with those from medical schools located in high-risk regions (p < 0.0001), both during and in the post-hostilities phases. Also, students of medical schools re-located to lesser-risk regions scored significantly less than those of medical schools located in high-risk regions (p < 0.0001), but their scores remained inferior to that of students of medical schools that were originally located in lower-risk regions (p < 0.0001)., Conclusion: Academic performance of final year medical students can be adversely affected by crises and conflicts, with a clear tendency to recovery upon crises resolution. The study underscores the importance of maintaining and safeguarding the infrastructure of educational institutions, especially during times of crisis. Governments and educational authorities should prioritise resource allocation to ensure that medical schools have access to essential services, learning resources, and teaching personnel., (© 2024 Association for the Study of Medical Education and John Wiley & Sons Ltd.)

INTRODUCTION At Maastricht University, the integration of knowledge, skills and attitudes in preclinical medical education is promoted by an 'Adoption Programme', where students carry out assignments in a general practice setting. The assignments are explained and discussed in practical medical coaching groups. The aim of this study was to examine the relationships between the elements that affect learning in the programme. METHOD Data from the evaluation questionnaires of 188 medical students were used to test a causal model of learning in the programme. A distinction was made between 'action' and 'reflection on action'. We examined the relationships between perceived instructiveness (the programme's value as a learning experience) and the following variables: comprehensibility; feasibility and execution of the assignments; time spent on the assignments, and the direct and indirect influences of the hosting general practitioner (GP) and the practical medical coaching group. RESULT Performing the assignments had little effect on perceived instructiveness. Coaching by the hosting GP influenced the execution of assignments mainly by its effect on feasibility. Coaching by the GP and in the practical medical coaching groups barely affected perceived instructiveness. DISCUSSION AND CONCLUSION The model gives insight into the contribution of aspects of the Adoption Programme to student learning. The results are probably negatively influenced by the insufficient priority given to the Adoption Programme in the practical medical coaching groups and by inadequate instruction given to the GPs for the purposes of their coaching role. More careful planning of patient-related assignments is recommended. Reflection on assignments and feedback on procedures are needed to lift practical experience onto a higher educational level. [ABSTRACT FROM AUTHOR]

Background: Concerning students' difficulties with fractions, many explanatory approaches are based on the distinction between conceptual knowledge and procedural knowledge. For further research in this field, it is thus crucial to make these constructs accessible to valid measurement., Aims: In this study, we aim at developing a test instrument that affords valid measurement of students' conceptual and procedural fraction knowledge, including in particular empirical validation of this distinction., Sample: The data used in this study were from 8th- and 9th-grade students (N = 235) in Germany., Methods: Facilitated by expert discussions, items from previous studies were developed further and assigned to either a conceptual scale or procedural scale. Confirmatory factor analysis was used to investigate the underlying structure of the data including model comparisons (1-dimensional; conceptual-procedural, verbal-non-verbal). Further analyses in terms of validation focused on reliability and on correlations of the knowledge types with general cognitive abilities., Results: It was found that the theoretically assumed 2-dimensional model fitted the data best. Correlations of the two knowledge types with general cognitive abilities differed significantly. Furthermore, the latent constructs could be reliably estimated from its indicators., Conclusions: Our findings indicate that the empirical separation of conceptual and procedural fraction knowledge is possible: A theoretically grounded test instrument was developed that allows measuring the knowledge types with a sufficient degree of validity. These findings address a research gap that was pointed out repeatedly and gives rise to further research into reasons and remedies for students' difficulties in dealing with fractions., (© 2019 The Authors. British Journal of Educational Psychology published by John Wiley & Sons Ltd on behalf of British Psychological Society.)

Objectives/hypothesis: The objective of this study were to determine if any of the 10 attributes of the standardized letter of recommendation (SLOR) correlate significantly with objective measures of each applicant and to determine whether professor characteristics were associated with reported SLOR attributes., Study Design: Retrospective cohort study., Methods: We reviewed 339 SLORs from 187 otolaryngology residency applicants. The 10 SLOR attributes were converted into a percentile score and statistically analyzed., Results: All 10 SLOR attributes had a mean ± standard deviation above the 80th percentile (0.82-0.93 ± 0.08-0.52) and negative skewness. United States Medical Licensing Examination step 1 and 2 scores correlated most strongly with match potential (B = 93.1,P < .001 and B = 96.3, P = .002, respectively). Students from top 40 National Institutes of Health-funded institutions had a significant difference in length of relationship with their letter writer (B = -0.51, P = .02) and interpersonal communication skills (B = -12.9, P = .011). Alpha Omega Alpha (AOA) and non-AOA members differed in medical knowledge (B = 21.0, P = .007), research (B = -8.80, P = .036), and commitment to otolaryngology (B = -25.89, P = .020). Total number of research experiences correlated most strongly with medical knowledge (B = -16.55, P = .011). There was a significant difference in means between length of relationship and professorial rank (F[4, 339] = 3.61, P = .023), administrative position (F[5, 339] = 5.43, P = .002), and institution of the letter writer (P < .001)., Conclusions: Overall, the lack of correlation between SLOR attributes and objective measures suggests that the SLOR adds little to the residency application. Professor characteristics were similarly not associated with any of the 10 SLOR attributes; however, there were significant differences in the length of relationship between the professor and students with respect to professor rank, administrative rank, and institution of the letter writer., Level of Evidence: NA Laryngoscope, 130:603-608, 2020., (© 2019 The American Laryngological, Rhinological and Otological Society, Inc.)

Context: High-stakes medical examinations seek to be fair to all candidates, including an increasing proportion of trainee doctors with specific learning differences. We aimed to investigate the performance of doctors declaring dyslexia in the clinical skills assessment (CSA), an objective structured clinical examination for licensing UK general practitioners., Methods: We employed a cross-sectional design using performance and attribute data from candidates taking the CSA between 2010 and 2017. We compared candidates who declared dyslexia ('early' before their first attempt or 'late' after failing at least once) with those who did not, using multivariable negative binomial regression investigating the effect of declaring dyslexia on passing the CSA, accounting for relevant factors previously associated with performance, including number of attempts, initial score, sex, place of primary medical qualification and ethnicity., Results: Of 20 879 CSA candidates, 598 (2.9%) declared that they had dyslexia. Candidates declaring dyslexia were more likely to be male (47.3% versus 37.8%; p < 0.001) and to have a non-UK primary medical qualification (26.9% versus 22.4%; p < 0.01), but were no different in ethnicity compared with those who never declared dyslexia. Candidates who declared dyslexia late were significantly more likely to fail compared with those candidates who declared dyslexia early (40.6% versus 9.2%; p < 0.001) and were more likely to have a non-UK medical qualification (79.3% versus 15.6%; p < 0.001) or come from a minority ethnic group (84.9% versus 39.2%; p < 0.001). The chance of passing was lower for candidates declaring dyslexia compared to those who never declared dyslexia and lower in those declaring late (incident rate ratio [IRR], 0.82; 95% confidence interval [CI], 0.70-0.96) compared with those declaring early (IRR, 0.95; 95% CI, 0.93-0.97)., Conclusions: A small proportion of candidates declaring dyslexia were less likely to pass the CSA, particularly if dyslexia was declared late. Further investigation of potential causes and solutions is needed., (© 2019 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Background: A holistic approach to performance assessment recognizes the theoretical complexity of multifaceted critical thinking (CT), a key objective of higher education. However, issues related to reliability, interpretation, and use arise with this approach., Aims and Method: Therefore, we take an analytic approach to scoring students' written responses on a performance assessment. We focus on the complementarity of holistic and analytic approaches and on whether theoretically developed analytical scoring rubrics can produce sub-scores that may measure the 'whole' performance in a holistic assessment., Sample: We use data from the Wind Turbines performance assessment, developed in the iPAL project this study where 55 students at a German university participated., Results: The (sub)scores generated from the scoring scheme empirically reproduced the theoretically assumed structure of CT, with valid and reliable scores in a three-dimensional model. The proposed interpretation of CT as assessed with a performance assessment and measured by the rating scheme was supported preliminarily., Conclusion: Our results support the complementarity of holistic and analytic approaches to assessing CT. When combined, they provide interpretable scores for a complex, multifaceted construct useful in diagnostic contexts., (© 2019 The British Psychological Society.)

Background and Aims: In educational measurement, performance assessments occupy a niche for offering a true-to-life format that affords the measurement of high-level cognitive competencies and the evidence to draw inferences about intellectual capital. However, true-to-life formats also introduce myriad complexities and can skew if not outright distort the accuracy of inferences. For validating claims about test-takers from performance assessments, the collection of evidence about response processes is a necessity of sufficient import that the validation process needs to be labelled a cognitive validation to ensure that the cognitive is not forgotten in the logic of the validation process., Analysis and Example: Cognitive validation is described as a three-pronged process of (1) identifying the knowledge, skills, and attributes associated with the intellectual capital of interest, (2) selecting and/or developing tasks to elicit intellectual capital, and (3) collecting substantive empirical evidence of examinee response processes as part of the overall validity argument. This three-pronged process is illustrated using the American Institute of CPA's (2018) practice analysis, task-based simulations (TBSs), and use of think-aloud interviews to evaluate claims., Conclusions: Although cognitive laboratories and think alouds are used to measure distinct types of response processes as test-takers interact with performance assessments, both methods are among the best for obtaining direct but differential evidence from test-takers. The labour and cost of collecting this evidence are often not done or not done well by many testing programmes. However, for performance assessments to succeed in measuring what they purport to measure, the investment of cognitive validation must be made., (© 2019 The British Psychological Society.)

Background: There is unrealized potential in higher education for greater use of performance assessment, particularly in support of teaching and learning: Well-designed performance tasks can elicit evidence regarding what students know and can do with respect to complex learning objectives. At the same time, there is some pressure, at least in the United States, to widen usage of standardized assessments in order to facilitate comparisons among institutions., Aims and Methods: We distinguish assessments requiring extended performances of some kind from those involving other response formats. Taking student-produced essays as a paradigmatic example, we examine the processes of assessment design and scoring as they pertain to the validity of the inferences drawn from student work products. We also briefly review the history of standardized assessments and the roles they have played in education. Further, we describe how advances in assessment theory and practice have permitted substantial relaxation in the implementation of standardization., Results: Our analysis of these issues in the higher education context draws on empirical evidence regarding both faculty usage of performance tasks (particularly as a means for enhancing the construct validity of an overall assessment) and faculty resistance to the imposition of externally developed, standardized assessments. We also describe efforts undertaken in the United States to strengthen faculty expertise in assessment design, scoring, and interpretation., Conclusions: We conclude that performance tasks have an important role to play in higher education, especially in view of the complexity and depth of the learning goals set for the students. For many purposes, they cannot be replaced by simpler, cheaper forms of assessment. Efforts to facilitate comparability in evaluating institutional outcomes, however, have not borne much fruit despite the substantial investments made. We argue that resources would be better directed at improving the quality of assessments developed and employed by college faculty and suggest how that might be accomplished., (© 2019 The British Psychological Society.)

Background: The International Performance Assessments of Learning (iPAL) project developed a framework for the construction of performance assessments for testing critical thinking, which proposes structuring them around a problem situation that sets an authentic (i.e., real-life-like) problem-solving task for the test-taker. Its critical thinking construct - which reflects the dominant conception of critical thinking - focuses on assessment and use of information, of arguments, and of consequences, and on communication. However, it fails to acknowledge that perspectives on a problem situation are systemically articulated in networks and derive from worldviews, which should also be critically examined., Aims: We aim to elaborate on the iPAL framework and enhance its critical thinking construct, incorporating a new facet of perspective analysis. We also intend to show what design considerations the construction of a performance assessment appropriate for this task entails., Development: Based on Werner Ulrich's Critical Systems Heuristics, which provides a systemic way of analysing perspectives about a problem situation, we argue the need to introduce perspective analysis in the critical thinking construct, and we discuss the considerations necessary for designing critical thinking performance assessments that test perspective analysis skills: about the kind of problem situation the test-taker will face and about the expectations she may have about her task or role in that problem situation. We illustrate how performance assessments of this kind can be constructed by means of an example of an actual test., (© 2019 British Psychological Society.)

Some dental students preferentially schedule graded assessments with adjunct faculty members in expectation of obtaining a higher grade. The aim of this pilot study was to compare the grades given by full-time and adjunct faculty members in clinical periodontics to determine if a difference in grading existed. Grades for students given by seven full-time and 11 adjunct faculty members for various technical and competency assessments in D3 and D4 clinical periodontics courses in academic year 2015-16 were compared. Associations between grades and faculty type were sought using mixed model regression to account for correlations among a student's scores. A total of 845 technical assessment and competency scores from 208 students in the two classes were included in the analysis. Scores were for four procedures: periodontal examination and treatment planning, scaling and root planing, evaluation of initial therapy, and periodontal maintenance. Full-time faculty members graded 58% of the assessments, and 42% were scored by adjunct faculty members. Grades for the adjunct faculty were on average 0.14 (out of 5) higher than those for the full-time faculty (p<0.0001). The number of graded assessments was also significantly associated with a higher average score, suggesting potential student bias toward selecting faculty members who grade more favorably (p<0.0001). These students generally received higher grades from adjunct faculty members, whereas the grades given by full-time faculty members were closer in average and more consist with each other. The results point to the need for more calibration training for adjunct faculty members.

Self-assessments were introduced into a third-year clinical periodontics course to enable dental students to self-assess and receive faculty feedback prior to technical assessment. The aims of this study were to determine if there was an improvement in the pass rate and average grade on the technical assessments following implementation of self-assessments and to analyze the accuracy of the student self-assessments by comparing them with faculty assessments. Scores from 106 third-year dental students for four periodontal procedures were included in the analysis. Faculty and student self-assessment scores, along with self-assessment and final technical assessments scores, were compared, as were the average number of self-assessment attempts for the various procedures and the pass rates. The average number of self-assessments per student was 6.5 compared to 4.2 for the technical assessments. There was no significant difference between students' self-assessment scores and faculty scores (average difference=0.005; p=0.7894). However, there was a significant increase in overall scores between students' self-assessments and technical assessments (average difference: 0.30 out of 5; p=0.0001). There was also a significant difference in pass rate between self-assessments and technical assessments (92.3% and 99.7%, respectively). The individual technical assessment with the greatest improvement was for scaling and root planing (average difference=0.47; p=0.0001), which also had significantly more self-assessments (2.3 vs. 1.2-1.7, p=0.0001). These results suggest that self-assessment and the subsequent faculty feedback provided students with information that enabled them to improve their performance on technical assessments.

Background: Although averaging across multiple examiners' judgements reduces unwanted overall score variability in objective structured clinical examinations (OSCE), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner-cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner-cohort effects on students' scores., Methods: We developed video-based examiner score comparison and adjustment (VESCA): volunteer students were filmed 'live' on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station-specific common-comparator videos, producing partial crossing between examiner cohorts. Many-facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner-cohort effects on students' scores., Results: After accounting for students' ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students' global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6-9.5% students' scores were altered by at least 0.5 standard deviations of student ability., Conclusions: Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness., (© 2018 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Context: Ethnicity-related differences in clinical grades exist. Broad sampling in assessment of clinical competencies involves multiple assessments used by multiple assessors across multiple moments. Broad sampling in assessment potentially reduces irrelevant variances and may therefore mitigate ethnic disparities in clinical grades., Objectives: Research question 1 (RQ1): to assess whether the relationship between students' ethnicity and clinical grades is weaker in a broadly sampled versus a global assessment. Research question 2 (RQ2): to assess whether larger ethnicity-related differences in grades occur when supervisors are given the opportunity to deviate from the broadly sampled assessment score., Methods: Students' ethnicity was classified as Turkish/Moroccan/African, Surinamese/Antillean, Asian, Western, and native Dutch. RQ1: 1667 students (74.3% native Dutch students) were included, who entered medical school between 2002 and 2004 (global assessment, 818 students) and between 2008 and 2010 (broadly sampled assessment, 849 students). The main outcome measure was whether or not students received ≥3 times a grade of 8 or higher on a scale from 1 to 10 in five clerkships. RQ2: 849 students (72.4% native Dutch students) were included, who were assessed by broad sampling. The main outcome measure was the number of grade points by which supervisors had deviated from broadly sampled scores. Both analyses were adjusted for gender, age, (im)migration status and average bachelor grade., Results: Research question 1: ethnicity-related differences in clinical grades were smaller in broadly sampled than in global assessment, and this was also seen after adjustments. More specifically, native Dutch students had reduced probabilities (0.87-0.65) in broadly sampled as compared with global assessment, whereas Surinamese (0.03-0.51) and Asian students (0.21-0.30) had increased probabilities of having ≥3 times a grade of 8 or higher in five clerkships. Research question 2: when supervisors were allowed to deviate from original grades, ethnicity-related differences in clinical grades were reintroduced., Conclusions: Broadly sampled assessment reduces ethnicity-related differences in grades., (© 2019 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Context: As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society., Objectives: We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role., Methods: We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies., Discussion: As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians., (© 2018 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Context: Single-best-answer questions (SBAQs) have been widely used to test knowledge because they are easy to mark and demonstrate high reliability. However, SBAQs have been criticised for being subject to cueing., Objectives: We used a novel assessment tool that facilitates efficient marking of open-ended very-short-answer questions (VSAQs). We compared VSAQs with SBAQs with regard to reliability, discrimination and student performance, and evaluated the acceptability of VSAQs., Methods: Medical students were randomised to sit a 60-question assessment administered in either VSAQ and then SBAQ format (Group 1, n = 155) or the reverse (Group 2, n = 144). The VSAQs were delivered on a tablet; responses were computer-marked and subsequently reviewed by two examiners. The standard error of measurement (SEM) across the ability spectrum was estimated using item response theory., Results: The review of machine-marked questions took an average of 1 minute, 36 seconds per question for all students. The VSAQs had high reliability (alpha: 0.91), a significantly lower SEM than the SBAQs (p < 0.001) and higher mean item-total point biserial correlations (p < 0.001). The VSAQ scores were significantly lower than the SBAQ scores (p < 0.001). The difference in scores between VSAQs and SBAQs was attenuated in Group 2. Although 80.4% of students found the VSAQs more difficult, 69.2% found them more authentic., Conclusions: The VSAQ format demonstrated high reliability and discrimination and items were perceived as more authentic. The SBAQ format was associated with significant cueing. The present results suggest the VSAQ format has a higher degree of validity., (© 2018 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Assessments are widely used in dental education to record the academic progress of students and ultimately determine whether they are ready to begin independent dental practice. Whilst some would consider this a "rite-of-passage" of learning, the concept of assessments in education is being challenged to allow the evolution of "assessment for learning." This serves as an economical use of learning resources whilst allowing our learners to prove their knowledge and skills and demonstrating competence. The Association for Dental Education in Europe and the American Dental Education Association held a joint international meeting in London in May 2017 allowing experts in dental education to come together for the purposes of Shaping the Future of Dental Education. Assessment in a Global Context was one topic in which international leaders could discuss different methods of assessment, identifying the positives, the pitfalls and critiquing the method of implementation to determine the optimum assessment for a learner studying to be a healthcare professional. A post-workshop survey identified that educators were thinking differently about assessment, instead of working as individuals providing isolated assessments; the general consensus was that a longitudinally orientated systematic and programmatic approach to assessment provide greater reliability and improved the ability to demonstrate learning., (© 2018 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd.)

Context: Optimising student learning and academic performance is a continuous challenge for medical schools. The assessment policy may influence both learning and performance. Previously, the joint contribution of self-regulated learning (SRL) and participation in scheduled learning activities towards academic performance has been reported. However, little is known about the relationships between SRL, participation and academic performance under different assessment policies., Objectives: The goal of this study was to investigate differences in average scores of SRL, participation and academic performance of students under two assessment policies: (i) a conjunctive lower stakes, lower performance standard (old) assessment policy and (ii) a compensatory higher stakes, higher performance standard (new) assessment policy. In addition, this research investigated whether the relationships between academic performance, SRL and participation are similar across both assessment policies., Methods: Year-1 medical students (i) under the old assessment policy (n = 648) and (ii) under the new assessment policy (n = 529) completed the Motivated Strategies for Learning Questionnaire on SRL, and additional items on participation. Year-1 performance was operationalised as students' average Year-1 course examination grades. manova and structural equation modelling were used for analyses., Results: Generally, students under the new assessment policy showed significantly higher Year-1 performance, SRL and participation, compared with students under the old assessment policy. The relationships between Year-1 performance, SRL and participation were similar across assessment policies., Conclusions: This study indicates that the higher academic performance under a compensatory higher stakes, higher performance standard assessment policy, results from higher SRL and participation, but not from altered relationships between SRL, participation and performance. In sum, assessment policies have the potential to optimise student learning and performance., (© 2017 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Dental hygiene students' performance in oral radiology courses may give an early indication of their readiness prior to taking the National Board Dental Hygiene Examination (NBDHE). The aim of this study was to determine the relationship between dental hygiene students' performance in an oral radiology lecture course and their performance on the NBDHE. Data were collected for all 117 dental hygiene students at Texas A&M University College of Dentistry from 2006 to 2009 who took the NBDHE during their second year of the program. Their final grades and scores on three written section examinations in an oral radiology course taken in their first year were compared with their overall NBDHE scores and raw scores on the oral radiology and case study sections. Moderate correlations (0.3

In this study, we manipulated mastery criterion form (rolling or block) and stringency (across 6 or 12 trials) and measured the emergence of derived relations. College students learned neuroanatomy equivalence classes and experienced one of two rolling mastery criteria (6 or 12 consecutive correct responses) or a block mastery criterion (12 trials in a block) during training. The study found that block and rolling mastery criteria produced similar outcomes. Effectiveness was hampered when the criterion was less stringent., (© 2017 Society for the Experimental Analysis of Behavior.)

Background: Student affect plays a considerable role in mathematical problem solving performance, yet is rarely formally assessed. In this manuscript, an instrument and its properties are discussed to enable educational psychologists the opportunity to assess student affect., Aims: The study was conducted to norm the CAIMPS (instrument) with gifted students. In so doing, educational psychologists are informed of the process and the instrument's properties., Sample: The sample was comprised of 160 middle-grade (7 and 8) students, identified as gifted, in the United States., Methods: After completing one of four model-eliciting activities (MEAs), all participants completed the CAIMPS (Chamberlin Affective Instrument for Mathematical Problem Solving). Data were analysed using confirmatory factor analysis to ascertain the number of factors in the instrument. The normed fit index (0.6939), non-normed fit index (0.8072), and root mean square error approximation (.076) were at or near the acceptable levels. Alpha levels for factors were also robust (.637-.923)., Results and Conclusion: Data suggest that the instrument was a good fit for use with mathematics students in middle grades when solving problems. Perhaps the most impressive characteristic of the instrument was that the four factors (AVI: anxiety, value, and interest), SS (self-efficacy and self-esteem), ASP (aspiration), and ANX (anxiety) did not correlate highly with one another, which defies previous hypotheses in educational psychology., (© 2017 The British Psychological Society.)

In 2012, when the National Board Dental Examination (NBDE) was changed from a numerical scoring system to pass/fail, advanced dental education programs lost a metric widely used for differentiating applicants to those programs. The American Dental Association (ADA) has developed the Advanced Dental Admission Test (ADAT) to address this issue. Implementation of the ADAT began in 2016 with a pilot program, which has not yet been widely accepted in the overall admissions process. This Point/Counterpoint explores the benefits and challenges of using the ADAT for postgraduate admissions. Viewpoint 1 supports use of the ADAT, arguing that the test provides a viable, long-term solution to this immediate need. In contrast, Viewpoint 2 questions the need for and appropriateness of this additional academic measure for postgraduate admissions.

Background: A fundamental aim of standardized educational assessment is to achieve reliable discrimination between students differing in the knowledge, skills and abilities assessed. However, questions of the purity with which these tests index students' genuine abilities have arisen. Specifically, literacy and numeracy assessments may also engage unintentionally assessed capacities., Aims: The current study investigated the extent to which domain-general processes - working memory (WM) and non-verbal reasoning - contribute to students' standardized test performance and the pathway(s) through which they exert this influence., Sample: Participants were 91 Grade 2 students recruited from five regional and metropolitan primary schools in Australia., Methods: Participants completed measures of WM and non-verbal reasoning, as well as literacy and numeracy subtests of a national standardized educational assessment., Results: Path analysis of Rasch-derived ability estimates and residuals with domain-general cognitive abilities indicated: (1) a consistent indirect pathway from WM to literacy and numeracy ability, through non-verbal reasoning; (2) direct paths from phonological WM and literacy ability to numeracy ability estimates; and (3) a direct path from WM to spelling test residuals., Conclusions: Results suggest that the constitution of this nationwide standardized assessment confounded non-targeted abilities with those that were the target of assessment. This appears to extend beyond the effect of WM on learning more generally, to the demands of different assessment types and methods. This has implications for students' abilities to demonstrate genuine competency in assessed areas and the educational supports and provisions they are provided on the basis of these results., (© 2016 The British Psychological Society.)

Multidimensional computerized adaptive testing (MCAT) has received increasing attention over the past few years in educational measurement. Like all other formats of CAT, item replenishment is an essential part of MCAT for its item bank maintenance and management, which governs retiring overexposed or obsolete items over time and replacing them with new ones. Moreover, calibration precision of the new items will directly affect the estimation accuracy of examinees' ability vectors. In unidimensional CAT (UCAT) and cognitive diagnostic CAT, online calibration techniques have been developed to effectively calibrate new items. However, there has been very little discussion of online calibration in MCAT in the literature. Thus, this paper proposes new online calibration methods for MCAT based upon some popular methods used in UCAT. Three representative methods, Method A, the 'one EM cycle' method and the 'multiple EM cycles' method, are generalized to MCAT. Three simulation studies were conducted to compare the three new methods by manipulating three factors (test length, item bank design, and level of correlation between coordinate dimensions). The results showed that all the new methods were able to recover the item parameters accurately, and the adaptive online calibration designs showed some improvements compared to the random design under most conditions., (© 2017 The British Psychological Society.)

Introduction: Assessment of practical skills in undergraduate dental education has difficulties, including a lack of transparency/objectivity in student evaluations. This prospective study investigated whether agreement between student- and faculty-based assessments increased when students were trained to use the assessment criteria., Materials and Methods: Assessment criteria were available for 6 tasks in the Phantom course of Conservative Dentistry: Class II cavity preparation, composite restoration, gold and ceramic partial crown preparation, trepanation and access cavity preparation, and root canal filling. Forty-two students were allocated to three different assessment training groups. Students' self-assessments of practical tests were compared with instructors' assessments, depending on training intensity and task type (Pearson's rho, ANOVA, Tukey's post hoc test, Kruskal-Wallis test). Students/instructors were questioned regarding benefits/drawbacks of the criteria., Results: Student self-assessments showed either consent (24.2%), overestimation (31%) or underestimation (44.8%). Gender differences were negligible. Regarding passing/failing grades, more intensive training yielded significantly increased agreement only for preparation of gold partial crowns (rho = +0.313, P = 0.044). Ratings exhibited significant differences amongst tasks (P = 0.002), for example, trepanation assessment had significantly lower differences than gold or ceramic preparation assessments. These discrepancies decreased with increased training. Students consistently reported benefitting most from learning the criteria for gold preparations, followed by composite and ceramic preparations. They also reported that learning the criteria was beneficial for knowledge transfer/feedback. Instructors rated the criteria as helpful for task evaluation and feedback efficiency., Conclusions: Pre-defined assessment criteria may increase consistency between student self-assessment and instructor assessment depending on the task, improving transparency and feedback in dental education., (© 2015 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd.)

Multisource assessment (MSA) uses multiple assessors to provide feedback. Little is known about the validity of using MSA feedback for improving students' ability to self-assess in a preclinical environment. Therefore, the aim of this study was to measure the validity of using a defined reflective process involving an MSA tool for building skill in dental students' self-evaluation of caries excavation on extracted teeth. As part of this process, 104 first-year students at one U.S. dental school used a self-generated study plan (SGSP) for structured reflection on MSA feedback during the 2013-14 academic year. Interrater agreement, determined through calculation of percentage-agreements in scoring, was measured among three assessor groups (self-, peer, and expert assessors) in formative assessment and between two assessor groups (self- and expert assessors) in summative assessment two weeks apart, allowing for reflective practice and completion of an SGSP between assessments. Validity for improving self-assessment was determined by measuring significance in positive shifts of agreement between self- and expert assessors. The results showed that interrater agreement between the self- and expert assessors increased significantly: from a 28% agreement in formative assessment to a 60% agreement in summative assessment. Significance in percentage shifts between assessments was demonstrated with a McNemar score of 0.26 (p<0.001). These results suggest that the described MSA tool and reflective process in an SGSP may be valid methods for improving skill in student self-evaluation of competence in caries excavation on extracted teeth.

Protecting the security of examination questions is an important task for high-stakes examining boards/agencies and university programs. To maintain the security of questions, examining boards and university programs use a combination of prevention, detection, and enforcement strategies. A common prevention strategy is to establish a number of controls on access to questions; however, restricting access can motivate examinees to try harder to reconstruct questions that may appear on future versions of the test. Moreover, access to study materials by some groups and not others can present a challenge to the fairness of examinations. The aim of this study was to evaluate the empirical stability of test characteristics. Specifically, the primary research objective was to investigate the empirical stability of the items and test forms of a written examination before and after a specific policy decision was implemented. As a response to both of these concerns, this article describes a study that evaluated how psychometric (i.e., statistical) properties of test forms and individual questions might be affected by publicly releasing a larger number of questions from an item question pool. A series of analyses were conducted, including item drift to evaluate stability of the characteristics. The results suggest that empirical characteristics of the test forms and individual questions have remained relatively stable since the release policy was implemented. Specifically, statistical properties of the test forms have continued to perform similarly to test forms that were constructed prior to the release. Although the results of this study were promising, the context of this specific testing program may have offered additional protections such as a limited number of administrations that others may not. Therefore, testing/examining agencies and university programs may want to consider this strategy with appropriate caution.

The aims of this study were to identify the level of interexaminer agreement among preclinical operative dentistry faculty members when grading Class II preparations performed by first-year dental students; to develop discrimination exercises for specific preparation components where interexaminer agreement was poor; and to evaluate if the discrimination exercises were able to improve inter- and intraexaminer agreement. In the preliminary phase of this study, 13 components of 32 Class II cavity preparations were assessed by eight course faculty members at one U.S. dental school. Analysis of average interexaminer agreement on these components revealed that six were below 60%. These were proximal contact clearance, retention groove placement, retention groove depth, preparation walls, preparation margins, and preparation toilet/debris. A 30-minute calibration session was subsequently developed to provide discrimination exercises utilizing 3-D models and digital images of various levels of student performance for five of the six components. Immediately following calibration, the course faculty assessed the same 32 preparations (Phase I) followed by a delayed assessment without calibration (Phase II) approximately six months later. The results showed that overall interexaminer reliability improved after calibration. Although there was a decline in interexaminer reliability after an interval of six months (Phase II), the degree of variation among examiners was lower than in the preliminary assessment. These findings support the use of discrimination exercises for preclinical operative dentistry course faculty to increase interexaminer agreement and thereby improve the consistency of faculty-student communication.

Context: In the UK, supervised learning events (SLE) replaced traditional workplace-based assessments for foundation-year trainees in 2012. A key element of SLEs was to incorporate trainee reflection and assessor feedback in order to drive learning and identify training issues early. Few studies, however, have investigated the value of SLEs in predicting doctors in difficulty. This study aimed to identify principles that would inform understanding about how and why SLEs work or not in identifying doctors in difficulty (DiD)., Methods: A retrospective case-control study of North West Foundation School trainees' electronic portfolios was conducted. Cases comprised all known DiD. Controls were randomly selected from the same cohort. Free-text supervisor comments from each SLE were assessed for the four domains defined in the General Medical Council's Good Medical Practice Guidelines and each scored blindly for level of concern using a three-point ordinal scale. Cumulative scores for each SLE were then analysed quantitatively for their predictive value of actual DiD. A qualitative thematic analysis was also conducted., Results: The prevalence of DiD in this sample was 6.5%. Receiver operator characteristic curve analysis showed that Team Assessment of Behaviour (TAB) was the only SLE strongly predictive of actual DiD status. The Educational Supervisor Report (ESR) was also strongly predictive of DiD status. Fisher's test showed significant associations of TAB and ESR for both predicted and actual DiD status and also the health and performance subtypes. None of the other SLEs showed significant associations. Qualitative data analysis revealed inadequate completion and lack of constructive, particularly negative, feedback. This indicated that SLEs were not used to their full potential., Conclusions: TAB and the ESR are strongly predictive of DiD. However, SLEs are not being used to their full potential, and the quality of completion of reports on SLEs and feedback needs to be improved in order to better identify and manage DiD., (© 2016 John Wiley & Sons Ltd.)

The 2015 American College of Clinical Pharmacy (ACCP) Educational Affairs Committee was charged with developing a self-assessment guide for residency programs to quantitatively and qualitatively evaluate the outcomes of resident teaching curricula. After extensively reviewing the literature, the committee developed assessment rubrics modeled after the 2013 ACCP white paper titled "Guidelines for Resident Teaching Experiences" and the revised American Society of Health-System Pharmacists (ASHP) 2014 accreditation standards for PGY1 residencies, which place greater emphasis on the teaching and learning curriculum (TLC) than the previous accreditation standards. The self-assessment guide developed by the present committee can serve as an assessment tool for both basic and expanded TLCs. It provides the criteria for program goals, mentoring, directed readings with topic discussions, teaching experiences, and assessment methodology. For an expanded TLC, the committee has provided additional guidance on developing a teaching philosophy, becoming involved in interactive seminars, expanding teaching experiences, developing courses, and serving on academic committees. All the guidelines listed in the present paper use the measures "not present," "developing," and "well developed" so that residency program directors can self-assess along the continuum and identify areas of excellence and areas for improvement. Residency program directors should consider using this new assessment tool to measure program quality and outcomes of residency teaching experiences. Results of the assessment will help residency programs focus on areas within the TLC that will potentially benefit from additional attention and possible modification., (© 2016 Pharmacotherapy Publications, Inc.)

Test items created for dentistry examinations are often individually written by content experts. This approach to item development is expensive because it requires the time and effort of many content experts but yields relatively few items. The aim of this study was to describe and illustrate how items can be generated using a systematic approach. Automatic item generation (AIG) is an alternative method that allows a small number of content experts to produce large numbers of items by integrating their domain expertise with computer technology. This article describes and illustrates how three modeling approaches to item content-item cloning, cognitive modeling, and image-anchored modeling-can be used to generate large numbers of multiple-choice test items for examinations in dentistry. Test items can be generated by combining the expertise of two content specialists with technology supported by AIG. A total of 5,467 new items were created during this study. From substitution of item content, to modeling appropriate responses based upon a cognitive model of correct responses, to generating items linked to specific graphical findings, AIG has the potential for meeting increasing demands for test items. Further, the methods described in this study can be generalized and applied to many other item types. Future research applications for AIG in dental education are discussed.

Context: As a contribution to this special issue commemorating the journal's 50th volume, this paper seeks to explore directions for national licensing examinations (NLEs) in medicine. Increases in the numbers of new medical schools and the mobility of doctors across national borders mean that NLEs are becoming even more important to ensuring physician competence., Objectives: The purpose of this paper is to explore the use of NLEs in the future in the context of global changes in medical education and health care delivery., Methods: Because the literature related to NLEs is so large, we have not attempted a comprehensive review, but have focused instead on a small number of topics on which we think we have something useful to say. The paper is organised around five predicted trends for NLEs., Discussion: The first section discusses reasons why we think the use of NLEs will increase in the coming years. The second section discusses the ongoing problem of content specificity and its implications for the design of NLEs. The third section examines the evolution of large-scale, standardised cognitive assessments in NLEs and suggests some future directions. Reflecting the fact that NLEs are, increasingly, attempting to assess more than just knowledge, the fourth section addresses the future of large-scale clinical skills assessments in NLEs, predicting both increases in their use and some shifts in the nature of the stations used. The fifth section discusses workplace-based assessments, predicting increases in their use for formative assessment and identifying some limitations in their direct application in NLEs. The concluding section discusses the cost of NLEs and indulges in some further speculations about their evolution., (© 2015 John Wiley & Sons Ltd.)

The European Board of Ophthalmology (EBO) aims to establish common European standards in ophthalmology. The comprehensive EBO Diploma (EBOD) examination has been officially used since 1995 by an increasing number of European countries to assess the level of education in ophthalmology of their residents at the end of their training. The current comprehensive EBOD examination ensures a minimum standard of knowledge. We present an update on the EBO plans to establish a subspecialty advanced EBOD (AdEBOD) examination in order to award ophthalmologists who complete subspecialty training with formal recognition of their expertise. The purpose, structure and process of the AdEBOD examination are presented., (© 2015 Acta Ophthalmologica Scandinavica Foundation. Published by John Wiley & Sons Ltd.)

Long training workshops on the writing of exam questions have been shown to be effective; however, the effectiveness of short workshops needs to be demonstrated. The aim of this study was to evaluate the impact of a one-day, seven-hour faculty development workshop at the College of Dentistry, King Saud University, Saudi Arabia, on the quality of multiple-choice questions (MCQs). Kirkpatrick's four-level evaluation model was used. Participants' satisfaction (Kirkpatrick's Level 1) was evaluated with a post-workshop questionnaire. A quasi-experimental, randomized separate sample, pretest-posttest design was used to assess the learning effect (Kirkpatrick's Level 2). To evaluate transfer of learning to practice (Kirkpatrick's Level 3), MCQs created by ten faculty members as a result of the training were assessed. To assess Kirkpatrick's Level 4 regarding institutional change, interviews with three key leaders of the school were conducted, coded, and analyzed. A total of 72 course directors were invited to and attended some part of the workshop; all 52 who attended the entire workshop completed the satisfaction form; and 22 of the 36 participants in the experimental group completed the posttest. The results showed that all 52 participants were highly satisfied with the workshop, and significant positive changes were found in the faculty members' knowledge and the quality of their MCQs with effect sizes of 0.7 and 0.28, respectively. At the institutional level, the interviews demonstrated positive structural changes in the school's assessment system. Overall, this one-day item-writing faculty workshop resulted in positive changes at all four of Kirkpatrick's levels; these effects suggest that even a short training session can improve a dental school's assessment of its students.

The aim of this study was to analyze multiple-choice questions used in teaching prosthodontics to dental students based on the items' distractor effectiveness and their difficulty and discrimination capacity. The overall goals were to assist the prosthodontics department in modifying its existing question bank and creating a set of prevalidated questions of known difficulty and discrimination capacity. The items selected for this study were from the three formative examinations in prosthodontics for fourth-year BDS students at Sinhgad Dental College and Hospital, Pune, Maharastra, India, during academic year 2011-12. As all the students' completed exams and scores were available, the task was to evaluate the efficacy of items already administered. Based on the item analysis of the questions, the following results were noted. None of the three examinations had items that were too difficult or defective. However, all three examinations had large percentages of easy items and many items had a low discrimination index. More than 60-70% of the items on all three examinations included ineffective distractors. All items therefore needed modifications based on their distractor effectiveness and difficulty and discrimination indices. After the study was completed, the results were discussed in a meeting with all teachers in the prosthodontics department and the test constructors, and suggestions were made with regard to improving the items for better evaluation of students.