Search

This study examines the legality and appropriateness of keeping the multiple-choice question items of the Korean Medical Licensing Examination (KMLE) confidential. Through an analysis of cases from the United States, Canada, and Australia, where medical licensing exams are conducted using item banks and computer-based testing, we found that exam items are kept confidential to ensure fairness and prevent cheating. In Korea, the Korea Health Personnel Licensing Examination Institute (KHPLEI) has been disclosing KMLE questions despite concerns over exam integrity. Korean courts have consistently ruled that multiple-choice question items prepared by public institutions are non-public information under Article 9(1)(v) of the Korea Official Information Disclosure Act (KOIDA), which exempts disclosure if it significantly hinders the fairness of exams or research and development. The Constitutional Court of Korea has upheld this provision. Given the time and cost involved in developing high-quality items and the need to accurately assess examinees' abilities, there are compelling reasons to keep KMLE items confidential. As a public institution responsible for selecting qualified medical practitioners, KHPLEI should establish its disclosure policy based on a balanced assessment of public interest, without influence from specific groups. We conclude that KMLE questions qualify as non-public information under KOIDA, and KHPLEI may choose to maintain their confidentiality to ensure exam fairness and efficiency.

Introduction: Online open book assessment has been a common alternative to a traditional invigilated test or examination during the COVID-19 pandemic. However, its unsupervised nature increases ease of cheating, which is an academic integrity concern. This study's purpose was to evaluate the integrity of two online open book assessments with different formats (1. Tightly time restricted - 50 min for mid-semester and 2. Take home - any 4 h within a 24-h window for end of semester) implemented in a radiologic pathology unit of a Bachelor of Science (Medical Radiation Science) course during the pandemic., Methods: This was a retrospective study involving a review and analysis of existing information related to the integrity of the two radiologic pathology assessments. Three integrity evaluation approaches were employed. The first approach was to review all the Turnitin plagiarism detection software reports with use of 'seven-words-in-a-row' criterion to identify any potential collusion. The second approach was to search for highly irrelevant assessment answers during marking for detection of other cheating types. Examples of highly irrelevant answers included those not addressing question requirements and stating patients' clinical information not from given patient histories. The third approach was an assessment score statistical analysis through descriptive and inferential statistics to identify any abnormal patterns that might suggest cheating occurred. An abnormal pattern example was high assessment scores. The descriptive statistics used were minimum, maximum, range, first quartile, median, third quartile, interquartile range, mean, standard deviation, fail and full mark rates. T-test was employed to compare mean scores between the two assessments in this year (2020), between the two assessments in the last year (2019), between the two mid-semester assessments in 2019 and 2020, and between this and last years' end of semester assessments. A p-value of less than 0.05 was considered statistically significant., Results: No cheating evidence was found in all Turnitin reports and assessment answers. The mean scores of the end of semester assessments in 2019 (88.2%) and 2020 (90.9%) were similar (p = 0.098). However, the mean score of the online open book mid-semester assessment in 2020 (62.8%) was statistically significantly lower than that of the traditional invigilated mid-semester assessment in 2019 (71.8%) with p < 0.0001., Conclusion: This study shows the use of the online open book assessments with tight time restrictions and the take home formats in the radiologic pathology unit did not have any academic integrity issues. Apparently, the strict assessment time limit played an important role in maintaining their integrity., (Copyright © 2020. Published by Elsevier Inc.)

The COVID-19 pandemic has had a profound effect on society and higher education in Australia. In just a few weeks, entire courses have been re-structured and are now delivered online. The need to adapt rapidly has prompted many innovative changes that will ultimately have long-term benefits for medical education in Australia and New Zealand., (© 2020 Royal Australasian College of Physicians.)

There is inconclusive evidence suggesting which standard setting method yields the highest validity for pass/fail decisions in examinations. The Objective Borderline Method 2 (OBM2) is a decision-making tool for reclassification of borderline grades to clear pass or clear fail grades to resolve examiner uncertainty for high-stakes pass/fail decisions. This study evaluated the predictive validity of OBM2 pass/fail decisions, using consecutive years' Objective Structured Clinical Examination (OSCE) results within a medical cohort (n=271) at the University of New South Wales, Australia. OBM2 decisions in one OSCE (n=687) were compared to marks obtained in a subsequent OSCE via independent samples T-tests and analysis of variance (ANOVA). The extent of the relationship between these two variables determines the predictive validity of OBM2 decisions, given that past student grades are capable of predicting future performance. OBM2 decisions in an initial OSCE were found to have a statistically significant predictive nature for subsequent OSCE marks (p=.005). For initial decisions which reclassified to a pass grade, subsequent OSCE marks were significantly higher than for the cases where initial decisions were reclassified to a fail grade. Stronger associations were identified between related assessment domains/criteria compared to unrelated domains/criteria (Cohen's d=.469 vs Cohen's d=.388 respectively). Through demonstrating the OBM2 decisions' predictive association across exams there is support for the OBM2's predictive validity, deeming it a promising method to be used for resolving examiner uncertainty when making pass/fail decisions within OSCEs., Competing Interests: The authors declare that they have no competing interests., (Copyright © 2020 Klein Nulend et al.)

Background: Development of an entry-level physiotherapy curriculum in China currently follows the World Confederation for Physical Therapy (WCPT) guidelines, however there is no standard, validated, assessment tool for physiotherapy practice in use in China. This article reports the process of translation of the "Assessment of Physiotherapy Practice" (APP), a validated assessment instrument adopted by all universities in Australia and New Zealand, into Chinese (APP-Chinese) and its implementation by Chinese physiotherapy clinical educators (CEs) and students during clinical placements., Methods: The process of forward and backward translation of the APP was undertaken by a team of academics from universities in Shanghai, Hong Kong, United States and Australia. An APP-Chinese version was produced and used for assessment of the clinical performance of 4th year students at a university in Shanghai. Feedback on the implementation of the APP-Chinese was solicited from students and CEs using the same two questionnaires employed to assess implementation of the original APP., Results: All CEs agreed that the rules used to score the APP-Chinese were helpful in assessing student performance. Over 90% of the CEs considered the APP-Chinese was pragmatic for use in the clinical environment in China. All students agreed with the rating of their performance on the APP-Chinese marked by their educators, and that the performance indicators were useful in guiding their expected performance behaviour., Conclusion: The APP-Chinese is the first standardised assessment tool for evaluation of clinical performance of physiotherapy students in China and was shown to be well accepted by both students and CEs in the clinical education unit and university involved in this study.

Introduction: Relatively few general practice (GP) workplace-based assessment instruments have been psychometrically evaluated. This study aims to establish the content validity and internal consistency of the General Practice Registrar Competency Assessment Grid (GPR-CAG). Methods: The GPR-CAG was constructed as a formative assessment instrument for Australian GP registrars (trainees). GPR-CAG items were determined by an iterative literature review, expert opinion and pilot-testing process. Validation data were collected, between 2014 and 2016, during routine clinical teaching visits within registrars' first two general practice training terms (GPT1 and GPT2) for registrars across New South Wales and the Australian Capital Territory. Factor analysis and expert consensus were used to refine items and establish GPR-CAG's internal structure. GPT1 and GPT2 competencies were analysed separately. Results: Data of 555 registrars undertaking GPT1 and 537 registrars undertaking GPT2 were included in analyses. A four-factor, 16-item solution was identified for GPT1 competencies (Cronbach's alpha range: 0.71-0.83) and a seven-factor 27-item solution for GPT2 competencies (Cronbach's alpha: 0.63-0.84). The emergent factor structures were clinically characterisable and resonant with existing medical education competency frameworks. Discussion: This study establishes initial evidence for the content validity and internal consistency of GPR-CAG. GPR-CAG appears to have utility as a formative GP training WBA instrument.

Introduction: Outcomes-based education requires active student learning with assessment strategies that foster deep approaches to learning, which are often influenced by students' perceptions of assessment. We aimed to investigate the perceptions of pharmacy students at an Australian university about their experiences of assessment and its impact on their learning., Methods: A mixed-methods study was conducted involving a self-administered questionnaire and semi-structured focus groups. Descriptive statistics were used to rate the perceived value of different assessment tools and confirmed by themes arising from the focus groups. Examination results over a five-year period were also collated to assess congruence between perceptions and academic performance., Results: From the 123 questionnaire and nine focus group participants, short-answer questions were the most positively-received form of assessment due to students being able to demonstrate and receive marks for partial knowledge. Multiple-choice questions received mixed response as they were cited as being useful in assessing student knowledge but potentially difficult to interpret/answer correctly. Reflective pieces received the lowest ratings and were considered the least beneficial. Key identified themes were ensuring quality assurance of assessment processes, use of authentic assessment, timely feedback, and appropriate match between workload and assessment weightings. Overall, there was congruence between students' exam scores and their perceptions of the different assessment types., Conclusions: Strategic planning and delivery of correctly-weighted authentic assessments with the provision of constructive feedback are key elements for active engagement of students and achievement of life-long learning outcomes., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Objectives: The formative aspect of the mini-clinical evaluation exercise (mini-CEX) in postgraduate medical workplace-based assessment is intended to afford opportunities for active learning. Yet, there is little understanding of the perceived relationship between the mini-CEX and how trainees self-regulate their learning. Our objective was to explore trainees' perceptions of their mini-CEX experiences from a learning perspective, using Zimmerman's self-regulated learning theoretical framework as an interpretive lens., Design: Qualitative, using semi-structured interviews conducted in 2017. The interviews were analysed thematically., Setting: Geriatric medicine training., Participants: Purposive sampling was employed to recruit geriatric medicine trainees in Melbourne, Australia. Twelve advanced trainees participated in the interviews., Results: Four themes were found with a cyclical inter-relationship between three of these themes: namely, goal setting, task translation and perceived outcome. These themes reflect the phases of the self-regulated learning framework. Each phase was influenced by the fourth theme, supervisor co-regulation. Goal setting had motivational properties that had significant impact on the later phases of the cycle. A 'tick box' goal aligned with an opportunistic approach and poorer perceived educational outcomes. Participants reported that external feedback following assessment was critical for their self-evaluation, affective responses and perceived outcomes., Conclusions: Trainees perceived the performance of a mini-CEX as a complex, inter-related cyclical process, influenced at all stages by the supervisor. Based on our trainee perspectives of the mini-CEX, we conclude that supervisor engagement is essential to support trainees to individually regulate their learning in the clinical environment., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Failing to fail nursing students in clinical assessments is an internationally acknowledged issue that necessitates thorough examination. Despite the gravity of 'mis-grading' nursing students, current research exploring this subject is limited. Understanding the experiences of assessors when faced with an underperforming student may help illuminate some of the broad issues associated with failure to fail., Objectives: To describe both tertiary and industry based assessors' experiences of grading nursing student performances in clinical courses when that performance was not a clear pass or fail., Design: A pilot study using a descriptive survey design was employed., Setting: Participants were drawn from all states and territories in Australia., Participants: Academic and industry assessors of undergraduate nursing students across Australian universities participated in this study., Methods: A newly developed and validated survey was conducted via an online platform. Descriptive data were collected on assessors' experiences across the domains of the Invitational Theory: people; processes; programmes; policies and places., Results: Participants clearly had a strong duty of care to patients and the nursing profession. However, 23.5% of participants had given the benefit of the doubt to student performances. They reported failing student performances nevertheless reported passing students. Some participants experienced a number of coercive student behaviours. They perceived that the culture of the ward impacted on student performances in assessments. Inadequate time to assess students in clinical practice was reported by 44% of participants. Participants reported no differentiation between theoretical and clinical assessments in their organisations' assessment policies., Conclusions: Assessors reported many challenges when assessing students who were not performing at requisite standards. This study furthered the extant literature around the impact of people and processes on assessors and failure to fail. For the first time, assessors' experiences of the assessment environment, programme design and organisational policies associated with grading marginal student performances are reported., (Copyright © 2019 Elsevier Ltd. All rights reserved.)

Objective: The objective of this collaborative study was to compare current practices of conducting high-stake, exit-level Objective Structured Clinical Examinations (OSCEs) at all Australian medical schools. We aimed to document similarities and differences between schools, and compare existing practice against available gold standard, evidence-based practice. We also aimed to identify areas where gold standards do not currently exist, and could be developed in the future. Methods: A 72-item semi-structured questionnaire was sent to all 19 Australian medical schools with graduating students. Results: A total of 18/19 schools responded. Of these, 16/18 schools had summative exit-level OSCEs representing content from multiple medical specialties. The total number of OSCE stations varied from 8 to 16, with total OSCE testing time ranging from 70 to 160 min. All schools blueprinted their OSCE to their curriculum, and trained simulated patients and examiners. There was variation in the format of marking rubric used. Conclusions: This study has provided insight into the current OSCE practices of the majority of medical schools in Australia. Whilst the comparative data reveal a wide variation in OSCE practices between schools, many recommended "gold standard" OSCE practices are implemented. The collective awareness of our similarities and differences provides us with a baseline platform, as well as an impetus for iterative quality improvement. Such discourse also serves to develop new gold standards in practice where none have previously existed.

Purpose: Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions., Methods: We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments., Results: We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively., Conclusions: Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.

Background: Reliable interpretation of the Assessment of Physiotherapy Practice (APP) tool is necessary for consistent assessment of physiotherapy students in the clinical setting. However, since the APP was implemented, no study has reassessed how consistently a student performance is evaluated against the threshold standards. Therefore, the primary aim of this study was to determine the consistency among physiotherapy educators when assessing a student performance using the APP tool., Methods: Physiotherapists (n = 153) from Australia with a minimum 3 years clinical experience and who had supervised a physiotherapy student within the past 12-months were recruited. Three levels of performance (not adequate, adequate, good/excellent) were scripted and filmed across outpatient musculoskeletal, neurorehabilitation, cardiorespiratory and inpatient musculoskeletal. In the initial phase of the study, scripts were written by academic staff and reviewed by an expert panel (n = 8) to ensure face and content validity as well as clinical relevance prior to filming. In the second phase of the study, pilot testing of the vignettes was performed by clinical academics (n = 16) from Australian universities to confirm the validity of each vignette. In the final phase, study participants reviewed one randomly allocated vignette, in their nominated clinical area and rated the student performance including a rationale for their decision. Participants were blinded to the performance level. Percentage agreement between participants was calculated for each vignette with an a priori percentage agreement of 75% considered acceptable., Results: Consensus among educators across all areas was observed when assessing a performance at either the 'not adequate' (97%) or the 'good/excellent' level (89%). When assessing a student at the 'adequate' level, consensus reduced to 43%. Similarly, consensus amongst the 'not adequate' and 'good/excellent' ranged from 83 to 100% across each clinical area; while agreement was between 33 and 46% for the 'adequate' level. Percent agreement between clinical educators was 89% when differentiating 'not adequate' from 'adequate' or better., Conclusion: Consistency is achievable for 'not adequate' and 'good/excellent' performances, although, variability exists at an adequate level. Consistency remained when differentiating an 'adequate' or better from a 'not adequate' performance.

Academic English tests are used to ascertain if international English as a Second Language nursing students have sufficient language skills to commence their nursing degrees, and later, if they have sufficient English for nursing registration. However, an academic English test may not be appropriate for clinical contexts. This study examines the relationship between two types of English test and the performance of forty-nine undergraduate international nursing students in both their first year of theory-centred academic topics and practice-centred clinical topics. An academic English test, called the International English Language Testing System (IELTS), and a general English proficiency/processing speed test using a variation of the cloze-test (C-test) format were administered at the commencement of the students' course of study. At the end of one year, grade percentages were collected. It was found that both the IELTS test and the C-test were significantly correlated to both types of topic, albeit with different patterns. The two English tests were also tested for similarities in the constructs they measured, with a significant overlap found. The implications are to rethink the way English tests are applied to entry in university degrees involving a clinical component and, by extension, to direct universities to rethink how nursing students are supported during their degree. The question is also raised about the practice of using academic English tests for professional nursing registration purposes. The benefits of the two testing approaches are also considered, particularly the large differences in monetary outlay and time found between the two tests, given their performance in explaining the variance in grade outcomes., (Copyright © 2018. Published by Elsevier Ltd.)

International migration of healthcare professionals has increased substantially in recent decades. In order to practice medicine in the recipient country, International Medical Graduates (IMG) are required to fulfil the requirements of their new countries medical registration authorities. The purpose of this project was to compare the final fellowship exit examination in Orthopaedic Surgery for the UK, Australia, Canada and South Africa. The curriculum of the Australian Orthopaedic Association (SET) was selected as a baseline reference. The competencies and technical modules specified in the training syllabus, as well as the specifics of the final fellowship examination as outlined in SET, were then compared between countries. Of the nine competencies outlined in SET, the curricula of the UK, South Africa and Canada were all compatible with the Australian syllabus, and covered 97.7%, 86% and 93%, respectively, of all competencies and sub-items. The final fellowship examinations of Australia, South Africa and the UK were all highly similar in format and content. The examination in Canada was substantially different, and had two written sessions but combined the oral and clinical component into a structured OSCE using standardized patients and the component included unmanned stations. There were no significant differences for completion certificate of training and/or board certification observed between these countries. The results of this study strongly suggest that core and technical competencies outlined in the training and education curriculum and the final fellowship examination in Orthopaedic Surgery in Australia, South Africa and the UK are compatible. Between country reciprocal recognition of these fellowship examinations should not only be considered by the relevant Colleges, but should also be regulated by the individual countries health practitioner registration boards and governing bodies.

Capstone experiences facilitate consolidation and application of previous learning, strengthening professional identity and competency. This study evaluates the effectiveness and acceptability of a capstone assessment item, in the form of an e-portfolio, designed to identify and demonstrate preparedness for professional midwifery practice. A sequential explanatory strategy occurred within a mixed method research design. Final year Bachelor of Midwifery students at an Australian university, having completed the e-portfolio assessment, participated in two phases of data collection; an initial online-survey, followed by in-depth exploration of emergent concepts within a focus group. Analysis of the quantitative data identified completing the e-portfolio assessment increased students' skills, knowledge and confidence and promoted reflection and critical thinking. Three themes emerged from the qualitative data; acknowledging growth and development; transitioning to practice, and knowing 'who I am and where I am going'. The e-portfolio assessment meets the aims of a capstone assessment and provides an appropriate framework and authentic opportunity for students to identify and demonstrate their level of preparedness for professional practice, determine their ongoing learning needs and develop strategies for achieving them. The assessment item provides an opportunity to develop and articulate a personal practice philosophy and embeds the principles of lifelong learning., (Copyright © 2018 Elsevier Ltd. All rights reserved.)

Aim: Health professionals seeking employment in foreign countries are commonly required to undertake competency assessment in order to practice. The present study aims to outline the development and validation of a written examination for Dietetic Skills Recognition (DSR), to assess the knowledge, skills, capabilities and professional judgement of overseas-educated dietitians against the competency standards applied to dietetic graduates in Australia., Methods: The present study reviews the design, rationale, validation and outcomes of a multiple choice question (MCQ) written examination for overseas-educated dietitians based on 5 years of administration. The validity of the exam is evaluated using Messick's validity framework, which focuses on five potential sources of validity evidence-content, internal structure, relationships with other variables, response process and consequences. The reference point for the exam pass mark or "cutscore" is the minimum standard required for safe practice., Results: In total, 114 candidates have completed the MCQ examination at least once, with an overall pass rate of 52% on the first attempt. Pass rates are higher from countries where dietetic education more closely reflects the Australian model. While the pass rate for each exam tends to vary with each cohort, the cutscore has remained relatively stable over eight administrations., Conclusions: The findings provide important data supporting the validity of the MCQ exam. A more complete evaluation of the validity of the exam must be sought within the context of the whole DSR program of assessment. The DSR written component may serve as a model for use of the MCQ format for dietetic and other professional credentialing organisations., (© 2018 Dietitians Association of Australia.)

Background: There is no current validated clinical assessment tool to measure the attainment of midwifery student competence in the midwifery practice setting. The lack of a valid assessment tool has led to a proliferation of tools and inconsistency in assessment of, and feedback on student learning., Objective: This research aimed to develop and validate a tool to assess competence of midwifery students in practice-based settings., Design: A mixed-methods approach was used and the study implemented in two phases. Phase one involved the development of the AMSAT tool with qualitative feedback from midwifery academics, midwife assessors of students, and midwifery students. In phase two the newly developed AMSAT tool was piloted across a range of midwifery practice settings and ANOVA was used to compare scores across year levels, with feedback being obtained from assessors., Findings: Analysis of 150 AMSAT forms indicate the AMSAT as: reliable (Cronbach alpha greater than 0.9); valid-data extraction loaded predominantly onto one factor; and sensitivity scores indicating level of proficiency increased across the three years. Feedback evaluation forms (n=83) suggest acceptance of this tool for the purpose of both assessing and providing feedback on midwifery student's practice performance and competence., Conclusion: The AMSAT is a valid, reliable and acceptable midwifery assessment tool enables consistent assessment of midwifery student competence. This assists benchmarking across midwifery education programs., (Copyright © 2017 Australian College of Midwives. Published by Elsevier Ltd. All rights reserved.)

Purpose: The biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established., Methods: We compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance., Results: Five hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores., Conclusion: We suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.

Background: Evaluations of clinical assessments that use judgement-based methods have frequently shown them to have sub-optimal reliability and internal validity evidence for their interpretation and intended use. The aim of this study was to enhance that validity evidence by an evaluation of the internal validity and reliability of competency constructs from supervisors' end-of-term summative assessments for prevocational medical trainees., Methods: The populations were medical trainees preparing for full registration as a medical practitioner (74) and supervisors who undertook ≥2 end-of-term summative assessments (n = 349) from a single institution. Confirmatory Factor Analysis was used to evaluate assessment internal construct validity. The hypothesised competency construct model to be tested, identified by exploratory factor analysis, had a theoretical basis established in workplace-psychology literature. Comparisons were made with competing models of potential competency constructs including the competency construct model of the original assessment. The optimal model for the competency constructs was identified using model fit and measurement invariance analysis. Construct homogeneity was assessed by Cronbach's α. Reliability measures were variance components of individual competency items and the identified competency constructs, and the number of assessments needed to achieve adequate reliability of R > 0.80., Results: The hypothesised competency constructs of "general professional job performance", "clinical skills" and "professional abilities" provides a good model-fit to the data, and a better fit than all alternative models. Model fit indices were χ2/df = 2.8; RMSEA = 0.073 (CI 0.057-0.088); CFI = 0.93; TLI = 0.95; SRMR = 0.039; WRMR = 0.93; AIC = 3879; and BIC = 4018). The optimal model had adequate measurement invariance with nested analysis of important population subgroups supporting the presence of full metric invariance. Reliability estimates for the competency construct "general professional job performance" indicated a resource efficient and reliable assessment for such a construct (6 assessments for an R > 0.80). Item homogeneity was good (Cronbach's alpha = 0.899). Other competency constructs are resource intensive requiring ≥11 assessments for a reliable assessment score., Conclusion: Internal validity and reliability of clinical competence assessments using judgement-based methods are acceptable when actual competency constructs used by assessors are adequately identified. Validation for interpretation and use of supervisors' assessment in local training schemes is feasible using standard methods for gathering validity evidence.

Objective: The objective of this article is to explore the tension between what should be complementary notions of medical education and professional identity formation, referencing a major recent change to the assessment program of the Royal Australian and New Zealand College of Psychiatrists (RANZCP). The College recently determined that the observed clinical interview (OCI) is to cease with equivalent skills assessed by multiple workplace-based assessments within a competency-based education program. The literature relating to the change is briefly reviewed. The shortcomings of the OCI include concerns regarding reliability, validity, sustainability and transparency. Arguments in favour of keeping the OCI are then considered, including those made on grounds of reliability of the new instruments, validity of the new assessment program as a whole, the importance of professional identity formation and cultural considerations within the specialty., Conclusion: Education and professional identity formation are not synonymous. Applying educational evidence alone to assessment design may mean an opportunity for trainees to demonstrate qualities previously assessed at the OCI is lost. This may have negative consequences for the quality of RANZCP trained psychiatrists and for our patients. The OCI is a flawed instrument but the new system will be flawed in different and possibly more important ways., (© The Royal Australian and New Zealand College of Psychiatrists 2015.)

Background: Benchmarking among medical schools is essential, but may result in unwanted effects., Aim: To apply a conceptual framework to selected benchmarking activities of medical schools., Methods: We present an analogy between the effects of assessment on student learning and the effects of benchmarking on medical school educational activities. A framework by which benchmarking can be evaluated was developed and applied to key current benchmarking activities in Australia and New Zealand., Results: The analogy generated a conceptual framework that tested five questions to be considered in relation to benchmarking: what is the purpose? what are the attributes of value? what are the best tools to assess the attributes of value? what happens to the results? and, what is the likely "institutional impact" of the results? If the activities were compared against a blueprint of desirable medical graduate outcomes, notable omissions would emerge., Conclusion: Medical schools should benchmark their performance on a range of educational activities to ensure quality improvement and to assure stakeholders that standards are being met. Although benchmarking potentially has positive benefits, it could also result in perverse incentives with unforeseen and detrimental effects on learning if it is undertaken using only a few selected assessment tools.

Background/aim: The Student Practice Evaluation Form-Revised Edition (SPEF-R) is widely used in Australian universities to evaluate occupational therapy students' performance in practice education. Reliable completion of the SPEF-R by practice educators is critical for students and universities from a quality assurance perspective. This study used standardised video vignettes to examine the inter-rater and test-retest reliability of practice educators when completing the SPEF-R., Methods: Nine vignettes were developed with three levels of student performance (below expectations, adequate-to-proficient, and outstanding) for each of three scenarios which depicted a multidisciplinary team meeting, a supervision session, and a home visit respectively. Seventy-five occupational therapy practice educators viewed the vignettes and completed an online survey to rate student performance in each vignette using five selected SPEF-R items and using the five-point rating scale. Twenty of these practice educators completed the rating process twice for test-retest reliability purposes. Percentage agreement, t-tests and Rasch Measurement Model were used for analysis., Results: Practice educators exhibited consistent ratings on most of the SPEF-R items, except for rating adequate-to-proficient student performance on three items for inter-rater reliability and eight items for test-retest reliability. Rasch analysis found that the majority of practice educators (96-98.7%) exhibited consistency in their use of the SPEF-R rating scales. Practice educators also demonstrated satisfactory test-retest agreement of severity/leniency in rating student performance in the multidisciplinary team meeting scenario., Conclusion: The study findings suggest that the SPEF-R could be used reliably and interpreted consistently by practice educators with diverse backgrounds and levels of experience., (© 2014 Occupational Therapy Australia.)

Objectives: To compare the achievement goal orientations of first-year with those of third-year undergraduate Australian pharmacy students and to examine the relationship of goal orientations to academic achievement., Methods: The Achievement Goal Questionnaire was administered to first-year and third-year students during class time. Students' grades were obtained from course coordinators., Results: More first-year students adopted performance-approach and mastery-approach goals than did third-year students. Performance-approach goals were positively correlated with academic achievement in the first year. Chinese Australian students scored the highest in adopting performance-approach goals. Vietnamese Australian students adopted mastery-avoidance goals more than other ethnicities. First-year students were more strongly performance approach goal-oriented than third-year students., Conclusion: Adopting performance-approach goals was positively correlated with academic achievement, while adopting avoidance goals was not. Ethnicity has an effect on the adoption of achievement goals and academic achievement.

Objective: To determine trainee perspectives of the utility of a trainee research project (TRP) or the coursework pathway (CP) in completing the Trainee Research Requirement of the ACEM., Methods: A survey based on the ACEM learning objectives for research was sent to all trainees and Fellows who had completed or intending to complete the Trainee Research Requirement between 2010 and 2012. Participants were asked to rate the value of the TRP or CP on a scale of 1 to 10 (1 = 'not useful' and 10 = 'invaluable'). In addition, open-ended questions were asked for qualitative assessment., Results: Survey response was 142/621 (23%). Most participants had undertaken the CP (113/142, 79%). Median scores were better for the CP compared with the TRP, and when results were dichotomised to scores of 1-5 as being not useful and 6-10 as being useful, a significantly higher proportion of participants rated the CP as being useful across all learning objectives (P ≤ 0.01). There was a bimodal distribution of scores for TRPs, with a minority of trainees reporting a very poor TRP experience. Cost was the most commonly cited negative aspect of the CP, whereas the TRP was perceived as time consuming and being difficult to attain., Conclusions: More ACEM trainees are undertaking the CP compared with a TRP. The CP was associated with better self-reported fulfilment of the ACEM learning objectives for research. There is scope for the ACEM to improve the experience of trainees wishing to undertake their own research projects., (© 2014 Australasian College for Emergency Medicine and Australasian Society for Emergency Medicine.)

Grading group work assessments so that students perceive the grade to be fair to all group members is sometimes challenging. This is particularly important in a higher education environment that is increasingly concerned with student perceptions of teaching quality and satisfaction. This article reports on research that compared undergraduate nursing students perceptions of two different approaches to the grading of group work assessment. A survey design was used to identify students' perspectives and preferences for different group work assessment methods. Participants were undergraduate bachelor of nursing students from a large, metropolitan university in Australia. Data analysis indicated that the perceptions of students around group work assessments changed little as they progressed across the program, although students who had experienced the calculation of individual grades for a group assessment preferred this approach. Many believed the grading of group assessments penalised good students and were less reliable than individual assessments. Students maintained the belief that teamwork skills were essential for the registered nurse role. In conclusion group work assessment should only be used when it is the best assessment method to demonstrate student learning of specific objectives. The weighted mark approach is the group work assessment grading approach of choice., (Copyright © 2013 Elsevier Ltd. All rights reserved.)

Background: The Final Examination for Orthopaedic Surgeons in Australia comprises seven components of which two are written responses and five are oral examinations involving separate pairs of examiners., Methods: The reliability of the examination process and close marking system was assessed by using a standardized DVD of a sham examination. Fifteen current examiners viewed the DVD and scored the viewed examination independently., Results: The results demonstrated consistency in scoring and reliability. This validation is likely to be further increased by the current system of the pairing of examiners. Additionally, the patient-based examination segments were found to be as reliably scored as the criterion-referenced computer-based examination stations., Conclusions: Candidates for the Fellowship Examination should be reassured that this study has shown that the examination process has inter-examiner reliability and consistency., (© 2012 The Author. ANZ Journal of Surgery © 2012 Royal Australasian College of Surgeons.)

Context: Education in simulated learning environments (SLEs) has grown rapidly across health care professions, yet no substantive randomised controlled trial (RCT) has investigated whether SLEs can, in part, substitute for traditional clinical education., Methods: Participants were physiotherapy students (RCT 1, n = 192; RCT 2, n = 178) from six Australian universities undertaking clinical education in an ambulatory care setting with patients with musculoskeletal disorders. A simulated learning programme was developed as a replica for clinical education in musculoskeletal practice to replace 1 week of a 4-week clinical education placement. Two SLE models were designed. Model 1 provided 1 week in the SLE, followed by 3 weeks in clinical immersion; Model 2 offered training in the SLE in parallel with clinical immersion during the first 2 weeks of the 4-week placement. Two single-blind, multicentre RCTs (RCT 1, Model 1; RCT 2, Model 2) were conducted using a non-inferiority design to determine if the clinical competencies of students part-educated in SLEs would be any worse than those of students educated fully in traditional clinical immersion. The RCTs were conducted simultaneously, but independently. Within each RCT, students were stratified on academic score and randomised to either the SLE group or the control ('Traditional') group, which undertook 4 weeks of traditional clinical immersion. The primary outcome measure was a blinded assessment of student competency conducted over two clinical examinations at week 4 using the Assessment of Physiotherapy Practice (APP) tool., Results: Students' achievement of clinical competencies was no worse in the SLE groups than in the Traditional groups in either RCT (Margin [Δ] ≥ 0.4 difference on APP score; RCT 1: 95% CI - 0.07 to 0.17; RCT 2: 95% CI - 0.11 to 0.16)., Conclusions: These RCTs provide evidence that clinical education in an SLE can in part (25%) replace clinical time with real patients without compromising students' attainment of the professional competencies required to practise., (© Blackwell Publishing Ltd 2012.)

One of the key responsibilities of professional bodies, such as the Royal Australian and New Zealand College of Ophthalmologists, is to determine, teach and assess the competencies required for trainees to reach an expert level. Vocational training programs (VTP) need to incorporate advances in educational research and reflect changes in generational thinking and learning styles to provide the most optimal learning environment to meet the desired educational outcomes. This paper seeks to introduce some of the important concepts of adult educational theory and to explain how they connect to four strategic areas in the development and implementation of the VTP: 1 What are the learning needs of trainees? 2 What educational methods best address these needs? 3 What assessment methods best test the acquisition of the desired learning outcomes? 4 What are the needs of supervisors and teachers?, (© 2011 The Authors. Clinical and Experimental Ophthalmology © 2011 Royal Australian and New Zealand College of Ophthalmologists.)

The learning objectives, curriculum content, and assessment standards for distributed medical education programs must be aligned across the health care systems and community contexts in which their students train. In this article, the authors describe their experiences at Monash University implementing a distributed medical education program at metropolitan, regional, and rural Australian sites and an offshore Malaysian site, using four different implementation models. Standardizing learning objectives, curriculum content, and assessment standards across all sites while allowing for site-specific implementation models created challenges for educational alignment. At the same time, this diversity created opportunities to customize the curriculum to fit a variety of settings and for innovations that have enriched the educational system as a whole.Developing these distributed medical education programs required a detailed review of Monash's learning objectives and curriculum content and their relevance to the four different sites. It also required a review of assessment methods to ensure an identical and equitable system of assessment for students at all sites. It additionally demanded changes to the systems of governance and the management of the educational program away from a centrally constructed and mandated curriculum to more collaborative approaches to curriculum design and implementation involving discipline leaders at multiple sites.Distributed medical education programs, like that at Monash, in which cohorts of students undertake the same curriculum in different contexts, provide potentially powerful research platforms to compare different pedagogical approaches to medical education and the impact of context on learning outcomes.

The Education Committee of the Australian Psychoanalytic Society, inspired by David Tuckett's research, relating to the measurement of candidate competence, conducted a similar study that, they believe, added value to his earlier work. The indicators of analytic competence were more numerous and specific, and greater emphasis was placed on how these criteria could further be operationalized. Besides being used in the measuring of candidate competence, the author makes a case for how the criteria can be adapted for use in the selection of prospective trainees; the defining of teaching and supervisory objectives and processes; and as a basis for further research. The close resonance that the qualitative research method used has with analytic thought and practice is highlighted. It is proposed that this synchrony reduces the resistance that analysts often have in participating in such research. There were definite advantages that arose out of the Australian project aside from producing a society-specific schema of candidate capacities. The research process opened up a space for dialogue, debate and experimentation that increased group cohesion and society identity. Including the candidates in the survey facilitated the increase of transparency between candidates and members, contributing to the strengthening of the educational alliance., (Copyright © 2011 Institute of Psychoanalysis.)

Background: There is a lack of acceptable, reliable, and valid survey instruments to measure conceptual research utilization (CRU). In this study, we investigated the psychometric properties of a newly developed scale (the CRU Scale)., Methods: We used the Standards for Educational and Psychological Testing as a validation framework to assess four sources of validity evidence: content, response processes, internal structure, and relations to other variables. A panel of nine international research utilization experts performed a formal content validity assessment. To determine response process validity, we conducted a series of one-on-one scale administration sessions with 10 healthcare aides. Internal structure and relations to other variables validity was examined using CRU Scale response data from a sample of 707 healthcare aides working in 30 urban Canadian nursing homes. Principal components analysis and confirmatory factor analyses were conducted to determine internal structure. Relations to other variables were examined using: (1) bivariate correlations; (2) change in mean values of CRU with increasing levels of other kinds of research utilization; and (3) multivariate linear regression., Results: Content validity index scores for the five items ranged from 0.55 to 1.00. The principal components analysis predicted a 5-item 1-factor model. This was inconsistent with the findings from the confirmatory factor analysis, which showed best fit for a 4-item 1-factor model. Bivariate associations between CRU and other kinds of research utilization were statistically significant (p < 0.01) for the latent CRU scale score and all five CRU items. The CRU scale score was also shown to be significant predictor of overall research utilization in multivariate linear regression., Conclusions: The CRU scale showed acceptable initial psychometric properties with respect to responses from healthcare aides in nursing homes. Based on our validity, reliability, and acceptability analyses, we recommend using a reduced (four-item) version of the CRU scale to yield sound assessments of CRU by healthcare aides. Refinement to the wording of one item is also needed. Planned future research will include: latent scale scoring, identification of variables that predict and are outcomes to conceptual research use, and longitudinal work to determine CRU Scale sensitivity to change.

Purpose: Although the global nursing faculty shortage has led to increasing reliance upon sessional staff, limited research has explored the impact of these sessional staff on the quality of teaching in higher education. We aim to examine differences in (a) student satisfaction with sessional and tenured staff and (b) assessment scores awarded by sessional and tenured staff in students' written assignments., Design: A comparative study method was used. Participants were recruited from students enrolled in the three nursing practice subjects across the 3 years of the baccalaureate program in an Australian university during the second semester of 2008., Methods: This study collected student data via an online version of the Perceptions of Teaching and Course Satisfaction scale and compared the grades awarded by sessional and tenured academics for a written assessment in a single assignment in each of the nursing practice subjects. Of the 2,045 students enrolled in the nursing practice subjects across the 3 years of the bachelor of nursing (BN) program, 566 (28%) completed the online teaching and course satisfaction survey, and 1,972 assignment grades (96%) were available for analysis., Findings: Compared with tenured academics, sessional teachers received higher rating on students' perception on teaching satisfaction by students in Year 1 (p= .021) and Year 2 (p= .002), but not by students in Year 3 (p= .348). Following the same trend, sessional teachers awarded higher assignment grades to students in Year 1 (p < .001) and Year 2 (p < .001) than tenured academics, with no significant disparity in grades awarded to students in Year 3., Conclusions: The higher grades awarded by sessional teachers to 1st- and 2nd-year students could be one explanation for why these teachers received higher student ratings than tenured teachers. Not discounting the possibility of grade inflation by sessional staff, it could be that tenured teachers have a higher expectation for the quality of students' work, and hence were more stringent in their assessment grading. Sessional teachers did not receive a higher rating from 3rd-year students, and this could be attributed to a change in student perception as they progress through the course, valuing a broader and more professional aspect of nursing knowledge, which is more likely to be the strength of tenured staff., Clinical Relevance: These findings highlight a need for the development and implementation of strategies to facilitate the inclusion of sessional staff teaching in a BN program, in order to prepare graduate nurses that are well-equipped for clinical practice., (© 2010 Sigma Theta Tau International.)

The identification and specification of competency-based standards in speech-language pathology has provided practitioners, educators, employers, and government regulators with information and guidance. This paper reports the outcomes of workshops that provided familiarization with the new competency-based assessment tool, COMPASS®, which was introduced for the assessment of speech-language pathology (SLP) students across all 13 SLP professional preparation programs in Australia during 2007. An anonymous evaluation was administered before and after the first eight familiarization workshops held nationally, involving 240 clinical educators. Quantitative data were analysed descriptively, and qualitative data were entered into NVivo qualitative analysis software for content analysis. Post-workshop, results indicated partial or full uptake of the main concepts involved in the new approach to assessment. Least uptake was observed for the need for direct observation of competence in workplace performance. Qualitatively, post-workshop, formative assessment was more apparent within student goals formulated in response to a hypothetical scenario. A possible contributor to this outcome is suggested to be the alignment between the tool and the professional community of practice, due to the collaborative process of its development. Research into the longer term impact of the new assessment in the context of everyday practice is suggested.

Context: There is increasing agreement that graduates who finish tertiary education with the full complement of skills and knowledge required for their designated profession are not 'work-ready' unless they also acquire interpersonal, collaborative practice and team-working capabilities. Health workers are unable to contribute to organisational culture in a positive way unless they too attain these capabilities. These capabilities have been shown to improve health care in terms of patient safety, worker satisfaction and health service efficiency. Given the importance of interprofessional learning (IPL) which seeks to address these capabilities, why is IPL not consistently embedded into the education of undergraduates, postgraduates and vocationally qualified personnel through formal assessment?, Methods: This paper offers an argument for the formal assessment of IPL. It illustrates how the interests of the many stakeholders in IPL can benefit from, and contribute to, the integration of IPL into mainstream professional development and tertiary education. It offers practical examples of assessment in IPL which could drive learning and offer authentic, contextual teaching and learning experiences to undergraduates and health workers alike., Conclusions: Assessment drives learning and without formal assessment IPL will continue to be viewed as an optional topic of little relative importance for learners. In order to make the next step forward, IPL needs to be recognised and endorsed through formal assessment, both at the tertiary education level and within the workplace environment. This is supported by workforce initiatives and tertiary education policy which can be used to specify the capabilities or generic skills necessary for effective teamwork and collaborative practice.

This article explores the use of the objective structured clinical examination (OSCE) in undergraduate nursing education. The advantages and limitations of this assessment approach are discussed and various applications of the OSCE are described. Attention is given to the complexities of evaluating some psychosocial competency components. The issues are considered in an endeavour to delineate the competency components, or skill sets, that best lend themselves to assessment by the OSCE. We conclude that OSCEs can be used most effectively in nurse undergraduate curricula to assess safe practice in terms of performance of psychomotor skills, as well as the declarative and schematic knowledge associated with their application. OSCEs should be integrated within a curriculum in conjunction with other relevant student evaluation methods.

Introduction: Accreditation of the Australian College of Rural and Remote Medicine (ACRRM) as a standards and training provider, by the Australian Medical Council (AMC) in 2007, is the first time in the world that a peak professional organisation for rural and remote medical education has been formally recognised. As a consequence, the Australian Government provided rural and remote medicine with formal recognition under Medicare as a generalist discipline. This accreditation was based on the ability of ACRRM to meet the AMC's guidelines for its training and assessment program., Methods: The methodology was a six-step process that included: developing an assessment blueprint and a classification scheme; identifying an assessment model; choosing innovative summative and formative assessment methods that met the needs of rural and remote located medical practitioner candidates; 21 rural doctors and academics developing the assessment items as part of a week-long writing workshop; investigating the feasibility of purchasing assessment items; and 48 rural candidates piloting three of the assessment items to ensure they would meet the guidelines for national accreditation., Results: The project resulted in an innovative formative and summative assessment program that occurs throughout 4 years of vocational training, using innovative, reliable, valid and acceptable methods with educational impact. The piloting process occurred for 3 of the 6 assessment tools. Structured Assessment Using Multiple Patient Scenarios (StAMPS) is a new assessment method developed as part of this project. The StAMPS pilot found that it was reliable, with a generalisability coefficient of >0.76 and was a valid, acceptable and feasible assessment tool with desired educational impact. The multiple choice question (MCQ) examination pilot found that the applied clinical nature of the questions and their wide range of scenarios proved a very acceptable examination to the profession. The web based in-training assessment examination pilot revealed that it would serve well as a formative process until ACRRM can further develop their MCQ database., Conclusions: The ACRRM assessment program breaks new ground for assessing rural and remote doctors in Australia, and provides new evidence regarding how a comprehensive and contemporary assessment system can work within a postgraduate medical setting.

In Australia and Norway final examinations to determine eligibility for registration as a nurse were discontinued during the period when nurse education moved into the higher education sector. In response to recent calls for the reintroduction of final examinations we explore the range of knowledge needs for the practice of nursing. These various forms of knowledge demand different forms of mediation and acquisition as well as assessment. There are numerous problems identified in the literature about the shortcomings of examinations as the foundation of assessing clinically based professions. There is a need to develop systems of appropriate assessment to ensure that graduates of nursing demonstrate adequate knowledge and competence to enter their profession.

Graduate-entry medical programmes have been introduced in many Australian states. This has coincided with the introduction of learner-centred, problem-orientated medical courses and has resulted in a need to develop new selection pathways. Medical schools have complemented their primary selection tool of academic performance with tests of personal attributes considered important in future doctors. Schools with graduate-entry programmes now employ a combination of the results of the applicant's undergraduate degree, a semistructured interview and a psychometric test (the Graduate Australian Medical School Admissions Test) to select students. Rationale for the change to graduate entry and the outcomes of this change will be discussed.

Objectives: This study aims to identify whether non-science graduates perform as well as science graduates in Basic and Clinical Sciences (B & CS) assessments during Years 1-3 of a four-year graduate-entry programme at the University of Sydney (the 'USydMP')., Methods: Students were grouped into five categories: Health Professions (HP), Biomedical Sciences (BMS), Other Biology (BIOL), Physical Sciences (PHYS) or Non-Science (NONS). We examined the performance rank of students in each of the five groups for single best answer (SBA) and modified essay (MEQ) assessments separately, and also calculated the relative risk of failure in the summative assessments in Years 2 and 3., Results: Students with science-based prior degrees performed better in the SBA assessments. The same occurred initially in the MEQs, but the effect diminished with time. The HP students performed consistently better but converged with other groups over time, particularly in the MEQs. Relative performance by the NONS students improved with time in both assessment formats. Overall, differences between the highest and lowest groups were small and very few students failed to meet the overall standard for the summative assessments. HP and BMS students had the lowest failure rate. NONS students were more likely to fail the assessments in Year 2 and 3, but their pass rates were still high. Female students performed significantly better overall at the end of Year 2 and in Year 3. There were only minor differences between Australian resident and International students., Conclusion: While there are small differences in performance in B & CS early in the programme, these lessen with time. The study results will inform decisions regarding timing of summative assessments, selection policy and for providing additional support to students who need it to minimize their risk of failure. Readers should note that this paper refers to student performance in only one of the four curriculum themes, where health professional and science graduates would be expected to have a significant advantage.

Introduction: The introduction of graduate entry medical courses requires attention to models of decision-making and change. Much of the educational literature describes change as either centralised or decentralised with the latter claimed to be more effective. Recently Fullan has argued for the importance of an ongoing culture of change. This paper examines the change culture of two medical schools adopting graduate entry courses; Flinders University in South Australia and St George's Hospital Medical School, London., Method: Comparative case study was used. Key informants were interviewed and documents and records analysed. Data were cross-checked and categorised to generate models of change., Results: There were four components of the change culture at Flinders but they were not sufficient on their own to generate change. The process was triggered by a significant external event. The nature of the change was also important. The descriptive model developed suggested a complex interplay of factors rather than attributing the success of the change to the change mechanisms adopted alone. The model was tested for explanatory potential at St George's. The culture there was described as both 'macro-innovative' and 'micro-conservative'. External events were also important but they exerted a positive force. A more centralised approach was adopted., Discussion: The models developed represent change as 'dynamic, complex and open' rather than a simple centralised or decentralised dichotomy. While some of the elements of a change culture were evident at both schools there were longer term questions of sustainability. This has implications for development of all programmes but particularly for graduate entry schools.

The pass rate in the Australasian College for Emergency Medicine Fellowship exams between 1996 and 2003 inclusive averaged 61%, substantially lower than that of other specialties with comparable training structures. The explanation for this pass rate, which many would judge as unacceptably low, is likely to be multi-factorial. Possible factors that should be considered include trainee selection, training programme structure, the impact of the Director of Emergency Medicine Training, examination preparation, examination validity and examination reliability. Each of these potential factors needs to be addressed in a systematic fashion in the context of inevitable and increasing internal and external scrutiny of the outcomes of our training programme.

Background: The move from discipline-based to problem-based learning (PBL) at Adelaide University in 2000 offered exciting opportunities to integrate the teaching and learning of the basic and clinical sciences for medical undergraduates. However, several cohorts of students still needed to progress through the first 3 years of the more traditional curriculum. Paradoxically, their readiness to function in the integrated learning and assessment environment of the last 3 years was assessed in 7 separate discipline-based examinations at the end of third year. When considerable examination-related stress was noted in the 1997 cohort and students petitioned formally for a reduced examination load, it was considered to be time for assessment to lead the way in integrating the disciplines., Aim: After introducing third year integrated written assessments in 1998, we aimed to develop an integrated practical examination (IPE) linking theory to practice, and evaluate its impact on staff and students., Methods: After extensive staff collaboration, a structured objective multistation IPE was developed and administered in 1999 and 2000. Its utility was evaluated using a model proposed earlier., Results: Assessment validity was maximised by an extensive item review process. Reliability, as measured by Cronbach's alpha, was 0.79 and 0.80 in 1999 and 2000, respectively. An independent evaluation yielded qualitative data on the examination's educational impact, cost and acceptability., Conclusions: Investing time in changing from discipline-based to integrated assessment, integrating theory and practice, resulted in gains in assessment reliability, validity and educational impact on both staff and students.

Background: Computerised learning clearly offers exciting potential for improving student learning, either as an aid to or as a replacement for traditional formats, or for the development of innovative approaches. However, rigorous evaluation of the utility of computer-aided learning (CAL) in enhancing student learning can be difficult. Many studies have compared CAL to more traditional learning formats, but there is little evidence to show which style of CAL leads to the best learning outcomes., Aim: This study aimed to test the hypothesis that a CAL tutorial, in which the learner actively interacts with the computer, will result in superior learning (ability to apply and retain knowledge) to that obtained in more passive CAL formats., Methods: Third year medical undergraduates at Adelaide University, South Australia were randomly assigned to 4 groups. Following a pretest, only students in the "didactic", "problem-based" and "free text" groups had 2 weeks of free access to a neuroradiology CAL tutorial in their assigned format. Tutorial access was denied to all students 2 weeks before post-testing. Learning was quantified by comparing the post- to pretest scores for each of the 4 groups., Results: After active interaction with the computer material, students in the free text group demonstrated a statistically significant improvement in their ability to apply and retain knowledge compared to the control group, but no advantage compared to the didactic group., Conclusions: While users of an interactive CAL tutorial demonstrated significant learning gains compared to non-CAL users, these gains were not superior to those achieved from non-interactive CAL. When evaluating education interventions such as CAL packages, it is important to use a valid assessment tool to measure learning.

Background: Debate abounds regarding the most appropriate candidates to admit to medical school. This paper examines whether there is any advantage to admitting 'graduate' entrants over secondary school leavers on selected medical school and practice outcomes., Aim: To compare the medical school experiences, research and academic achievements and practice outcomes of graduates who entered 1 medical school in Australia directly from high school (secondary school entry) to those of graduates who entered with tertiary level education (tertiary entry)., Design and Methods: Cross-sectional study using a mail-out survey to graduates from the first 16 graduating years (1983-98 inclusive) of the University of Newcastle Medical School., Results: Secondary school entrants were, on average, 8 years younger than tertiary entrants and were less likely to have received rural-based schooling. However, there were no differences with respect to gender or type of secondary school attended (public or private). Motivations for studying medicine did not generally differ according to entry type, except that more secondary students were motivated by parental expectations and more tertiary entrants were motivated by the need for professional independence and the desire to prevent disease. A greater proportion of tertiary entrants experienced stress at medical school. However, secondary students experienced more stress due to doubts about being a doctor, while tertiary entrants experienced more stress due to lack of leisure time, finances and balancing commitments. There were no significant differences between the groups in terms of academic performance (as measured by the award of medical school honours) or research outcomes (as measured by completion of a research degree during or after medical school training, publication of scientific papers or holding career posts in the research sciences). There were no differences in career positions held by clinicians, choice of general practice or another specialty as a career, practice location (rural or urban) or employment sector (public or private)., Conclusion: There is no clear advantage, at least on the outcomes measured in this study, to limiting medical school entry to either those candidates from secondary school or those with tertiary backgrounds. Medical schools could reasonably broaden their selection criteria to include more graduate entry candidates in addition to secondary school leavers without compromising medical school and practice outcomes.

Aim: The study had two aims. The first aim was to examine the relationship between final year medical students' ethnicity and their attitudes towards consultation skills. The second aim was to investigate the relationship between ethnicity and final year performance., Method: A cross-sectional study was undertaken of all final year medical students in 1995 and 1996, both before and after their attachment through the Department of General Practice at Monash University, Australia. A questionnaire was designed to assess students' attitudes (views of importance and confidence) towards consultation skills. Also, records of final year performance were obtained from the University., Results: Five clusters of consultation skills were formed through factor analysis: communication skills, difficult consultations, traditional diagnostic methods, routine management and life threatening conditions. There were no significant differences in students' attitudes towards these consultation skills based on country of birth, language, or student status preattachment. The only significant postattachment difference was that students born in non-Western countries placed a significantly higher importance on communication skills and the traditional diagnostic method than students born in Western countries. There were significant differences in final year performance between students across all three parameters of ethnic diversity. Non-Western born students performed worse than Western born students. Students who preferred using a language other than English also performed worse than those students preferring English. International students performed worse than students with Australian citizenship or permanent residency.,, Conclusions: The only significant difference regarding attitudes to consultation skills was that non-Western born students placed significantly greater importance on communication skills and the traditional diagnostic method postattachment than Western born students. No significant differences were found in attitudes preattachment. There were significant differences in performance, with students of ethnic background performing consistently poorer in all the final year assessment parameters analysed.