Search

Information about several papers discussed at the 14th Ottawa Conference held by the Ottawa Conference Planning Commitee in Miami, Florida from May 15 to 19, 2010 is presented. Topics include the assessment of competence in medicine and the healthcare professions. The conference featured several speakers including chairperson John Norcini.

The 16th International Ottawa Conference/Canadian Conference on Medical Education (2014) featured a keynote deconstructing the rising discourse of competence-as-reflection in medical education. This paper, an elaborated version of the presentation, is an investigation into the theoretical roots of the diverse forms of reflective practice that are being employed by medical educators. It also raises questions about the degree to which any of these practices is compatible with assessment. [ABSTRACT FROM AUTHOR]

Over the past 25 years, professionalism has emerged as a substantive and sustained theme, the operationalization and measurement of which has become a major concern for those involved in medical education. However, how to go about establishing the elements that constitute appropriate professionalism in order to assess them is difficult. Using a discourse analysis approach, the International Ottawa Conference Working Group on Professionalism studied some of the dominant notions of professionalism, and in particular the implications for its assessment. The results presented here reveal different ways of thinking about professionalism that can lead towards a multi-dimensional, multi-paradigmatic approach to assessing professionalism at different levels: individual, inter-personal, societal--institutional. Recommendations for research about professionalism assessment are also presented. [ABSTRACT FROM AUTHOR]

The tutor plays an important role in facilitating learning in a problem-based learning (PBL) curriculum. This paper explored the ways that novice tutors were educated in a PBL programme at McMaster University. Thirteen novice tutors were interviewed in this qualitative, ethnographic study to identify their learning needs and culture at the entry phase of ‘becoming a tutor’. Ten tutor guides were also interviewed to provide additional information and perspectives regarding the data generated by the novice tutors. Categories that emerged were: (1) benefiting from the experience, (2) managing the challenges, (3) transitioning to a new role, (4) uncovering learning opportunities, (5) maintaining vigilance, and (6) explicating the implicit. The overarching framework that wove the categories together was that of the theme of storytelling in the teaching–learning process. Implications for practice for tutor training are addressed considering the oral tradition. [ABSTRACT FROM AUTHOR]

In response to the competitive examination culture that pervaded medical education in the 1940s and 1950s the founders of McMaster University's new MD Programme created an assessment system based on group functioning within the tutorial. While the tutorial has served the educational process well, 30 years of experience has highlighted its deficiencies as an assessment tool. This paper describes the accumulation of evidence that led to the awareness of the weakness of tutorial assessment and the attempts to provide reliable assessment by the reintroduction of examinations, but in novel formats which would not alter the goals of the curriculum. [ABSTRACT FROM AUTHOR]

Dual-process theory characterizes clinical reasoning (CR) as Type 1 (intuitive) and Type 2 (analytical) thinking. This study examined CR on a summative clinical clerkship structured clinical oral examination (SCOE). 511 clinical clerks at the University of Toronto underwent SCOEs. Type 1, Type 2, and Global CR performance were compared to other internal medicine clerkship assessments using descriptive statistics and Spearman correlations. Clinical clerks achieved mean marks >75% on the three clinical reasoning stations, on Type 1 and 2 CR tasks, and the overall SCOE. Performance on the SCOE CR stations correlated with each of the other clerkship assessments: written examination, inpatient, and ambulatory clinic assessments. The correlation of performance between Type 1 and Type 2 clinical reasoning tasks was statistically significant but weak (rs = 0.28). This suggests that defined measures of Type 1 and Type 2 reasoning were indeed assessing distinct constructs. Clinical clerks used both Type 1 and Type 2 reasoning with success. This study's characterization of Type 1 and Type 2 CR as separate domains, distinct from existing measures on the SCOE as well as the other clerkship assessments, can suggest a further addition to multimodal clerkship assessment. [ABSTRACT FROM AUTHOR]

There may be no competency more shrouded in uncertainty than health advocacy (HA), raising questions about the robustness of advocacy training in postgraduate medical education. By understanding how programs currently train HA, we can identify whether trainees' learning needs are being met. From 2017 to 2019, we reviewed curricular documents across nine direct-entry specialties at all Ontario medical schools, comparing content for the HA and communicator roles to delineate role-specific challenges. We then conducted semi-structured interviews with trainees (n = 9) and faculty (n = 6) to review findings and discuss their impact. Data were analyzed using thematic content analysis. Curricular documents revealed vague objectives and ill-defined modes of assessment for both intrinsic roles. This uncertainty was perceived as more problematic for HA, in part because HA seemed both undervalued in, and disconnected from, clinical learning. Trainees felt that the onus was on them to figure out how to develop and demonstrate HA competence, causing many to turn their learning attention elsewhere. Lack of curricular focus seems to create the perception that advocacy isn't valuable, deterring trainees—even those keen to become competent advocates—from developing HA skills. Such ambivalence may have troubling downstream effects for both patient care and trainees' professional development. [ABSTRACT FROM AUTHOR]

Selection and recruitment into healthcare education and practice is a key area of interest for educators with significant developments in research, policy, and practice in recent years. This updated consensus statement, developed through a multi-stage process, examines future opportunities and challenges in selection and recruitment. There is both a gap in the literature around and a compelling case for further theoretical and empirical literature to underpin the development of overall selection philosophes and policies and their enactment. More consistent evidence has emerged regarding the quality of different selection methods. Approaches to selection are context-dependent, requiring the consideration of an institution's philosophy regarding what they are trying to achieve, the communities it purports to serve, along with the system within which they are used. Diversity and globalization issues continue to be critically important topics. Further research is required to explore differential attainment and explain why there are substantial differences in culturally acceptable ways of approaching diversity and widening access. More sophisticated evaluation approaches using multi-disciplinary theoretical frameworks are required to address the issues. Following a discussion of these areas, 10 recommendations are presented to guide future research and practice and to encourage debate between colleagues across the globe. [ABSTRACT FROM AUTHOR]

Background: Few new Residency Program Directors (PD) are formally trained for the demands and responsibilities of the leadership aspect of their role. Currently, there are no comprehensive frameworks that describe specific leadership competencies that can inform PD self-reflection or faculty development. Methods: The authors developed a Postgraduate Program Director Competency Inventory (PPDCI) in order to frame the performance of PDs for a multisource feedback (MSF) program. The development of the PPDCI occurred in five phases which involved: development of an initial inventory, implementation of a key informant survey of national opinion leaders, execution of a validity survey with postgraduate education leaders and committee members and implementation of a further refined inventory with 17 PD and 147 raters as part of a pilot MSF program. Outcomes: Five distinct domains of leadership competence were identified which included: Communication and relationship management, leadership, professionalism and self-management, environmental engagement, and management skills and knowledge. The content validity of the PPDCI was endorsed by 85% of the key informants. The validity survey indicated strong endorsement of the PPDCI domains and recognition of its utility for both orientation of new PD as well as a frame for self-assessment. The pilot MSF program yielded a further refined and reduced inventory of 26 items of competence as well as recommendations for its utility. Conclusions: Use of this leadership inventory has the potential to ensure effective leadership of postgraduate programs. [ABSTRACT FROM AUTHOR]

Purpose: The purpose of this study was to explore the use of an objective structured clinical examination for Internal Medicine residents (IM-OSCE) as a progress test for clinical skills. Methods: Data from eight administrations of an IM-OSCE were analyzed retrospectively. Data were scaled to a mean of 500 and standard deviation (SD) of 100. A time-based comparison, treating post-graduate year (PGY) as a repeated-measures factor, was used to determine how residents’ performance progressed over time. Results: Residents’ total IM-OSCE scores (n = 244) increased over training from a mean of 445 (SD = 84) in PGY-1 to 534 (SD = 71) in PGY-3 (p < 0.001). In an analysis of sub-scores, including only those who participated in the IM OSCE for all three years of training (n = 46), mean structured oral scores increased from 464 (SD = 92) to 533 (SD = 83) (p < 0.001), physical examination scores increased from 464 (SD = 82) to 520 (SD = 75) (p < 0.001), and procedural skills increased from 495 (SD = 99) to 555 (SD = 67) (p = 0.033). There was no significant change in communication scores (p = 0.97). Conclusions: The IM-OSCE can be used to demonstrate progression of clinical skills throughout residency training. Although most of the clinical skills assessed improved as residents progressed through their training, communication skills did not appear to change. [ABSTRACT FROM PUBLISHER]

Background: Portfolios are widely used for meeting new accreditation standards in the age of competency-based medicine. However, the method of learning through portfolio has been suggested to be vulnerable. Aim: The aim of this study was to explore conditions affecting the experience of teaching and learning from the perspective of both students and mentors in a reflective writing-based portfolio initiative. Method: Using mixed-methods rooted in grounded theory, 139 students and 13 mentors completed questionnaires, 23 students participated in four focus groups and 9 mentors in individual interviews. Results: The overarching theme in our data was student–mentor engagement. Our results confirm previous literature describing portfolio as a vulnerable method of learning, extend this concept by identifying and categorizing specific points of vulnerability, and contribute new knowledge regarding acts of adaptability, which serve to strengthen the student–mentor relationship. Conclusion: Engagement is central to the success of portfolio and is shaped by a dynamic interaction between points of vulnerability and acts of adaptability. We propose a model of engagement in portfolio that can be used for faculty development to optimize student–mentor engagement. [ABSTRACT FROM AUTHOR]

The article presents news briefs regarding medical education, including announcements regarding the Miriam Friedman Ben David New Educator Award, the 17th Ottawa Conference on Assessment of Competence/Capability across the Continuum of Health Professional Education to be held in Perth, Australia from March 19-23, 2016, and questions and answers on medical education practices.

Background: Although many medical learners and teachers are using mobile technologies within medical education, there has been little evidence presented describing how they use mobile devices across a whole curriculum. Methods: The Northern Ontario School of Medicine (NOSM) introduced a new mobile device program in 2010. Incoming undergraduate medical learners received a laptop and an iPad and learners entering year three of the four-year program received a laptop and an iPhone. A survey was sent to all learners to gather information on their use of and attitudes toward these devices. A combination of quantitative and qualitative methods was used to analyze the data and to generate a series of themes that synthesized student behaviors, perceptions and attitudes. Results: Context and learner autonomy were found to be important factors with learners using multiple devices for different purposes and adopting strategic approaches to learning using these devices. The expectation that school-issued devices would be regularly and enthusiastically used to replace more traditional study media was not reflected in practice. Conclusions: Learners' approaches to using mobile devices are heterogeneous as is the extent to which they use them. Learners adapt their use of mobile devices to the learning cultures and contexts they find themselves in. [ABSTRACT FROM AUTHOR]

Background: Since a disproportionate amount of medical education still occurs in hospitals, there are concerns that medical school graduates are not fully prepared to deliver efficient and effective care in ambulatory settings to increasingly complex patients. Aims: To understand the current extent of scholarship in this area. Method: A scoping review was conducted by searching electronic databases and grey literature sources for articles published between 2001 and 2011 that identified key challenges and models of practice for undergraduate teaching of ambulatory care. Relevant articles were charted and assigned key descriptors, which were mapped onto Canadian recommendations for the future of undergraduate medical education. Results: Most of the relevant articles originated in the United States, Australia, or the United Kingdom. Recommendations related to faculty development, learning contexts and addressing community needs had numerous areas of scholarly activity while scholarly activity was lacking for recommendations related to inter-professional practice, the use of technology, preventive medicine, and medical leadership. Conclusions: Systems should be established to support education and research collaboration between medical schools to develop best practices and build capacity for change. This method of scoping the field can be applied using best practices and recommendations in other countries. [ABSTRACT FROM AUTHOR]

Purpose: This study explored the views of junior faculty toward informing mentorship program development. Method: Mixed sampling methodologies including questionnaires ( n = 175), focus groups (female, n = 4; male, n = 4), and individual interviews (female n = 10; male, n = 9) of junior faculty were conducted in clinical departments at one academic health sciences center. Results: Questionnaire results indicated that having role models increased commitment to an academic career; mentorship experience during residency training was a high incentive to pursue an academic career; and junior faculty did have identifiable mentorship experiences. Focus group results revealed that mentoring as well as the presence of role models a few years ahead of the junior faculty would promote career development. Females preferred similar age role models who spoke the same language, particularly in the area of promotion. Females identified several challenges and issues including a lack of researcher role models, a range of perceptions regarding the merits of formal versus informal mentoring, and the idea that mentors should provide advice on promotion and grants. Males valued advice on finances while females wanted advice on work-life balance. Conclusions: Mentorship emerged as an important factor in academic faculty recruitment and retention, with varying perceptions of how it should be institutionalized. Role models were viewed as important for retention, and a paucity of mid-career, female researcher role models suggests a gap to be filled in future programmatic efforts. [ABSTRACT FROM AUTHOR]

The article focuses on the recommendations and consensus statement for the performance assessments in medical education based from a conference in Ottawa, Ontario. It defines competence as the clinical practice ability of an individual. It mentions the recommendations for individual competency test which include ensuring the clear purpose, assessment development, and scoring format usage. Moreover, the diagram of G. E. Miller's model pyramid of performance assessment is presented.

The uptake of information and communication technologies (ICTs) in health professions education can have far-reaching consequences on assessment. The medical education community still needs to develop a deeper understanding of how technology can underpin and extend assessment practices. This article was developed by the 2010 Ottawa Conference Consensus Group on technology-enabled assessment to guide practitioners and researchers working in this area. This article highlights the changing nature of ICTs in assessment, the importance of aligning technology-enabled assessment with local context and needs, the need for better evidence to support use of technologies in health profession education assessment, and a number of challenges, particularly validity threats, that need to be addressed while incorporating technology in assessment. Our recommendations are intended for all practitioners across health professional education. Recommendations include adhering to principles of good assessment, the need for developing coherent institutional policy, using technologies to broaden the competencies to be assessed, linking patient-outcome data to assessment of practitioner performance, and capitalizing on technologies for the management of the entire life-cycle of assessment. [ABSTRACT FROM AUTHOR]

Assessment for selection in medicine and the health professions should follow the same quality assurance processes as in-course assessment. The literature on selection is limited and is not strongly theoretical or conceptual. For written testing, there is evidence of the predictive validity of Medical College Admission Test (MCAT) for medical school and licensing examination performance. There is also evidence for the predictive validity of grade point average, particularly in combination with MCAT for graduate entry but little evidence about the predictive validity of school leaver scores. Interviews have not been shown to be robust selection measures. Studies of multiple mini-interviews have indicated good predictive validity and reliability. Of other measures used in selection, only the growing interest in personality testing appears to warrant future work. Widening access to medical and health professional programmes is an increasing priority and relates to the social accountability mandate of medical and health professional schools. While traditional selection measures do discriminate against various population groups, there is little evidence on the effect of non-traditional measures in widening access. Preparation and outreach programmes show most promise. In summary, the areas of consensus for assessment for selection are small in number. Recommendations for future action focus on the adoption of principles of good assessment and curriculum alignment, use of multi-method programmatic approaches, development of interdisciplinary frameworks and utilisation of sophisticated measurement models. The social accountability mandate of medical and health professional schools demands that social inclusion, workforce issues and widening of access are embedded in the principles of good assessment for selection. [ABSTRACT FROM AUTHOR]

Medical education research in general is a young scientific discipline which is still finding its own position in the scientific range. It is rooted in both the biomedical sciences and the social sciences, each with their own scientific language. A more unique feature of medical education (and assessment) research is that it has to be both locally and internationally relevant. This is not always easy and sometimes leads to purely ideographic descriptions of an assessment procedure with insufficient general lessons or generalised scientific knowledge being generated or vice versa. For medical educational research, a plethora of methodologies is available to cater to many different research questions. This article contains consensus positions and suggestions on various elements of medical education (assessment) research. Overarching is the position that without a good theoretical underpinning and good knowledge of the existing literature, good research and sound conclusions are impossible to produce, and that there is no inherently superior methodology, but that the best methodology is the one most suited to answer the research question unambiguously. Although the positions should not be perceived as dogmas, they should be taken as very serious recommendations. Topics covered are: types of research, theoretical frameworks, designs and methodologies, instrument properties or psychometrics, costs/acceptability, ethics, infrastructure and support. [ABSTRACT FROM AUTHOR]

In this article, we outline criteria for good assessment that include: (1) validity or coherence, (2) reproducibility or consistency, (3) equivalence, (4) feasibility, (5) educational effect, (6) catalytic effect, and (7) acceptability. Many of the criteria have been described before and we continue to support their importance here. However, we place particular emphasis on the catalytic effect of the assessment, which is whether the assessment provides results and feedback in a fashion that creates, enhances, and supports education. These criteria do not apply equally well to all situations. Consequently, we discuss how the purpose of the test (summative versus formative) and the perspectives of stakeholders (examinees, patients, teachers-educational institutions, healthcare system, and regulators) influence the importance of the criteria. Finally, we offer a series of practice points as well as next steps that should be taken with the criteria. Specifically, we recommend that the criteria be expanded or modified to take account of: (1) the perspectives of patients and the public, (2) the intimate relationship between assessment, feedback, and continued learning, (3) systems of assessment, and (4) accreditation systems. [ABSTRACT FROM AUTHOR]

Purpose: To develop components of a curriculum for teaching and evaluating Residents as health advocates. Method: Modeled on the Delphi technique, the first step involved a multidisciplinary panel of 10 Queen's University health care providers with expertize in education and patient advocacy. In the context of four Advocacy questions: What is it?, Who does it?, How to teach it?, and How to evaluate it?, they discussed a curriculum framework including graded education, scholarly activity, role modeling, and case examples. In the second step, 24 faculty experts addressed two goals: (1) to identify attributes discussed by the expert panel in step 1 and corresponding measurable behaviours and (2) to refine the curriculum framework proposed in step 1 with emphasis on content and evaluation. Results: Six attributes of a health advocate were identified; knowledgeable, altruistic, honest, assertive, resourceful, and up-to date. Behaviours that reflect these attributes were identified as desirable or undesirable and means of teaching were matched to the attributes. For most residents, skills would be developed in a graded fashion, progressing from advocating for the individual to society as a whole. Conclusions: This study provides a general framework from which specialty-specific curriculums for training health advocates can be developed. [ABSTRACT FROM AUTHOR]

CURATA is a multifaceted continuing medical education (CME) intervention, developed with input from 12 healthcare organizations to address the gap between current and recommended osteoarthritis (OA) treatment of general practitioners in Québec, Canada. Focusing on appropriate prescription of non-steroidal anti-inflammatory drugs, including cyclooxygenase-2 selective inhibitors (coxibs), the intervention comprised small-group, case-based workshops modelled after the Script Concordance test, and a decision tool reflecting current evidence-based clinical practice guidelines. A self-reported questionnaire measured knowledge of recommended OA treatment on an eight-point scale. Participants ( n = 381) showed a mean 10.1% improvement in questionnaire score immediately following the workshop (15.2% improvement relative to mean pre-workshop score). Knowledge was maintained for three months post-workshop. [ABSTRACT FROM AUTHOR]

The Canadian Memorial Chiropractic College is in the fourth year of implementing a new four-year Integrative Curriculum. In order to determine how the students experienced the educational environment generated by the new curriculum, the Dundee Ready Education Environment (DREEM) Inventory was administered on the same day to 148 (96%) first-year, 131 (87%) second-year, and 128 (84%) third-year students ( n = 407). Resultant scores indicated many areas of concern. However, bimodal and large numbers of 'uncertain' responses also occurred. In order to establish and compare areas of most concern between the three year groups, responses were calculated as percentages indicating for each item those who agree, disagree or are uncertain. This result clearly indicated areas of joint concern for each of the five domains addressed by the DREEM Inventory. Analysis of these concerns formed the basis of strategic planning in order to institute remedial action, and on which to focus institutional resources. [ABSTRACT FROM AUTHOR]

For over 20 years the occupational therapy programmes offered by McMaster University and Mohawk College, Hamilton, Ontario have used small-group, problem-based learning tutorials as a major component of their curriculum. These programmes were among the first occupational therapy programmes in the world to use a problem-based tutorial format. The inclusion as tutors of both full-time faculty and clinicians, from all clinical practice areas, was central to the design of the problem-based learning courses. A survey of all tutors from the last 20 years collected information about why health professionals are motivated to tutor and what they see as challenges to maintaining this educational role. Three primary themes emerged from the data: being an educator; being a learner and present and future challenges to continuing with the tutoring role. Within the educator theme there was a secondary theme of professional duty or obligation. In addition, the participants identified suggestions for enhanced support and continuing education for tutors. This article summarizes the findings of the survey. [ABSTRACT FROM AUTHOR]

The article focuses on the author's insights on the impact of the inappropriate use of electronic media on medical education in Ontario. She points out that several medical schools have reported untoward incidents related to misuse of social networking sites such as Facebook including posting of inappropriate comments and photographs, negatively impacting the reputation of institutions by association with undesirable acts. She emphasizes the significance of developing some principles for digital professionalism wherein digital skills, attitudes, and knowledge should be embodied and modeled by teachers, physicians, and others involved in health professional education to students and colleagues.

In Ontario massage therapy is a regulated health profession, and it has been speculated that massage therapy students are motivated primarily by altruistic values, as has been documented in medicine. Students at Sutherland-Chan School and Teaching Clinic in Ontario were surveyed regarding their motivation to enter massage therapy education, with the intention of assessing the influence of certain value complexes on their decision. The results indicate that their decision was influenced more strongly by intrinsic values related to helping and working with people than by the prospect of extrinsic rewards ( p < 0.0005). This supports a belief commonly held within the professional community, and bodes well for the future of massage therapy as a caring profession. [ABSTRACT FROM AUTHOR]

The article presents the author's reflections on assessment practices in medical education, suggesting the need for reform. Topics addressed include the 16th Ottawa and 12th Canadian Conference on Medical Education, entitled "Transforming Healthcare through Excellence in Assessment and Evaluation," held in Ottawa, Canada in April 2014, a review of scholarly literature focusing on educational assessment, and the "Pumpkin Plan" paradigm proposed by strategist Mike Michalowicz.