Search

IntroductionAs adults age, their senses tend to decline and a large portion of those most affected by sensory decline reside in long-term care. At present, the creation of a sense-sensitive environment in long-term care is a difficult task as there is minimal evidence or tools available to guide this process. The 5Senses screening tool was developed to measure the sense-sensitivity of a particular environment, with a focus on long-term care. The purpose of this paper is to describe a study protocol to assess the psychometric properties of the newly developed 5Senses screening tool.Methods and analysisWe will conduct a psychometric evaluation of the 5Senses screening tool in long-term care based on the Standards for Educational and Psychological Testing Framework. In phase I, we will seek input from international content experts (n=20) to assess the content validity of all sections of the tool. In phase II, we will invite auditors (n=3-9), residents (n=3-9) and staff (n=3-9) to partake in think-aloud sessions to assess response process validity. In phase III, we will conduct field testing of the revised 5Senses screening tool with auditors (n=100), residents (n=100) and staff (n=100) to evaluate additional measures including acceptability, inter-rater reliability, internal structure validity and internal consistency reliability, where possible.Ethics and disseminationEthical approval was obtained from the University of Ottawa Research Ethics Board. Findings will be disseminated through a peer-reviewed manuscript, through a dedicated website, through presentations in long-term care communities and through presentations at research conferences.

The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.

The interest of psychologists in the study of safety in the educational environment and its psychological characteristics is increasing. The aim of this study was to select, substantiate, and disclose the psychological factors in the educational environment that influence the psychological well-being of pupils. There were three stages in our research: clarifying the level of psychological safety in the schools in the study; revealing the consistency of the teachers’ and the students’ evaluations of the level of psychological safety in those schools; determining the nature of the influence of the psychological safety of the educational environment on the psychological well-being of the students. The study involved 172 teachers and 876 students in Moscow and St. Petersburg schools. Psychological safety is a condition of educational environments that are free from psychological violence in the interactions of the people in them; psychological safety contributes to the satisfaction of the needs for personal trust and communication, creates for the participants a sense of belonging (the referential importance of the environment), and contributes to their mental health. The empirical study revealed that (1) the level of the psychological safety of the educational environment in the evaluations of the teachers was higher and had more expressed differentiation than did the level in the students’ evaluations; (2) the psychological well-being of the students was closely correlated with the level of psychological safety for the teachers; (3) such components of the well-being of the students as emotional comfort, self-confidence, a higher level of cognitive activity were developed in those schools with high levels of psychological safety for the teachers. The results can be useful in educational psychology and environmental psychology as well as when creating technologies to support the safety of school environments and of the people in them, including situations related to minimizing safety violations and reducing risks and threats.

Background and Purpose. In this paper, we present a protocol for advanced psychometric assessments of surveys based on the Standards for Educational and Psychological Testing. We use the Alberta Context Tool (ACT) as an exemplar survey to which this protocol can be applied.Methods. Data mapping, acceptability, reliability, and validity are addressed. Acceptability is assessed with missing data frequencies and the time required to complete the survey. Reliability is assessed with internal consistency coefficients and information functions. A unitary approach to validity consisting of accumulating evidence based on instrument content, response processes, internal structure, and relations to other variables is taken. We also address assessing performance of survey data when aggregated to higher levels (e.g., nursing unit).Discussion. In this paper we present a protocol for advanced psychometric assessment of survey data using the Alberta Context Tool (ACT) as an exemplar survey; application of the protocol to the ACT survey is underway. Psychometric assessment of any survey is essential to obtaining reliable and valid research findings. This protocol can be adapted for use with any nursing survey.

This book provides, from an international perspective, an independent analysis of major issues facing the educational evaluation and assessment framework, current policy initiatives, and possible future approaches in Luxembourg.

Background Organizational context has the potential to influence the use of new knowledge. However, despite advances in understanding the theoretical base of organizational context, its measurement has not been adequately addressed, limiting our ability to quantify and assess context in healthcare settings and thus, advance development of contextual interventions to improve patient care. We developed the Alberta Context Tool (the ACT) to address this concern. It consists of 58 items representing 10 modifiable contextual concepts. We reported the initial validation of the ACT in 2009. This paper presents the second stage of the psychometric validation of the ACT. Methods We used the Standards for Educational and Psychological Testing to frame our validity assessment. Data from 645 English speaking healthcare aides from 25 urban residential long-term care facilities (nursing homes) in the three Canadian Prairie Provinces were used for this stage of validation. In this stage we focused on: (1) advanced aspects of internal structure (e.g., confirmatory factor analysis) and (2) relations with other variables validity evidence. To assess reliability and validity of scores obtained using the ACT we conducted: Cronbach's alpha, confirmatory factor analysis, analysis of variance, and tests of association. We also assessed the performance of the ACT when individual responses were aggregated to the care unit level, because the instrument was developed to obtain unit-level scores of context. Results Item-total correlations exceeded acceptable standards (> 0.3) for the majority of items (51 of 58). We ran three confirmatory factor models. Model 1 (all ACT items) displayed unacceptable fit overall and for five specific items (1 item on adequate space for resident care in the Organizational Slack-Space ACT concept and 4 items on use of electronic resources in the Structural and Electronic Resources ACT concept). This prompted specification of two additional models. Model 2 used the 7 scaled ACT concepts while Model 3 used the 3 count-based ACT concepts. Both models displayed substantially improved fit in comparison to Model 1. Cronbach's alpha for the 10 ACT concepts ranged from 0.37 to 0.92 with 2 concepts performing below the commonly accepted standard of 0.70. Bivariate associations between the ACT concepts and instrumental research utilization levels (which the ACT should predict) were statistically significant at the 5% level for 8 of the 10 ACT concepts. The majority (8/10) of the ACT concepts also showed a statistically significant trend of increasing mean scores when arrayed across the lowest to the highest levels of instrumental research use. Conclusions The validation process in this study demonstrated additional empirical support for construct validity of the ACT, when completed by healthcare aides in nursing homes. The overall pattern of the data was consistent with the structure hypothesized in the development of the ACT and supports the ACT as an appropriate measure for assessing organizational context in nursing homes. Caution should be applied in using the one space and four electronic resource items that displayed misfit in this study with healthcare aides until further assessments are made.

Background There is a lack of acceptable, reliable, and valid survey instruments to measure conceptual research utilization (CRU). In this study, we investigated the psychometric properties of a newly developed scale (the CRU Scale). Methods We used the Standards for Educational and Psychological Testing as a validation framework to assess four sources of validity evidence: content, response processes, internal structure, and relations to other variables. A panel of nine international research utilization experts performed a formal content validity assessment. To determine response process validity, we conducted a series of one-on-one scale administration sessions with 10 healthcare aides. Internal structure and relations to other variables validity was examined using CRU Scale response data from a sample of 707 healthcare aides working in 30 urban Canadian nursing homes. Principal components analysis and confirmatory factor analyses were conducted to determine internal structure. Relations to other variables were examined using: (1) bivariate correlations; (2) change in mean values of CRU with increasing levels of other kinds of research utilization; and (3) multivariate linear regression. Results Content validity index scores for the five items ranged from 0.55 to 1.00. The principal components analysis predicted a 5-item 1-factor model. This was inconsistent with the findings from the confirmatory factor analysis, which showed best fit for a 4-item 1-factor model. Bivariate associations between CRU and other kinds of research utilization were statistically significant (p < 0.01) for the latent CRU scale score and all five CRU items. The CRU scale score was also shown to be significant predictor of overall research utilization in multivariate linear regression. Conclusions The CRU scale showed acceptable initial psychometric properties with respect to responses from healthcare aides in nursing homes. Based on our validity, reliability, and acceptability analyses, we recommend using a reduced (four-item) version of the CRU scale to yield sound assessments of CRU by healthcare aides. Refinement to the wording of one item is also needed. Planned future research will include: latent scale scoring, identification of variables that predict and are outcomes to conceptual research use, and longitudinal work to determine CRU Scale sensitivity to change.

Certification, as defined by the American Board of Nursing Specialties (ABNS), “is the formal recognition of the specialized knowledge, skills, and experience demonstrated by the achievement of standards identified by a nursing specialty to promote optimal health outcomes” (ABNS, 2006). The American Board of Neuroscience Nursing (ABNN) was established to design, implement, and evaluate a certification program for professional nurses involved in the specialty practice of neuroscience nursing (American Association of Neuroscience Nurses [AANN], 2006). Since 1978, the certification granted by ABNN to nurses practicing in this specialty is the Certified Neuroscience Registered Nurse (CNRN) designation. Studies of role delineation within nursing specialties are used to provide evidence that a certification examination provided by a certification program is reliable and valid. The Joint Standards for Educational and Psychological Testing (American Psychological Association, 1999) state that “the content domain to be covered by a credentialing test should be defined clearly and justified in terms of imiportance of the content for the credential-worthy performance in an occupation or profession” (p. 161). In 2002 ABNS awarded a 5-year accreditation to the CNRN examination, signifying that the examination demonstrated compliance with certification standards. ABNS Standard 7 states, “The certifying organization has conducted validation studies to assure that inferences made on the basis of test scores are appropriate and justified,” and generally recommends that job analyses be conducted within 5 years of the last full analysis (ABNS, 2004). ABNN conducted its most recent role delineation study (RDS) in 2001 (Blissitt, Roberts, Hinkle, & Kopp, 2003). Repeating the process within 5 years is consistent with both the ABNS standard and the practice of other nursing specialty organizations (McMillan, Heusinkveld, Chai, Miller-Murphy, & Huang, 2002; Muenzen, Greenberg, & Pirro, 2004). This article describes the background, methodology, results, and implications for certification of the 2005 study of role delineation in neuroscience nursing undertaken at the direction of ABNN. The study was conducted by a task force appointed by ABNN.

It is an honor to be asked to give this address at GERA. To acknowledge that honor briefly, and with my tongue in my cheek a bit, let me start with a brief account of my own contact with GERA, and its parent organization, AERA. This account is offered to indicate how one like me who does historical research in education has interacted with the movers and shakers in our professional educational research organizations, who largely do not do such research, over the past three decades.

Given the high stakes of teacher testing, there is no doubt that every teacher test should meet the industry guidelines set forth in the Standards for Educational and Psychological Testing. Unfortunately, however, there is no public or private business or governmental agency that serves to certify or in any other formal way declare that any teacher test does, in fact, meet the psychometric recommendations stipulated in the Standards. Consequently, there are no legislated penalties for faulty products (tests) nor are there opportunities for test takers simply to raise questions about a test and to have their questions taken seriously by an impartial panel. The purpose of this article is to highlight some of the psychometric results reported by National Evaluation Systems (NES) in their 1999 Massachusetts Educator Certification Test (MECT) Technical Report, and more specifically, to identify those technical characteristics of the MECT that are inconsistent with the Standards. A second purpose of this article is to call for the establishment of a standing test auditing organization with investigation and sanctioning power. The significance of the present analysis is twofold: a) psychometric results for the MECT are similar in nature to psychometric results presented as evidence of test development flaws in an Alabama class-action lawsuit dealing with teacher certification (an NES-designed testing system); and b) there was no impartial enforcement agency to whom complaints about the Alabama tests could be brought, other than the court, nor is there any such agency to whom complaints about the Massachusetts tests can be brought. I begin by reviewing NES's role in Allen v. Alabama State Board of Education, 81-697-N. Next I explain the purpose and interpretation of standard item analysis procedures and statistics. Finally, I present results taken directly from the 1999 MECT Technical Report and compare them to procedures, results, and consequences of procedures followed by NES in Alabama.

In 2000, Lawrence Lerner and the Thomas B. Fordham Foundation reviewed state science standards in 49 states and the District of Columbia, specifically with respect to the coverage of evolution. We repeat his survey for all current standards publicly available as of May 2009 and discuss the history and role of state science standards in American public education. Our survey indicates that science standards tend to cover evolution more extensively than they did a decade ago, and that the average quality of the treatment has increased. However, certain types of creationist language are also becoming more common in state standards. We also discuss the history and role of state science standards in American public education.

In this edition of Perspectives on Medical Education, Nesbitt et al. explore the role of simulation in the development of surgical skills [1]. The author’s main purpose of interest was the literature concerning endovascular simulation. The acquisition of motor skills is, however, universal and therefore their findings can be combined with research from other surgical specialities and in the end generalized into evidence-based guidelines. An extensive review from 2011 compared technology-enhanced simulation training with no intervention and found large effects for knowledge, skills and behaviours, and moderate effects for patient-related outcomes [2]. The fact that simulation-based training is better than no training might not be a big surprise; to quote Geoff Norman in his editorial Data dredging, salami-slicing, and other successful strategies to ensure rejection: twelve tips on how to not get your paper published: ‘We don’t need to compare something with nothing…We’ll accept without proof that some education is better than none’ [3]. However, a recent randomized controlled trial found that approximately 4 h of virtual-reality simulator training was significantly more efficient than a half day of supervised apprenticeship training on patients [4]. It is no longer of question if we should practice on simulators but how simulation-based training should be implemented. First of all, it is important to acknowledge that simulation-based training cannot replace traditional apprenticeship training. New surgical trainees must be supervised by more experienced colleagues during their first real procedures just as simulator trained pilots start out as co-pilots. Furthermore, we need to rethink the extreme focus on simulation equipment that has been the hallmark of the literature concerning advanced technical skills training. Nesbitt et al. carefully list the different pros and cons of synthetic models, animal models, virtual reality simulators, and human cadavers and even describe an ideal endovascular training model. Rightfully, they end up concluding that: ‘The biggest current barrier to the routine integration of simulation into endovascular training is the lack of an agreed curriculum’—not the lack of equipment. There is no doubt that the ‘boys and their toys’ phenomenon has resulted in many expensive simulators being covered in dust once the initial enthusiasm has disappeared. A viable training programme based on the best available evidence should be planned before investing time and money in simulation-based training. Creating professional simulation centres that are used by many departments and hospitals is an efficient way of countering potential problems with costs, licenses, logistics etc [5]. However, we totally agree with Nesbitt et al. that: ‘Simulation should not be a one-off training exercise’. Distributed learning where the training sessions are spaced out over several days is more efficient than massed practice which limits the maximum practical distance between the trainees and the simulation centre [6]. Fortunately, two trainees can share a simulator (dyad training) and a busy (and expensive) consultant does not need to be present at all times [7]. Directed, self-regulated learning where the trainees are allowed to make their own experiences and learn from their mistakes can improve retention compared with instructor-led training [8]. No matter how the training is conducted it is essential to end each programme with a simulation-based test. Not all trainees learn at the same pace and training to a pre-defined criterion is the only way to ensure basic competency before performance on patients. Furthermore, final testing is motivating and improves retention. Mastery learning produces strong and lasting effects and mandatory training and certification programmes are necessary to ensure the maximum gain from simulation-based skills training [9]. Simulation-based tests with solid evidence of validity and defensible pass/fail scores are a prerequisite for mastery learning. The Standards for Educational and Psychological Testing recommend to view validity as a unitary concept and to abandon the historical nomenclature (face validity, construct validity etc.) [10]. These recommendations have been around for more than 15 years and it is about time surgeons replace the outdated framework of validity with a contemporary one, such as Messick’s or Kane’s [11]. The design of the test is also important. Evidence suggests that global rating scales are better than checklists in capturing nuanced elements of expertise and that incompetent trainees can achieve high checklist scores despite committing serious procedural errors [12]. In conclusion, there is overwhelming evidence for the efficacy of simulation-based training in clinical skills training. Simulation must be integrated in the training curriculum as distributed training sessions with the possibility of directed, self-regulated learning in professional training facilities. Simulation-based training to proficiency should be mandatory before trainees are allowed to perform procedures on patients.