Search

Assessment and evaluation at all levels of educational systems have become policy priorities for many countries. Two common reasons for this are student learning expectations and accountability. Although much effort has been put into the creation and refinement of content standards, standardized tests, and methods for using testing results, there has been less attention paid to the development of performance standards (proficiency levels) that greatly affect policy decision making. The present study investigates the Objective Standard Setting Model (OSS) as an improved criterion-referenced method for setting multilevel performance standards. To demonstrate how OSS can be employed for performance standard setting, our study used data from preservice teachers completing an assessment literacy test in a pre- and posttest environment. Using OSS, performance standard levels of proficient and excellent were established with clear content-related descriptions of growth in student content mastery. [ABSTRACT FROM AUTHOR]

Objectives: The authors outline the difference between content and performance standards and the rationale for standard setting at a medical college. The principles of the college's standard setting processes for the written and objective structured clinical examination summative assessments are discussed in greater detail.Conclusion: There is no evidence of any single standard setting method to be the best. Multiple methods exist and will have varied results when applied. The judgement of a panel of subject experts remains an important component of the standard setting process. [ABSTRACT FROM AUTHOR]

In current research on achievement goal theory, most researchers differentiate between performance-approach and performance-avoidance goal orientations. Evidence from prior research and from several previously published data sets is used to highlight that the correlation is often rather large, with a number of studies reporting correlations above .50. The large magnitude of this correlation raises questions and warrants further investigation. The size of the correlation also varies substantially across studies; thus, several potential moderators were considered. Minimal evidence for moderation was found, with little variability in relations as a function of fear of failure, culture, and specificity of the goal assessment. There was some evidence of variability in the correlation based on age, perceived competence, and assessment instrument. The article concludes by highlighting theoretical, methodological, and instructional questions that arise as a result of the large correlation and making recommendations and guidance for research, instructional practice, and theory advancement. [ABSTRACT FROM PUBLISHER]

In several earlier analyses of two tests of academic literacy - the Test of Academic Literacy Levels (TALL) and its Afrikaans counterpart, the Toets vir Akademiese Geletterdheidsvlakke (TAG) - we have adopted an approach to the problem that tests may be abused (and therefore used to harm people) by discussing various antidotes to this, so as to ensure fairness and consistency in the tests we use, as well as by demonstrating how the process of test development and implementation might be made more transparent. We have pointed out that a true measure of the stability of such tests may only become apparent when we have data stretching across a number of years. This article reports on an investigation of differing measures of performance on the two tests in question across several of their administrations between 2005 and 2008. We point to a number of ways in which empirical measures may be used to achieve benign, enabling effects on those tested. [ABSTRACT FROM AUTHOR]

The paper analyses how teachers view state-mandated tests in Baden-Wurttemberg, a large state in the south-west of Germany. For the first time in 2006, public secondary schools were bound by law to administer Vergleichsarbeiten [state-wide tests] in up to three relevant subjects after Grade 6. Scholars in the field of school improvement emphasise that there is a gap between performance feedback provided by external tests and how this information is internally processed. To bridge this gap, a crucial precondition is that teachers must accept state-wide testing as a useful instrument for improving the performance of schools. To examine how teachers view the pedagogical relevance of performance feedback information, 256 secondary schools were approached, and an anonymous questionnaire was completed by a representative sample of 307 teachers with Grade 6 classes. Analysis of the quantitative data reveals that teachers in general secondary schools (the lower level of the academically streamed secondary system) are more open-minded towards the tests than their colleagues in intermediate schools or in grammar schools (the highest level). Even so, only a small percentage of teachers in general secondary schools acknowledge the pedagogical relevance of performance feedback information for improving teaching strategies. This result suggests that state-mandated testing in Baden-Wurttemberg is highly unlikely to initiate or support school improvement activities for the time being. [ABSTRACT FROM AUTHOR]

A one-year follow-up study examined the role of developmental challenges, personality (dependency, self-criticism, and personal efficacy), and support systems in adaptation among Israeli emerging adults (N = 236) participating in a preparatory academic program. Participants were assessed during their enrollment in the preparatory academic program and one year later, after graduation, when their academic success or failure could be determined. Personal efficacy predicted higher levels of goal investment and goal progress, as well as more positive life events. In contrast, self-criticism predicted lower goal investment, higher goal stress, and a reduced number of positive events. Self-criticism also predicted elevated levels of a motivation and an increase in depressive and anxiety symptoms. Paternal social support predicted young people's more favorable adaptation one year later. Among young adult males, academic failure predicted later level of symptoms and more negative events. These findings encourage further examination of the role of personality and family relationships in the transition to adulthood. [ABSTRACT FROM AUTHOR]

The article focuses on the overall results released by the National Assessment of Educational Progress (NAEP) regarding the mathematical skills and problem-solving ability of students whose ages range from nine to 17. It states that Long-Term Trend (LTT) assessment was used to indicate the change in performance of students over time. It poses several items employed for the assessment including Released Items, Computational Items, and Metric system knowledge items. It also lays various charts and figures which show the summary of the results.

Interest in end-of-year accountability exams has increased dramatically since the passing of the No Child Left Behind Act in 2001. With this increased interest comes a desire to use student data collected throughout the year to estimate student proficiency and predict how well they will perform on end-of-year exams. This article uses student performance on the Assistment System, an online mathematics tutor, to show that replacing percentage correct with an Item Response Theory estimate of student proficiency leads to better fitting prediction models. In addition, it uses other tutor performance metrics to further increase prediction accuracy. Prediction error bounds are also calculated to attain an absolute measure to which the models can be compared. [ABSTRACT FROM AUTHOR]

The No Child Left Behind Act's emphasis on universal standards of academic performance, defined as proficiency in grade-level core curriculum and state academic assessments, challenges general education program personnel to higher standards of professional practice. Because schools must report student achievement by subgroups, including students with disabilities and English language learners (ELLs), reform efforts have increasingly focused on closing the achievement gap between these subgroups and mainstream peers. However, little information is available about the qualifications of teachers who serve ELLs who also have disabilities. This article delineates critical teacher competencies to address the needs of these students and of ELLs who are at risk for special education placement. [ABSTRACT FROM AUTHOR]

This paper draws on semi-structured interview data and participant observations of senior secondary Physical Education (PE) teachers and students at two school sites across 20 weeks of the school year. The data indicated that the teachers in this study made progressive judgements about students' level of achievement across each unit of work without explicit or overt reference to the criteria and standards represented in the schools' work programmes and in the Senior PE syllabus. The teachers' justification for such an approach was that the criteria and standards had become for them sufficiently 'internalised'. Determining students' levels of achievement was for the teachers somewhat 'intuitive', being reliant on their memory of students' performances, and influenced by the construct-irrelevant affective characteristics of the students. It is argued in this paper that such construct-irrelevance compromised the construct validity and possible inter-rater reliability of the decisions made and advantaged some students and marginalised others on the basis of characteristics that were not specifically related to the learning expected from following the syllabus. The potential inequities of such an approach are discussed and suggestions are made for the consolidation of the validity and reliability of teachers' judgements. [ABSTRACT FROM AUTHOR]

This 2006 NCME Career Award Address presents observations made over a career in the field of standard setting, summarizes the status of current research, and makes recommendations for future research. [ABSTRACT FROM AUTHOR]

Standard-setting methods are widely used to determine cut scores on a test that examinees must meet for a certain performance standard. Because standard setting is a measurement procedure, it is important to evaluate variability of cut scores resulting from the standard-setting process. Generalizability theory is used in this study to estimate standard errors of cut scores resulting from two standard-setting methods: item rating (Angoff-based) and mapmark (bookmark-based) methods. In this study, two different generalizability (G) study designs and four different decision (D) study designs were examined, and the impact of varying different aspects of the study design and universes of generalization was examined. Results suggest that cut scores were generally consistent for both methods. The first round standard setting contributed the most to the overall variability for the mapmark method. Also, it is clear that there is no one standard error associated with a certain cut score. [ABSTRACT FROM AUTHOR]

In a sample of industrial job applicants, relationships among scores on an integrity test, a job knowledge measure, and maximal performance as assessed by a work sample measure were investigated. The observed correlation between the personality-based integrity test and maximal performance was .27, indicating that integrity tests can be predictive of maximal performance. Furthermore, integrity test scores correlated .14 with job knowledge and job knowledge scores had a validity of .36 for maximal performance. Theoretical implications for the maximal/typical performance distinction are discussed. [ABSTRACT FROM AUTHOR]

The collision of classical teaching methods with technology was inevitable; now how do we use technology to buttress the goals of analytical thinking and problem solving? This study juxtaposes the dialectical method of Socrates with current technology to demonstrate that time tested techniques can be delivered via new technologies. A pre-test, post-test study (n=239) employing case method concepts was conducted over five semesters. The learning module was created using modified hierarchy with associative structures which allows moderate learner control. Findings indicate that significant learning occurred among all levels of students, and student response was universally positive. Providing some learner control with structure seemed to accommodate student learning in terms of achievement and interest. [ABSTRACT FROM AUTHOR]

One of the factors which has been cited to affect whether multimedia is effective at improving learning performance is the difficulty of the material being learned. The general notion is that multimedia tends to be ineffective for learning easy materials. However, it's not clear at which point the use of multimedia starts becoming beneficial to learning performance and therefore when to and when not to use multimedia. In a sense all this may suggest that there is a relationship between material difficulty and the effectiveness of multimedia at improving learning that may benefit from investigation. In this study, this possible relationship was investigated. The learning performance scores of three groups of 12 students who used multimedia-based learning, computer-based learning, or paper-based learning were compared over varying levels of material difficulty. The mean test score of the multimedia group was found to be significantly more than those of the other two groups only when the level of material difficulty was at the highest. The implication of this finding is discussed. [ABSTRACT FROM AUTHOR]

Goal orientation research has developed two branches. One branch uses a two-factor dispositional model. The other branch uses a three-factor domain-specific model. The present multiwave study examines dispositional goal orientation constructs through both two- and three-factor models using repeated-measure confirmatory factor analysis. Results indicate dispositional performance goal orientation of the two-factor model can be better partitioned into the Prove and Avoid factors of the three-factor model. Further evidence is presented indicating the potential for more than three factors by further partitioning the Avoid factor into two separate constructs. Consistent with both branches of goal orientation research, learning goal orientation is largely independent of Performance, Prove, and Avoid factors in all tested models. [ABSTRACT FROM AUTHOR]

Current preschool assessments lack the ability to capture performance variance and complex thinking skills. This study examined the relationship among student performance, student feelings, and teacher perceptions as they apply to using a formal performance assessment in a preschool setting. A formal performance assessment was administered to 71 preschool-aged children (56–71 months of age). The data were analyzed using descriptive and inferential statistics. As hypothesized, there was a significant and positive correlation between the students’ performance and the teachers’ perceptions. Furthermore, there was not a significant relationship between either the students’ feelings and the teachers’ perceptions or the students’ feelings and the students’ performance. These findings and theoretical study indicate support for the use of formal performance assessment to evaluate preschool students. Limitations and implications of the results are discussed. [ABSTRACT FROM AUTHOR]

The Educational Commission for Foreign Medical Graduates has been conducting a high-stakes, high-volume Clinical Skills Assessment for over one year and has assessed several thousand candidates. In the process, a great deal has been learned about the demands of testing in such a demanding environment.These lessons may be of benefit to those embarking on similar assessment programs. [ABSTRACT FROM AUTHOR]

Special education teachers are routinely taught to collect and analyze student performance data. Data collection and analysis are assumed to be necessary for accurate analysis of three possible trends: (a) an upward trend (when performance trend (when performance worsens). Few studies have examined the accuracy of trend analysis when teachers simply observe consecutive training sessions, nor has there been sufficient comparison of the accuracy of trend analysis based on various forms of data (e.g., raw data vs. data in graphic form). The present study examined the effects of observation and various forms of data on teachers' ability to accurately analyze trends. A 2-factor experimental design was used with 4 groups of randomly assigned subjects. Subjects analyzed 15 samples of videotaped student performance. All subjects viewed the tapes with or without additional forms of data and made judgment regarding the trend of student performance following exposure to each sample. The data were analyzed using a two-way ANOVA. Both the main effect for form of data and the interaction between form of data and type of trends were statistically significant beyond the .01 level. All subjects were able to accurately analyze upward trends in performance, but data were required to accurately analyze level and downward trends. These results are interpreted as support for the practice of training teachers to collect and analyze student performance data. [ABSTRACT FROM AUTHOR]

Introduces the articles about person-fit analysis, published in the January 1996 issue of the periodical 'Applied Measurement in Education.' General introduction to person-fit analysis; Conceptual and methodological topics concerning person-fit in the context of typical performance measures.

This article looks at assessment in medical education. In one way or another physicians are involved in the assessment of trainees, peers and other health professionals. The author examines educational assessment tools and emerging methods of assessment such as the assessment of competence and performance, written examinations, supervising clinician assessments, direct observation or video review, clinical simulations, multisource assessments and portfolios. The new domains of assessment include multimethod and longitudinal assessment. The author believes that multiple methods of assessment implemented longitudinally can provide the data needed to assess trainees' learning needs and to remediate suboptimal performance by clinicians.