Search

Item banking involves tests being constructed by selecting from a bank of pre-written questions. There are various examples of multiple-choice tests where item banking is used, but few examples inv...

This paper analyses the psychometric paradigm for understanding educational measurement and standard setting by considering the extent to which that understanding is based on metaphors. I argue tha...

Comparative Judgement (CJ) is an increasingly widely investigated method in assessment for creating a scale, for example of the quality of essays. One area that has attracted attention in C...

This study compared models of assessment structure for achieving differentiation across the range of examinee attainment in the General Certificate of Secondary Education (GCSE) examination taken by 16-year-olds in England. The focus was on the “adjacent levels” model, where papers are targeted at three specific non-overlapping ranges of grades. Examinees enter for a pair of papers at adjacent levels and receive the highest grade achieved. There is no aggregation of marks across papers. This study used simulation, based on data from a GCSE Mathematics exam, to compare the adjacent levels model with two other tiering models in terms of: (1) suitability of grade boundary locations; (2) score distributions; and (3) reliability. The adjacent levels model led to lower reliability but arguably improved two aspects of validity: the strength of the inference about what examinees with a given grade would know and be able to do; and the removal of the ambiguity about overlapping grades inherent in the curre...

The five chapters that make up the second section of the Handbook are grouped under the heading ‘Content’. At first sight, the fifth of these (Chapter 11) about ‘Setting performance standards on te...

For many years, question choice has been used in some UK public examinations, with students free to choose which questions they answer from a selection (within certain parameters). There has been little published research on choice of exam questions in recent years in the UK. In this article we distinguish different scenarios in which choice arises, explore the arguments for and against using optional questions, and exploit the item level data that has recently become available from on-screen marking of examinations to exemplify methods for investigating the (statistical) comparability of optional questions. We conclude that unless there is a very good reason for allowing question choice it should be avoided.

This research investigated the accuracy (agreement with the original marking and grading) of examiners’ holistic judgements of the quality of examination scripts that were close together in overall mark. For a History and a Physics exam, examiners considered pairs of scripts (with marks removed) and made three types of judgement: (1) Absolute – which grade each script was worth; (2) Relative – which of the pair was better in terms of overall quality; (3) Confidence – how confident they were about judgements (1) and (2). In both subjects, relative judgements were more accurate than absolute judgements, and judgements rated as ‘very confident’ were more accurate than other judgements. In Physics, the further apart the two scripts in terms of overall mark the greater was the likelihood of a correct relative judgement, but in History this expected pattern was not found. Despite differences between the research setting and the use of expert judgement in grading the live examinations, these results suggest that...

This paper discusses the issues involved in calculating indices of composite reliability for ‘modular’ or ‘unitised’ assessments of the kind used in GCSEs, AS and A level examinations in England. The increasingly widespread use of on-screen marking has meant that the item-level data required for calculating indices of reliability is now routinely available for most (but not all) units of unitised assessments. Whilst it is relatively straightforward to obtain indices of reliability at unit level, it is far more complex to obtain indices at overall assessment level because of problems created by: (i) the number of different possible ‘routes’ to the final assessment; (ii) the different knowledge, skills and understanding assessed in different units; (iii) the wide variety in item type and size within and across units; (iv) the fact that the item-level data required for calculating reliability indices is not available (or does not exist) for certain units and (v) the different intended weighting of different ...

In this paper we develop an evidence-based framework for considering many of the factors affecting marker agreement in GCSEs and A levels. A logical analysis of the demands of the marking task sugg...

The rank‐ordering method for standard maintaining was designed for the purpose of mapping a known cut‐score (e.g. a grade boundary mark) on one test to an equivalent point on the test score scale of another test, using holistic expert judgements about the quality of exemplars of examinees’ work (scripts). It is a novel application of an old technique (Thurstone’s paired comparison method for scaling psychological stimuli), and one that can be applied when the more familiar methods of statistical equating or item banking are not possible. How should a method like this be evaluated? If the correct mapping were known, then the outcome of a rank‐ordering exercise could be compared against that. However, in the contexts for which the method was designed, there is no ‘right answer’. This paper presents an evaluation of the rank‐ordering method in terms of its rationale, its psychological validity and the stability of the outcome when various factors incidental to the method are varied (e.g. the number of judges...

Background: A recent article published in Educational Research on the reliability of results in National Curriculum testing in England (Newton, The reliability of results from national curriculum testing in England, Educational Research 51, no. 2: 181–212, 2009) suggested that: (1) classification accuracy can be calculated from classification consistency; and (2) classification accuracy on a single test administration is higher than classification consistency across two tests. Purpose: This article shows that it is not possible to calculate classification accuracy from classification consistency. It then shows that, given reasonable assumptions about the distribution of measurement error, the expected classification accuracy on a single test administration is higher than the expected classification consistency across two tests only in the case of a pass–fail test, but not necessarily for tests that classify test-takers into more than two categories. Main argument and conclusion: Classification accuracy is...

A new judgemental method of equating raw scores on two tests, based on rank‐ordering scripts from both tests, has been developed by Bramley. The rank‐ordering method has potential application as a judgemental standard‐maintaining mechanism, because given a mark on one test (e.g. the A grade boundary mark), the equivalent mark (i.e. at the same judgemental standard) on the other test can be determined. If the two tests come from different years then the standard from the earlier year can be applied to the later year. The current standard maintaining method used by Awarding Bodies in England, Wales and Northern Ireland, is the ‘awarding meeting’. Here expert judgement takes place within the context of a variety of statistical information, including score distributions and hence likely pass rates. The rank‐ordering method, in contrast, involves harnessing expert judgement independently of any statistical information. The aim of this study was to investigate the extent to which the outcome of an awarding meet...

In setting the cut-scores on National Curriculum tests it is important to maintain standards. In the process of test development, both within and across years, changes are made to the style of the questions in order to increase their ‘accessibility’. This raises the question of whether a more accessible test should have higher cut-scores. Purely statistical definitions of equating are blind to differences between ‘accessibility’ and ‘easiness’ and cut-scores derived from statistical equating methods will be higher for a more accessible test. Arguments about the increased validity of the more accessible test are sometimes used to justify not raising the cut-scores as much as would be indicated by statistical methods. These arguments are shown to be equivalent to postulating that changing the accessibility is changing the construct measured by the test. Using a statistical measurement model can provide a rational basis for understanding accessibility and identifying types of question where accessibility iss...

In GCSE mathematics there are many question parts (items) worth 2 or 3 marks. The mark distribution in some of these items is U-shaped - for example in a 2-mark item where the majority of candidates score either 2 or 0. Such items are effectively single-mark items being given double weight. This contributes to the spread of raw scores and therefore makes the test appear to be more discriminating than perhaps it is. The source of the problem is in the design of the mark scheme, which has to allocate the marks to fit a tight specification of content coverage and ability targeting. After identifying such items, analysis of their mark schemes showed that the most common causes are awarding an intermediate mark for correct working, and awarding marks for repetition of a single process. The items were re-scored to remove under-used middle categories. Several comparisons with the original scores, including a simulation of test scores, showed that the amount of measurement error was reduced and hence the quality ...

This paper describes a new method of comparing the raw mark scales on two tests using expert judgment. The two tests do not need to have any common items, nor to be taken by common groups of candidates. This study used scripts (i.e., the complete work of a candidate on the test) from England's National Curriculum Test for Reading at Key Stage 3 (14-year olds) in 2003 and 2004. Each member of a panel of 12 experts was given four packs each containing ten scripts--five scripts from each year's test. Marks and annotations from these scripts had been removed. Their task was to put the ten scripts into a single rank order, based on a holistic judgment of the level of performance exhibited in each. Because the design of the study linked scripts across judges and packs it was possible to construct a single latent trait of judged quality of performance. This was done using two different analytical methods: the Rasch formulation of Thurstone paired comparisons, and the Rasch Partial Credit model. Relating the two raw mark scales to the single latent scale allowed the two years' tests to be equated. The merits of using this standard-maintaining method as opposed to a standard-setting method in this particular context are discussed.