Search

This study investigated the effect on student scores of using anchor sets that differed in dimensionality in item response theory (IRT) scaled tests. Real data from a mathematics achievement test that had been documented to have dimensions aligned with item format were used. Item responses were available from a representative sample of approximately 2,600 fifth graders taking a mathematics test from a large state assessment. The use of anchor set items that differed in the degree their items loaded on each of the two salient factors resulted in varying amounts of equating effort. The equatings using anchors that contained items loading more heavily on the first or the second dimension had standard errors (square root of total error) of 6.46 and 3.44, representing 13% and 7% respectively of the approximate 50 scale score standard deviation. The standard error of equating anchors with items that were balanced in terms of their loadings on the two dimensions was 2.72, 5%, of a standard deviation. Bias constituted a substantial portion of the total error of the equatings using the unbalanced F1 and F2 anchor sets. (Contains 7 tables, 6 figures, and 17 references.) (SLD)

The purpose of this research was to study the effect of the three different ways of increasing the number of points contributed by constructed response (CR) items on the reliability of test scores from mixed-item-format tests. The assumption of unidimensionality that underlies the accuracy of item response theory model-based standard error predictions of reliability was initially evaluated for these tests. Large samples of students who had taken mixed-format field tests in mathematics at grades 5 and 8 and writing at grades 3 and 8 were available from a state criterion-referenced testing program. The selection of subsets of items from test-blueprint-representative forms of similar content and difficulty permitted an evaluation of the effects of weighting CR items on total test scores relative to criterion scores of putatively greater generalizability. As expected, there was a cost in terms of precision of having fewer, though weighted, CR items across a wide range of ability. The increment in standard error attributed to weighting was predictably less in the middle of the scale where the forms were targeted. The magnitude of the increase in error and the particular portion of the scale where it occurs are determined by the locations and amount of information contributed by the deleted CR items relative to those that are retained. Implications of different approaches to weighting are discussed. (Contains 5 tables, 10 figures, and 10 references.) (SLD)

This study investigated the practice of weighting a type of test item, such as constructed response, more than other types of items, such as selected response, to compute student scores for a mixed-item type of test. The study used data from statewide writing field tests in grades 3, 5, and 8 and considered two contexts, that in which a single extended response writing prompt is "intentionally" or "purposefully" weighted twice in computing student scores (ER context) and one in which a set of constructed response items, including one extended response item, is intentionally weighted twice (CR context). The weighting option was compared with the use of no weighting. In either context, the criterion for the two options (weighting and no weighting with a shorter form) was the administration of twice as many items as the items that are deliberately weighted, combined with no use of weighting. The three options were compared in terms of student scores as well as raw-score-to-scale score conversion tables. The state uses number-correct scoring as opposed to pattern scoring. Either intentionally weighted or un-weighted scores on the shorter form are, on average, very comparable to the criterion un-weighted scores from the longer form. On the level of individual student scores, as compared with the un-weighted shorter-form scores, more of the weighted shorter-form scores (2-5% more in the ER context and 1-9% more in the CR context) differ from the target scores by more than 10 points. The ramifications of the small decreases in individual score accuracy associated with purposeful weighting would depend on the purposes for which the scores are used and other factors. However, it is clear that purposeful weighting can never compensate for the loss of score accuracy caused by the shorter length of an actual test taken. Two appendixes contain a discussion of the item response models used in the study and a sample graph for one test item. (Contains 13 tables, 24 figures, and 12 references.) (Author/SLD)

This study was designed to investigate several scoring modalities in terms of the accuracy of scores and the efficiency of scoring procedures. Five scoring modalities (SM) are conceptualized by considering tasks and raters as sources of error where "p,""t," and "r" represent person, task, and rater, respectively. These are: (1) SM1 [p x t x r]; (2) SM2 [p x (t:r)]; (3) SM3 [p x (r:t)]; (4) SM4 [(p:r) x t]; and (5) SM5 [r:(p x t)]. SMI and SM5 seem to be somewhat unrealistic for practical use in large-scale performance assessments. SM2 would be least preferable among the five scoring modalities conceptualized in this study for large-scale performance assessments. SM3 could be the best option when a high level of score accuracy is required and a high level of rater variation might be expected. If considering both score accuracy and efficiency of scoring procedures, SM4 should be considered for use in large-scale performance assessments. (Contains 2 figures, 3 tables, and 10 references.) (Author/SLD)

A study was conducted to evaluate the effect of different modes (modalities) of assigning raters to test items. The impact on total constructed response (c.r.) score, and subsequently on total test score, of assigning a single versus multiple raters to an examination reading of a student's set of c.r. responses was evaluated for several mixed-item format tests. Samples of approximately 2,000 students were obtained from a state mathematics field test at each of grades 5, 8, and 10 and from a reading field test at each of grades 4, 8, and 10. Item responses for c.r. items for each selected student in the six samples were allocated to raters three different ways: (1) single rater reading of all responses (SM1); (2) assignment of each c.r. response to a different rater (SM2); and (3) splitting the c.r. items into thirds, with a different rater for each portion (SM3). SSM1 readings produced average total c.r. scores that were greater than the average total of c.r. scores produced in the SM2 condition. Average total c.r. scores when students' responses were allocated to three different raters (SM3) were similar to those of the SM2 condition. Results suggest that for tests with relatively large numbers of c.r. items, the use of as few as three raters to score a student's examination could produce scores that were similar in magnitude and scale to those obtained by assigning a different rater to each item. (Contains 7 tables and 11 references.) (SLD)

A common procedure for obtaining multiple readings (ratings) for a constructed response item, especially in high-stakes tests, is to have two readers read the papers independently, with a third reading if the results differ by more than one point. This necessitates a scoring rule that specifies how the ratings will be aggregated into a single item score. Two plausible scoring rules involve averaging the readings and rounding either to the nearest half point or the nearest integer, but it is not known which results in a greater precision of measurement. This study investigated the precision and accuracy of ability estimates obtained under the two scoring rules for mixed format tests calibrated under an item response theory model. Eleventh-grade reading, mathematics, and science test results and a fifth-grade mathematics test result were analyzed, with more than 1,200 students available for each form. There was little substantive difference in score information or the standard errors of ability estimates due to the type of rounding (integer versus half point), above the floors of three of the four tests, but in the fourth (11th grade reading) there was less error in the integer-rounded ability estimates at the lower portion of the scale. Integer-rounded estimates generally produce slightly larger predicted percent of maximum (test) scores, though not throughout the entire ability range of all the four tests studied. The expected larger positive differences or rounding bias for number correct estimates were observed. Within-subject differences between scale score estimates derived using integer versus half-point scores were generally small for both pattern and number correct ability estimates. The lack of substantive improvement in measurement precision that could be attributed to half-point rounding, coupled with the documented instance of increased error induced by that type of rounding in a portion of the ability range of students taking one test, would seem to argue for rounding average ratings to the nearest integer. Rounding up gives the preponderance of students the benefit of the doubt concerning the acceptability of their responses. (Contains two tables, four figures, and eight references.) (SLD)

Whether the presence of bidimensionality has any effect on the adaptive recalibration of test items was studied through live-data simulation of computer adaptive testing (CAT) forms. The source data were examinee responses to the 298 scored multiple choice items of a licensure examination in a health care profession. Three 75-item part-forms, differing in the degree of bidimensionality, were constructed from the examination forms. The dimensionality of each item was determined based on principal factor loadings using W. F. Stout's (1987) procedure. Samples of 2,100 examinees from an U.S.-educated group and a predominantly foreign-educated group were used, with samples sizes of 100, 200, 400, and 1,000 used for the analyses. As expected, an increase in sample size resulted in greater agreement between calibration and bank b-values. It appeared that dimensionality affected adaptive recalibration. Results from this study varied depending on the dimensionality of the part-forms. Results also affirm the importance of a well-defined reference group for recalibration, since samples from varying ability ranges sometimes produced considerably different b-values. Three tables present details of the analyses. (Contains six references.) (SLD)

Responses to previously calibrated items administered in a computerized adaptive testing (CAT) mode may be used to recalibrate the items. This live-data simulation study investigated the possibility, and limitations, of on-line adaptive recalibration of precalibrated items. Responses to items of a Rasch-based paper-and-pencil licensure examination were used to simulate CAT and paper-and-pencil administrations, defining CAT forms of varying difficulty levels and samples of various sizes. Forms were calibrated and new b-values were compared with the bank b-values obtained from responses of the reference group taking the paper-and-pencil examination. Results indicate that bank b-values were not well replicated when difficult items were calibrated using responses from able examinees and easy items were calibrated using responses from less able examinees. On-line adaptive recalibration as simulated in this study has limitations. However, a "modified" on-line adaptive recalibration may still be a possibility as long as: (1) a reasonably large CAT recalibration sample is predefined from the reference group; (2) the sample has a mean ability similar to that of the reference group; and (3) items to be recalibrated together are relatively heterogeneous in Rasch difficulty. Eight tables are included. (Contains 3 references.) (Author/SLD)

The purpose of this study was to investigate whether systematic, non-zero differences between pairs of item bank b-values have occurred in the recent history of two licensure examinations. Licensing examinations were studied for two related health care professions (Program 1 and Program 2). A series of analysis of covariance models was fit to the data in order to investigate the magnitude of changes in item bank b-values and the relationship of any changes to variables indexing factors that have been documented to affect the stability of item parameters. For Program 1, one 300-item form was used, and for Program 2, two 203-item forms were used to obtain a larger number of items. Analyses were first performed to evaluate the relationship, if any, between the dependent variable and the covariates. For both examinations, b-value differences across pairs of item administrations were not influenced by the changing position of the item in different forms. There were systematic changes as a function of time elapsed from a baseline, as discussed in the paper. Item parameter or scale drift was noted for one examination, while bank or pool drift was found for both. Factors that moderated bank drift, and differences between actual form cutscores and cutscores adjusted for documented bank and scale drift ranged from one to five points for the two examinations. Nine tables and five figures present details of the analyses. (SLD)

A part-form methodology was used to study the effect of varying degrees of multidimensionality on the consistency of pass/fail classification decisions obtained from simulated unidimensional item response theory (IRT) based licensure examinations. A control on the degree of form multidimensionality permitted an assessment throughout the range of multidimensionality of any potential effect on Rasch item parameters and pass/fail classifications obtained from scores derived from them. Four full-length (300-item) forms of a licensure examination produced by CTB Macmillan/McGraw Hill were used to generate part-forms for four administrations in the summer of 1988, winter of 1989, summer of 1989, and winter of 1990, respectively. There were 2,000 examinees for each form. All four full-length forms had been demonstrated to be multidimensional, but could be made unidimensional by deleting no more than half the items with the largest absolute loadings on the second factor. Overall, failure concordance percentages did not differ between those pairs of part-forms that differed maximally in the degree of predicted multidimensionality and those pairs of part-forms where members were both predicted to be multidimensional. Results suggest that increased multidimensionality had no substantial effect on failure decision agreement. However, the failure concordance percentages for pairs of part-forms that were both most likely unidimensional were slightly higher than those for other pairs of part-forms. Four tables, 1 figure of study data and 12 references are included. (SLD)

The sources of multidimensionality found in several different forms of a licensure examination were studied. The relationship between one source of multidimensionality, differential item functioning (DIF) (or factors producing DIF), and content characteristics was explored in an attempt to isolate aspects of training or curriculum that could accound for the causes of multidimensionality in real data. A non-parametric approach for assessing unidimensionality developed by W. Stout (1987) was used to evaluate the dimensionality of several forms of a 300-item Rasch-based test used to license professionals in a health care profession. Quasi-random samples of 2,000 first-time candidates were selected for each of the 4 forms. The four forms were demonstrated to be multidimensional. The source of the multidimensionality could not be attributed to the presence of a large number of passage-like cases with associated multiple items. A second dimension was identified as the source of the multidimensionality through the magnitude of eigenvalue differences and the successful construction of part-forms made unidimensional by removal of items loading heavily on the second factor. An association between the second factor and items flagged for DIF was demonstrated. Items that loaded heavily on the second factor and were often flagged for DIF spanned content that involved knowledge and recall of physiological needs versus an evaluation/analysis of psychosocial needs. Six tables present study data, and there is a 16-item list of references. (SLD)

To investigate the psychometric feasibility of replacing a paper-and-pencil licensing examination with a computer-administered test, a validity study was conducted. The computer-administered test (Cadm) was a common set of items for all test takers, distinct from computerized adaptive testing, in which test takers receive items appropriate to their estimated abilities. The Cadm version, scheduled for implementation in 1990, would consist of 230 items. The validation study was designed to use the capabilities of item response theory (IRT) to produce shorter test forms of a specified reliability relative to the original test or the proposed version. Four combinations of paper-and-pencil and Cadm tests were administered to 418 licensure candidates, each of whom were administered 150 items each in a paper-and-pencil mode and in a Cadm mode. No effects of the mode of administration were found, and there were no signs of significant differences in overall performance across the administration modes. Assessment of multiple item forms and candidate samples reinforced the conclusion that there was no significant effect of computer administration on candidate performance. Eleven tables and eight figures present study results. (SLD)

The results of classifying test items on the basis of their Mantel-Haenszel (MH) alpha estimates were compared to the results of classifying these items using an item response theory (IRT) based procedure involving the comparison of item difficulties in the interest of identifying the alpha value that maximized the decision concordance between the two methods. The data consisted of candidates' responses to 299 scored items on an examination for professional licensure. A total of 68,458 candidates took this examination in 1988. The candidates' ethnicity was determined by their self-classifications into one of seven categories. A total of 47,573 candidates classified themselves in Ethnic Group 1; 6,486 in Ethnic Group 2; 5,466 in Ethnic Group 3; 2,004 in Ethnic Group 4; 1,014 in Ethnic Group 5; 486 in Ethnic Group 6; 307 in Ethnic Group 7; 726 as "other"; and the remaining 4,396 candidates did not specify their ethnicity. All candidates except the 4,396 who did not classify themselves were used for the MH analyses; for the IRT analyses, random samples of candidates were drawn from the four largest ethnic groups, while all candidates from the smaller groups were used. Candidates were divided into score groups. In both analyses, the majority ethnic group became the reference group and the other ethnic groups were designated focal. The MH estimate of alpha indicated that no differential item functioning was detected. Statistics resulting from IRT methods correlated highly with MH alpha values. Using the MH method in conjunction with IRT methods resulted in cutscores with a high level of decision concordance with advantages over traditional methods of using a significance level to establish an alpha criterion. Eight tables and eight figures provide study data. (SLD)