Search

The Rapid Online Assessment of Reading (ROAR) is a web-based lexical decision task that measures single-word reading abilities in children and adults without a proctor. Here we study whether item response theory (IRT) and computerized adaptive testing (CAT) can be used to create a more efficient online measure of word recognition. To construct an item bank, we first analyzed data taken from four groups of students (N = 1960) who differed in age, socioeconomic status, and language-based learning disabilities. The majority of item parameters were highly consistent across groups (r =.78–.94), and six items that functioned differently across groups were removed. Next, we implemented a JavaScript CAT algorithm and conducted a validation experiment with 485 students in grades 1–8 who were randomly assigned to complete trials of all items in the item bank in either (a) a random order or (b) a CAT order. We found that, to achieve reliability of 0.9, CAT improved test efficiency by 40%: 75 CAT items produced the same standard error of measurement as 125 items in a random order. Subsequent validation in 32 public school classrooms showed that an approximately 3-min ROAR-CAT can achieve high correlations (r =.89 for first grade, r =.73 for second grade) with alternative 5–15-min individually proctored oral reading assessments. Our findings suggest that ROAR-CAT is a promising tool for efficiently and accurately measuring single-word reading ability. Furthermore, our development process serves as a model for creating adaptive online assessments that bridge research and practice. [ABSTRACT FROM AUTHOR]

This study aims to investigate the effect of content balancing, which involves equal and different weighting of content areas in dichotomous items in computerized adaptive testing (CAT), on measurement precision under different measurement conditions. Conducted as a simulation study, small sample sizes were set at 250, while large sample sizes comprised 500 individuals. The ability parameters of the individuals forming the sample were generated to display a normal distribution within the range of -3 to +3 for each sample. Using the three-parameter logistic (3PL) item response model, a pool of 750 items spanning five different content areas was developed for dichotomous items. The study considered different sample sizes, ability estimation methods (Maximum Likelihood Estimation and Expected A Posteriori), and termination rules (20 items, 60 items, and SE≤.30) as significant factors in the CAT algorithm for examining the effect of content balancing. For each CAT application, measurement precision was assessed by calculating the root mean square error (RMSE), bias, and fidelity coefficients, and these were analyzed comparatively. The results showed that bias values were close to zero under all conditions. RMSE values were lowest when the test was terminated at 60 items across all conditions, while standard error termination rules and situations where the test terminated at 20 items produced similar values. Considering all conditions, the highest fidelity coefficient was observed when the test terminated at 60 items. The fidelity coefficient did not vary significantly with other variables. Implementing content balancing in conditions using different ability estimation methods increased the average number of items by approximately one item. While the average number of items in the test slightly increased with content balancing, measurement precision was maintained. Overall, the maximum item exposure rate decreased with content balancing when content areas were weighted equally, whereas it increased when they were weighted disproportionately. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) aims to present items that statistically optimize the assessment process by considering the examinee's responses and estimated trait levels. Recent developments in reinforcement learning and deep neural networks provide CAT with the potential to select items that utilize more information across all the items on the remaining tests, rather than just focusing on the next several items to be selected. In this study, we reformulate CAT under the reinforcement learning framework and propose a new item selection strategy based on the deep Q-network (DQN) method. Through simulated and empirical studies, we demonstrate how to monitor the training process to obtain the optimal Q-networks, and we compare the accuracy of the DQN-based item selection strategy with that of five traditional strategies—maximum Fisher information, Fisher information weighted by likelihood, Kullback‒Leibler information weighted by likelihood, maximum posterior weighted information, and maximum expected information—on both simulated and real item banks and responses. We further investigate how sample size and the distribution of the trait levels of the examinees used in training affect DQN performance. The results show that DQN achieves lower RMSE and MAE values than traditional strategies under simulated and real banks and responses in most conditions. Suggestions for the use of DQN-based strategies are provided, as well as their code. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) is a widely embraced approach for delivering personalized educational assessments, tailoring each test to the real‐time performance of individual examinees. Despite its potential advantages, CAT�s application in small‐scale assessments has been limited due to the complexities associated with calibrating the item bank using sparse response data and small sample sizes. This study addresses these challenges by developing a two‐step item bank calibration strategy that leverages the 1‐bit matrix completion method in conjunction with two distinct incomplete pretesting designs. We introduce two novel 1‐bit matrix completion‐based imputation methods specifically designed to tackle the issues associated with item calibration in the presence of sparse response data and limited sample sizes. To demonstrate the effectiveness of these approaches, we conduct a comparative assessment against several established item parameter estimation methods capable of handling missing data. This evaluation is carried out through two sets of simulation studies, each featuring different pretesting designs, item bank structures, and sample sizes. Furthermore, we illustrate the practical application of the methods investigated, using empirical data collected from small‐scale assessments. [ABSTRACT FROM AUTHOR]

Accumulating evidence supports the presence of a general psychopathology dimension, the p factor ('p'). Despite growing interest in the p factor, questions remain about how p is assessed. Although multi-informant assessment of psychopathology is commonplace in clinical research and practice with children and adolescents, almost no research has taken a multi-informant approach to studying youth p or has examined the degree of concordance between parent and youth reports. Further, estimating p requires assessment of a large number of symptoms, resulting in high reporter burden that may not be feasible in many clinical and research settings. In the present study, we used bifactor multidimensional item response theory models to estimate parent- and adolescent-reported p in a large community sample of youth (11–17 years) and parents (N = 5,060 dyads). We examined agreement between parent and youth p scores and associations with assessor-rated youth global functioning. We also applied computerized adaptive testing (CAT) simulations to parent and youth reports to determine whether adaptive testing substantially alters agreement on p or associations with youth global functioning. Parent-youth agreement on p was moderate (r =.44) and both reports were negatively associated with youth global functioning. Notably, 7 out of 10 of the highest loading items were common across reporters. CAT reduced the average number of items administered by 57%. Agreement between CAT-derived p scores was similar to the full form (r =.40) and CAT scores were negatively correlated with youth functioning. These novel results highlight the promise and potential clinical utility of a multi-informant p factor approach. [ABSTRACT FROM AUTHOR]

Item position effect (IPE) refers to situations where an item performs differently when it is administered in different positions on a test. The majority of previous research studies have focused on investigating IPE under linear testing. There is a lack of IPE research under adaptive testing. In addition, the existence of IPE might violate Item Response Theory (IRT)’s item parameter invariance assumption, which facilitates applications of IRT in various psychometric tasks such as computerized adaptive testing (CAT). Ignoring IPE might lead to issues such as inaccurate ability estimation in CAT. This article extends research on IPE by proposing and evaluating approaches to controlling position effects under an item‐level computerized adaptive test via a simulation study. The results show that adjusting IPE via a pretesting design (approach 3) or a pool design (approach 4) results in better ability estimation accuracy compared to no adjustment (baseline approach) and item‐level adjustment (approach 2). Practical implications of each approach as well as future research directions are discussed as well. [ABSTRACT FROM AUTHOR]

The given paper discusses an original method of the evaluation of outcomes of adaptive testing in the case of the strategy of the multilevel testing. The multiplicity/set of the outcomes of testing consists of atypical different-dimensional elements. The given paper defines the criteria of their comparison, describes the principles of ordering of the given multiplicity and draws the getting of a final score. The paper presented considers the ordering method of outcome set for multi-stage testing (MST) of 1-3-3 model. The ordering method of outcome set is used for the estimation of results of computerized adaptive testing (CAT). This method is not tied to a specific testing procedure. Acknowledgment of this is its usage for the 1-3-3 model, which is described in the paper. To sort the set of testing outcomes the function-criteria described in the initial article are used here and a comparative analysis of obtained results is performed. The criterion of ordering of the outcomes of testing may not be the only option. The given paper illustrates this fact through a comparative discussion of two samples. An original procedure of testing is used for the presentation of the essence of the method. The given procedure is aimed to be illustrative, because a described method of assessment can be used for similar strategies. The ordered outcome set is estimated by a hundred-point system according to the normal distribution. Applied results of our scientific research is developed as "Adaptester" portal and available on the following address: https://adaptester.com. [ABSTRACT FROM AUTHOR]

Background: Computerized adaptive testing (CAT) is an effective way to reduce time, repetitious redundancy, and respond burden, and has been used to measure outcomes in many diseases. This study aimed to develop and validate a comprehensive disease-specific CAT for chronic obstructive pulmonary disease (COPD) patient-reported outcome measurement. Methods: The discrimination and difficulty of the items from the modified patient-reported outcome scale for COPD (mCOPD-PRO) were analyzed using item response theory. Then the initial item, item selection method, ability estimation method, and stopping criteria were further set based on Concerto platform to form the CAT. Finally, the reliability and validity were validated. Results: The item discrimination ranged from 1.05 to 2.71, and the item difficulty ranged from − 3.08 to 3.65. The measurement reliability of the CAT ranged from 0.910 to 0.922 using random method, while that ranged from 0.910 to 0.924 using maximum Fisher information (MFI) method. The content validity was good. The correlation coefficient between theta of the CAT and COPD assessment test and modified Medical Research Council dyspnea scale scores using random method was 0.628 and 0.540 (P < 0.001; P < 0.001) respectively, while that using MFI method was 0.347 and 0.328 (P = 0.007; P = 0.010) respectively. About 11 items (reducing by 59.3%) on average were tested using random method, while about seven items (reducing by 74.1%) on average using MFI method. The correlation coefficient between theta of the CAT and mCOPD-PRO total scores using random method was 0.919 (P < 0.001), while that using MFI method was 0.760 (P < 0.001). Conclusions: The comprehensive disease-specific CAT for COPD patient-reported outcome measurement is well developed with good psychometric properties, which can provide an efficient, accurate, and user-friendly measurement for patient-reported outcome of COPD. [ABSTRACT FROM AUTHOR]

With the development of information technology and the widespread application of computerized adaptive testing (CAT), response time data during the testing process has become increasingly accessible. It's application in the field of psychological measurement has also become more widespread. Applying response time data and models to automated test assembly (ATA) holds significant importance for improving testing efficiency, precision of ability estimation for examinees during testing, and promoting educational equity. This paper systematically evaluates the test assembly methods and research results in CAT, comprehensively presenting test assembly methods that incorporate response time data. It explored how such methods address issues related to test duration constraints, testing efficiency, test bank security, and differences in examinees' response times. Further research on the ATA methods that incorporate response time data, as well as the application of research findings to large-scale assessment projects in China, are important research directions to be explored in the future. [ABSTRACT FROM AUTHOR]

This study aimed to examine the performances of item selection algorithms in terms of measurement accuracy and computational time, using factors such as test length, number of attributes, and item quality in fixed-length CDCAT and average test lengths and computational time, using factors such as number of attributes and item quality in variable-length CD-CAT. In the research, two different simulation studies were conducted for the fixed and variable-length tests. Item responses were generated according to the DINA model. Two item banks, which consisted of 480 items for 5 and 6 attributes, were generated, and the item banks were used for both the fixed and variable-length tests. Q-matrix was generated item by item and attribute by attribute. In the study, 3000 examinees were generated in such a way that each examinee had a 50% chance of achieving each attribute. The cognitive patterns of the examinees were estimated by usingMAP. In the variable-length CD-CAT, the first-highest posterior probability threshold is 0.80, and the second-highest posterior probability threshold is 0.10. The CD-CAT administration and other analyses were conducted using R 3.6.1.At the end of the study in which the fixed-length CD-CAT was used, it was concluded that an increase in the number of attributes resulted in a decrease in the pattern recovery rates of item selection algorithms. Conversely, these rates improved with higher item quality and longer test lengths. The highest values in terms of pattern recovery rate were obtained from JSD and MPWKL algorithms. In the variable-length CD-CAT, it was concluded that the average test length increased with the number of attributes and decreased with higher item quality. Across all conditions, the JSD algorithm yielded the shortest average test length. Additionally, It has been determined that GDI algorithm had the shortest computation time in all scenarios, whereas the MPWKL algorithm exhibited the longest computation time. [ABSTRACT FROM AUTHOR]

Computerized adaptive tests (CATs) are highly efficient assessment tools that couple low patient and clinician time burden with high diagnostic accuracy. A CAT for substance use disorders (CAT-SUD-E) has been validated in adult populations but has yet to be tested in adolescents. The purpose of this study was to perform initial evaluation of the K-CAT-SUD-E (i.e., Kiddy-CAT-SUD-E) in an adolescent sample compared to a gold-standard diagnostic interview. Adolescents (N = 156; aged 11–17) with diverse substance use histories completed the K-CAT-SUD-E electronically and the substance related disorders portion of a clinician-conducted diagnostic interview (K-SADS) via tele-videoconferencing platform. The K-CAT-SUD-E assessed both current and lifetime overall SUD and substance-specific diagnoses for nine substance classes. Using the K-CAT-SUD-E continuous severity score and diagnoses to predict the presence of any K-SADS SUD diagnosis, the classification accuracy ranged from excellent for current SUD (AUC = 0.89, 95% CI = 0.81, 0.95) to outstanding (AUC = 0.93, 95% CI = 0.82, 0.97) for lifetime SUD. Regarding current substance-specific diagnoses, the classification accuracy was excellent for alcohol (AUC = 0.82), cannabis (AUC = 0.83) and nicotine/tobacco (AUC = 0.90). For lifetime substance-specific diagnoses, the classification accuracy ranged from excellent (e.g., opioids, AUC = 0.84) to outstanding (e.g., stimulants, AUC = 0.96). K-CAT-SUD-E median completion time was 4 min 22 s compared to 45 min for the K-SADS. This study provides initial support for the K-CAT-SUD-E as a feasible accurate diagnostic tool for assessing SUDs in adolescents. Future studies should further validate the K-CAT-SUD-E in a larger sample of adolescents and examine its acceptability, feasibility, and scalability in youth-serving settings. [ABSTRACT FROM AUTHOR]

The computerized adaptive form of cognitive diagnostic testing, CD-CAT, has gained increasing attention in the domain of personalized measurements for its ability to categorize individual mastery status of fine-grained attributes more accurately and efficiently through administering items tailored to one's ability progressively. How to select the next item based on previous response(s) is crucial for the success of CD-CAT. Previous item selection strategies for CD-CAT have often followed a greedy or semi-greedy approach, which makes it difficult to strike a balance between diagnostic performance and item bank utilization. To address this issue, this study takes a graph perspective and transforms the item selection problem in CD-CAT into a path-searching problem, in which paths refer to possible test construction and nodes refer to individual items. A heuristic function is defined to predict the prospect of a path, indicating how well the corresponding test can diagnose the current examinee. Two search mechanisms with different biases towards item exposure control are proposed to approximate the optimal path with the best prospect. The first unused item on the resulting path is selected as the next item. The above components compose a novel CD-CAT item selection framework based on heuristic search. Simulation studies are conducted under a variety of conditions regarding bank designs, bank-quality conditions, and testing scenarios. The results are compared with different types of classic item selection strategies in CD-CAT, showing that the proposed framework can enhance bank utilization at a smaller cost of diagnostic performance. [ABSTRACT FROM AUTHOR]

Questionnaires to detect emotional and behavioral (EB) problems in preventive child healthcare (PCH) should be short; this potentially affects their validity and reliability. Computerized adaptive testing (CAT) could overcome this weakness. The aim of this study was to (1) develop a CAT to measure EB problems among pre-school children and (2) assess the efficiency and validity of this CAT. We used a Dutch national dataset obtained from parents of pre-school children undergoing a well-child care assessment by PCH (n = 2192, response 70%). Data regarded 197 items on EB problems, based on four questionnaires, the Strengths and Difficulties Questionnaire (SDQ), the Child Behavior Checklist (CBCL), the Ages and Stages Questionnaire: Social Emotional (ASQ:SE), and the Brief Infant–Toddler Social and Emotional Assessment (BITSEA). Using 80% of the sample, we calculated item parameters necessary for a CAT and defined a cutoff for EB problems. With the remaining part of the sample, we used simulation techniques to determine the validity and efficiency of this CAT, using as criterion a total clinical score on the CBCL. Item criteria were met by 193 items. This CAT needed, on average, 16 items to identify children with EB problems. Sensitivity and specificity compared to a clinical score on the CBCL were 0.89 and 0.91, respectively, for total problems; 0.80 and 0.93 for emotional problems; and 0.94 and 0.91 for behavioral problems. Conclusion: A CAT is very promising for the identification of EB problems in pre-school children, as it seems to yield an efficient, yet high-quality identification. This conclusion should be confirmed by real-life administration of this CAT. What is Known: • Studies indicate the validity of using computerized adaptive test (CAT) applications to identify emotional and behavioral problems in school-aged children. • Evidence is as yet limited on whether CAT applications can also be used with pre-school children. What is New: • The results of this study show that a computerized adaptive test is very promising for the identification of emotional and behavior problems in pre-school children, as it appears to yield an efficient and high-quality identification. [ABSTRACT FROM AUTHOR]

The article presents adaptive testing strategies for polytomously scored technology-enhanced innovative items. We investigate item selection methods that match examinee's ability levels in location and explore ways to leverage test-taking speeds during item selection. Existing approaches to selecting polytomous items are mostly based on information measures and tend to experience an item pool usage problem. In this study, we introduce location indices for polytomous items and show that location-matched item selection significantly improves the usage problem and achieves more diverse item sampling. We also contemplate matching items' time intensities so that testing times can be regulated across the examinees. Numerical experiment from Monte Carlo simulation suggests that location-matched item selection achieves significantly better and more balanced item pool usage. Leveraging working speed in item selection distinctly reduced the average testing times as well as variation across the examinees. Both the procedures incurred marginal measurement cost (e.g., precision and efficiency) and yet showed significant improvement in the administrative outcomes. The experiment in two test settings also suggested that the procedures can lead to different administrative gains depending on the test design. [ABSTRACT FROM AUTHOR]

In computerized adaptive testing, overexposure of items in the bank is a serious problem and might result in item compromise. We develop an item selection algorithm that utilizes the entire bank well and reduces the overexposure of items. The algorithm is based on collaborative filtering and selects an item in two stages. In the first stage, a set of candidate items whose expected performance matches the examinee's current performance is selected. In the second stage, an item that is approximately matched to the examinee's observed performance is selected from the candidate set. The expected performance of an examinee on an item is predicted by autoencoders. Experiment results show that the proposed algorithm outperforms existing item selection algorithms in terms of item exposure while incurring only a small loss in measurement precision. [ABSTRACT FROM AUTHOR]

Background: Cancer patients may experience a decrease in cognitive functioning before, during and after cancer treatment. So far, the Quality of Life Group of the European Organisation for Research and Treatment of Cancer (EORTC QLG) developed an item bank to assess self-reported memory and attention within a single, cognitive functioning scale (CF) using computerized adaptive testing (EORTC CAT Core CF item bank). However, the distinction between different cognitive functions might be important to assess the patients' functional status appropriately and to determine treatment impact. To allow for such assessment, the aim of this study was to develop and psychometrically evaluate separate item banks for memory and attention based on the EORTC CAT Core CF item bank. Methods: In a multistep process including an expert-based content analysis, we assigned 44 items from the EORTC CAT Core CF item bank to the memory or attention domain. Then, we conducted psychometric analyses based on a sample used within the development of the EORTC CAT Core CF item bank. The sample consisted of 1030 cancer patients from Denmark, France, Poland, and the United Kingdom. We evaluated measurement properties of the newly developed item banks using confirmatory factor analysis (CFA) and item response theory model calibration. Results: Item assignment resulted in 31 memory and 13 attention items. Conducted CFAs suggested good fit to a 1-factor model for each domain and no violations of monotonicity or indications of differential item functioning. Evaluation of CATs for both memory and attention confirmed well-functioning item banks with increased power/reduced sample size requirements (for CATs ≥ 4 items and up to 40% reduction in sample size requirements in comparison to non-CAT format). Conclusion: Two well-functioning and psychometrically robust item banks for memory and attention were formed from the existing EORTC CAT Core CF item bank. These findings could support further research on self-reported cognitive functioning in cancer patients in clinical trials as well as for real-word-evidence. A more precise assessment of attention and memory deficits in cancer patients will strengthen the evidence on the effects of cancer treatment for different cancer entities, and therefore contribute to shared and informed clinical decision-making. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) can improve test efficiency, but it also causes the problem of unbalanced item usage within a pool. The effect of uneven item exposure rates can not only induce a test security problem due to overexposed items but also raise economic concerns about item pool development due to underexposed items. Therefore, this study proposes a two-stage Sympson–Hetter (TSH) method to enhance balanced item pool utilization by simultaneously controlling the minimum and maximum item exposure rates. The TSH method divides CAT into two stages. While the item exposure rates are controlled above a prespecified level (e.g., r min) in the first stage to increase the exposure rates of the underexposed items, they are controlled below another prespecified level (e.g., r max) in the second stage to prevent items from overexposure. To reduce the effect on trait estimation, TSH only administers a minimum sufficient number of underexposed items that are generally less discriminating in the first stage of CAT. The simulation study results indicate that the TSH method can effectively improve item pool usage without clearly compromising trait estimation precision in most conditions while maintaining the required level of test security. [ABSTRACT FROM AUTHOR]

This study introduces two new statistics for measuring the score comparability of computerized adaptive tests (CATs) based on comparing conditional standard errors of measurement (CSEMs) for examinees that achieved the same scale scores. One statistic is designed to evaluate score comparability of alternate CAT forms for individual scale scores, while the other statistic is designed to evaluate the overall score comparability of alternate CAT forms. The effectiveness of the new statistics is illustrated using data from grade 3 through 8 reading and math CATs. Results suggest that both CATs demonstrated reasonably high levels of score comparability, that score comparability was less at very high or low scores where few students score, and that using random samples with fewer students per grade did not have a big impact on score comparability. Results also suggested that score comparability was sometimes higher when the bottom 20% of scorers were used to calculate overall score comparability compared to all students. Additional discussion related to applying the statistics in different contexts is provided. [ABSTRACT FROM AUTHOR]

Objective The main aims of this study were: (1) to create a patient-reported outcome measure (PROM) item bank for measuring the impact of lower quadrant edema (LQE) on physical function using item response theory and (2) to assess reliability, validity, and administration efficiency of LQE PROM scores based on computerized adaptive test (CAT) and the reliability of a 10-item short form (SF). Methods This retrospective study included data from patients treated in outpatient rehabilitation clinics for lower quadrant edema who responded to all 30 candidate items at intake. Item response theory model assumptions of unidimensionality, local item independence, item fit, and presence of differential item functioning (DIF) were evaluated. LQE-CAT-generated scores were assessed for reliability, validity, and administration efficiency. LQE-SF-generated scores were assessed for reliability. Results The total cohort included 4894 patients (mean [SD] age = 65 [14] years; range = 14–89 years). A set of 20 items was selected for the item bank based on support for its unidimensionality and fit to the item response theory model, with reliability estimates greater than 0.92 for CAT and SF administration modes. No items demonstrated DIF with respect to tested variables. After controlling for scores at intake, scores discriminated among multiple patient groups in clinically logical ways with better outcomes observed for patients who were younger with less chronic symptoms and fewer comorbidities. Scores were responsive to change but the effect size was small (0.4). There were negligible floor and ceiling effects. CAT administration of the item bank required an average of 6.1 items (median = 5). Scores correlated highly with full-bank scores (Pearson correlation coefficient = 0.98). Conclusion Scores on the LQE PROM were reliable, valid, and efficient for assessing perceived physical function of patients with lower quadrant edema. The LQE, CAT, and SF are suitable for research and routine clinical care. Reasons for the small effect size for change scores should be studied. Impact The newly developed LQE PROM was reliable and valid and offered efficient administration modes for assessing perceived physical function of patients with LQE, both for research and routine clinical care in busy outpatient rehabilitation settings. As an item response theory-based measure, the LQE PROM allows administration of condition-specific functional questions with low response burden for patients. The 10-item LQE-SF offers a feasible alternative administration mode when CAT administration is not available. This study supports a transition to PROMs that are based on modern measurement approaches to achieve the combined benefits of high accuracy and efficiency. [ABSTRACT FROM AUTHOR]

In recent years, Computerized Multistage Testing (MST), with their versatile benefits, have found themselves a wide application in large scale assessments and have increased their popularity. The fact that forms can be made ready before the exam application, such as a linear test, and that they can be adapted according to the test taker's ability level, such as computerized adaptive tests, has brought MST to the forefront. It is observed that simulation studies are often used in research on MST. The R programming language used in the conduct of simulation studies is widely used for statistical calculation, data visualization, and Monte Carlo research. Researchers can perform their analysis according to their own research questions, both by writing their own code in R and by using the packages in the library. This study aims to demonstrate the design and implementation of MST simulation examples using the R programming language. In this context, first of all, the basic components of MST were discussed, then R packages written on MST were examined in terms of advantages, disadvantages and analysis facility. Then, three different MST simulation examples were designed with the R programming language. It is considered that this study will be useful to those who are interested in MST. [ABSTRACT FROM AUTHOR]

Purpose: Based on the general formula, which depends on the length and difficulty of the test, the number of respondents and the number of ability levels, this study aims to provide a closed formula for the adaptive tests with medium difficulty (probability of solution is p = 1/2) to determine the accuracy of the parameters for each item and in the case of calibrated items, determine the required test length given number of respondents. Design/methodology/approach: Empirical results have been obtained on computerized or multistage adaptive implementation. Simulation studies and classroom/experimental results show that adaptive tests can measure test subjects' ability to the same quality over half the test length compared to linear versions. Due to the complexity of the problem, the authors discuss a closed mathematical formula: the relationship between the length of the tests, the difficulty of solving the items, the number of respondents and the levels of ability. Findings: The authors present a closed formula that provides a lower bound for the minimum test length in the case of adaptive tests. The authors also present example calculations using the formula, based on the assessment framework of some student assessments to show the similarity between the theoretical calculations and the empirical results. Originality/value: With this formula, we can form a connection between theoretical and simulation results. [ABSTRACT FROM AUTHOR]

The current methods for measuring patient-reported outcomes for amphetamine (speed) craving have limitation ability to adapt to the needs of individual patients while maintaining consistency in their scores. This study aimed to investigate whether the 40-item Desires for Speed Questionnaire (DSQ) could be improved for assessing clinical subjects using computerized adaptive testing (CAT). A sample of 677 participants from four drug addiction treatment centers in China was utilized in the study. Two types of analysis were conducted using the response data. First, the psychometric properties of all items were evaluated to meet the requirements of CAT. Second, multiple CAT simulations were carried out using real response data. The results indicated that the CAT method, which only required a small number of items (50%–75%), produced results that were only slightly different from the full DSQ assessment in terms of measuring amphetamine craving and criterion validity. In conclusion, this study suggests that developing a DSQ CAT for clinical subjects is useful as it leads to more efficient measurement without compromising the reliability of the test outcomes. Public Significance Statement: This study highlights the potential benefits of using computerized adaptive testing (CAT) to improve the efficiency and effectiveness of measuring amphetamine craving in clinical subjects. By reducing the number of items needed while maintaining reliability, this approach could ultimately lead to more personalized and targeted treatments for individuals with drug addiction. [ABSTRACT FROM AUTHOR]

Purpose: To assess the psychometric properties of glaucoma-specific health-related quality of life (HRQoL) item banks (IBs), and explore their efficiency using computerized adaptive testing (CAT) simulations. Methods: In this cross-sectional, clinical study, 300 Asian glaucoma patients answered 221 items within seven IBs: Ocular Comfort Symptoms (OS); Activity Limitation (AL); Lighting (LT); Mobility (MB); Glaucoma Management (GM); Psychosocial (PSY); and Work (WK). Rasch analysis was conducted to assess each IB's psychometric properties (e.g., item "fit" to the construct; unidimensionality) and a set of analytic performance criteria guiding decision making relating to retaining or dropping domains and items was employed. CAT simulations determined the mean number of items for 'high' and 'moderate' measurement precision (stopping rule: SEM 0.3 and 0.387, respectively). Results: Participants' mean age was 67.2 ± 9.2 years (62% male; 87% Chinese). LT, MB, and GM displayed good psychometric properties overall. To optimize AL's psychometric properties, 16 items were deleted due to poor "fit", high missing data, item bias, low discrimination and/or a low clinical/patient importance rating. To resolve multidimensionality in PSY, we rehomed 16 items into a "Concern (CN)" domain. PSY and CN required further amendment, including collapsing of response categories, and removal of poorly functioning items (N = 7). Due to poor measurement precision, low applicability and high ceiling effect, low test information indices, and low item separation index the WK IB was not considered further. In CAT simulations on the final seven IBs (n = 182 items total), an average of 12.1 and 15.7 items per IB were required for moderate and high precision measurement, respectively. Conclusions: After reengineering our seven IBs, they displayed robust psychometric properties and good efficiency in CAT simulations. Once finalized, GlauCAT™-Asian may enable comprehensive assessment of the HRQoL impact of glaucoma and associated treatments. [ABSTRACT FROM AUTHOR]

This paper presents an item selection method in CAT (Computerized Adaptive Testing) based on association rules. According to the correct (1) or incorrect (0) answer of the examinee to each item, association rules are searched with the answered item and its result (1 or 0) as antecedent, and items with greater or lower complexity level than the responded item as a consequent, respectively. The election of the item is performed considering rules with the highest confidence and support. Finally, we include an example to demonstrate the efficacy of the proposed method. [ABSTRACT FROM AUTHOR]

Objective The aims of this study were to calibrate the original 16 items from the Activities-Specific Balance Confidence (ABC) Scale to create an item response theory (IRT)-based item bank and scoring metric of balance confidence (BC) and to assess psychometric properties of a computerized adaptive test (BC-CAT) and 6-item short-form (BC-SF) administration modes. Methods This retrospective study included data from patients who were treated in outpatient rehabilitation clinics and assessed for balance impairments by responding to the full ABC Scale at intake. IRT model assumptions of unidimensionality, local item independence, item fit, and presence of differential item functioning (DIF) were evaluated. BC-CAT-generated scores were assessed for reliability, validity, and administration efficiency, and the newly developed BC-SF was assessed for reliability. Results Total cohort included 20,354 patients (mean age [SD] = 66 [16] years; range = 14–89). All 16 items were retained in the final item bank based on support for unidimensionality and fit to the IRT model. No items demonstrated DIF. Reliability estimates were 0.95, 0.96, and 0.98 for the BC-SF, BC-CAT, and the full item bank, respectively. Scores discriminated among patient groups in clinically logical ways. After controlling for scores at intake, better outcomes were achieved for patients who were younger, had more acute symptoms, exercised more, and had fewer comorbidities. Scores were responsive to change with a moderate effect size, with negligible floor and ceiling effects. CAT scores were generated using an average of 4.7 items (median = 4) and correlated highly with full-bank scores (Pearson correlation coefficient = 0.99). Conclusion The IRT–based BC patient-reported outcome measure (PROM) was reliable, valid, moderately responsive to change, and efficient, with excellent score coverage. The measure is suitable for research and routine clinical administration using the BC-CAT or BC-SF administration modes. The full ABC Scale can be administered for increased clinical content when appropriate. Impact The newly developed BC-PROM was reliable and valid for assessing perceived BC. In addition, the BC-PROM has efficient administration modes with low patient response burden, which enhances feasibility and promotes use during routine clinical practice in busy rehabilitation settings. This study supports a transition to PROMs that are based on modern measurement approaches to achieve the combined benefits of high accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) offers an efficient and highly accurate method for estimating examinees' abilities. In this article, the free version of Concerto Software for CAT was reviewed, dividing our evaluation into three sections: software implementation, the Item Response Theory (IRT) features of CAT, and user experience. Overall, Concerto is an excellent tool for researchers seeking to create computerized adaptive (or non-adaptive) tests, providing a robust platform with comprehensive IRT capabilities and a user-friendly interface. [ABSTRACT FROM AUTHOR]

Although many studies have been carried out on the psychometric aspects of computerized adaptive testing (CAT), its psychological aspects are less researched. Early studies claimed that CAT can be more motivating and induce less anxiety than traditional fixed-item tests (FIT). The purpose of this systematic review and meta-analysis was to gain a comprehensive understanding of the effects of CAT on motivation and anxiety in comparison to traditional fixed-item testing. Seven databases were examined. Articles were eligible if they employed an empirical study containing a direct comparison between CAT and FIT. Meta-analytical results showed no overall effect of test type on anxiety and motivation when comparing CAT with FIT (k = 11, g+ = 0.06, p =.28). However, easier CAT had positive effect compared with FIT (k = 2, g+ =.22, p <.001). Certain modifications in CAT administration can provide positive psychological effects for test-takers. [ABSTRACT FROM AUTHOR]

Sibling items developed through automatic item generation share similar but not identical psychometric properties. However, considering sibling item variations may bring huge computation difficulties and little improvement on scoring. Assuming identical characteristics among siblings, this study explores the impact of item model parameter variations (i.e., within-family variation between siblings) on person parameter estimation in linear tests and Computerized Adaptive Testing (CAT). Specifically, we explore (1) what if small/medium/large within-family variance is ignored, (2) if the effect of larger within-model variance can be compensated by greater test length, (3) if the item model pool properties affect the impact of within-family variance on scoring, and (4) if the issues in (1) and (2) are different in linear vs. adaptive testing. Related sibling model is used for data generation and identical sibling model is assumed for scoring. Manipulated factors include test length, the size of within-model variation, and item model pool characteristics. Results show that as within-family variance increases, the standard error of scores remains at similar levels. For correlations between true and estimated score and RMSE, the effect of the larger within-model variance was compensated by test length. For bias, scores are biased towards the center, and bias was not compensated by test length. Despite the within-family variation is random in current simulations, to yield less biased ability estimates, the item model pool should provide balanced opportunities such that "fake-easy" and "fake-difficult" item instances cancel their effects. The results of CAT are similar to that of linear tests, except for higher efficiency. [ABSTRACT FROM AUTHOR]

The article presents a new computerized adaptive testing (CAT) procedure for use with batteries of unidimensional tests. At each step of testing, the estimate of a certain ability is updated on the basis of the response to the latest administered item and the current estimates of all other abilities measured by the battery. The information deriving from these abilities is incorporated into an empirical prior that is updated each time that new estimates of the abilities are computed. In two simulation studies, the performance of the proposed procedure is compared with that of a standard procedure for CAT with batteries of unidimensional tests. The proposed procedure yields more accurate ability estimates in fixed-length CATs, and a reduction of test length in variable-length CATs. These gains in accuracy and efficiency increase with the correlation between the abilities measured by the batteries. [ABSTRACT FROM AUTHOR]

Several forced-choice (FC) computerized adaptive tests (CATs) have emerged in the field of organizational psychology, all of them employing ideal-point items. However, despite most items developed historically follow dominance response models, research on FC CAT using dominance items is limited. Existing research is heavily dominated by simulations and lacking in empirical deployment. This empirical study trialed a FC CAT with dominance items described by the Thurstonian Item Response Theory model with research participants. This study investigated important practical issues such as the implications of adaptive item selection and social desirability balancing criteria on score distributions, measurement accuracy and participant perceptions. Moreover, nonadaptive but optimal tests of similar design were trialed alongside the CATs to provide a baseline for comparison, helping to quantify the return on investment when converting an otherwise-optimized static assessment into an adaptive one. Although the benefit of adaptive item selection in improving measurement precision was confirmed, results also indicated that at shorter test lengths CAT had no notable advantage compared with optimal static tests. Taking a holistic view incorporating both psychometric and operational considerations, implications for the design and deployment of FC assessments in research and practice are discussed. [ABSTRACT FROM AUTHOR]

In computerized adaptive testing (CAT), examinees see items targeted to their ability level. Postoperational data have a high degree of missing information relative to designs where everyone answers all questions. Item responses are observed over a restricted range of abilities, reducing item-total score correlations. However, if the adaptive item selection depends only on observed responses, the data are missing at random (MAR). We simulated data from three different testing designs (common items, randomly selected items, and CAT) and found that it was possible to re-estimate both person and item parameters from postoperational CAT data. In a multidimensional CAT, we show that it is necessary to include all responses from the testing phase to avoid violating missing data assumptions. We also observed that some CAT designs produced “reversals” where item discriminations became negative causing dramatic under and over-estimation of abilities. Our results apply to situations where researchers work with data drawn from adaptive testing or from instructional tools with adaptive delivery. To avoid bias, researchers must make sure they use all the data necessary to meet the MAR assumptions. [ABSTRACT FROM AUTHOR]

International student assessment tests in recent years have changed their structures to implement the adaptive format. The digital ENEM makes it possible to restructure the test by adhering to adaptive tests. This article proposes an adaptive multistage test (MST) for the ENEM test based on the 2019 edition in Mathematics. The ENEM items were analyzed through the Item Response Theory, and those items were used to build the MST modules and stages. The routing between the modules was defined by a study of the optimal cutoff point for the estimated latent trait, exhaustively tested to find the one that brought the best result compared to the complete exam. Ultimately, a multistage test architecture with specific cutoff values for routing was proposed. It was found that the adaptive test reduced the number of test items by 44.4%, and the estimation of skills was kept close to the estimation with the complete test. It was also noted that the ENEM is aimed at evaluating higher levels on the skill scale, making the estimation of skills impaired for less proficient individuals. [ABSTRACT FROM AUTHOR]

The focus of this paper is on the empirical relationship between item difficulty and item discrimination. Two studies—an empirical investigation and a simulation study—were conducted to examine the association between item difficulty and item discrimination under classical test theory and item response theory (IRT), and the effects of the association on various quantities of interest. Results from the empirical investigation show that item difficulty and item discrimination are negatively correlated under classical test theory, mostly negatively correlated under the two‐parameter logistic model, and mostly positively correlated under the three‐parameter logistic model; the magnitude of the correlation varied over the different data sets. Results from the simulation study reveal that a failure to incorporate the correlation between item difficulty and item discrimination in IRT simulations may provide the investigator with inaccurate values of important quantities of interest, and may lead to incorrect operational decisions. Implications to practice and future directions are discussed. [ABSTRACT FROM AUTHOR]

Adaptive classification testing (ACT) is a variation of computerized adaptive testing (CAT) that is developed to efficiently classify examinees into multiple groups based on predetermined cutoffs. In multidimensional multiclassification (i.e., more than two categories exist along each dimension), grid classification is proposed to classify each examinee into one of the grids encircled by cutoffs (lines/surfaces) along different dimensions so as to provide clearer information regarding an examinee's relative standing along each dimension and facilitate subsequent treatment and intervention. In this article, the sequential probability ratio test (SPRT) and confidence interval method were implemented in the grid multiclassification ACT. In addition, two new termination criteria, the grid classification generalized likelihood ratio (GGLR) and simplified grid classification generalized likelihood ratio were proposed for grid multiclassification ACT. Simulation studies, using a simulated item bank, and a real item bank with polytomous multidimensional items, show that grid multiclassification ACT is more efficient than classification based on measurement CAT that focuses on trait estimate precision. In the context of a high-quality bank, GGLR was found to most efficiently terminate the grid multiclassification ACT and classify examinees. [ABSTRACT FROM AUTHOR]

We aimed to develop an item bank of computerized adaptive testing for eating disorders (CAT-ED) in Chinese university students to increase measurement precision and improve test efficiency. A total of 1,025 Chinese undergraduate respondents answered a series of questions about eating disorders in a paper-pencil test. A total of 133 items from four well-validated Chinese-version scales of eating disorders were used to construct the item bank of CAT-ED with the following analysis. First, unidimensionality, model fit, local independence, item fit, discrimination and differential item functioning (DIF) were tested. Then, two simulation studies were applied to test the CAT-ED's effectivity and rationality by calculating concurrent criterion-related validity, sensitivity and specificity. The final item bank comprised 77 items, which met the requirements of local independence, item fit, high discrimination and no differential item functioning in CAT. The mean number of administered items in CAT with the stopping rule fixed at SE ≤ 0.3 was 11 items. The obtained results showed that CAT-ED had acceptable reliability, validity, sensitivity and specificity. [ABSTRACT FROM AUTHOR]

With increasing migration and mobility, German at the workplace has become an important topic in the field of German as a foreign Language/German as a second language. Vocational language skills can be measured with the help of various tests for the profession. The Goethe-Test PRO: German for professionals (GTP) is one of those tests that provides evidence of vocational language skills. This article first deals with the vocational orientation, the test format and the adaptive test procedure of the GTP. The second part of this paper is devoted to the description of its psychometric quality. Finally, the results are discussed with regard to the relevance of vocational tests for German as a foreign language. [ABSTRACT FROM AUTHOR]

Language proficiency testing serves an important function of classifying examinees into different categories of ability. However, misclassification is to some extent inevitable and may have important consequences for stakeholders. Recent research suggests that classification efficacy may be enhanced substantially using computerized adaptive testing (CAT). Using real data simulations, the current study investigated the classification performance of CAT on the reading section of an English language proficiency test and made comparisons with the paper-based version of the same test. Classification analysis was carried out to estimate classification accuracy (CA) and classification consistency (CC) by applying different locations and numbers of cutoff points. The results showed that classification was suitable when a single cutoff score was used, particularly for high- and low-ability test takers. Classification performance declined significantly when multiple cutoff points were simultaneously employed. Content analysis also raised important questions about construct coverage in CAT. The results highlight the potential for CAT to serve classification purposes and outline avenues for further research. [ABSTRACT FROM AUTHOR]

Adaptive measurement of change (AMC) is a psychometric method for measuring intra-individual change on one or more latent traits across testing occasions. Three hypothesis tests—a Z test, likelihood ratio test, and score ratio index—have demonstrated desirable statistical properties in this context, including low false positive rates and high true positive rates. However, the extant AMC research has assumed that the item parameter values in the simulated item banks were devoid of estimation error. This assumption is unrealistic for applied testing settings, where item parameters are estimated from a calibration sample before test administration. Using Monte Carlo simulation, this study evaluated the robustness of the common AMC hypothesis tests to the presence of item parameter estimation error when measuring omnibus change across four testing occasions. Results indicated that item parameter estimation error had at most a small effect on false positive rates and latent trait change recovery, and these effects were largely explained by the computerized adaptive testing item bank information functions. Differences in AMC performance as a function of item parameter estimation error and choice of hypothesis test were generally limited to simulees with particularly low or high latent trait values, where the item bank provided relatively lower information. These simulations highlight how AMC can accurately measure intra-individual change in the presence of item parameter estimation error when paired with an informative item bank. Limitations and future directions for AMC research are discussed. [ABSTRACT FROM AUTHOR]

Using classical test theory and item response theory, this study applied sequential procedures to a real operational item pool in a variable-length computerized adaptive testing (CAT) to detect items whose security may be compromised. Moreover, this study proposed a hybrid threshold approach to improve the detection power of the sequential procedure while controlling the Type I error rate. The hybrid threshold approach uses a local threshold for each item in an early stage of the CAT administration, and then it uses the global threshold in the decision-making stage. Applying various simulation factors, a series of simulation studies examined which factors contribute significantly to the power rate and lag time of the procedure. In addition to the simulation study, a case study investigated whether the procedures are applicable to the real item pool administered in CAT and can identify potentially compromised items in the pool. This research found that the increment of probability of a correct answer (p -increment) was the simulation factor most important to the sequential procedures' ability to detect compromised items. This study also found that the local threshold approach improved power rates and shortened lag times when the p -increment was small. The findings of this study could help practitioners implement the sequential procedures using the hybrid threshold approach in real-time CAT administration. [ABSTRACT FROM AUTHOR]

The new generation of tests not only focuses on the general ability but also the process of finer-grained skills. Under the guidance of this thought, researchers have developed a dual-purpose CD-CAT (Dual-CAT). In the existing Dual-CAT, the models used in overall ability estimation are unidimensional IRT models, which cannot apply to the multidimensional tests. This article intends to develop a multidimensional Dual-CAT to improve its applicability. To achieve this goal, this article firstly proposes some item selection methods for the multidimensional Dual-CAT, and then verifies the estimation accuracy and exposure rate of these methods through both simulation study and a real item bank study. The results show that the established multidimensional Dual-CAT is effective and the new proposed methods outperform the traditional methods. Finally, this article discusses the future direction of the Dual-CAT. [ABSTRACT FROM AUTHOR]

A computerized adaptive testing (CAT) solution for tests with multidimensional pairwise-comparison (MPC) items, aiming to measure career interest, value, and personality, is rare. This paper proposes new item selection and exposure control methods for CAT with dichotomous and polytomous MPC items and present simulation study results. The results show that the procedures are effective in selecting items and controlling within-person statement exposure with no loss of efficiency. Implications are discussed in two applications of the proposed CAT procedures: a work attitude test with dichotomous MPC items and a career interest assessment with polytomous MPC items. [ABSTRACT FROM AUTHOR]

Cognitive diagnosis computerized adaptive testing (CD-CAT) aims to identify each examinee's strengths and weaknesses on latent attributes for appropriate classification into an attribute profile. As the cost of a CD-CAT misclassification differs across user needs (e.g., remedial program vs. scholarship eligibilities), item selection can incorporate such costs to improve measurement efficiency. This study proposes such a method, minimum expected risk (MER), based on Bayesian decision theory. According to simulations, using MER to identify examinees with no mastery (MER-U0) or full mastery (MER-U1) showed greater classification accuracy and efficiency than other methods for these attribute profiles, especially for shorter tests or low quality item banks. For other attribute profiles, regardless of item quality or termination criterion, MER methods, modified posterior-weighted Kullback–Leibler information (MPWKL), posterior-weighted CDM discrimination index (PWCDI), and Shannon entropy (SHE) performed similarly and outperformed posterior-weighted attribute-level CDM discrimination index (PWACDI) in classification accuracy and test efficiency, especially on short tests. MER with a zero-one loss function, MER-U0, MER-U1, and PWACDI utilized item banks more effectively than the other methods. Overall, these results show the feasibility of using MER in CD-CAT to increase the accuracy for specific attribute profiles to address different user needs. [ABSTRACT FROM AUTHOR]

Objective: The scales measuring latent variables are used to gain information about the characteristic (θ) levels of individuals. Scales can be investigated with classical methods as well as the Computer Adaptive Testing (CAT) method. In this study, the performance of machine learning algorithms, including Classification and Regression Tree (CART), Random Forest (RF), Gradient Boosting Machines (GBM) and Extreme Gradient Boosting Machines (XGBoost), were tested as a new approach to a CAT application with the algorithm routinely used in CAT in simulation data derived from different scenarios from the Rasch model. Material and Methods: In the CAT application, the Rasch model was used as the probabilistic model and the question selection based on the information criterion was used as the question selection criterion. The performances of the methods were compared, based on the average number of items, the square root of the mean squared error (RMSE), the intraclass correlation coefficient (ICC) and the average prediction value obtained from these predictions. Results: Different methods become superior to others as category numbers increased. When the number of items was considered, the CART method made good predictions with the least number of items. When RMSE values were analysed, both GBM and XGBoost methods had low RMSE values. The methods compared have a good ICC value in estimating total scores. Conclusion: As a result of general comparisons, we recommend that a person planning a new study use machine learning methods which are frequently used in different fields recently as an alternative to the CAT method in accordance with its purpose. [ABSTRACT FROM AUTHOR]

Purpose: We are developing an age-related macular degeneration (AMD) health-related quality of life (HRQoL) item bank, applicable to Western and Asian populations. We report primarily on content generation and refinement, but also compare the HRQoL issues reported in our study with Western studies and current AMD-HRQoL questionnaires. Methods: In this cross-sectional, qualitative study of AMD patients attending the Singapore National Eye Centre (May–December 2019), items/domains were generated from: (1) AMD-specific questionnaires; (2) published articles; (3) focus groups/semi-structured interviews with AMD patients (n = 27); and (4) written feedback from retinal experts. Following thematic analysis, items were systematically refined to a minimally representative set and pre-tested using cognitive interviews with 16 AMD patients. Results: Of the 27 patients (mean ± standard deviation age 67.9 ± 7.0; 59.2% male), 18 (66.7%), two (7.4%), and seven (25.9%) had no, early-intermediate, and late/advanced AMD (better eye), respectively. Whilst some HRQoL issues, e.g. activity limitation, mobility, lighting, and concerns were similarly reported by Western patients and covered by other questionnaires, others like anxiety about intravitreal injections, work tasks, and financial dependency were novel. Overall, 462 items within seven independent HRQoL domains were identified: Activity limitation, Lighting, Mobility, Emotional, Concerns, AMD management, and Work. Following item refinement, items were reduced to 219, with 31 items undergoing amendment. Conclusion: Our 7-domain, 219-item AMD-specific HRQoL instrument will undergo psychometric testing and calibration for computerized adaptive testing. The future instrument will enable users to precisely, rapidly, and comprehensively quantify the HRQoL impact of AMD and associated treatments, with item coverage relevant across several populations. [ABSTRACT FROM AUTHOR]

Purpose: This study aimed to evaluate and improve the accuracy and efficiency of the QuickDASH for use in assessment of limb function in patients with upper extremity lymphedema using modern psychometric techniques. Method: We conducted confirmative factor analysis (CFA) and Mokken analysis to examine the assumption of unidimensionality for IRT model on data from 285 patients who completed the QuickDASH, and then fit the data to Samejima's graded response model (GRM) and assessed the assumption of local independence of items and calibrated the item responses for CAT simulation. Results: Initial CFA and Mokken analyses demonstrated good scalability of items and unidimensionality. However, the local independence of items assumption was violated between items 9 (severity of pain) and 11 (sleeping difficulty due to pain) (Yen's Q3 = 0.46) and disordered thresholds were evident for item 5 (cutting food). After addressing these breaches of assumptions, the re-analyzed GRM with the remaining 10 items achieved an improved fit. Simulation of CAT administration demonstrated a high correlation between scores on the CAT and the QuickDash (r = 0.98). Items 2 (doing heavy chores) and 8 (limiting work or daily activities) were the most frequently used. The correlation among factor scores derived from the QuickDASH version with 11 items and the Ultra-QuickDASH version with items 2 and 8 was as high as 0.91. Conclusion: By administering just these two best performing QuickDash items we can obtain estimates that are very similar to those obtained from the full-length QuickDash without the need for CAT technology. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) is a context-based adaptive assessment. However, the assessment result may not be valid because the examinee might cheat or guess the answers. Although there are many guessing detection methods, there are not many discussions about their implementation into CAT. Therefore, this paper presents an example of a modification of an existing software so the newly modified software can detect guessed answers and be able to select questions adaptively. The system can detect assuming behavior by recording the examinee's answer time. Also, the designed system can like questions adaptively by connecting Fuzzy logic, which calculates what level the question should select for the next iteration. The system is responded well by elementary and college students. A total of 56.6% felt the system was straightforward to use. The detection methods can detect guessing behavior of about 73%. However, the system's sensitivity is low if the method is forced to classify answers which answered in a long response time / general guessing. Nevertheless, when we limit the data classified within 10s response time (rapid-guessing), the method's sensitivity rises to 68.78%. [ABSTRACT FROM AUTHOR]

Although many educational and psychological tests are labeled as computerized adaptive test (CAT), not all tests show the same level of adaptivity – some tests might not have much adaptation because of various constraints imposed by test developers. Researchers have proposed some indices to measure the amount of adaption for an adaptive test. This article shows some limitations of the existing indices. A new index of adaptivity is proposed in this article. Its performance was evaluated in a simulation. The results show that the new index was able to overcome some of the limitations of the existing indices in the simulated scenarios. [ABSTRACT FROM AUTHOR]

Objective: The main aims of this study were to (1) create a patient-reported outcome measure (PROM) item bank for measuring the impact of upper quadrant edema (UQE) on physical function by calibrating responses to newly developed items; and (2) assess reliability, validity, and administration efficiency of scores based on computerized adaptive test (CAT) and 10-item short-form (SF) administration modes. Methods: This was a retrospective study including data from patients treated in outpatient rehabilitation clinics for UQE that responded to all 27 candidate items at intake. Item response theory model assumptions of unidimensionality, local item independence, item fit, and presence of differential item functioning were evaluated. UQE-CAT– and UQE-SF–generated scores were assessed for reliability, validity, and administration efficiency. Results: The total cohort included 3486 patients (mean [SD] age = 61 [13] years; range = 14–89 years). After removing 2 items, a 25-item solution was supported for its unidimensionality and fit to the item response theory model with reliability estimates of more than 0.93 for scores based on both CAT and SF administration modes. No items demonstrated differential item functioning. Scores discriminated among multiple patient groups in clinically logical ways and were moderately responsive to change with negligible floor and acceptable ceiling effects. CAT scores were generated using an average of 5.6 items (median = 5). Conclusion: Scores on the UQE PROM were reliable, valid, and efficient for assessing perceived physical function of patients with upper quadrant edema; thus, the measure is suitable for research and routine clinical administration. Impact: The newly developed UQE PROM is reliable and valid and offers efficient administration modes for assessing perceived physical function of patients with UQE caused by lymphatic and venous disorders, both for research and routine clinical care in busy outpatient rehabilitation settings. As an item response theory–based measure, the UQE PROM allows administration of condition-specific functional questions with low response burden for patients. This study supports a transition to PROMs that are based on modern measurement approaches to achieve high accuracy and efficiency. [ABSTRACT FROM AUTHOR]

This study aimed to validate the PROMIS® pediatric v2.0 Anger scale in the Dutch general population, provide reference data, and compare reliability and relative efficiency between the full-length scale, its short-form, computerized adaptive test (CAT), and Pediatric Quality of Life Inventory (PedsQLTM) emotional functioning (EF) subscale scores. Children (N = 1,328), representative of the Dutch population, were asked to complete the PROMIS pediatric Anger scale (8–18 years) and PedsQLTM (8–17 years). A graded response model (GRM) was fit to the data. Structural validity was assessed by checking item-fit statistics (S−X2, p <.001 = misfit). For construct validity, a moderate correlation (Pearson's r > 0.50) was expected between the Anger scale and PedsQLTM EF subscale score. Dutch mean T score based on the U.S. model was calculated to provide reference data and cut-offs. Standard error of measurement (SE(θ)) was used to assess reliability (SE(θ) <.32 =.90 reliability). Relative efficiency was calculated (1 − SE(θ)2/N items) to compare how good the measures performed relative to the amount of items administered. In total, 527 children completed the PROMIS pediatric Anger scale, of which 482 completed the PedsQLTM. Structural validity was sufficient as no items displayed misfit (S−X2 = 22.9–40.3, p >.001). The Anger scale score correlated moderately (Pearson's r =.64) with the PedsQLTM EF subscale score. Dutch mean T score was 44.20 (SD = 11.39), with cut-offs of >52.2 for moderate and ≥62.3 for severe symptoms. Reliable measurements were obtained at the population mean and >2SD in the clinically relevant direction. CAT outperformed all other measures in efficiency. The PROMIS pediatric Anger scale displayed sufficient psychometric properties within the Dutch population and reference data are available. Public Significance Statement: This study provides evidence that the PROMIS pediatric Anger scale has good psychometric properties in the Dutch general population. This scale can thus be implemented in clinical practice and for pediatric research in the Netherlands to solve the problem that validated Patient-Reported Outcome Measures (PROMs) assessing anger in a pediatric population are missing, improve interpretability and comparison of scores and reduce the burden of completing PROMs for pediatric patients. [ABSTRACT FROM AUTHOR]

Research on computerized adaptive testing (CAT) has provided evidence of the psychometric advantages of CAT over traditional fixed-item tests (FIT). However, evidence on test-taking experience is limited. This study investigated the psychometric and psychological impacts of CAT in the context of a multidimensional fluid reasoning test. In an online experimental study, we tested 286 participants, randomly assigning them to either the CAT or FIT conditions. We used two types of measures to assess participants' experience: self-report and time-based measures. The results showed that CAT outperforms FIT in terms of measurement precision but had a minimal impact on the test-taking experience. At the same time, FIT resulted in a more optimistic evaluation: when completing FIT, participants believed they had answered more items correctly than when completing CAT. Yet response time analysis indicates that participants invested more effort when working in CAT. Rapid guessing increased as the test progressed in both CAT and FIT, particularly in the FIT condition with the most difficult items. Providing information about how CAT works or starting the test with easier items may potentially enhance test-takers' experiences. [ABSTRACT FROM AUTHOR]