Search

The Rapid Online Assessment of Reading (ROAR) is a web-based lexical decision task that measures single-word reading abilities in children and adults without a proctor. Here we study whether item response theory (IRT) and computerized adaptive testing (CAT) can be used to create a more efficient online measure of word recognition. To construct an item bank, we first analyzed data taken from four groups of students (N = 1960) who differed in age, socioeconomic status, and language-based learning disabilities. The majority of item parameters were highly consistent across groups (r =.78–.94), and six items that functioned differently across groups were removed. Next, we implemented a JavaScript CAT algorithm and conducted a validation experiment with 485 students in grades 1–8 who were randomly assigned to complete trials of all items in the item bank in either (a) a random order or (b) a CAT order. We found that, to achieve reliability of 0.9, CAT improved test efficiency by 40%: 75 CAT items produced the same standard error of measurement as 125 items in a random order. Subsequent validation in 32 public school classrooms showed that an approximately 3-min ROAR-CAT can achieve high correlations (r =.89 for first grade, r =.73 for second grade) with alternative 5–15-min individually proctored oral reading assessments. Our findings suggest that ROAR-CAT is a promising tool for efficiently and accurately measuring single-word reading ability. Furthermore, our development process serves as a model for creating adaptive online assessments that bridge research and practice. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) aims to present items that statistically optimize the assessment process by considering the examinee's responses and estimated trait levels. Recent developments in reinforcement learning and deep neural networks provide CAT with the potential to select items that utilize more information across all the items on the remaining tests, rather than just focusing on the next several items to be selected. In this study, we reformulate CAT under the reinforcement learning framework and propose a new item selection strategy based on the deep Q-network (DQN) method. Through simulated and empirical studies, we demonstrate how to monitor the training process to obtain the optimal Q-networks, and we compare the accuracy of the DQN-based item selection strategy with that of five traditional strategies—maximum Fisher information, Fisher information weighted by likelihood, Kullback‒Leibler information weighted by likelihood, maximum posterior weighted information, and maximum expected information—on both simulated and real item banks and responses. We further investigate how sample size and the distribution of the trait levels of the examinees used in training affect DQN performance. The results show that DQN achieves lower RMSE and MAE values than traditional strategies under simulated and real banks and responses in most conditions. Suggestions for the use of DQN-based strategies are provided, as well as their code. [ABSTRACT FROM AUTHOR]

Accumulating evidence supports the presence of a general psychopathology dimension, the p factor ('p'). Despite growing interest in the p factor, questions remain about how p is assessed. Although multi-informant assessment of psychopathology is commonplace in clinical research and practice with children and adolescents, almost no research has taken a multi-informant approach to studying youth p or has examined the degree of concordance between parent and youth reports. Further, estimating p requires assessment of a large number of symptoms, resulting in high reporter burden that may not be feasible in many clinical and research settings. In the present study, we used bifactor multidimensional item response theory models to estimate parent- and adolescent-reported p in a large community sample of youth (11–17 years) and parents (N = 5,060 dyads). We examined agreement between parent and youth p scores and associations with assessor-rated youth global functioning. We also applied computerized adaptive testing (CAT) simulations to parent and youth reports to determine whether adaptive testing substantially alters agreement on p or associations with youth global functioning. Parent-youth agreement on p was moderate (r =.44) and both reports were negatively associated with youth global functioning. Notably, 7 out of 10 of the highest loading items were common across reporters. CAT reduced the average number of items administered by 57%. Agreement between CAT-derived p scores was similar to the full form (r =.40) and CAT scores were negatively correlated with youth functioning. These novel results highlight the promise and potential clinical utility of a multi-informant p factor approach. [ABSTRACT FROM AUTHOR]

The given paper discusses an original method of the evaluation of outcomes of adaptive testing in the case of the strategy of the multilevel testing. The multiplicity/set of the outcomes of testing consists of atypical different-dimensional elements. The given paper defines the criteria of their comparison, describes the principles of ordering of the given multiplicity and draws the getting of a final score. The paper presented considers the ordering method of outcome set for multi-stage testing (MST) of 1-3-3 model. The ordering method of outcome set is used for the estimation of results of computerized adaptive testing (CAT). This method is not tied to a specific testing procedure. Acknowledgment of this is its usage for the 1-3-3 model, which is described in the paper. To sort the set of testing outcomes the function-criteria described in the initial article are used here and a comparative analysis of obtained results is performed. The criterion of ordering of the outcomes of testing may not be the only option. The given paper illustrates this fact through a comparative discussion of two samples. An original procedure of testing is used for the presentation of the essence of the method. The given procedure is aimed to be illustrative, because a described method of assessment can be used for similar strategies. The ordered outcome set is estimated by a hundred-point system according to the normal distribution. Applied results of our scientific research is developed as "Adaptester" portal and available on the following address: https://adaptester.com. [ABSTRACT FROM AUTHOR]

The number of mobile phone users worldwide has increased in recent years. As people spend more time on their phones, negative effects such as problematic mobile phone use (PMPU) have become more pronounced. Many researchers have dedicated their efforts to developing the questionnaires and revising the tools to more accurately evaluate PMPU. Previous studies had demonstrated that CAT-PMPU for adults could significantly enhance measurement accuracy and efficiency. However, most of the items in this scenario was developed for adults, and there were notable differences between adults and adolescents, making some items potentially unsuitable for the latter. Thus, this study aimed to generalize the CAT-PMPU adult version to make it suitable for both adult and adolescent populations. A total of 740 Chinese adolescents and 980 Chinese adults participated in this study, completing online or paper-pencil questionnaires. Empirical data was then used to simulate CAT-PMPU, and the measurement efficiency, accuracy and reliability were compared between adults and adolescents in different stopping rules. The results showed that generalized CAT-PMPU in this study had promising measurement efficiency and accuracy, which was consistent with adult version. In conclusion, the CAT-PMPU developed in this study not only exhibited satisfactory test reliability but also emerged as a novel technical support for evaluation of PMPU in both adolescent and adult populations, which demonstrated the potential applicability in practice. [ABSTRACT FROM AUTHOR]

The computerized adaptive form of cognitive diagnostic testing, CD-CAT, has gained increasing attention in the domain of personalized measurements for its ability to categorize individual mastery status of fine-grained attributes more accurately and efficiently through administering items tailored to one's ability progressively. How to select the next item based on previous response(s) is crucial for the success of CD-CAT. Previous item selection strategies for CD-CAT have often followed a greedy or semi-greedy approach, which makes it difficult to strike a balance between diagnostic performance and item bank utilization. To address this issue, this study takes a graph perspective and transforms the item selection problem in CD-CAT into a path-searching problem, in which paths refer to possible test construction and nodes refer to individual items. A heuristic function is defined to predict the prospect of a path, indicating how well the corresponding test can diagnose the current examinee. Two search mechanisms with different biases towards item exposure control are proposed to approximate the optimal path with the best prospect. The first unused item on the resulting path is selected as the next item. The above components compose a novel CD-CAT item selection framework based on heuristic search. Simulation studies are conducted under a variety of conditions regarding bank designs, bank-quality conditions, and testing scenarios. The results are compared with different types of classic item selection strategies in CD-CAT, showing that the proposed framework can enhance bank utilization at a smaller cost of diagnostic performance. [ABSTRACT FROM AUTHOR]

Questionnaires to detect emotional and behavioral (EB) problems in preventive child healthcare (PCH) should be short; this potentially affects their validity and reliability. Computerized adaptive testing (CAT) could overcome this weakness. The aim of this study was to (1) develop a CAT to measure EB problems among pre-school children and (2) assess the efficiency and validity of this CAT. We used a Dutch national dataset obtained from parents of pre-school children undergoing a well-child care assessment by PCH (n = 2192, response 70%). Data regarded 197 items on EB problems, based on four questionnaires, the Strengths and Difficulties Questionnaire (SDQ), the Child Behavior Checklist (CBCL), the Ages and Stages Questionnaire: Social Emotional (ASQ:SE), and the Brief Infant–Toddler Social and Emotional Assessment (BITSEA). Using 80% of the sample, we calculated item parameters necessary for a CAT and defined a cutoff for EB problems. With the remaining part of the sample, we used simulation techniques to determine the validity and efficiency of this CAT, using as criterion a total clinical score on the CBCL. Item criteria were met by 193 items. This CAT needed, on average, 16 items to identify children with EB problems. Sensitivity and specificity compared to a clinical score on the CBCL were 0.89 and 0.91, respectively, for total problems; 0.80 and 0.93 for emotional problems; and 0.94 and 0.91 for behavioral problems. Conclusion: A CAT is very promising for the identification of EB problems in pre-school children, as it seems to yield an efficient, yet high-quality identification. This conclusion should be confirmed by real-life administration of this CAT. What is Known: • Studies indicate the validity of using computerized adaptive test (CAT) applications to identify emotional and behavioral problems in school-aged children. • Evidence is as yet limited on whether CAT applications can also be used with pre-school children. What is New: • The results of this study show that a computerized adaptive test is very promising for the identification of emotional and behavior problems in pre-school children, as it appears to yield an efficient and high-quality identification. [ABSTRACT FROM AUTHOR]

This paper presents a new method for accurately measuring the vocabulary size of second language (L2) learners. Traditional vocabulary size tests (VSTs) are limited in capturing a tester's vocabulary and are often population-specific. To overcome these issues, we propose an intelligent vocabulary size measurement method that utilizes massive robot testers. They are equipped with randomized and word-frequency-based vocabularies to simulate L2 learners' variant vocabularies. An intelligent vocabulary size test (IVST) is developed to precisely measure vocabulary size for any population. The robot testers "take" the IVST, which dynamically generates quizzes with varying levels of difficulty adapted to the estimated tester's vocabulary size in real-time using an artificial neural network (ANN) through iterative learning. The effectiveness of the IVST is factually verified by their visible vocabularies. Additionally, we apply a long short-term memory (LSTM) model to further enhance the method's performance. The proposed method has demonstrated high reliability and effectiveness, achieving accuracies of 98.47% for the IVST and 99.87% for the IVST with LSTM. This novel approach provides a more precise and reliable method for measuring vocabulary size in L2 learners compared to traditional VSTs, offering potential benefits to language learners and educators. [ABSTRACT FROM AUTHOR]

Computerized adaptive testing (CAT) targets to accurately assess the student's proficiency in the required subject/area. The key issue is how to design a question selector that adaptively selects the best-suited questions for each student based on previous performance step by step. Most existing question selectors execute via greedy metric functions (e.g., question information and uncertainty), which can not effectively capture data characteristics. There also exist learning-based question selectors that redefine the CAT problem as a bilevel optimization problem, where the parameter learning of the question selector and the student proficiency estimation model are coupled, which is not flexible enough. To this end, in this paper, we propose a novel CAT framework with Decoupled Learning selector (DL-CAT). Specifically, we first use the currently estimated student ability and question characteristics as input and design a deep learning-based question selector to predict question selection scores. Then, to address the issue that there is no ground truth to measure the quality of the selected question, an approximate ground-truth and a pairwise rank loss function are specially designed to update the parameters of the question selector independently. Extensive experiments on two real datasets demonstrate that our proposed DL-CAT has certain advantages in effectiveness and significant advantages in efficiency. [ABSTRACT FROM AUTHOR]

Purpose: To assess the psychometric properties of glaucoma-specific health-related quality of life (HRQoL) item banks (IBs), and explore their efficiency using computerized adaptive testing (CAT) simulations. Methods: In this cross-sectional, clinical study, 300 Asian glaucoma patients answered 221 items within seven IBs: Ocular Comfort Symptoms (OS); Activity Limitation (AL); Lighting (LT); Mobility (MB); Glaucoma Management (GM); Psychosocial (PSY); and Work (WK). Rasch analysis was conducted to assess each IB's psychometric properties (e.g., item "fit" to the construct; unidimensionality) and a set of analytic performance criteria guiding decision making relating to retaining or dropping domains and items was employed. CAT simulations determined the mean number of items for 'high' and 'moderate' measurement precision (stopping rule: SEM 0.3 and 0.387, respectively). Results: Participants' mean age was 67.2 ± 9.2 years (62% male; 87% Chinese). LT, MB, and GM displayed good psychometric properties overall. To optimize AL's psychometric properties, 16 items were deleted due to poor "fit", high missing data, item bias, low discrimination and/or a low clinical/patient importance rating. To resolve multidimensionality in PSY, we rehomed 16 items into a "Concern (CN)" domain. PSY and CN required further amendment, including collapsing of response categories, and removal of poorly functioning items (N = 7). Due to poor measurement precision, low applicability and high ceiling effect, low test information indices, and low item separation index the WK IB was not considered further. In CAT simulations on the final seven IBs (n = 182 items total), an average of 12.1 and 15.7 items per IB were required for moderate and high precision measurement, respectively. Conclusions: After reengineering our seven IBs, they displayed robust psychometric properties and good efficiency in CAT simulations. Once finalized, GlauCAT™-Asian may enable comprehensive assessment of the HRQoL impact of glaucoma and associated treatments. [ABSTRACT FROM AUTHOR]

Quantifying the severity of neurological disorders, such as attention-deficit/hyperactivity disorder (ADHD), is encouraged in practice. To explore the methodological properties of data-driven classification, this study aimed to develop computerized adaptive testing (CAT) based on Bayesian networks to classify ADHD symptom severity for individual subjects efficiently. The Current Symptoms Scale was used to build an ADHD symptom severity classifier in a sample of 892 college students, and the CAT making use of information gains from conditional probabilities of the Bayesian network was evaluated through a simulation study. The results showed that distributing 8 out of 18 items on average to an individual subject would result in 85% classification accuracy, supporting the adaption of surveying ADHD to digital platforms demanding much-shortened scales. The severity classifier based on Bayesian networks can assist diagnostic tasks via a data-driven mechanism. Further, integrating CAT into the classifier can shorten the scale while maintain acceptable measurement quality. The ADHD symptoms were used in the paper; however, the proposed method applies to other psychological symptoms as well. The impact and limitations of the CAT for ADHD were discussed, and several suggestions for future research were provided. [ABSTRACT FROM AUTHOR]

The replacement of existing technology or the introduction of novel technology into the day-to-day routines of higher education institutions is not a trivial task. Currently, many higher education institutions are faced with the challenge of replacing existing procedures for administering written exams with e-exams. To guide this process, this paper proposes the novel technology-based exams acceptance model (TEAM) and empirically evaluates its model structure and usefulness from the perspective of higher education teachers. The model can be used to guide the transition from paper-based exams to e-exams and the implementation of innovative (e.g., adaptive) e-exam formats. The model includes perceived usefulness, computer self-efficacy, computer anxiety, prior experience, facilitating conditions, and subjective norm as predictors of the behavioral intention to use e-exams. To test the model empirically, the responses of 992 teachers at 63 German universities to a standardized online questionnaire were analyzed using structural equation modeling. The model fit was acceptable. With 77% (conventional e-exams) and 82% (adaptive e-exams), a large proportion of the variance of the intention to use these types of exams was explained. With TEAM, a highly predictive model for explaining the behavioral intention to use e-exams is now available. It offers a theoretical basis that can be used for the successful implementation of e-exams in higher education. [ABSTRACT FROM AUTHOR]

Background: A glaucoma-specific health-related quality of life (HRQoL) item bank (IB) and computerized adaptive testing (CAT) system relevant to Asian populations is not currently available. We aimed to develop content for an IB focusing on HRQoL domains important to Asian people with glaucoma; and to compare the content coverage of our new instrument with established glaucoma-specific instruments. Methods: In this qualitative study of glaucoma patients recruited from the Singapore National Eye Centre (November 2018-November 2019), items/domains were generated from: (1) glaucoma-specific questionnaires; (2) published articles; (3) focus groups/semi-structured interviews with glaucoma patients (n = 27); and (4) feedback from glaucoma experts. Data were analyzed using the constant comparative method. Items were systematically refined to a concise set, and pre-tested using cognitive interviews with 27 additional glaucoma patients. Results: Of the 54 patients (mean ± standard deviation [SD] age 66.9 ± 9.8; 53.7% male), 67 (62.0%), 30 (27.8%), and 11 (10.2%) eyes had primary open angle glaucoma, angle closure glaucoma, and no glaucoma respectively. Eighteen (33.3%), 11 (20.4%), 8 (14.8%), 12 (22.2%), and 5 (9.3%) patients had no, mild, moderate, severe, or advanced/end-stage glaucoma (better eye), respectively. Initially, 311 items within nine HRQoL domains were identified: Visual Symptoms, Ocular Comfort Symptoms, Activity Limitation, Driving, Lighting, Mobility, Psychosocial, Glaucoma management, and Work; however, Driving and Visual Symptoms were subsequently removed during the refinement process. During cognitive interviews, 12, 23 and 10 items were added, dropped and modified, respectively. Conclusion: Following a rigorous process, we developed a 221-item, 7-domain Asian glaucoma-specific IB. Once operationalised using CAT, this new instrument will enable precise, rapid, and comprehensive assessment of the HRQoL impact of glaucoma and associated treatment efficacy. [ABSTRACT FROM AUTHOR]

The computerized adaptive tests (CAT) apply an adaptive process in which the items are tailored to individuals' ability scores. The multidimensional CAT (MCAT) designs differ in terms of different item selection, ability estimation, and termination methods being used. This study aims at investigating the performance of the MCAT designs used to measure the language ability of students and to compare the results of MCAT designs with the outcomes of corresponding paper–pencil tests. For this purpose, items in the English Proficiency Tests (EPT) were used to create a multi-dimensional item pool that consists of 599 items. The performance of the MCAT designs was evaluated and compared based on the reliability coefficients, root means square error (RMSE), test-length, and root means squared difference (RMSD) statistics, respectively. Therefore, 36 different conditions were investigated in total. The results of the post-hoc simulation designs indicate that the MCAT designs with the A-optimality item selection method outperformed MCAT designs with other item selection methods by decreasing the test length and RMSD values without any sacrifice in test reliability. Additionally, the best error variance stopping rule for each MCAT algorithm with A-optimality item selection could be considered as 0.25 with 27.9 average test length and 30 items for the fixed test-length stopping rule for the Bayesian MAP method. Overall, MCAT designs tend to decrease the test length by 60 to 65 percent and provide ability estimations with higher precision compared to the traditional paper–pencil tests with 65 to 75 items. Therefore, it is suggested to use the A-optimality method for item selection and the Bayesian MAP method for ability estimation for the MCAT designs since the MCAT algorithm with these specifications shows better performance than others. [ABSTRACT FROM AUTHOR]

Purpose: We are developing an age-related macular degeneration (AMD) health-related quality of life (HRQoL) item bank, applicable to Western and Asian populations. We report primarily on content generation and refinement, but also compare the HRQoL issues reported in our study with Western studies and current AMD-HRQoL questionnaires. Methods: In this cross-sectional, qualitative study of AMD patients attending the Singapore National Eye Centre (May–December 2019), items/domains were generated from: (1) AMD-specific questionnaires; (2) published articles; (3) focus groups/semi-structured interviews with AMD patients (n = 27); and (4) written feedback from retinal experts. Following thematic analysis, items were systematically refined to a minimally representative set and pre-tested using cognitive interviews with 16 AMD patients. Results: Of the 27 patients (mean ± standard deviation age 67.9 ± 7.0; 59.2% male), 18 (66.7%), two (7.4%), and seven (25.9%) had no, early-intermediate, and late/advanced AMD (better eye), respectively. Whilst some HRQoL issues, e.g. activity limitation, mobility, lighting, and concerns were similarly reported by Western patients and covered by other questionnaires, others like anxiety about intravitreal injections, work tasks, and financial dependency were novel. Overall, 462 items within seven independent HRQoL domains were identified: Activity limitation, Lighting, Mobility, Emotional, Concerns, AMD management, and Work. Following item refinement, items were reduced to 219, with 31 items undergoing amendment. Conclusion: Our 7-domain, 219-item AMD-specific HRQoL instrument will undergo psychometric testing and calibration for computerized adaptive testing. The future instrument will enable users to precisely, rapidly, and comprehensively quantify the HRQoL impact of AMD and associated treatments, with item coverage relevant across several populations. [ABSTRACT FROM AUTHOR]

Purpose: This study aimed to evaluate and improve the accuracy and efficiency of the QuickDASH for use in assessment of limb function in patients with upper extremity lymphedema using modern psychometric techniques. Method: We conducted confirmative factor analysis (CFA) and Mokken analysis to examine the assumption of unidimensionality for IRT model on data from 285 patients who completed the QuickDASH, and then fit the data to Samejima's graded response model (GRM) and assessed the assumption of local independence of items and calibrated the item responses for CAT simulation. Results: Initial CFA and Mokken analyses demonstrated good scalability of items and unidimensionality. However, the local independence of items assumption was violated between items 9 (severity of pain) and 11 (sleeping difficulty due to pain) (Yen's Q3 = 0.46) and disordered thresholds were evident for item 5 (cutting food). After addressing these breaches of assumptions, the re-analyzed GRM with the remaining 10 items achieved an improved fit. Simulation of CAT administration demonstrated a high correlation between scores on the CAT and the QuickDash (r = 0.98). Items 2 (doing heavy chores) and 8 (limiting work or daily activities) were the most frequently used. The correlation among factor scores derived from the QuickDASH version with 11 items and the Ultra-QuickDASH version with items 2 and 8 was as high as 0.91. Conclusion: By administering just these two best performing QuickDash items we can obtain estimates that are very similar to those obtained from the full-length QuickDash without the need for CAT technology. [ABSTRACT FROM AUTHOR]

Purpose: This study aimed to validate the PROMIS Pediatric item bank v2.0 Peer Relationships and compare reliability of the full item bank to its short form, computerized adaptive test (CAT) and the social functioning (SF) subscale of the Pediatric Quality of Life Inventory (PedsQL™). Methods: Children aged 8–18 (n = 1327), representative of the Dutch population completed the Peer Relationships item bank. A graded response model (GRM) was fit to the data. Structural validity was assessed by checking item-fit statistics (S-X2, p < 0.001 = misfit). For construct validity, a moderately strong correlation (> 0.50) was expected between Peer Relationships and the PedsQL SF subscale. Cross-cultural DIF between U.S. and NL was assessed using logistic regression, where an item with McFadden's pseudo R2 > 0.02 was considered to have DIF. Percentage of participants reliably measured was assessed using the standard error of measurement (SEM) < 0.32 as a criterion (reliability of 0.90). Relative efficiency ((1-SEM2)/nitems) was calculated to compare how well the instruments performed relative to the amount of items administered. Results: In total, 527 (response rate: 39.7%) children completed the PROMIS v2.0 Peer Relationships item bank (nitems = 15) and the PedsQL™ (nitems = 23). Structural validity of the Peer Relationships item bank was sufficient, but one item displayed misfit in the GRM model (S-X2 < 0.001); 5152R1r ("I played alone and kept to myself"). The item 733R1r ("I was a good friend") was the only item that displayed cross-cultural DIF (R2 = 0.0253). The item bank correlated moderately high (r = 0.61) with the PedsQL SF subscale Reliable measurements were obtained at the population mean and > 2SD in the clinically relevant direction. CAT outperformed all other measures in efficiency. Mean T-score of the Dutch general population was 46.9(SD 9.5). Conclusion: The pediatric PROMIS Peer Relationships item bank was successfully validated for use within the Dutch population and reference data are now available. [ABSTRACT FROM AUTHOR]

Copyright of English Teaching & Learning is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

To further advance assessment of patient-reported outcomes, the European Organisation of Research and Treatment of Cancer (EORTC) Quality of Life Group has developed computerized adaptive test (CAT) versions of all EORTC Quality of Life Core Questionnaire (QLQ-C30) scales/items. The aim of this study was to develop and evaluate an item bank for CAT measurement of insomnia (CAT-SL). In line with the EORTC guidelines, the developmental process comprised four phases: (I) defining the concept insomnia and literature search, (II) selection and formulation of new items, (III) pre-testing and (IV) field-testing, including psychometric analyses of the final item bank. In phase I, the literature search identified 155 items that were compatible with our conceptualisation of insomnia, including both quantity and quality of sleep. In phase II, following a multistep-approach, this number was reduced to 15 candidate items. Pre-testing of these items in cancer patients (phase III) resulted in an item list of 14 items, which were field-tested among 1094 patients in phase IV. Psychometric evaluations showed that eight items could be retained in a unidimensional model. The final item bank yielded greater measurement precision than the original QLQ-C30 insomnia item. It was estimated that administering two or more items from the insomnia item bank with CAT results in a saving in sample size between approximately 15–25%. The 8-item EORTC CAT-SL item bank facilitates precise and efficient measurement of insomnia as part of the EORTC CAT system of health-related quality life assessment in both clinical research and practice. [ABSTRACT FROM AUTHOR]

An adaptive task difficulty assignment method which we reckon as balanced difficulty task finder (BDTF) is proposed in this paper. The aim is to recommend tasks to a learner using a trade-off between skills of the learner and difficulty of the tasks such that the learner experiences a state of flow during the learning. Flow is a mental state that psychologists refer to when someone is completely immersed in an activity. Flow state is a multidisciplinary field of research and has been studied not only in psychology, but also neuroscience, education, sport, and games. The idea behind this paper is to try to achieve a flow state in a similar way as Elo's chess skill rating (Glickman in Am Chess J 3:59–102) and TrueSkill (Herbrich et al. in Advances in neural information processing systems, 2006) for matching game players, where "matched players" should possess similar capabilities and skills in order to maintain the level of motivation and involvement in the game. The BDTF draws analogy between choosing an appropriate opponent or appropriate game level and automatically choosing an appropriate difficulty level of a learning task. This method, as an intelligent tutoring system, could be used in a wide range of applications from online learning environments and e-learning, to learning and remembering techniques in traditional methods such as adjusting delayed matching to sample and spaced retrieval training that can be used for people with memory problems such as people with dementia. [ABSTRACT FROM AUTHOR]

Likert-type measures have been criticized in psychological assessment because they are vulnerable to response biases, including central tendency, acquiescence, leniency, halo, and socially desirable responding. As an alternative, multidimensional forced choice (MFC) testing has been proposed to address these concerns. A number of researchers have developed item response theory (IRT) models for MFC data and have examined latent trait estimation with tests of different dimensionality and length. Research has also explored the advantages of computerized adaptive testing (CAT) with MFC pair tests having as many as 25 dimensions, but there have been no published studies on CAT with MFC triplets or tetrads. Thus, in this research we aimed to address that issue. We used recently developed item information functions for an MFC ranking model to compare the benefits of CAT with MFC pair, triplet, and tetrad tests. A simulation study showed that CAT substantially outperformed nonadaptive testing for latent trait estimation across MFC formats. More importantly, CAT with MFC pairs provided estimation accuracy similar to or better than that from tests of equivalent numbers of nonadaptive MFC triplets. On the basis of these findings, implications and recommendations are further discussed for constructing MFC measures to use in psychological contexts. [ABSTRACT FROM AUTHOR]

Purpose: To compare the results from a simulated computerized adaptive test (CAT) for the 28-item Impact of Vision Impairment (IVI) questionnaire and the original paper-pencil version in terms of efficiency (main outcome), defined as percentage item reduction.Methods: Using paper-pencil IVI data from 832 participants across the spectrum of vision impairment, item calibrations of the 28-item IVI instrument and its associated 20-item vision-specific functioning (VSF) and 8-item emotional well-being (EWB) subscales were generated with Rasch analysis. Based on these calibrations, CAT simulations were conducted on 1000 cases, with 'high' and 'moderate' precision stopping rules (standard error of measurement [SEM] 0.387 and 0.521, respectively). We examined the average number of items needed to satisfy the stopping rules and the corresponding percentage item reduction, level of agreement between person measures estimated from the full IVI item bank and from the CAT simulations, and item exposure rates (IER).Results: For the overall IVI-CAT, 5 or 9.7 items were required, on average, to obtain moderate or high precision estimates of vision-related quality of life, corresponding to 82.1 and 65.4% item reductions compared to the paper-pencil IVI. Agreement was high between the person measures generated from the full IVI item bank and the IVI-CAT for both the high precision simulation (mean bias, - 0.004 logits; 95% LOA - 0.594 to 0.587) and moderate precision simulation (mean bias, 0.014 logits; 95% LOA - 0.828 to 0.855). The IER for the IVI-CAT in the moderate precision simulation was skewed, with six EWB items used > 40% of the time.Conclusion: Compared to the paper-pencil IVI instrument, the IVI-CATs required fewer items without loss of measurement precision, making them potentially attractive outcome instruments for implementation into clinical trials, healthcare, and research. Final versions of the IVI-CATs are available. [ABSTRACT FROM AUTHOR]

In a cognitive diagnostic computerized adaptive testing (CD-CAT) exam, an item pool that consists of items with calibrated item parameters is used for item selection and attribute estimation. The parameter estimates for the items in the item pool are often treated as if they were the true population parameters, and therefore, the calibration errors are ignored. The purpose of this study was to investigate the effects of calibration errors on the attribute classification accuracy, the measurement precision of attribute mastery classification, and the test information under the log-linear cognitive diagnosis model (LCDM) framework. The deterministic input, noisy “and” gate (DINA) model and the compensatory re-parameterized unified model (C-RUM) were used in fixed-length CD-CAT simulations. The results showed that high levels of calibration errors were associated with low classification accuracy, low test information, and misleading estimation of measurement precision. The effects of calibration errors decreased as the test length increased, and the DINA model appeared to be more vulnerable in the presence of calibration errors. The C-RUM was less influenced by calibration errors because of its additive characteristics in the LCDM framework. The same conclusions applied when item exposure control was incorporated and when different item selection methods were used. Finally, the use of a larger calibration sample size to calibrate the item pool was found to reduce the magnitudes of error variances and increase the attribute classification accuracy. [ABSTRACT FROM AUTHOR]

Purpose: The purpose of this paper is to demonstrate the procedures to develop and implement a computerized adaptive patient-reported outcome (PRO) measure using secondary analysis of a dataset and items from fixed-format legacy measures.Methods: We conducted secondary analysis of a dataset of responses from 1429 persons with work-related lower extremity impairment. We calibrated three measures of physical functioning on the same metric, based on item response theory (IRT). We evaluated efficiency and measurement precision of various computerized adaptive test (CAT) designs using computer simulations.Results: IRT and confirmatory factor analyses support combining the items from the three scales for a CAT item bank of 31 items. The item parameters for IRT were calculated using the generalized partial credit model. CAT simulations show that reducing the test length from the full 31 items to a maximum test length of 8 items, or 20 items is possible without a significant loss of information (95, 99% correlation with legacy measure scores).Conclusions: We demonstrated feasibility and efficiency of using CAT for PRO measurement of physical functioning. The procedures we outlined are straightforward, and can be applied to other PRO measures. Additionally, we have included all the information necessary to implement the CAT of physical functioning in the electronic supplementary material of this paper. [ABSTRACT FROM AUTHOR]

Purpose: Our understanding of the coping strategies used by people with visual impairment to manage stress related to visual loss is limited. This study aims to develop a sophisticated coping instrument in the form of an item bank implemented via Computerised adaptive testing (CAT) for hereditary retinal diseases.Methods: Items on coping were extracted from qualitative interviews with patients which were supplemented by items from a literature review. A systematic multi-stage process of item refinement was carried out followed by expert panel discussion and cognitive interviews. The final coping item bank had 30 items. Rasch analysis was used to assess the psychometric properties. A CAT simulation was carried out to estimate an average number of items required to gain precise measurement of hereditary retinal disease-related coping.Results: One hundred eighty-nine participants answered the coping item bank (median age = 58 years). The coping scale demonstrated good precision and targeting. The standardised residual loadings for items revealed six items grouped together. Removal of the six items reduced the precision of the main coping scale and worsened the variance explained by the measure. Therefore, the six items were retained within the main scale. Our CAT simulation indicated that, on average, less than 10 items are required to gain a precise measurement of coping.Conclusions: This is the first study to develop a psychometrically robust coping instrument for hereditary retinal diseases. CAT simulation indicated that on an average, only four and nine items were required to gain measurement at moderate and high precision, respectively. [ABSTRACT FROM AUTHOR]

Working memory tasks designed for children usually present trials in order of ascending difficulty, with testing discontinued when the child fails a particular level. Unfortunately, this procedure comes with a number of issues, such as decreased engagement from high-ability children, vulnerability of the scores to temporary mind-wandering, and large between-subjects variations in number of trials, testing time, and proactive interference. To circumvent these problems, the goal of the present study was to demonstrate the feasibility of assessing working memory using an adaptive testing procedure. The principle of adaptive testing is to dynamically adjust the level of difficulty as the task progresses to match the participant's ability. We used this method to develop an adaptive complex span task (the ACCES) comprising verbal and visuo-spatial subtests. The task presents a fixed number of trials to all participants, allows for partial credit scoring, and can be used with children regardless of ability level. The ACCES demonstrated satisfying psychometric properties in a sample of 268 children aged 8-13 years, confirming the feasibility of using adaptive tasks to measure working memory capacity in children. A free-to-use implementation of the ACCES is provided. [ABSTRACT FROM AUTHOR]

Purpose: Multidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.Methods: Prompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.Results: The commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank's efficiency and precision.Discussion: Multidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation. [ABSTRACT FROM AUTHOR]

Objective: Traditional patient-reported physical function instruments often poorly differentiate patients with mild-to-moderate disability. We describe the development and psychometric evaluation of a generic item bank for measuring everyday activity limitations in outpatient populations.Study Design and Setting: Seventy-two items generated from patient interviews and mapped to the International Classification of Functioning, Disability and Health (ICF) domestic life chapter were administered to 1128 adults representative of the Dutch population. The partial credit model was fitted to the item responses and evaluated with respect to its assumptions, model fit, and differential item functioning (DIF). Measurement performance of a computerized adaptive testing (CAT) algorithm was compared with the SF-36 physical functioning scale (PF-10).Results: A final bank of 41 items was developed. All items demonstrated acceptable fit to the partial credit model and measurement invariance across age, sex, and educational level. Five- and ten-item CAT simulations were shown to have high measurement precision, which exceeded that of SF-36 physical functioning scale across the physical function continuum. Floor effects were absent for a 10-item empirical CAT simulation, and ceiling effects were low (13.5%) compared with SF-36 physical functioning (38.1%). CAT also discriminated better than SF-36 physical functioning between age groups, number of chronic conditions, and respondents with or without rheumatic conditions.Conclusion: The Rasch assessment of everyday activity limitations (REAL) item bank will hopefully prove a useful instrument for assessing everyday activity limitations. T-scores obtained using derived measures can be used to benchmark physical function outcomes against the general Dutch adult population. [ABSTRACT FROM AUTHOR]

Background: The European Organisation of Research and Treatment of Cancer (EORTC) Quality of Life Group is developing computerized adaptive testing (CAT) versions of all EORTC Quality of Life Questionnaire (QLQ-C30) scales with the aim to enhance measurement precision. Here we present the results on the field-testing and psychometric evaluation of the item bank for cognitive functioning (CF).Methods: In previous phases (I-III), 44 candidate items were developed measuring CF in cancer patients. In phase IV, these items were psychometrically evaluated in a large sample of international cancer patients. This evaluation included an assessment of dimensionality, fit to the item response theory (IRT) model, differential item functioning (DIF), and measurement properties.Results: A total of 1030 cancer patients completed the 44 candidate items on CF. Of these, 34 items could be included in a unidimensional IRT model, showing an acceptable fit. Although several items showed DIF, these had a negligible impact on CF estimation. Measurement precision of the item bank was much higher than the two original QLQ-C30 CF items alone, across the whole continuum. Moreover, CAT measurement may on average reduce study sample sizes with about 35-40% compared to the original QLQ-C30 CF scale, without loss of power.Conclusion: A CF item bank for CAT measurement consisting of 34 items was established, applicable to various cancer patients across countries. This CAT measurement system will facilitate precise and efficient assessment of HRQOL of cancer patients, without loss of comparability of results. [ABSTRACT FROM AUTHOR]

In this article, we propose a simplified version of the maximum information per time unit method (MIT; Fan, Wang, Chang, & Douglas, Journal of Educational and Behavioral Statistics 37: 655-670, 2012), or MIT-S, for computerized adaptive testing. Unlike the original MIT method, the proposed MIT-S method does not require fitting a response time model to the individual-level response time data. It is also computationally efficient. The performance of the MIT-S method was compared against that of the maximum information (MI) method in terms of measurement precision, testing time saving, and item pool usage under various item response theory (IRT) models. The results indicated that when the underlying IRT model is the two- or three-parameter logistic model, the MIT-S method maintains measurement precision and saves testing time. It performs similarly to the MI method in exposure control; both result in highly skewed item exposure distributions, due to heavy reliance on the highly discriminating items. If the underlying model is the one-parameter logistic (1PL) model, the MIT-S method maintains the measurement precision and saves a considerable amount of testing time. However, its heavy reliance on time-saving items leads to a highly skewed item exposure distribution. This weakness can be ameliorated by using randomesque exposure control, which successfully balances the item pool usage. Overall, the MIT-S method with randomesque exposure control is recommended for achieving better testing efficiency while maintaining measurement precision and balanced item pool usage when the underlying IRT model is 1PL. [ABSTRACT FROM AUTHOR]

Computerized Adaptive Testing (CAT) is a field of research originating in psychometrics that has been adopted in the last years for formative and summative assessment activities in educational settings in general and in online learning in particular. While a variety of platforms is available for designing and deploying CAT the challenge of providing the flexibility in test and item design required for domain-specific assessment formats in education has hardly been addressed so far. The present article introduces a software architecture to fill this gap and enable the development of fully customizable CAT tools with respect to domain-specific item design and visualization as well as deployed CAT algorithms. A prototypical implementation of the architecture and a set of domain-specific item types are presented to demonstrate the feasibility of the proposed approach and outline future directions of development and research. [ABSTRACT FROM AUTHOR]

We assessed differential item functioning (DIF) based on computerized adaptive testing (CAT) to examine how perinatal mood disorders differ from adult psychiatric disorders. The CAT-Mental Health (CAT-MH) was administered to 1614 adult psychiatric outpatients and 419 perinatal women with IRB approval. We examined individual item-level differences using logistic regression and overall score differences by scoring the perinatal data using the original bifactor model calibration based on the psychiatric sample data and a new bifactor model calibration based on the perinatal data and computing their correlation. To examine convergent validity, we computed correlations of the CAT-MH with contemporaneously administered Edinburgh Postnatal Depression Scales (EPDS). The rate of major depression in the perinatal sample was 13 %. Rates of anxiety, mania, and suicide risk were 5, 6, and 0.4 %, respectively. One of 66 depression items, one of 69 anxiety items, and 15 of 53 mania items exhibited DIF (i.e., failure to discriminate between high and low levels of the disorder) in the perinatal sample based on the psychiatric sample calibration. Removal of these items resulted in correlations of the original and perinatal calibrations of r = 0.983 for depression, r = 0.986 for anxiety, and r = 0.932 for mania. The 91.3 % of cases were concordantly categorized as either 'at-risk' or 'low-risk' between the EPDS and the perinatal calibration of the CAT-MH. There was little evidence of DIF for depression and anxiety symptoms in perinatal women. This was not true for mania. Now calibrated for perinatal women, the CAT-MH can be evaluated for longitudinal symptom monitoring. [ABSTRACT FROM AUTHOR]

Background: An innovative measurement system using a computerized adaptive testing technique based on the item response theory (CAT) has been expanding to measure mental health status. However, little is known about details in its measurement properties based on the empirical data. Moreover, the response time (RT) data, which are not available by a paper-and-pencil measurement but available by a computerized measurement, would be worth investigating for exploring the response behavior. Purpose: We aimed at constructing the CAT to measure depressive symptomatology in a community population and exploring its measurement properties. Also, we examined the relationships between RTs, individual item responses, and depressive levels. Method: For constructing the CAT system, responses of 2061 workers and university students to 24 depression scale plus four negatively revised positive affect items were subjected to a polytomous IRT analysis. The stopping rule was set for standard error of estimation < 0.30 or the maximum 15 items displayed. The CAT and non-adaptive computer-based test (CBT) were administered to 209 undergraduates, and 168 of them administered again after 1 week. Results: On average, the CAT was converged by 10.4 items. The θ values estimated by CAT and CBT were highly correlated ( r = 0.94 and 0.95 for the 1st and 2nd measurements) and with the traditional scoring procedures ( r's > 0.90). The test-retest reliability was at a satisfactory level ( r = 0.86). RTs to some items significantly correlated with the θ estimates. The mean RT varied by the item contents and wording, i.e., the RT to positive affect items required additional 2 s or longer than the other subscale items. Conclusion: The CAT would be a reliable and practical measurement tool for various purposes including stress check at workplace. [ABSTRACT FROM AUTHOR]

Purpose: Goldberg's General Health Questionnaire (GHQ) items are frequently used to assess psychological distress but no study to date has investigated the GHQ-30's potential for adaptive administration. In computerized adaptive testing (CAT) items are matched optimally to the targeted distress level of respondents instead of relying on fixed-length versions of instruments. We therefore calibrate GHQ-30 items and report a simulation study exploring the potential of this instrument for adaptive administration in a longitudinal setting.Methods: GHQ-30 responses of 3445 participants with 2 completed assessments (baseline, 7-year follow-up) in the UK Health and Lifestyle Survey were calibrated using item response theory. Our simulation study evaluated the efficiency of CAT administration of the items, cross-sectionally and longitudinally, with different estimators, item selection methods, and measurement precision criteria.Results: To yield accurate distress measurements (marginal reliability at least 0.90) nearly all GHQ-30 items need to be administered to most survey respondents in general population samples. When lower accuracy is permissible (marginal reliability of 0.80), adaptive administration saves approximately 2/3 of the items. For longitudinal applications, change scores based on the complete set of GHQ-30 items correlate highly with change scores from adaptive administrations.Conclusions: The rationale for CAT-GHQ-30 is only supported when the required marginal reliability is lower than 0.9, which is most likely to be the case in cross-sectional and longitudinal studies assessing mean changes in populations. Precise measurement of psychological distress at the individual level can be achieved, but requires the deployment of all 30 items. [ABSTRACT FROM AUTHOR]

This article describes the evolution and current state of the domain-independent Siette assessment environment. Siette supports different assessment methods-including classical test theory, item response theory, and computer adaptive testing-and integrates them with multidimensional student models used by intelligent educational systems. Teachers can use an authoring tool to create large item pools of different types of questions, including multiple choice, open answer, generative questions, and complex tasks. Siette can be used for formative and summative assessment and incorporates different learning elements, including scaffolding features, such as hints, feedback, and misconceptions. It includes numerous other features covering different educational needs and techniques, such as spaced repetition, collaborative testing, or pervasive learning. Siette is designed as a web-based assessment component that can be semantically integrated with intelligent systems or with large LMSs, such as Moodle. This article reviews the evolution of the Siette system, presents information on its use, and analyses this information from a broader and critical perspective on the use of intelligent systems in education. [ABSTRACT FROM AUTHOR]

Interest in just-in-time adaptive interventions (JITAI) has rapidly increased in recent years. One core challenge for JITAI is the efficient and precise measurement of tailoring variables that are used to inform the timing of momentary intervention delivery. Ecological momentary assessment (EMA) is often used for this purpose, even though EMA in its traditional form was not designed specifically to facilitate momentary interventions. In this article, we introduce just-in-time adaptive EMA (JITA-EMA) as a strategy to reduce participant response burden and decrease measurement error when EMA is used as a tailoring variable in JITAI. JITA-EMA builds on computerized adaptive testing methods developed for purposes of classification (computerized classification testing, CCT), and applies them to the classification of momentary states within individuals. The goal of JITA-EMA is to administer a small and informative selection of EMA questions needed to accurately classify an individual’s current state at each measurement occasion. After illustrating the basic components of JITA-EMA (adaptively choosing the initial and subsequent items to administer, adaptively stopping item administration, accommodating dynamically tailored classification cutoffs), we present two simulation studies that explored the performance of JITA-EMA, using the example of momentary fatigue states. Compared with conventional EMA item selection methods that administered a fixed set of questions at each moment, JITA-EMA yielded more accurate momentary classification with fewer questions administered. Our results suggest that JITA-EMA has the potential to enhance some approaches to mobile health interventions by facilitating efficient and precise identification of momentary states that may inform intervention tailoring. [ABSTRACT FROM AUTHOR]

Objective: Quality of life (QoL) measurements are considered important outcome measures both for research on multiple sclerosis (MS) and in clinical practice. Computerized adaptive testing (CAT) can improve the precision of measurements made using QoL instruments while reducing the burden of testing on patients. Moreover, a cross-cultural approach is also necessary to guarantee the wide applicability of CAT. The aim of this preliminary study was to develop a calibrated item bank that is available in multiple languages and measures QoL related to mental health by combining one generic (SF-36) and one disease-specific questionnaire (MusiQoL). Methods: Patients with MS were enrolled in this international, multicenter, cross-sectional study. The psychometric properties of the item bank were based on classical test and item response theories and approaches, including the evaluation of unidimensionality, item response theory model fitting, and analyses of differential item functioning (DIF). Convergent and discriminant validities of the item bank were examined according to socio-demographic, clinical, and QoL features. Results: A total of 1992 patients with MS and from 15 countries were enrolled in this study to calibrate the 22-item bank developed in this study. The strict monotonicity of the Cronbach's alpha curve, the high eigenvalue ratio estimator (5.50), and the adequate CFA model fit (RMSEA = 0.07 and CFI = 0.95) indicated that a strong assumption of unidimensionality was warranted. The infit mean square statistic ranged from 0.76 to 1.27, indicating a satisfactory item fit. DIF analyses revealed no item biases across geographical areas, confirming the cross-cultural equivalence of the item bank. External validity testing revealed that the item bank scores correlated significantly with QoL scores but also showed discriminant validity for socio-demographic and clinical characteristics. Conclusion: This work demonstrated satisfactory psychometric characteristics for a QoL item bank for MS in multiple languages. This work may offer a common measure for the assessment of QoL in different cultural contexts and for international studies conducted on MS. [ABSTRACT FROM AUTHOR]

Objectives: The objectives of the present study are to investigate the precision of static (fixed-length) short forms versus computerized adaptive testing (CAT) administration, response pattern scoring versus summed score conversion, and test-retest reliability (stability) of the Patient-Reported Outcomes Measurement Information System (PROMIS) pediatric self-report scales measuring the latent constructs of depressive symptoms, anxiety, anger, pain interference, peer relationships, fatigue, mobility, upper extremity functioning, and asthma impact with polytomous items. Methods: Participants ( N = 331) between the ages of 8 and 17 were recruited from outpatient general pediatrics and subspecialty clinics. Of the 331 participants, 137 were diagnosed with asthma. Three scores based on item response theory (IRT) were computed for each respondent: CAT response pattern expected a posteriori estimates, short-form response pattern expected a posteriori estimates, and short-form summed score expected a posteriori estimates. Scores were also compared between participants with and without asthma. To examine test-retest reliability, 54 children were selected for retesting approximately 2 weeks after the first assessment. Results: A short CAT (maximum 12 items with a standard error of 0.4) was found, on average, to be less precise than the static short forms. The CAT appears to have limited usefulness over and above what can be accomplished with the existing static short forms (8-10 items). Stability of the scale scores over a 2-week period was generally supported. Conclusion: The study provides further information on the psychometric properties of the PROMIS pediatric scales and extends the previous IRT analyses to include precision estimates of dynamic versus static administration, test-retest reliability, and validity of administration across groups. Both the positive and negative aspects of using CAT versus short forms are highlighted. [ABSTRACT FROM AUTHOR]

Objectives: To develop a paper-and-pencil semi-adaptive test for 5 domains of health-related quality of life (PAT-5D-QOL) based on item response theory (IRT). Methods: The questionnaire uses items from previously developed item banks for 5 domains: (1) walking, (2) handling objects, (3) daily activities, (4) pain or discomfort, and (5) feelings. For each domain, respondents are initially classified into 4 functional levels. Depending on the level, they are instructed to respond to a different set of 5 additional questions. IRT scores for each domain and overall health utility scores are obtained using a simple spreadsheet. The questions were selected using psychometric and conceptual criteria. The format of the questionnaire was developed through focus groups and cognitive interviews. Feasibility was tested in two population surveys. A simulation study was conducted to compare PAT-5D-QOL with a computerized adaptive test (CAT-5D-QOL) and a fixed questionnaire, developed from the same item banks, in terms of accuracy, bias, precision, and ceiling and floor effects. Results: Close to 90 % of the participants in feasibility studies followed the skip instructions properly. In a simulation study, scores on PAT-5D-QOL for all domains tended to be more accurate, more precise, less biased, and less affected by a ceiling effect than scores on a fixed IRT-based questionnaire of the same length. PAT-5D-QOL was slightly inferior to a fully adaptive instrument. Conclusions: PAT-5D-QOL is a novel, semi-adaptive, IRT-based measure of health-related quality of life with a broad range of potential applications. [ABSTRACT FROM AUTHOR]

Purpose: Most multidimensional patient-reported outcomes (PRO) measures are lengthy to complete. Computerized adaptive testing (CAT) that selects the most informative items can potentially reduce respondent burden without sacrificing measurement accuracy. The commonly used maximum Fisher information item selection method has been reported to lead to highly unbalanced item bank usage and potentially imprecise trait estimation. This study employs the content-balancing strategy in a bifactor-modeled CAT item selection and examines its impact on measurement accuracy and item bank usage. Methods: Item responses from a population-based SF-36 survey were first calibrated using the bifactor graded response model. Four post hoc CATs using items and responses from the SF-36 data set were then created. The content-balancing strategy was adopted in the item selection procedure of the bifactor-modeled CAT. The measurement accuracy and usage of items of the CAT were compared between the tests with and without the content-balancing strategy. Results: The results indicate that the CAT implemented with the content-balancing strategy offers a better overall measurement accuracy of both the general health status and the two health domains (physical and mental) of the SF-36. Conclusions: The content-balancing strategy helps the CAT-PRO to balance the selection of items and achieve improved measurement accuracy. Its implementation in real-time CAT administration to measure multidimensional PRO traits merits further studies. [ABSTRACT FROM AUTHOR]

In this article, four item selection methods in computerized adaptive testing are examined in terms of classification accuracy and consistency, including two popular heuristics for constraint management, the maximum priority index (MPI) method and the weighted deviation modeling method, as well as the widely known maximum Fisher information method and randomized item selection as baselines. Results suggest that the MPI method is able to meet constraints and keep test overlap rate low. Among the four methods, it is the only one that manages to produce parallel forms in terms of content coverage and, consequently, the only method to which the idea of classification consistency applies. With tests as short as 12 items, the MPI method does fairly well in classifying examinees accurately and consistently. Its performance improves with longer tests. The effects of number of decision categories and cut score locations are also examined. Recommendations are made in the Discussion section. [ABSTRACT FROM AUTHOR]

Purpose: We provide detailed instructions for analyzing patient-reported outcome (PRO) data collected with an existing (legacy) instrument so that scores can be calibrated to the PRO Measurement Information System (PROMIS) metric. This calibration facilitates migration to computerized adaptive test (CAT) PROMIS data collection, while facilitating research using historical legacy data alongside new PROMIS data. Methods: A cross-sectional convenience sample ( n = 2,178) from the Universities of Washington and Alabama at Birmingham HIV clinics completed the PROMIS short form and Patient Health Questionnaire (PHQ-9) depression symptom measures between August 2008 and December 2009. We calibrated the tests using item response theory. We compared measurement precision of the PHQ-9, the PROMIS short form, and simulated PROMIS CAT. Results: Dimensionality analyses confirmed the PHQ-9 could be calibrated to the PROMIS metric. We provide code used to score the PHQ-9 on the PROMIS metric. The mean standard errors of measurement were 0.49 for the PHQ-9, 0.35 for the PROMIS short form, and 0.37, 0.28, and 0.27 for 3-, 8-, and 9-item-simulated CATs. Conclusions: The strategy described here facilitated migration from a fixed-format legacy scale to PROMIS CAT administration and may be useful in other settings. [ABSTRACT FROM AUTHOR]

We examined the presence and impact of differential item functioning (DIF) in a set of knee-specific functional status (FS) items administered using computerized adaptive testing (CAT) among English (United States) and Hebrew (Israel) speaking patients receiving therapy for knee impairments. DIF occurs in an item if probabilities of endorsing responses differ across groups after controlling for the FS measured. We analyzed data from 28,320 patients (14,160 U.S., 14,160 Israel) who completed the knee-specific CAT. Items were assessed for DIF by gender, age, symptom acuity, surgical history, exercise history, and language spoken using a hybrid technique that combines multiple ordinal logistic regression and item response theory FS estimates. Several items had non-uniform DIF for covariates including language, but unadjusted and DIF-adjusted functional status estimates were in strong concordance [ICC(2,1) values ≥0.97], and differences between unadjusted and adjusted FS scores represented <0.4% of the unadjusted FS standard deviation. Statistically significant DIF was identified in some items but represented negligible clinical impact. Results suggested no need to adjust items for DIF when assessing FS outcomes across groups of patients with knee impairments who answer the knee CAT items in English in the United States or Hebrew in Israel. These findings suggest negligible differences in cultural perceptions between English and Hebrew wording of these knee-specific CAT FS items. [ABSTRACT FROM AUTHOR]

The purposes of this study were to apply a bi-factor model for the determination of test dimensionality and a multidimensional CAT using computer simulations of real data for the assessment of a new global physical health measure for children with cerebral palsy (CP). Parent respondents of 306 children with cerebral palsy were recruited from four pediatric rehabilitation hospitals and outpatient clinics. We compared confirmatory factor analysis results across four models: (1) one-factor unidimensional; (2) two-factor multidimensional (MIRT); (3) bi-factor MIRT with fixed slopes; and (4) bi-factor MIRT with varied slopes. We tested whether the general and content (fatigue and pain) person score estimates could discriminate across severity and types of CP, and whether score estimates from a simulated CAT were similar to estimates based on the total item bank, and whether they correlated as expected with external measures. Confirmatory factor analysis suggested separate pain and fatigue sub-factors; all 37 items were retained in the analyses. From the bi-factor MIRT model with fixed slopes, the full item bank scores discriminated across levels of severity and types of CP, and compared favorably to external instruments. CAT scores based on 10- and 15-item versions accurately captured the global physical health scores. The bi-factor MIRT CAT application, especially the 10- and 15-item versions, yielded accurate global physical health scores that discriminated across known severity groups and types of CP, and correlated as expected with concurrent measures. The CATs have potential for collecting complex data on the physical health of children with CP in an efficient manner. [ABSTRACT FROM AUTHOR]

We tested the item response theory (IRT) model assumptions of the original item bank, and evaluated the practical and psychometric adequacy, of a computerized adaptive test (CAT) for patients with foot or ankle impairments seeking rehabilitation in outpatient therapy clinics. Data from 10,287 patients with foot or ankle impairments receiving outpatient physical therapy were analyzed. We first examined the unidimensionality, fit, and invariance IRT assumptions of the CAT item bank. Then we evaluated the efficiency of the CAT administration and construct validity and sensitivity of change of the foot/ankle CAT measure of lower-extremity functional status (FS). Results supported unidimensionality, model fit, and invariance of item parameters and patient ability estimates. On average, the CAT used seven items to produce precise estimates of FS that adequately covered the content range with negligible floor and ceiling effects. Patients who were older, had more chronic symptoms, had more surgeries, had more comorbidities, and did not exercise prior to receiving rehabilitation reported worse discharge FS. Seventy-one percent of patients obtained statistically significant change at follow-up. Change of 8 FS units (scale 0–100) represented minimal clinically important improvement. We concluded that the foot/ankle item bank met IRT assumptions and that the CAT FS measure was precise, valid, and responsive, supporting its use in routine clinical application. [ABSTRACT FROM AUTHOR]

Item banks and Computerized Adaptive Testing (CAT) have the potential to greatly improve the assessment of health outcomes. This review describes the unique features of item banks and CAT and discusses how to develop item banks. In CAT, a computer selects the items from an item bank that are most relevant for and informative about the particular respondent; thus optimizing test relevance and precision. Item response theory (IRT) provides the foundation for selecting the items that are most informative for the particular respondent and for scoring responses on a common metric. The development of an item bank is a multi-stage process that requires a clear definition of the construct to be measured, good items, a careful psychometric analysis of the items, and a clear specification of the final CAT. The psychometric analysis needs to evaluate the assumptions of the IRT model such as unidimensionality and local independence; that the items function the same way in different subgroups of the population; and that there is an adequate fit between the data and the chosen item response models. Also, interpretation guidelines need to be established to help the clinical application of the assessment. Although medical research can draw upon expertise from educational testing in the development of item banks and CAT, the medical field also encounters unique opportunities and challenges. [ABSTRACT FROM AUTHOR]

The use of item banks and computerized adaptive testing (CAT) begins with clear definitions of important outcomes, and references those definitions to specific questions gathered into large and well-studied pools, or “banks” of items. Items can be selected from the bank to form customized short scales, or can be administered in a sequence and length determined by a computer programmed for precision and clinical relevance. Although far from perfect, such item banks can form a common definition and understanding of human symptoms and functional problems such as fatigue, pain, depression, mobility, social function, sensory function, and many other health concepts that we can only measure by asking people directly. The support of the National Institutes of Health (NIH), as witnessed by its cooperative agreement with measurement experts through the NIH Roadmap Initiative known as PROMIS (www.nihpromis.org), is a big step in that direction. Our approach to item banking and CAT is practical; as focused on application as it is on science or theory. From a practical perspective, we frequently must decide whether to re-write and retest an item, add more items to fill gaps (often at the ceiling of the measure), re-test a bank after some modifications, or split up a bank into units that are more unidimensional, yet less clinically relevant or complete. These decisions are not easy, and yet they are rarely unforgiving. We encourage people to build practical tools that are capable of producing multiple short form measures and CAT administrations from common banks, and to further our understanding of these banks with various clinical populations and ages, so that with time the scores that emerge from these many activities begin to have not only a common metric and range, but a shared meaning and understanding across users. In this paper, we provide an overview of item banking and CAT, discuss our approach to item banking and its byproducts, describe testing options, discuss an example of CAT for fatigue, and discuss models for long term sustainability of an entity such as PROMIS. Some barriers to success include limitations in the methods themselves, controversies and disagreements across approaches, and end-user reluctance to move away from the familiar. [ABSTRACT FROM AUTHOR]

Successful integration of modern psychometrics and advanced informatics in patient-reported outcomes (PRO) measurement and management can potentially maximize the value of health outcomes research and optimize the delivery of quality patient care. Unlike the traditional labor-intensive paper-and-pencil data collection method, item response theory-based computerized adaptive testing methodologies coupled with novel technologies provide an integrated environment to collect, analyze and present ready-to-use PRO data for informed and shared decision-making. This article describes the needs, challenges and solutions for accurate, efficient and cost-effective PRO data acquisition and dissemination means in order to provide critical and timely PRO information necessary to actively support and enhance routine patient care in busy clinical settings. [ABSTRACT FROM AUTHOR]

Depression is one of the most prevalent mental health problems and measuring depressive symptoms becomes increasingly important in science as well as medical practice. Computer Adaptive Tests (CAT) based on the Item Response Theory (IRT) promise to enhance measurement precision and reduce respondent's burden. Our aim was to develop a CAT application to measure depressive symptoms. Three thousand two hundred seventy psychosomatic patients answered an overall of 11 mental health questionnaires at the University Clinic in Berlin. Three independent reviewers rated 144 items out of these questionnaires as indicative of depressive symptoms. All items underwent six empirical steps to analyze unidimensionality, local independence and item discrimination. Finally 64 items could be used to calculate item parameters applying a Generalized Partial Credit Model (GPCM). CAT scores were estimated using an 'expected a posteriori' algorithm (EAP). Two simulation experiments showed that for theta values within the range of 2SD around the mean (98% of the cases), the latent trait can be estimated out of approximately six items with a predefined standard error of [Symbol: see text] 0.32 (reliability rho [Symbol: see text] 0.90). The CAT-scores correlated high with scores of all depression items (r = 0.95), with the Beck Depression Inventory (r = 0.79) and with a CES-D 8 item short form (r = 0.76). We conclude that the Depression-CAT measures depressive symptoms with high precision and low respondent burden. [ABSTRACT FROM AUTHOR]

Background: Measurement of headache impact is important in clinical trials, case detection, and the clinical monitoring of patients. Computerized adaptive testing (CAT) of headache impact has potential advantages over traditional fixed-length tests in terms of precision, relevance, real-time quality control and flexibility.Objective: To develop an item pool that can be used for a computerized adaptive test of headache impact.Methods: We analyzed responses to four well-known tests of headache impact from a population-based sample of recent headache sufferers (n = 1016). We used confirmatory factor analysis for categorical data and analyses based on item response theory (IRT).Results: In factor analyses, we found very high correlations between the factors hypothesized by the original test constructers, both within and between the original questionnaires. These results suggest that a single score of headache impact is sufficient. We established a pool of 47 items which fitted the generalized partial credit IRT model. By simulating a computerized adaptive health test we showed that an adaptive test of only five items had a very high concordance with the score based on all items and that different worst-case item selection scenarios did not lead to bias.Conclusion: We have established a headache impact item pool that can be used in CAT of headache impact. [ABSTRACT FROM AUTHOR]