Search

The COVID-19 pandemic has had a significant impact on high-stakes testing, including the national benchmark tests in South Africa. Current linear testing formats have been criticized for their limitations, leading to a shift towards Computerized Adaptive Testing [CAT]. Assessments with CAT are more precise and take less time. Evaluation of CAT programs requires simulation studies. To assess the feasibility of implementing CAT in NBTs, SimulCAT, a simulation tool, was utilized. The SimulCAT simulation involved creating 10,000 examinees with a normal distribution characterized by a mean of 0 and a standard deviation of 1. A pool of 500 test items was employed, and specific parameters were established for the item selection algorithm, CAT administration rules, item exposure control, and termination criteria. The termination criteria required a standard error of less than 0.35 to ensure accurate abilities estimation. The findings from the simulation study demonstrated that fixed-length tests provided higher testing precision without any systematic error, as indicated by measurement statistics like CBIAS, CMAE, and CRMSE. However, fixed-length tests exhibited a higher item exposure rate, which could be mitigated by selecting items with fewer dependencies on specific item parameters (a-parameters). On the other hand, variable-length tests demonstrated increased redundancy. Based on these results, CAT is recommended as an alternative approach for conducting NBTs due to its capability to accurately measure individual abilities and reduce the testing duration. For high-stakes assessments like the NBTs, fixed-length tests are preferred as they offer superior testing precision while minimizing item exposure rates.

A modified and improved inductive inferential approach to evaluate item discriminations in a conditional maximum likelihood and Rasch modeling framework is suggested. The new approach involves the derivation of four hypothesis tests. It implies a linear restriction of the assumed set of probability distributions in the classical approach that represents scenarios of different item discriminations in a straightforward and efficient manner. Its improvement is discussed, compared to classical procedures (tests and information criteria), and illustrated in Monte Carlo experiments as well as real data examples from educational research. The results show an improvement of power of the modified tests of up to 0.3.

Given the frequent presence of slipping and guessing in item responses, models for the inclusion of their effects are highly important. Unfortunately, the most common model for their inclusion, the four-parameter item response theory model, potentially has severe deficiencies related to its possible unidentifiability. With this issue in mind, the dyad four-parameter normal ogive (Dyad-4PNO) model was developed. This model allows for slipping and guessing effects by including binary augmented variables--each indicated by two items whose probabilities are determined by slipping and guessing parameters--which are subsequently related to a continuous latent trait through a two-parameter model. Furthermore, the Dyad-4PNO assumes uncertainty as to which items are paired on each augmented variable. In this way, the model is inherently exploratory. In the current article, the new model, called the Set-4PNO model, is an extension of the Dyad-4PNO in two ways. First, the new model allows for more than two items per augmented variable. Second, these item sets are assumed to be fixed, that is, the model is confirmatory. This article discusses this extension and introduces a Gibbs sampling algorithm to estimate the model. A Monte Carlo simulation study shows the efficacy of the algorithm at estimating the model parameters. A real data example shows that this extension may be viable in practice, with the data fitting a more general Set-4PNO model (i.e., more than two items per augmented variable) better than the Dyad-4PNO, 2PNO, 3PNO, and 4PNO models.

Evaluating scale comparability in international large-scale assessments depends on measurement invariance (MI). The root mean square deviation (RMSD) is a standard method for establishing MI in several programs, such as the Programme for International Student Assessment and the Programme for the International Assessment of Adult Competencies. Previous research showed that the RMSD was unable to detect departures from MI when the latent trait distribution was far from item difficulty. In this study, we developed three alternative approaches to the original RMSD: equal, item information, and b-norm weighted RMSDs. Specifically, we considered the item-centered normalized weight distributions to compute the item characteristic curve difference in the RMSD procedure more efficiently. We further compared all methods' performance via a simulation study and the item information and b-norm weighted RMSDs showed the most promising results. An empirical example is demonstrated, and implications for researchers are discussed.

A Monte Carlo simulation study was conducted to compare fit indices used for detecting the correct latent class in three dichotomous mixture item response theory (IRT) models. Ten indices were considered: Akaike's information criterion (AIC), the corrected AIC (AICc), Bayesian information criterion (BIC), consistent AIC (CAIC), Draper's information criterion (DIC), sample size adjusted BIC (SABIC), relative entropy, the integrated classification likelihood criterion (ICL-BIC), the adjusted Lo-Mendell-Rubin (LMR), and Vuong-Lo-Mendell-Rubin (VLMR). The accuracy of the fit indices was assessed for correct detection of the number of latent classes for different simulation conditions including sample size (2,500 and 5,000), test length (15, 30, and 45), mixture proportions (equal and unequal), number of latent classes (2, 3, and 4), and latent class separation (no-separation and small separation). Simulation study results indicated that as the number of examinees or number of items increased, correct identification rates also increased for most of the indices. Correct identification rates by the different fit indices, however, decreased as the number of estimated latent classes or parameters (i.e., model complexity) increased. Results were good for BIC, CAIC, DIC, SABIC, ICL-BIC, LMR, and VLMR, and the relative entropy index tended to select correct models most of the time. Consistent with previous studies, AIC and AICc showed poor performance. Most of these indices had limited utility for three-class and four-class mixture 3PL model conditions.

Forced-choice (FC) measures have been widely used in many personality or attitude tests as an alternative to rating scales, which employ comparative rather than absolute judgments. Several response biases, such as social desirability, response styles, and acquiescence bias, can be reduced effectively. Another type of data linked with comparative judgments is response time (RT), which contains potential information concerning respondents' decision-making process. It would be challenging but exciting to combine RT into FC measures better to reveal respondents' behaviors or preferences in personality measurement. Given this situation, this study aims to propose a new item response theory (IRT) model that incorporates RT into FC measures to improve personality assessment. Simulation studies show that the proposed model can effectively improve the estimation accuracy of personality traits with the ancillary information contained in RT. Also, an application on a real data set reveals that the proposed model estimates similar but different parameter values compared with the conventional Thurstonian IRT model. The RT information can explain these differences.

Likert-type item is the most popular response format for collecting data in social, educational, and psychological studies through scales or questionnaires. However, there is no consensus on whether parametric or non-parametric tests should be preferred when analyzing Likert-type data. This study examined the statistical power of parametric and non-parametric tests when each Likert-type item was analyzed independently in survey studies. The main purpose of the study is to examine the statistical power of Wilcoxon-Mann-Whitney, Welch's t, and Student's t tests for Likert-type data, which are pairwise comparison tests. For this purpose, a Monte Carlo simulation study was conducted. The statistical significance of the selected tests was examined under the conditions of sample size, group size ratio, and effect size. The results showed that the Wilcoxon-Mann-Whitney test was superior to its counterparts, especially for small samples and unequal group sizes. However, the Student's t-test for Likert-type data had similar statistical power to the Wilcoxon-Mann-Whitney test under conditions of equal group sizes when the sample size was 200 or more. Consistent with the empirical results, practical recommendations were provided for researchers on what to consider when collecting and analyzing Likert-type data.

To ensure the validity of the tests is to check that all items have similar results across different groups of individuals. However, differential item functioning (DIF) occurs when the results of individuals with equal ability levels from different groups differ from each other on the same test item. Based on Item Response Theory and Classic Test Theory, there are some methods, with different advantages and limitations to identify items that show DIF. This study aims to compare the performances of five methods for detecting DIF. The efficacies of Mantel-Haenszel (MH), Logistic Regression (LR), Crossing simultaneous item bias test (CSIBTEST), Lord's chi-square (LORD), and Raju's area measure (RAJU) methods are examined considering conditions of the sample size, DIF ratio, and test length. In this study, to compare the detection methods, power and Type I error rates are evaluated using a simulation study with 100 replications conducted for each condition. Results show that LR and MH have the lowest Type I error and the highest power rate in detecting uniform DIF. In addition, CSIBTEST has a similar power rate to MH and LR. Under DIF conditions, sample size, DIF ratio, test length and their interactions affect Type I error and power rates.

Among all, Cronbach's Alpha and McDonald's Omega are commonly used for reliability estimations. The alpha uses inter-item correlations while omega is based on a factor analysis result. This study uses simulated ordinal data sets to test whether the alpha and omega produce different estimates. Their performances were compared according to the sample size, number of items, and deviance from tau equivalence. Based on the result, the alpha and omega had similar results, except for the small sample size, the smaller number of items, and the low factor loading values. When there were 5 or more items in the scale and factor analysis which the omega was calculated from showed fit to the data set, using omega over alpha could be preferred. Also, as the number of items exceeds 5, the alpha and omega differences disappear. Since calculating the alpha is easier compared to the omega (omega requires fitting a factor model first) using alpha over omega can also be suggested. However, when the number of items and the correlations among the items were small, omega performed worse than alpha. Therefore, alpha should be used for the reliability estimations.

This article introduces an innovative sampling scheme, the median sampling (MS), utilizing individual observations over time to efficiently estimate the mean of a process characterized by a symmetric (non-uniform) probability distribution. The mean estimator based on MS is not only unbiased but also boasts enhanced precision compared to its simple random sampling-based counterpart. Moreover, a new EWMA chart based on the mean estimator within the MS scheme is proposed. The performance of the EWMA charts, derived from both simple random sampling (SRS) and MS schemes, is evaluated using the metrics of steady-state average run-length and average number of items-to-signal. The findings underscore the superiority of the EWMA-MS chart over the EWMA-SRS chart. Additionally, as the magnitude of ranking errors escalates, the behavior of the EWMA-MS chart converges toward that of the EWMA-SRS chart. The practical implementation of the newly introduced EWMA chart is demonstrated through an illustrative example.

Multisite cluster randomized trials (MCRTs), in which, the intermediate-level clusters (e.g., classrooms) are randomly assigned to the treatment or control condition within each site (e.g., school), are among the most commonly used experimental designs across a broad range of disciplines. MCRTs often align with the theory that programs are delivered at a cluster-level (e.g., teacher professional development) and provide opportunities to explore treatment effect heterogeneity across sites. In designing experimental studies, a critical step is the statistical power analysis and sample size determination. However, the statistical tools for power analysis of moderator effects in three-level MCRTs are not available. In this study, we derived formulas for calculating the statistical power and the minimum detectable effect size difference (MDESD) with confidence intervals for investigating the effects of various moderators in three-level MCRTs. We considered the levels of the moderators (level-1, -2, and -3), the scales of the moderators (binary and continuous), and random and nonrandomly varying slopes of the (moderated) treatment effects. We validated our formulas through Monte Carlo simulations. Finally, we conclude with directions for future work.

We have entered an era in which scientific evidence increasingly informs research practice and policy. As there is an exponential increase in the use of single-case experimental designs (SCEDs) to evaluate intervention effectiveness, there is accumulating evidence available for quantitative synthesis. Consequently, there is a growing interest in techniques suitable to meta-analyze SCED research. One technique that can be applied is individual patient data (IPD) meta-analysis. IPD is a flexible approach, allowing for a variety of modeling options such as modeling moderators to explain intervention heterogeneity. To date, no methodological research has been conducted to evaluate the statistical properties of effect estimates obtained by using IPD meta-analysis with the inclusion of moderators. This study is designed to address this by conducting a large-scale Monte Carlo study. Based on the results, specific recommendations are provided to indicate under which conditions the IPD meta-analysis including moderators is suitable.

Economic efficiency demands accurate assessment of individual ability for selection purposes. This study investigates Classical Test Theory (CTT) and Item Response Theory (IRT) for estimating true ability and ranking individuals. Two Monte Carlo simulations and real data analyses were conducted. Results suggest a slight advantage for IRT, but ability estimates from both methods were highly correlated (r=0.95), indicating similar outcomes. The Logistic two-parameter IRT model emerged as the most reliable and rigorous approach.

In this study, the accuracy of the artificial neural network (ANN) was assessed considering the uncertainties associated with the randomness of the data and the lack of learning. The Monte-Carlo algorithm was applied to simulate the randomness of the input variables and evaluate the output distribution. It has been shown that under certain conditions, the GUM framework for uncertainty evaluation may completely fail. The ANN modeling technique can be used as an alternative method for estimating the expectation value and evaluating the associated uncertainty. Furthermore, unlike the GUM and Monte-Carlo frameworks, the ANN models do not require mathematical expressions between the input and output variables. On the other hand, owing to the uncertainty associated with the lack of learning, the ANN model may produce unrealistic results, even if a global minimum is approached. This behavior is explained by Bayesian theory which assumes that the output values generated by various runs are normally distributed at each target. This may lead to an unrealistic output when the overall distribution of the target values has a different distribution than that presumed by Bayesian theory. To minimize this drawback, the ANN model output should be calculated from sufficiently large repeated runs with new starting values of the weights and biases.

The testlet design is very popular in educational and psychological assessments. This article proposes a new cognitive diagnosis model, the multiple-choice cognitive diagnostic testlet (MC-CDT) model for tests using testlets consisting of MC items. The MC-CDT model uses the original examinees' responses to MC items instead of dichotomously scored data (i.e., correct or incorrect) to retain information of different distractors and thus enhance the MC items' diagnostic power. The Markov chain Monte Carlo algorithm was adopted to calibrate the model using the WinBUGS software. Then, a thorough simulation study was conducted to evaluate the estimation accuracy for both item and examinee parameters in the MC-CDT model under various conditions. The results showed that the proposed MC-CDT model outperformed the traditional MC cognitive diagnostic model. Specifically, the MC-CDT model fits the testlet data better than the traditional model, while also fitting the data without testlets well. The findings of this empirical study show that the MC-CDT model fits real data better than the traditional model and that it can also provide testlet information.

Comprehensive evaluation of treatment effects is aided by considerations for moderated effects. In educational research, the combination of natural hierarchical structures and prevalence of group-administered or shared facilitator treatments often produces three-level partially nested data structures. Literature details planning strategies for a variety of experimental designs when moderation effects are of interest but has yet to establish power formulas for detecting moderation effects in three-level partially nested designs. To address this gap, we derive and assess the accuracy of power formulas for detecting the different types of moderation effects possible in these designs. Using Monte Carlo simulation studies, we probe power rates and adequate sample sizes for detecting the different moderation effects while varying common influential factors including variance in the outcome explained by covariates, magnitude of the moderation effect, and sample sizes. The power formulas developed improve the planning of experimental studies with partial nesting and encourage the inclusion of moderator variables to capture for whom and under what conditions a treatment is effective. Educational researchers also have some initial guidance regarding adequate sample sizes and the factors that influence detecting moderation effects in three-level partially nested designs.

This study builds on the simulation framework of a recent paper by Stanley and Doucouliagos ("Research Synthesis Methods" 2023;14;515--519). S&D use simulations to make the argument that meta-analyses using partial correlation coefficients (PCCs) should employ a "suboptimal" estimator of the PCC standard error when constructing weights for fixed effect and random effects estimation. We address concerns that their simulations and subsequent recommendation may give meta-analysts a misleading impression. While the estimator they promote dominates the "correct" formula in their Monte Carlo framework, there are other estimators that perform even better. We conclude that more research is needed before best practice recommendations can be made for meta-analyses with PCCs.

Questionnaires are by far the most common tool for measuring noncognitive constructs in psychology and educational sciences. Response bias may pose an additional source of variation between respondents that threatens validity of conclusions drawn from questionnaire data. We present a mixture modeling approach that leverages response time data from computer-administered questionnaires for the joint identification and modeling of two commonly encountered response bias that, so far, have only been modeled separately--careless and insufficient effort responding and response styles (RS) in attentive answering. Using empirical data from the Programme for International Student Assessment 2015 background questionnaire and the case of extreme RS as an example, we illustrate how the proposed approach supports gaining a more nuanced understanding of response behavior as well as how neglecting either type of response bias may impact conclusions on respondents' content trait levels as well as on their displayed response behavior. We further contrast the proposed approach against a more heuristic two-step procedure that first eliminates presumed careless respondents from the data and subsequently applies model-based approaches accommodating RS. To investigate the trustworthiness of results obtained in the empirical application, we conduct a parameter recovery study.

We present a method for Bayesian structural equation modeling of sample correlation matrices as correlation structures. The method transforms the sample correlation matrix to an unbounded vector using the matrix logarithm function. Bayesian inference about the unbounded vector is performed assuming a multivariate-normal likelihood, with a mean based on the transformed model-implied correlation matrix, and a covariance assumed to be of known form. Using Monte Carlo simulation, we examine the performance of the method with normal and ordinal indicators, as well as the capacity of the method to estimate models that account for misspecification. The performance of the approach is often adequate suggesting that the proposed method can be used for Bayesian analysis of correlation structures. We conclude with a discussion of potential applications of the approach, as well as future directions needed to further develop the method.

The primary objective of this investigation is the formulation of random intercept latent profile transition analysis (RI-LPTA). Our simulation investigation suggests that the election between LPTA and RI-LPTA for examination has negligible impact on the estimation of transition probability parameters when the population parameters are generated based on the LPTA model. However, an estimation bias in the transition probabilities is observed in all simulated conditions when LPTA is used for analysis and the generated population parameters follow the RI-LPTA model. Researchers constructing empirical longitudinal models are advised to prioritize the use of RI-LPTA in model construction, regardless of whether LPTA exhibits random intercept effect. This measure helps to reduce the possible bias in the estimation parameters which may occur if the random intercept effect of LPTA is not taken into account during model specification.

Accelerated longitudinal designs (ALDs) provide an opportunity to capture long developmental periods in a shorter time framework using a relatively small number of assessments. Prior literature has investigated whether univariate developmental processes can be characterized with data obtained from ALDs. However, many important questions in psychology and related sciences imply working with several variables that are intercorrelated as they unfold over time, such as cognitive and cortical development. Therefore, bivariate developmental models are required. This study aimed to assess the effectiveness of continuous-time bivariate Latent Change Score (CT-BLCS) models for recovering the trajectories of two interdependent developmental processes using data from diverse ALDs. Through a Monte Carlo simulation study, the efficacy of different sampling designs and sample sizes was examined. The study fills a gap in the literature by examining the performance of ALDs in bivariate systems, providing specific recommendations for future application of ALDs for studying interrelated developmental variables.

The Bayesian piecewise growth model (PGM) is a useful class of models for analyzing nonlinear change processes that consist of distinct growth phases. In applications of Bayesian PGMs, it is important to accurately capture growth trajectories and carefully consider knot placements. The presence of missing data is another challenge researchers commonly encounter. To address these issues, one could use model fit and selection indices to detect misspecified Bayesian PGMs, and should give care to the potential impact of missing data on model evaluation. Here we conducted a simulation study to examine the impact of model misspecification and missing data on the performance of Bayesian model fit and selection indices (PPP-value, BCFI, BTLI, BRMSEA, BIC, and DIC), with an additional focus on prior sensitivity. Results indicated that (a) increasing the degree of model misspecification and amount of missing data aggravated the performance of indices in detecting misfit, and (b) different prior specifications had negligible impact on model assessment. We provide practical guidelines for researchers to facilitate effective implementation of Bayesian PGMs.

Measurement invariance holds when a latent construct is measured in the same way across different levels of background variables (continuous or categorical) while controlling for the true value of that construct. Using Monte Carlo simulation, this paper compares the multiple indicators, multiple causes (MIMIC) model and MIMIC-interaction to a novel use of alignment optimization (AO) for detecting measurement noninvariance when the violator is a continuous variable. Results showed that MIMIC and MIMIC-interaction in sequential likelihood ratio tests and Wald tests with a Bonferroni correction provided a good balance between identifying invariant and noninvariant (linear violations) items when n=500 in terms of classification accuracy (CA). AO (CA = 0.86) was as competitive as MIMIC and MIMIC-interaction to linear invariance violations but was far better under nonlinear quadratic violations when n= 1,000 (i.e., 100 per group for 10 groups).

Generalized structured component analysis (GSCA) is a multivariate method for specifying and examining interrelationships between observed variables and components. Despite its data-analytic flexibility honed over the decade, GSCA always defines every component as a linear function of observed variables, which can be less optimal when observed variables for a component are nonlinearly related, often reducing the component's predictive power. To address this issue, we combine deep learning and GSCA into a single framework to allow a component to be a nonlinear function of observed variables without specifying the exact functional form in advance. This new method, termed deep learning generalized structured component analysis (DL-GSCA), aims to maximize the predictive power of components while their directed or undirected network remains interpretable. Our real and simulated data analyses show that DL-GSCA produces components with greater predictive power than those from GSCA in the presence of nonlinear associations between observed variables per component.

Within the frequentist structural equation modeling (SEM) framework, adjudicating model quality through measures of fit has been an active area of methodological research. Complicating this conversation is research revealing that a higher quality measurement portion of a SEM can result in poorer estimates of overall model fit than lower quality measurement models, given the same structural misspecifications. Through population analysis and Monte Carlo simulation, we extend the earlier research to recently developed Bayesian SEM measures of fit to evaluate whether these indices are susceptible to the same reliability paradox, in the context of using both uninformative and informative priors. Our results show that the reliability paradox occurs for RMSEA, and to some extent, gamma-hat and PPP (measures of absolute fit); but not CFI or TLI (measures of relative fit), across Bayesian (MCMC) and frequentist (maximum likelihood) SEM frameworks alike. Taken together, these findings indicate that the behavior of these newly adapted Bayesian fit indices map closely to their frequentist analogs. Implications for their utility in identifying incorrectly specified models are discussed.

In Savalei's (2011) simulation that evaluated the performance of polychoric correlation estimates in small samples, two methods for treating zero-frequency cells, adding 0.5 (ADD) and doing nothing (NONE), were compared. Savalei tentatively suggested using ADD for binary data and NONE for data with three or more categories. Yet, Savalei's suggestion could be explained by the skewness of the data distribution being severe for binary data and slight for three-category data. To rule out this alternative explanation, we extended Savalei's design by incorporating the degree of skewness into our simulation. With slightly skewed data, NONE is recommended due to its high-quality estimates. With severely skewed data, only ADD is recommended for binary data when the skewness of two variables is the same-signed and the underlying correlation is expected to be strong. Methods for improving the polychoric correlation estimates with severely skewed data merit further study.

Oblique bifactor models, where group factors are allowed to correlate with one another, are commonly used. However, the lack of research on the statistical properties of oblique bifactor models renders the statistical validity of empirical findings questionable. Therefore, the present study took the first step to examine the statistical properties of oblique bifactor models through Monte Carlo simulations. Study 1 showed that the classic oblique bifactor "measurement" models had severe convergence issues in many conditions. Even for converged replications, both factor loading and group factor correlation estimates were severely biased. Study 2 further showed that the classic oblique bifactor "predictive" models still had serious convergence problems, and structural parameters suffered from issues of severe estimation bias and low power. Fortunately, the augmentation approach, where one or multiple indicators are specified to load onto only the general factor but not any of the group factors, was useful in ameliorating these issues in both oblique bifactor measurement and predictive models. Tentative recommendations regarding the selection between oblique and orthogonal bifactor models and the approaches to finding augmenting indicators were provided.

The logistic and confined exponential curves are frequently used in studies of growth and learning. These models, which are nonlinear in their parameters, can be estimated using structural equation modeling software. This paper proposes a single combined model, a weighted combination of both models. Mplus, Proc Calis, and lavaan code for the model are provided. Monte Carlo simulations varying the number of measurement occasions (5, 10, and 15), internal consistency ([alpha] = 0.5, 0.7, and 0.8), and sample size (N = 1,000, 500, and 300) were examined to understand whether the model can be successfully fit with SEM software. Convergence failures were appreciable when model parameters were equal to special cases of logistic or confined exponential curves. At least ten measurement occasions and a moderate degree of reliability ([alpha] > 0.7) were required to identify the model as superior to its stand-alone alternatives.

Latent class analysis (LCA) is a widely used technique for detecting unobserved population heterogeneity in cross-sectional data. Despite its popularity, the performance of LCA is not well understood. In this study, we evaluate the performance of LCA with binary data by examining classification accuracy, parameter estimation accuracy, and coverage rates of confidence intervals (CIs) through Monte Carlo simulation studies. We address the issue of label switching with a distance-based relabeling approach and introduce an index to measure separation among latent classes. Our results show that classification accuracy, parameter estimation accuracy, and CI coverage rates are primarily influenced by class separation and the number of indicators used for LCA. We recommend using a large sample size to mitigate the effects of tiny class sizes. Additionally, the study finds that the parametric bootstrap CIs perform comparably well or better when compared with the CIs based on the standard maximum likelihood method.

Parallel process latent growth curve mediation models (PP-LGCMMs) are frequently used to longitudinally investigate the mediation effects of treatment on the level and change of outcome through the level and change of mediator. An important but often violated assumption in empirical PP-LGCMM analysis is the absence of omitted confounders of the relationships among treatment, mediator, and outcome. In this study, we analytically examined how omitting pretreatment confounders impacts the inference of mediation from the PP-LGCMM. Using the analytical results, we developed three sensitivity analysis approaches for the PP-LGCMM, including the frequentist, Bayesian, and Monte Carlo approaches. The three approaches help investigate different questions regarding the robustness of mediation results from the PP-LGCMM, and handle the uncertainty in the sensitivity parameters differently. Applications of the three sensitivity analyses are illustrated using a real-data example. A user-friendly Shiny web application is developed to conduct the sensitivity analyses.

We consider the problem of bias arising from conditioning on a post-outcome collider. We illustrate this with reference to Elwert and Winship (2014) but we go beyond their study to investigate the extent to which inverse probability weighting might offer solutions. We use linear models to derive expressions for the bias arising in different kinds of post-outcome confounding, and we show the specific situations in which inverse probability weighting will allow us to obtain estimates that are consistent or, if not consistent, less biased than those obtained via ordinary least squares regression.

Purpose: Many studies using machine learning (ML) in speech, language, and hearing sciences rely upon cross-validations with single data splitting. This study's first purpose is to provide quantitative evidence that would incentivize researchers to instead use the more robust data splitting method of nested k-fold cross-validation. The second purpose is to present methods and MATLAB code to perform power analysis for ML-based analysis during the design of a study. Method: First, the significant impact of different cross-validations on ML outcomes was demonstrated using real-world clinical data. Then, Monte Carlo simulations were used to quantify the interactions among the employed cross-validation method, the discriminative power of features, the dimensionality of the feature space, the dimensionality of the model, and the sample size. Four different cross-validation methods (single holdout, 10-fold, train-validation-test, and nested 10-fold) were compared based on the statistical power and confidence of the resulting ML models. Distributions of the null and alternative hypotheses were used to determine the minimum required sample size for obtaining a statistically significant outcome (5% significance) with 80% power. Statistical confidence of the model was defined as the probability of correct features being selected for inclusion in the final model. Results: ML models generated based on the single holdout method had very low statistical power and confidence, leading to overestimation of classification accuracy. Conversely, the nested 10-fold cross-validation method resulted in the highest statistical confidence and power while also providing an unbiased estimate of accuracy. The required sample size using the single holdout method could be 50% higher than what would be needed if nested k-fold cross-validation were used. Statistical confidence in the model based on nested k-fold cross-validation was as much as four times higher than the confidence obtained with the single holdout-based model. A computational model, MATLAB code, and lookup tables are provided to assist researchers with estimating the minimum sample size needed during study design. Conclusion: The adoption of nested k-fold cross-validation is critical for unbiased and robust ML studies in the speech, language, and hearing sciences.

In the current study, we compare propensity score (PS) matching methods for data with a cross-classified structure, where each individual is clustered within more than one group, but the groups are not hierarchically organized. Through a Monte Carlo simulation study, we compared sequential cluster matching (SCM), preferential within cluster matching (PWCM), greedy matching (GM), and optimal full matching (OFM), using propensity scores from four different models. The results indicated that the four matching methods performed well when PSs were estimated with logistic regression containing both level-1 and level-2 covariates. When the level-2 covariates were omitted in the logistic regression PS model, matching methods resulted in biased treatment effect estimates. However, omission of level-2 covariates did not result in biased estimates when the PS model was a logistic cross-classified random effects model (CCREM). SCM and PWCM outperformed GM and OFM with a logistic CCREM that included level-1 and level-2 covariates.

Parallel process latent growth curve mediation models (PP-LGCMMs) are frequently used to longitudinally investigate the mediation effects of treatment on the level and change of outcome through the level and change of mediator. An important but often violated assumption in empirical PP-LGCMM analysis is the absence of omitted confounders of the relationships among treatment, mediator, and outcome. In this study, we analytically examined how omitting pretreatment confounders impacts the inference of mediation from the PP-LGCMM. Using the analytical results, we developed three sensitivity analysis approaches for the PP-LGCMM, including the frequentist, Bayesian, and Monte Carlo approaches. The three approaches help investigate different questions regarding the robustness of mediation results from the PP-LGCMM, and handle the uncertainty in the sensitivity parameters differently. Applications of the three sensitivity analyses are illustrated using a real-data example. A user-friendly Shiny web application is developed to conduct the sensitivity analyses.

The aim of the study was to examine the common items in the mixed format (e.g., multiple-choices and essay items) contain parameter drifts in the test equating processes performed with the common item nonequivalent groups design. In this study, which was carried out using Monte Carlo simulation with a fully crossed design, the factors of test length (30 and 50), sample size (1000 and 3000), common item ratio (30 and 40%), ratio of items with item parameter drift (IPD) in common items (20 and 30%), location of common items in tests (at the beginning, randomly distributed, and at the end) and IPD size in multiple-choice items (low [0.2] and high [1.0]) were studied. Four test forms were created, and two test forms do not contain parameter drifts. After the parameter drift was performed on the first of the other two test forms, the parameter drift was again performed on the second test form. Test equating results were compared using the root mean squared error (RMSE) value. As a result of the research, ratio of items with IPD in common items, IPD size in multiple-choice items, common item ratio, sample size and test length on equating errors were found to be significant.

Cognitive diagnosis models (CDMs) are the statistical framework for cognitive diagnostic assessment in education and psychology. They generally assume that subjects' latent attributes are dichotomous--mastery or nonmastery, which seems quite deterministic. As an alternative to dichotomous attribute mastery, attention is drawn to the use of a continuous attribute mastery format in recent literature. To obtain subjects' finer-grained attribute mastery for more precise diagnosis and guidance, an equivalent but more explicit form of the partial-mastery-deterministic inputs, noisy "and" gate (DINA) model (termed continuous attribute profile [CAP]-DINA form) is proposed in this article. Its parameters estimation algorithm based on this form using Bayesian techniques with Markov chain Monte Carlo algorithm is also presented. Two simulation studies are conducted then to explore its parameter recovery and model misspecification, and the results demonstrate that the CAP-DINA form performs robustly with satisfactory efficiency in these two aspects. A real data study of the English test also indicates it has a better model fit than DINA.

Large-scale assessment survey (LSAS) data are collected via complex sampling designs with special features (e.g., clustering and unequal probability of selection). Multilevel models have been utilized to account for clustering effects whereas the probability weighting approach (PWA) has been used to deal with design informativeness derived from the unequal probability selection. However, the difficulty of applying PWA in multilevel models (MLM) has been generally underestimated and practical guidance is scarce. This study utilizes an empirical as well as a Monte Carlo simulation investigation to examine the performance of the multilevel pseudo maximum likelihood (MPML) estimation based on information derived from the Early Childhood Longitudinal Study Kindergarten cohort of 2010-2011 (ECLS-K:2011). Variance components and fixed effects estimators across four estimation methods including three MPML estimators (i.e., weighted without scaling, weighted size-scaled and weighted effective-scaled) and the unweighted estimator are provided. Practical guidance about the use of sampling weights in MLM analyses of LSAS data is also offered.

Although educational research and evaluation generally occur in multilevel settings, many analyses ignore cluster effects. Neglecting the nature of data from educational settings, especially in non-randomized experiments, can result in biased estimates with long-term consequences. Our manuscript improves the availability and understanding of artificial neural networks, an underutilized method trending in other disciplines. This method also shows promise for dealing with challenges faced by educational researchers, such as analyzing clustered data. Therefore, we simulated data to generalize the potential benefits of artificial neural networks to different data types. We also compared artificial neural networks to more familiar methods and investigated the time it demanded to perform each technique. Hence, readers can decide when it may be more appropriate to use one method instead of another.

Most researchers investigate the corrected item-total correlation of items when analyzing item discrimination in multi-dimensional structures under the Classical Test Theory, which might lead to underestimating item discrimination, thereby removing items from the test. Researchers might investigate the corrected item-total correlation with the factors to which that item belongs; however, getting a general overview of the entire test is impossible. Based on this problem, this study aims to recommend a new index to investigate item discrimination in two-dimensional structures through a Monte Carlo simulation. The new item discrimination index is evaluated by identifying sample size, item discrimination value, inter-factor correlation, and the number of categories. Based upon the results of the study it can be claimed that the proposed item discrimination index proves acceptable performance for two-dimensional structures. Accordingly, using this new item discrimination index could be recommended to researchers when investigating item discrimination in two-dimensional structures.

Recently, adaptive test approaches have become a viable alternative to traditional fixed-item tests. The main advantage of adaptive tests is that they reach desired measurement precision with fewer items. However, fewer items mean that each item has a more significant effect on ability estimation and therefore those tests are open to more consequential results from any flaw in an item. So, any items indicating differential item functioning (DIF) may play an important role in examinees' test scores. This study, therefore, aimed to investigate the effect of DIF items on the performance of computer adaptive and multi-stage tests. For this purpose, different test designs were tested under different test lengths and ratios of DIF items using Monte Carlo simulation. As a result, it was seen that computer adaptive test (CAT) designs had the best measurement precision over all conditions. When multi-stage test (MST) panel designs were compared, it was found that the 1-3-3 design had higher measurement precision in most of the conditions; however, the findings were not enough to say that 1-3-3 design performed better than the 1-2-4 design. Furthermore, CAT was found to be the least affected design by the increase of ratio of DIF items. MST designs were affected by that increment especially in the 10-item length test.

Simulation modeling is an effective tool for solving problems that cannot be explained analytically or when data cannot be collected. This is done by simulating the observed behavior of a problem under study using a computer program. In math education, this can develop knowledge and fundamental competencies of simulation modeling at a higher level and foster its applications in everyday life. This study created learning activities for secondary school students to simulate the probabilistic behavior of random events using Python. 28 grade 12 students took part in these activities using appropriate scaffolding strategies and a powerful mathematical tool, Python. After completing the activities, student competency in simulating the probabilistic behavior of random events with Python was evaluated using rubrics and the factors of student enjoyment, perceived value, interest, and self-efficacy were determined through a Likert-scale questionnaire. The assessment results showed that the activities had a positive effect on student competencies and emotions. The outcomes of the study can serve guidelines for teachers who are interested in expanding the results for further student development.

Item analysis is performed by developers as an integral part of the scale development process. Thus, items are excluded from the scale depending on the item analysis prior to the factor analysis. Existing item discrimination indices are calculated based on correlation, yet items with different response patterns are likely to have a similar item discrimination index. This study proposed a new item discrimination index that can be used in Likert type of scales and examined its effect on factor analysis results. For this purpose, simulative datasets were generated, and items were excluded from the analysis according to the 0.20, 0.30 and 0.35 item discrimination index criteria, and exploratory factor analysis was performed for a single factor. Accordingly, it was found that more variance could be explained by a single factor with fewer items compared to other discrimination indices when the 0.20 criterion of the slope coefficient was used as suggested in this study. Similar findings were obtained using the 0.35 criterion with other discrimination indices. In this context, it is recommended to use the slope coefficient as an additional discrimination index calculation method in the scale development process.

Financial professionals involved in divorce proceedings, whether for a client or an attorney, often use software to project the ability of a dependent spouse to earn income off of her separate estate. These projections have historically relied on static inputs and use a Monte Carlo simulation to illustrate the paths a portfolio might take. Within this study, the effects on dynamic income and expense changes on outcomes were examined. A comparison was made between the traditional Monte Carlo methods and Markov Chain Monte Carlo (MCMC) methods. Results using MCMC methods more closely approximated investment return distribution, and illustrated investable assets were the primary driver of long-term success, and not items such as spousal or child support. Practical implications for financial professionals, family law attorneys, judges, and clients are discussed as well as opportunities for future research.