Search

Effect sizes are important because they are an accessible way to indicate the practical importance of observed associations or differences. Standardized mean difference (SMD) effect sizes, such as Cohen's d, are widely used in education and the social sciences -- in part because they are relatively easy to calculate. However, SMD effect sizes assume normally distributed data, whereas most data in these fields are ordinal and/or non-normal. In these situations, SMD effect sizes can be biased, and a non-parametric measure such as Cliff's [delta] is more appropriate. This paper provides a practical guide on how to calculate Cliff's [delta]. First, we present a conceptual overview and a worked example. Then we present two methods of calculating Cliff's [delta]: (1) a web-based Shiny application developed to accompany this paper (https://cliffdelta.shinyapps.io/calculator; suitable for all users), and (2) an R tutorial (suitable for R users). This is intended to provide researchers and practitioners with an appropriate and accessible effect size measure for non-normal data.

Dual Enrollment (DE) or Dual Credit (DC) programs have become increasingly prevalent at both the high school and college settings. These programs enable students to earn both high school and college credit and get a head start on their college education. Additionally, students in these programs must take university core curriculum courses to fulfill their college education requirements. A course that several students take is Introductory to Biology, a STEM (Science, Technology, Engineering, and Mathematics) course. This course is an introductory science course and may present a challenge for students in that there is a large quantity of material discussed. Moreover, students must adapt to new study strategies excel in the course. Moreover, educational administrators or school leaders, administrators or principals must consider student performance and therefore must look at location and student performance to better improve student outcomes. Scant studies look at school leaders' perception on science dual enrollment courses. Moreover, administrators must have systems in place to promote high quality instruction and student success in dual enrollment courses. This study will look at students taking introductory biology courses and their performance at both the high school and non-high school locations (i.e. college environment) in Texas using the Mann-Whitney U statistical test. In addition, this study will provide recommendations to high school leaders such as administrators and principals regarding location and student outcomes for students in dual enrollment biology courses to ensure high quality instruction and student achievement are in place.

We conducted a descriptive analysis and meta-analysis of single-case research design (SCRD) studies investigating the effectiveness of the graduated guidance procedure. Once we identified studies through electronic databases and reference lists, we used What Works Clearinghouse (WWC) Standards to evaluate each study. Then, we described studies in terms of various descriptive variables, calculated effect sizes through three non-parametric effect size methods, and analyzed results across studies. Results showed 11 of the 27 studies met WWC Standards or met standards with reservation. Of the 11, seven studies resulted in a large effect. We found the graduated guidance procedure to be an evidence-based practice when evaluating the findings against contemporary evidence standards. However, this review also showed that the majority of the reviewed studies (n = 20) had no effects and only one third of the studies had moderate or strong effects. Implications for researchers and practitioners are discussed.

Likert-type item is the most popular response format for collecting data in social, educational, and psychological studies through scales or questionnaires. However, there is no consensus on whether parametric or non-parametric tests should be preferred when analyzing Likert-type data. This study examined the statistical power of parametric and non-parametric tests when each Likert-type item was analyzed independently in survey studies. The main purpose of the study is to examine the statistical power of Wilcoxon-Mann-Whitney, Welch's t, and Student's t tests for Likert-type data, which are pairwise comparison tests. For this purpose, a Monte Carlo simulation study was conducted. The statistical significance of the selected tests was examined under the conditions of sample size, group size ratio, and effect size. The results showed that the Wilcoxon-Mann-Whitney test was superior to its counterparts, especially for small samples and unequal group sizes. However, the Student's t-test for Likert-type data had similar statistical power to the Wilcoxon-Mann-Whitney test under conditions of equal group sizes when the sample size was 200 or more. Consistent with the empirical results, practical recommendations were provided for researchers on what to consider when collecting and analyzing Likert-type data.

A workshop in environmental sustainability was organized to develop the skills and competencies in mitigating climate change, adapting climate risk management, managing waste, adapting sustainable human settlements and using sustainable transport, and developing skills of reducing CO2 emissions, and conserving energy and ecosystem, & educating people. The study aimed to compare current opinion of participants in environmental sustainability. After a short notification in the print medias and websites; students, teachers, and local people expressed their interested and joined in an environmental sustainability workshop organized in the Department of Education, Assam University, Silchar, India. Non-parametric Kruskal-Wallis H test used to assess the mean differences in the opinion of the participants in environmental sustainability. It resulted that the participants who were actively participated have no differences in their opinion in skills and competencies in mitigating climate change, adapting climate risk management, managing waste, adapting sustainable human settlements and using sustainable transport, and developing skills of reducing CO2 emissions, and conserving energy and ecosystem, & educating people.

Objective: This study examined relationships between eating disorder risk (EDR), lifestyle variables (e.g., exposure to healthy eating media), and differences among male and female college students. Participants: College students (N = 323) completed survey questionnaires (Fall, 2016). Fifty-three participants retook the survey at a later time. Methods: Participants completed a survey measuring EDR using EDI-3 subscales and 10 college lifestyle variables. Female and male EDR and Time 1 and Time 2 EDR were compared. Relationships between EDR and college life-style variables were examined. Results: Exercise, fewer daily meals, less face-to-face interactions, more digital interactions, less exposure to healthy eating media, and having conversations about body image were associated with EDR risk. Male college students showed an increase in EDR over time. Conclusions: Several lifestyle factors predicted EDR in college females and males. These lifestyle factors are modifiable and may be addressed by colleges during orientation and within the campus environment.

Small-scale spatial abilities that involve the mental representation and transformation of two- and three-dimensional images and manipulation of objects at table-top have been studied extensively and are considered predictive of both interest and success in STEM disciplines. However, research investigating the relation of large-scale spatial abilities to STEM disciplines is sparse. The paper describes the design and implementation of a study for assessing individual differences (if any) in spatial abilities in both figural and environmental spaces between STEM experts (with over 10 years of experience) and non-experts (individuals without any studies in STEM fields). Participants' performance in 16 small-, 10 large-scale tasks, and one self-assessment questionnaire at environmental scale was evaluated to assess their corresponding abilities. Results indicate differences between experts and non-experts, which are mostly highlighted for small-scale abilities where experts outperform non-experts. At large scale, some significant differences are identified, which also favor experts. Correlations among the variables tested provide evidence that different abilities are prominent between experts and non-experts.

The questionnaire method has always been an important research method in psychology. The increasing prevalence of multidimensional trait measures in psychological research has led researchers to use longer questionnaires. However, questionnaires that are too long will inevitably reduce the quality of the completed questionnaires and the efficiency of collection. Computer adaptive testing (CAT) can be used to reduce the test length while preserving the measurement accuracy. However, it is more often used in aptitude testing and involves a large number of parametric assumptions. Applying CAT to psychological questionnaires often requires question-specific model design and preexperimentation. The present article proposes a nonparametric and item response theory (IRT)-independent CAT algorithm. The new algorithm is simple and highly generalizable. It can be quickly used in a variety of questionnaires and tests without being limited by theoretical assumptions in different research areas. Simulation and empirical studies were conducted to demonstrate the validity of the new algorithm in aptitude tests and personality measures.

C-Test is a gap-filling test designed to measure first and second language proficiency. Over the past four decades, researchers have shown the fit of C-Test data to parametric item response theory (IRT) models, but no study so far has shown the fit of C-Tests to nonparametric IRT models. The purpose of this study is to contribute to the ongoing C-Test validation project by providing evidence of fit to the Mokken scale analysis as a widely used nonparametric IRT model. A six-passage C-Test battery was analyzed using the monotone homogeneity model and the double monotonicity model of Mokken. Unidimensionality was evaluated using the automatic item selection procedure. Findings showed that the C-Test passages form a strong unidimensional scale, fit well to the monotone homogeneity model, and, after deleting one item, have a sufficient fit to the double monotonicity model. The findings also indicated that the items form a hierarchy, and persons can be located on an ordinal scale using their C-Test sum scores. Implications of the study for C-Test validity and application are discussed.

Researchers who use measurement models for evaluation purposes often select models with stringent requirements, such as Rasch models, which are parametric. Mokken Scale Analysis (MSA) offers a theory-driven nonparametric modeling approach that may be more appropriate for some measurement applications. Researchers have discussed using MSA as a preliminary procedure with which to evaluate data quality before applying a parametric model. However, the literature includes only a few examples in which researchers have integrated MSA techniques with parametric models throughout the analytic procedure. We consider a systematic approach for integrating results from nonparametric MSA techniques with parametric measurement models to evaluate measurement quality and construct scales with useful measurement properties. We use real-data illustrations and a simulation study to demonstrate and systematically explore our approach. We discuss implications for research and practice.

We discuss analysis of 5-grade Likert type data in the two-sample case. Analysis using two-sample "t" tests, nonparametric Wilcoxon tests, and ordinal regression methods, are compared using simulated data based on an ordinal regression paradigm. One thousand pairs of samples of size "n"=10 and "n"=30 were generated, with three different degrees of skewness. For all sample sizes and degrees of skewness, the ordinal probit model has highest power. This is not surprising since the data was generated with this model in mind. Slightly more surprising is that the "t" test has higher power than the Wilcoxon test in all studied situations, even for skewed data. For "n"=30, the differences between the methods are small.

This laboratory experiment was divided into four parts. In the first part, students evaluate previously published data and check their normality using histograms, Q-Q plots, the Shapiro-Wilk test, and boxplots. In the second part, two different groups were compared. First, data normality and homoskedasticity were checked by using the Shapiro-Wilk test and Levene's test, respectively. Then the group mean was compared using the Student, Welch, and Mann-Whitney tests. In the third part, the famous Iris flower data set was analyzed, where data normality and homoskedasticity were checked. Then, three independent groups were compared using ANOVA, without correction, and using the Welch and Brown-Forsythe corrections. The Iris flowers data set was also compared using the nonparametric version of ANOVA (Kruskal-Wallis test) and post hoc tests such as the Tukey test (parametric) and Dunn's test (nonparametric). In the fourth part, students measured the density of water using four pieces of glassware (a beaker, a graduated cylinder, a volumetric pipet, and a graduated pipet). After the experiment, the data' normality and homoskedasticity were checked. Then, the data was compared using ANOVA and post hoc tests. The results of these tests were visually interpreted using raincloud and discriminant plots. These data provide a starting point for teaching precision, accuracy, descriptive statistics, and hypothesis tests. Statistical analysis was carried out using Jeffreys's Amazing Statistics Program (JASP), which is free and open-source software that provides several graphical interpretations of the hypothesis test.

The dimensionality is one of the most investigated concepts in the psychological assessment, and there are many ways to determine the dimensionality of a measured construct. The Automated Item Selection Procedure (AISP) and the DETECT are non-parametric methods aiming to determine the factorial structure of a data set. In the current study, dimensionality results provided by the two methods were compared based on the original factorial structure defined by the scale developers. For the comparison of the two methods, the data was obtained by implementing a scale measuring academic dishonesty levels of bachelor students. The scale was conducted on junior students studying at a public and a private university. The dataset was analyzed by using the AISP and DETECT analyses. The "mokken" and "sirt" packages on the R program were utilized for the AISP and DETECT analyses, respectively. The similarities and differences between the findings provided by the methods were analyzed depending on the original factor structure of the scale verified by the scale developers.

Large-scale standardized ESL tests such as the International English Language Testing System (IELTS) are widely used around the world to measure the language proficiency of test-takers and make different decisions based on their scores. Reading comprehension is an integral part of such tests which requires test-takers to read passages and answer a set of questions. Although IELTS is a popular standardized test and is used for making critical decisions about test-takers, very few attempts have been made to explore the validity of the exam and especially the reading part of the General Training Module. With this in mind, the purpose of the present study was to use a nonparametric item response theory model, called Mokken Scale Analysis (MSA), to examine the validity of the reading part of the General Training module of IELTS. To this end, item responses of 352 test-takers to the reading comprehension test were analyzed. The results of item scalability, total scalability, and item-pair scalability showed that the reading part is a weak unidimensional scale. Using Monotone Homogeneity Model (MHM), the monotonicity results also indicated that there are some items which violate the monotonicity assumption, although their values are insignificant. The analysis of unidimensionality using the AISP revealed that there are two scales and four unscalable items in the reading part. Therefore, the Mokken scale analysis did not support the unidimensional structure of the reading part of the General Training module of IELTS.

A deep understanding of the learning level of online learners is a critical factor in promoting the success of online learning. Using knowledge structures as a way to understand learning can help analyze online students' learning levels. The study used concept maps and clustering analysis to investigate online learners' knowledge structures in a flipped classroom's online learning environment. Concept maps (n = 359) constructed by 36 students during one semester (11 weeks) through the online learning platform were collected as analysis objects of learners' knowledge structures. Clustering analysis was used to identify online learners' knowledge structure patterns and learner types, and a non-parametric test was used to analyze the differences in learning achievement among learner types. The results showed that (1) there were three online learners' knowledge structure patterns of increasing complexity, namely, spoke, small-network, and large-network patterns. Moreover, online learners with novice status mostly had spoke patterns in the context of flipped classrooms' online learning. (2) Two types of online learners were found to have different distributions of knowledge structure patterns, and the complex knowledge structure type of learners exhibited better learning achievement. The study explored a new way for educators to analyze knowledge structures by data mining automatically. The findings provide evidence in the online learning context for the relationship between complex knowledge structures and better learning achievement while suggesting the existence of inadequate knowledge preparedness for flipped classroom learners without a special instructional design.

The multiple-choice (MC) item format has been widely used in educational assessments across diverse content domains. MC items purportedly allow for collecting richer diagnostic information. The effectiveness and economy of administering MC items may have further contributed to their popularity not just in educational assessment. The MC item format has also been adapted to the cognitive diagnosis (CD) framework. Early approaches simply dichotomized the responses and analyzed them with a CD model for binary responses. Obviously, this strategy cannot exploit the additional diagnostic information provided by MC items. De la Torre's MC Deterministic Inputs, Noisy "And" Gate (MC-DINA) model was the first for the explicit analysis of items having MC response format. However, as a drawback, the attribute vectors of the distractors are restricted to be nested within the key and each other. The method presented in this article for the CD of DINA items having MC response format does not require such constraints. Another contribution of the proposed method concerns its implementation using a nonparametric classification algorithm, which predestines it for use especially in small-sample settings like classrooms, where CD is most needed for monitoring instruction and student learning. In contrast, default parametric CD estimation routines that rely on EM- or MCMC-based algorithms cannot guarantee stable and reliable estimates--despite their effectiveness and efficiency when samples are large--due to computational feasibility issues caused by insufficient sample sizes. Results of simulation studies and a real-world application are also reported.

This study investigated the scalability of a cognitive multiple-choice test through the Mokken package in the R programming language for statistical computing. A 2019 mathematics West African Examinations Council (WAEC) instrument was used to gather data from randomly drawn K-12 participants (N = 2866; Male = 1232; Female = 1634; Mean age = 16.5 years) in Education District I, Lagos State, Nigeria. The results showed that the monotone homogeneity model (MHM) was consistent with the empirical dataset. However, it was observed that the test could not be scaled unidimensionally due to the low scalability of some items. In addition, the test discriminated well and had low accuracy for item-invariant ordering (IIO). Thus, items seriously violated the IIO property and scalability criteria when the HT coefficient was estimated. Consequently, the test requires modification in order to provide monotonic characteristics. This has implications for public examining bodies when endeavouring to assess the IIO assumption of their items in order to boost the validity of testing.

The Kaufman Brief Intelligence Test--Second Edition (KBIT-2) is designed to measure verbal and nonverbal abilities in a wide range of individuals from 4 years 0 months to 90 years 11 months of age. This study examines both the advantages of using Mokken Scale Analysis (MSA) in intelligence tests and the hierarchical order of the items in the KBIT-2: Turkish form by estimating the parameters of each of the three subtests by testing the dimensionality of the KBIT2 subtests by using the Invariant Item Ordering (IIO) assumptions. 2850 people participated in the study, including children, adolescents, and adults. Participants' ages varied from 48 months (4 years 0 months) to 539 months (44 years 11 months). Automated Item Selection Procedure (AISP) was applied for the assessment of unidimensionality under three different lower bounds as 0.30, 0.40, and 0.55. The items of all three subtests formed a unidimensional scale. Backward Item Selection (BIS) procedure detected seven items in the Matrices and 17 items in the Verbal Knowledge, while six items in the Riddles subtest violated the IIO criteria. KBIT-2: Reliability values obtained using MSA analysis show that all three subtests have a high degree of internal consistency. However, care should be taken when IIO assumptions do not fit the intelligence scales in the original form.

An understanding of statistical concepts is necessary for a chemist with a complete education. Here, statistical tests were taught using the R Commander and the Factoshiny packages. These packages run on R software and have a graphical user interface (GUI), which allows students to do statistical tests quickly and easily. These packages were presented through 14 case studies. In each case study, a data set was provided in MS Excel format, and a series of questions were answered using these packages. Hypothesis tests and exploratory analysis (principal component analysis; PCA) were taught using R Commander and Factoshiny, respectively. Outlier results were found using boxplots. Data normality was checked using histograms and the Shapiro-Wilk test. Normally distributed data sets were compared using parametric hypothesis tests (t test, paired t test, one-way ANOVA, two-way ANOVA, one-way repeated measure ANOVA). Non-normally distrusted data sets were compared using nonparametric tests (Wilcoxon, Kruskal--Wallis, and Friedman tests). Results provided by these parametric and nonparametric tests were also verified using plots (plot of means and boxplots).

The purpose of this study is to illustrate the use of functional data analysis (FDA) as a general methodology for analyzing person response functions (PRFs). Applications of FDA to psychometrics have included the estimation of item response functions and latent distributions, as well as differential item functioning. Although FDA has been suggested for modeling PRFs, there has been relatively little research stressing this application. FDA offers an approach for diagnosing person responses that may be due to guessing and other sources of within-person multidimensionality. PRFs provide graphical displays that can be used to highlight unusual response patterns, and to identify persons that are not responding as expected to a set of test items. In addition to examining individual PRFs, functional clustering techniques can be used to identify subgroups of persons that may be exhibiting categories of misfit such as guessing. A small simulation study is conducted to illustrate how FDA can be used to identify persons exhibiting different levels of guessing behavior (5%, 10%, 15% and 20%). The methodology is also applied to real data from a 3rd grade science assessment used in a southeastern state. FDA offers a promising methodology for evaluating whether or not meaningful scores have been obtained for a person. Typical indices of psychometric quality, such as standard errors of measurement and person fit indices, are not sufficient for representing certain types of aberrance in person response patterns. Nonparametric graphical methods for estimating PRFs that are based FDA provide a rich source of validity evidence regarding the meaning and usefulness of each person's score.

This article reports on a trace-based assessment of approaches to learning used by middle school aged children who interacted with NASA Mars Mission science, technology, engineering and mathematics (STEM) games in "Whyville," an online game environment with 8 million registered young learners. The learning objectives of two games included awareness and knowledge of NASA missions, developing knowledge and skills of measurement and scaling, applying measurement for planetary comparisons in the solar system. Trace data from 1361 interactions were analysed with nonparametric multidimensional scaling methods, which permitted visual examination and statistical validation, and provided an example and proof of concept for the multidimensional scaling approach to analysis of time-based behavioural data from a game or simulation. Differences in approach to learning were found illustrating the potential value of the methodology to curriculum and game-based learning designers as well as other creators of online STEM content for pre-college youth. The theoretical framework of the method and analysis makes use of the Epistemic Network Analysis toolkit as a post hoc data exploration platform, and the discussion centres on issues of semantic interpretation of interaction end-states and the application of evidence centred design in post hoc analysis.

With the continuous development of education, personalized learning has attracted great attention. How to evaluate students' learning effects has become increasingly important. In information technology courses, the traditional academic evaluation focuses on the student's learning outcomes, such as "scores" or "right/wrong," which seldom reflects the development of students' cognitive level and lacks effective diagnostic information. This article proposes a non-parametric multi-level scoring cognitive diagnosis method based on the KNN and the characteristics of information technology courses named the EW-KNN (E-weight K-Nearest Neighbor). Compared with the KNN, the EW-KNN improved two key points. One is that it takes the number of IRP (Ideal Response Pattern) as the K value to adapt to different types of tests. The other is that the nearest neighbor distance is introduced to solve the problem of misjudgment of the categories. The Monte Carlo simulation method is used to test its performance. The results indicate that the EW-KNN has a higher accuracy rate and is suitable for information technology courses. Furthermore, the method is applied in information technology course to make a cognitive diagnosis of 120 students of high school in Shanghai. Results demonstrate that the EW-KNN can accurately diagnose each student's cognition levels and knowledge structure accurately.

Person-fit analyses are commonly used to detect aberrant responding in self-report data. Nonparametric person fit statistics do not require fitting a parametric test theory model and have performed well compared to other person-fit statistics. However, detection of aberrant responding has primarily focused on dominance response data, thus the effectiveness of person-fit statistics in detecting different aberrant behaviors in ideal point data is unclear. This study compares the performance of nonparametric person-fit statistics in unfolding and dominance model contexts. Results for dominance data indicate that increases in detection rates depend, among other factors, on type of aberrant responding and person-fit statistic used. The detection of aberrant responses in ideal point data was ineffective using four nonparametric person-fit statistics, with slightly higher type I error and power less than 0.25. Additional research is needed to identify or develop nonparametric or parametric person-fit statistics effective for aberrant behavior exhibited in ideal point data.

This study investigated dimensionality of Binary Response Items through a non-parametric technique of Item Response Theory measurement framework. The study used causal comparative research type of nonexperimental design. The sample consisted of 5,076 public senior secondary school examinees (SSS3) between the age of 14-16 years from 45 schools, which were drawn randomly from three senatorial districts of Osun State, Nigeria. Instrument used for this study was 2018 Osun State unified multiple-choice mathematics achievement test items with empirical reliability coefficient of 0.82. Data obtained were analysed using Non-linear factor analysis, Stout's Test of Essential Unidimensionality (STEU), Factor Analysis (FA), Full Information Factor Analysis (FIFA) and Bootstrap Modified Parallel Analysis Test (BMPAT). Results showed that both the BMPAT and STEU ascertained violation of unidimensionality assumption of the test items (the observed difference in the second eigenvalue of the observed data and that of second eigenvalue of the simulated data was statistically significant, p = 0.0099; Stout's test rejected the assumption of essential unidimensionality, T = 10.6260, p<0.05). Non-linear factor analysis and full information factor analysis revealed that four dimensions embedded in the test items and loadings of the items showed within-item multidimensionality respectively. The authors' concluded that modeling examinees' performance with unidimensional model when it was actually multidimensional in nature would affect performance of examinees adversely and could lead to blur judgment. Consequently, it is recommended that unidimensional scoring method of Osun State unified mathematics achievement test implicit in Classical Test Theory should be jettisoned and an appropriate scoring model (multidimensional) should be embraced.

Checking the normality assumption is necessary to decide whether a parametric or non-parametric test needs to be used. Different ways are suggested in literature to use for checking normality. Skewness and kurtosis values are one of them. However, there is no consensus which values indicated a normal distribution. Therefore, the effects of different criteria in terms of skewness values were simulated in this study. Specifically, the results of t-test and U-test are compared under different skewness values. The results showed that t-test and U-test give different results when the data showed skewness. Based on the results, using skewness values alone to decide about normality of a dataset may not be enough. Therefore, the use of non-parametric tests might be inevitable.

Mokken models have recently started to become the preferred method of researchers from different fields in studies of nonparametric item response theory (NIRT). Despite increasing application of these models, some features of this type of modelling need further study and explanation. Invariant item ordering (IIO) is one of these areas, which the present study aims to exemplify using a real dataset and comparing the findings of different methods. The main purpose of this study is to check the IIO assumption for a large scale test by using different methods. Data relating to the high school placement test (applied in 2016) was investigated. The sample size was determined as being 250, which complies with NIRT. Two different methods have been used for dichotomous items in the IIO testing: rest-scores and P-matrix methods. The H[superscript T] coefficients were also calculated in order to define the placement of item response functions. Findings show that the test battery is not suitable for Mokken scaling. IIO property was not met for any of the tests, and findings from the different methods were not consistent with each other. As for the results, the rest-score method defined more items violating IIO properties than the P-matrix method for all the tests. The H[superscript T] coefficients were also estimated from the critical value, which shows that the tests do not have IIO properties. The conflicting results indicate that there is a need for new studies to investigate IIO empirically.

Kelley's Discrimination Index (DI) is a simple and robust, classical non-parametric short-cut to estimate the item discrimination power (IDP) in the practical educational settings. Unlike item-total correlation, DI can reach the ultimate values of +1 and -1, and it is stable against the outliers. Because of the computational easiness, DI is specifically suitable for the rough estimation where the sophisticated tools for item analysis such as IRT modelling are not available as is usual, for example, in the classroom testing. Unlike most of the other traditional indices for IDP, DI uses only the extreme cases of the ordered dataset in the estimation. One deficiency of DI is that it suits only for dichotomous datasets. This article generalizes DI to allow polytomous dataset and flexible cut-offs for selecting the extreme cases. A new algorithm based on the concept of the characteristic vector of the item is introduced to compute the generalized DI (GDI). A new visual method for item analysis, the cut-off curve, is introduced based on the procedure called exhaustive splitting.

Raven's Standard Progressive Matrices (Raven 1941) is a widely used 60-item long measure of general mental ability. It was recently suggested that, for situations where taking this test is too time consuming, a shorter version, comprised of only the last series of the Standard Progressive Matrices (Myszkowski and Storme 2018) could be used, while preserving satisfactory psychometric properties (Garcia-Garzon et al. 2019; Myszkowski and Storme 2018). In this study, I argue, however, that some psychometric properties have been left aside by previous investigations. As part of this special issue on the reinvestigation of Myszkowski and Storme's dataset, I propose to use the non-parametric Item Response Theory framework of Mokken Scale Analysis (Mokken 1971, 1997) and its current developments (Sijtsma and van der Ark 2017) to shed new light on the SPM-LS. Extending previous findings, this investigation indicated that the SPM-LS had satisfactory scalability (H=0.469), local independence and reliability (MS=0.841, LCRC=0.874). Further, all item response functions were monotonically increasing, and there was overall evidence for invariant item ordering (H[subscript T]=0.475), supporting the Double Monotonicity Model (Mokken 1997). Item 1, however, appeared problematic in most analyses. I discuss the implications of these results, notably regarding whether to discard item 1, whether the SPM-LS sum scores can confidently be used to order persons, and whether the invariant item ordering of the SPM-LS allows to use a stopping rule to further shorten test administration.

The purpose of this study is to examine the validity of the scale for identifying gifted children, whose validity was proven by exploratory, confirmatory factor analysis, and whose reliability was proven the Cronbach alpha coefficient for identifying children in the 3-6 age group, using Mokken scaling based on nonparametric item response theory. The study group of the research consists of 253 students. As a result of the analysis of the 13-item 3-dimensional scale ("above average ability," "creativity" and "task commitment") with Monotone homogeneity Model, it is seen that a one-dimensional structure is obtained and the model fits when analyzed as a three-dimensional construct. For model data fit, when the discrimination and reliability values of the items are examined, it can be said that the one-dimensional structure of the scale is at an acceptable level, and relatively higher values are obtained for each criterion in the three-dimensional structure compared to the one-dimensional structure. Based on the findings, it can be interpreted that a parallel result was obtained in the validity study based on non-parametric item response theory of the scale, which was developed based on confirmatory factor analysis.

Randomised controlled trials of cancer treatments typically report progression free survival (PFS) and overall survival (OS) outcomes. Existing methods to synthesise evidence on PFS and OS either rely on the proportional hazards assumption or make parametric assumptions which may not capture the diverse survival curve shapes across studies and treatments. Furthermore, PFS and OS are not independent; OS is the sum of PFS and post-progression survival (PPS). Our aim was to develop a non-parametric approach for jointly synthesising evidence from published Kaplan-Meier survival curves of PFS and OS without assuming proportional hazards. Restricted mean survival times (RMST) are estimated by the area under the survival curves (AUCs) up to a restricted follow-up time. The correlation between AUCs due to the constraint that OS > PFS is estimated using bootstrap re-sampling. Network meta-analysis models are given for RMST for PFS and PPS and ensure that OS = PFS + PPS. Both additive and multiplicative network meta-analysis models are presented to obtain relative treatment effects as either differences or ratios of RMST. The methods are illustrated with a network meta-analysis of treatments for stage IIIA-N2 non-small cell lung cancer. The approach has implications for health economic models of cancer treatments, which require estimates of the mean time spent in the PFS and PPS health-states. The methods can be applied to a single time-to-event outcome, and so have wide applicability in any field where time-to-event outcomes are reported, the proportional hazards assumption is in doubt, and survival curve shapes differ across studies and interventions.

Researchers frequently use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, when they have relatively small samples of examinees. Researchers have provided some guidance regarding the minimum sample size for applications of MSA under various conditions. However, these studies have not focused on item-level measurement problems, such as violations of monotonicity or invariant item ordering (IIO). Moreover, these studies have focused on problems that occur for a complete sample of examinees. The current study uses a simulation study to consider the sensitivity of MSA item analysis procedures to problematic item characteristics that occur within limited ranges of the latent variable. Results generally support the use of MSA with small samples (N around 100 examinees) as long as multiple indicators of item quality are considered.

Conventional advice discourages controlling for postoutcome variables in regression analysis. By contrast, we show that controlling for commonly available postoutcome (i.e., future) values of the treatment variable can help detect, reduce, and even remove omitted variable bias (unobserved confounding). The premise is that the same unobserved confounder that affects treatment also affects the future value of the treatment. Future treatments thus proxy for the unmeasured confounder, and researchers can exploit these proxy measures productively. We establish several new results: Regarding a commonly assumed data-generating process involving future treatments, we (1) introduce a simple new approach and show that it strictly reduces bias, (2) elaborate on existing approaches and show that they can increase bias, (3) assess the relative merits of alternative approaches, and (4) analyze true state dependence and selection as key challenges. (5) Importantly, we also introduce a new nonparametric test that uses future treatments to detect hidden bias even when future-treatment estimation fails to reduce bias. We illustrate these results empirically with an analysis of the effect of parental income on children's educational attainment.

Item response theory (IRT) has so many advantages than its precedent Classical Test Theory (CTT) such as non-changing item parameters, ability parameter estimations free from the items. However, in order to get these advantages, some assumptions should be met and they are; unidimensionality, normality and local independence. However, it is not always so easy to be met these assumptions by datasets. Especially when the normality of data is not provided, another approach for IRT can be applied, which is Non-Parametric Item Response Theory (NIRT). NIRT provides more flexible methods to scale datasets and it is used when the assumptions of Parametric Item Response Theory (PIRT) are not met at a satisfactory level. The assumption of local independence, is one of the situations in which NIRT can be used more effectively than PIRT. In this study, by using a real dataset, taken from TIMSS 2011, the effect of local dependence on the item parameters was investigated. With this goal, a dataset composed of 1,000 students was formed randomly from the TIMSS 2011 eight grade Mathematic test. Firstly, the item parameters were calculated from data set according to the two approaches without any manipulation. After that, two items were arranged as local dependent by changing the response patterns completely the same and the item parameters have been estimated from each sample by using R program, "ltm" and "mokken" packages. Two sets of item parameters estimated from data set were compared and the differences of the parameters were analyzed with statistical test. By this way, the effect of local independence has been analyzed on the item parameters have been decided.

Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polynomial. Monotonic polynomial items can be used with traditional CAT item selection algorithms that use analytical derivatives. We compared these approaches in CAT simulations with a variety of item selection algorithms. Our simulations also varied the features of the calibration and item pool: sample size, the presence of missing data, and the percentage of nonstandard items. In general, the results support the use of semi- and nonparametric item response functions in a CAT.

Notwithstanding a large body of literature on log-linear models and odds ratios, no general marginal-free index of the association in a contingency table has gained a wide acceptance. Building on a framework developed by L. A. Goodman, we put into light the direct links between odds ratios, the Altham index, the intrinsic association coefficient, and coefficients in log-multiplicative models including Unidiff and row-column association models. We devise a normalized version of the latter coefficient varying between 0 and 1, which offers a simpler interpretation than existing indices similar to the correlation coefficient. We illustrate with the case of educational and socioeconomic homogamy among 149 European regions how this index can be used either alone in a non- or semiparametric approach or combined with models, and how it can protect against incorrect conclusions based on models which rely on strong assumptions to summarize the strength of association as a single parameter.

In cognitive diagnosis assessment (CDA), the impact of misspecified item-attribute relations (or "Q-matrix") designed by subject-matter experts has been a great challenge to real-world applications. This study examined parameter estimation of the CDA with the expert-designed Q-matrix and two refined Q-matrices for international large-scale data. Specifically, the G-DINA model was used to analyze TIMSS data for Grade 8 for five selected countries separately; and the need of a refined Q-matrix specific to the country was investigated. The results suggested that the two refined Q-matrices fitted the data better than the expert-designed Q-matrix, and the stepwise validation method performed better than the nonparametric classification method, resulting in a substantively different classification of students in attribute mastery patterns and different item parameter estimates. The results confirmed that the use of country-specific Q-matrices based on the G-DINA model led to a better fit compared to a universal expert-designed Q-matrix.

Purpose: A literature research conducted in education and agricultural education journals published during a period of 10 years revealed that 98% of the studies used parametric analyses. In general, model assumptions were not tested, and statistical criteria were not followed to apply the parametric approach. The objective of this paper is to persuade researchers to use the most appropriate statistical analysis for their data. Design/Methodology/approach: We present a case study in agricultural education where a parametric multiple linear regression (MLR) could be applied. A survey was designed to find out how Theory of Planned Behavior and Importance-Performance variables were associated to Behavioral Intent concerning landscape water conservation practices. Although model assumptions were not met, we initially carried out a MLR analysis based on the premise that the results could be reported descriptively if they were double cross-validated successfully. Findings: The double cross-validation of the MLR was not successful, and model assumptions were not held even though the sample size was large. A quantile regression (QR) model fitted the data well. Theory of Planned Behavior and Importance-Performance variables were good predictors of Behavioral Intent, excepting Attitude. Practical implications: Researchers must rely on statistical criteria to support decisions regarding the use of parametric or nonparametric procedures. Theoretical implications: The adherence to best practices in the utilization of statistical procedures must be discussed as an ethical matter in research across all fields of science. Originality/value: We demonstrate that imposing the Central Limit Theorem to use the MLR model is not the correct criterion to apply a parametric approach. We should use double cross-validation.

Many of the analytical models commonly used in educational research often aim to maximize explained variance and identify variable importance within models. These models are useful for understanding general ideas and trends, but give limited insight into the individuals within said models. Data envelopment analysis (DEA), is a method rooted in organizational management that makes such insights possible. Unlike models alluded to above, DEA does not explain variance. Instead, it explains how efficiently an individual utilizes their inputs to produce outputs, and identifies which input is not being utilized optimally. This paper provides a history and usages of DEA from fields outside of education, and describes the math and processes behind it. This paper then extends DEA's usage into the educational field using a study on child reading ability. Using students from the Project KIDS dataset (n = 1987), DEA is demonstrated using a simple view of reading framework, identifying individual efficiency levels in using reading-based skills to achieve reading comprehension, determining which skills are being underutilized, and classifying new subsets of readers. New subsets of readers were identified using this method, with implications for more targeted interventions.

To extend the existing research examining multiple text comprehension and its assessment, we developed a verification task approach to assessing of information that was "explicitly" and "implicitly" presented "within" and across nine texts. A nonparametric form of signal detection theory was used to analyse the performance of 47 middle school students using a rating response scale to judge the truth value of specific statements. The results showed that the four types of statements in the verification task (explicit and implicit; within- and across- text) behaved systematically: within-text explicit statements were the easiest for students to verify as true and across-text implicit statements were the most difficult. Practical implications of these results regarding assessment of complex comprehension processes are discussed.

When there are no personal consequences associated with test performance for examinees, rapid guessing (RG) is a concern and can differ between subgroups. To date, the impact of differential RG on item-level measurement invariance has received minimal attention. To that end, a simulation study was conducted to examine the robustness of the Mantel-Haenszel (MH), standardization index (STD), and logistic regression (LR) differential item functioning (DIF) procedures to type I error in the presence of differential RG. Sample size, test difficulty, group impact, and differential RG rates were manipulated. Findings revealed that the LR procedure was completely robust to type I errors, while slightly elevated false positive rates (< 1%) were observed for the MH and STD procedures. An applied analysis examining data from the Programme for International Student Assessment showed minimal differences in DIF classifications when comparing data in which RG responses were unfiltered and filtered. These results suggest that large rates of differences in RG rates between subgroups are unassociated with false positive classifications of DIF.

Listening comprehension in English, as one of the most fundamental skills, has an essential role in the process of learning English. Mokken scale analysis (MSA) is a probabilistic-nonparametric approach to item response theory (IRT) which determines the one-dimensionality and scalability of test. Mokken scaling techniques are a useful tool for researchers who wish to construct unidimensional tests or use questionnaires that comprise multiple binary or polytomous items. The current study utilized the MSA to examine one-dimensionality and scalability of TEOFL iBT listening test, administered on 400 EFL university students in the Iranian context. The results illustrated no main concerns in terms of item scalability. Monoton homogeneity model (MHM) fitted all the items of the test very well as measured by the scalability coefficients and restscore groups method. Considering the invariant item ordering (IIO), it was concluded that the ordering of items according to their mean is invariant across examinees, although HT was small. Dimensionality analysis results using the automated item selection procedure (AISP) depicted that the test is unidimensional-confirming evidence of the validity of the test in measuring a single ability dimension. The empirical application depicted a potential and feasible approach, whereby MSA could be used as a valuable method for exploring the behavior of scaled items in response to varying levels of a latent trait in education research.

Researchers are encouraged to report effect size statistics to quantify treatment effects or effects due to group differences. However, estimates of effect sizes, most commonly Cohen's "d," make assumptions about the distribution of data that are not always true. An alternative nonparametric estimate of effect size, relying on the median absolute deviation, is proposed. Comparison of this estimate to Cohen's "d" using (simulated) non-normally distributed data demonstrate that the nonparametric approach effect size may be a better estimate of effect size.

Multilevel latent class analysis (MLCA) has been increasingly used to investigate unobserved population heterogeneity while taking into account data dependency. Nonparametric MLCA has gained much popularity due to the advantage of classifying both individuals and clusters into latent classes. This study demonstrated the need to relax the assumption in specifying the nonparametric MLCA: item response probabilities varied only across level-1 latent classes, but not level-2 latent classes. An empirical demonstration with data from the Trends in International Mathematics and Science Study (TIMSS) 2011 showed that item response probabilities could vary across both level-1 and level-2 latent classes. This relaxed MLCA yielded better model fit and provided more nuanced understanding of the heterogeneous response patterns. Monte Carlo simulation was conducted to evaluate class enumeration and assignment accuracy of the relaxed MLCA. Based on the simulation results, we recommended the use of AIC in class enumeration and highlighted the benefits of having larger cluster size.

Bayesian Knowledge Tracing (BKT) is a commonly used approach for student modeling, and Long Short Term Memory (LSTM) is a versatile model that can be applied to a wide range of tasks, such as language translation. In this work, we directly compared three models: BKT, its variant Intervention-BKT (IBKT), and LSTM, on two types of student modeling tasks: post-test scores prediction and learning gains prediction. Additionally, while previous work on student learning has often used skill/knowledge components identified by domain experts, we incorporated an automatic skill discovery method (SK), which includes a nonparametric prior over the exercise-skill assignments, to all three models. Thus, we explored a total of six models: BKT, BKT+SK, IBKT, IBKT+SK, LSTM, and LSTM+SK. Two training datasets were employed, one was collected from a natural language physics intelligent tutoring system named Cordillera, and the other was from a standard probability intelligent tutoring system named Pyrenees. Overall, our results showed that BKT and BKT+SK outperformed the others on predicting post-test scores, whereas LSTM and LSTM+SK achieved the highest accuracy, F1-measure, and area under the ROC curve (AUC) on predicting learning gains. Furthermore, we demonstrated that by combining SK with the BKT model, BKT+SK could reliably predict post-test scores using only the earliest 50% of the entire training sequences. For learning gain early prediction, using the earliest 70% of the entire sequences, LSTM can deliver a comparable prediction as using the entire training sequences. The findings yield a learning environment that can foretell students' performance and learning gains early, and can render adaptive pedagogical strategy accordingly.