Search

This study investigates inter- and intracoder reliability, proposing a new approach based on social network analysis (SNA) and exponential random graph models (ERGM). During a recent exit poll, the responses of voters to two open-ended questions were recorded. A coding experiment was conducted where a group of coders coded a sample of text segments. Analyzing the data, we show that the proposed SNA/ERGM method extends significantly our analytical leverage, beyond what popular tools such as Krippendorff's [alpha] and Fleiss's K have to offer. The reliability of coding for individual coders differed significantly for the two questions although they were very similar and the same codebook was used. We conclude that the main advantages of the proposed SNA/ERGM method are the intuitive visualizations and the nuanced measurements. Detailed guidelines are provided for practitioners who would like to use the proposed method in operational settings.

Recruiting participants through gatekeepers has been widely discussed in qualitative research. However, when a sampling frame is not available, surveying through gatekeepers can also be important for quantitative studies. We used three sampling methods to survey guardians of University students: (a) a gatekeeper variant of the time-space sampling, (b) a direct postal survey to a random sample of guardians' addresses, and (c) a phone survey of the general population. The gatekeeper survey was the most efficient, but yielded a skewed sample. Albeit skewed, the gatekeeper sample yielded the same responses as the random sample, on five theoretically important scales (N = 29 questions). In-depth personal interviews and online semi-structured questionnaires with guardians and students revealed a two-way gatekeeping mechanism within families, regulating guardians' participation in the study. We discuss the indispensable contribution of qualitative data in our study and provide methodological suggestions for practitioners.

This longitudinal study (2002-2014) investigates the stability of rating characteristics of a large group of raters over time in the context of the writing paper of a national high-stakes examination. The study uses one measure of rater severity and two measures of rater consistency. The results suggest that the rating characteristics of individual raters are not stable. Thus, predictions from one administration to the next are difficult, although not impossible. In fact, as the membership of the group of raters changes from year to year, past data on rating characteristics become less useful. When the membership of the group of raters is retained, the community of raters develops more stable characteristics. However, "cultural shocks" (low retention of raters and large numbers of newcomers) destabilize the rating characteristics of the community and predictions become more difficult. We propose practical measures to increase the stability of rating across time and offer methodological suggestions for more efficient rater effect-related research designs and analyses.

Research has shown that consistent family support improves higher education (HE) students' chances for adjustment and graduation but family over-involvement negatively affects students' well-being. We theoretically bridge three largely disjointed bodies of literature (namely, family 'support', 'involvement' and 'over-involvement') and show that families of undergraduate students very often engage in roles traditionally reserved for lower levels of education. Through a large-scale, quantitative university student survey in two universities in Cyprus, we empirically show that the three bodies of literature should be unified, because family involvement in HE is better conceptualised and operationalised as a continuum. We further suggest that family involvement in HE can be split into two qualitatively different and empirically not significantly related constructs: rearguard and front line family involvement. In addition, the students seem not only to approve, but also encourage more family involvement (even 'over-involvement'), and state that they would like their families to provide them not just with more financial help, but with more emotional support as well. The discussion extends to include an analysis of students' perceptions of university actors' attitudes towards family involvement.

The way social protest affects electoral outcomes remains a lacuna. This article helps fill this gap by examining how social protest against far right actors affects their electoral standing. The article utilizes a unique dataset of 4,745 local protest events to investigate how mobilization against the far right in Greece affected its electoral performance. The article finds that protest activity depressed the electoral results of the far right Golden Dawn by as much as 16%, after controlling for a number of important variables. The article identifies and specifies the patterns through which protests against the far right affect its electoral standing. Protests are effective when following the "tango" pattern—when there is close interaction of far right and anti-far right events. The timing of protest is also important and the article shows how the synchronization of protest and electoral cycles affects electoral outcomes. The article uses the findings to discuss the varying impact of protest across electoral cycles. [ABSTRACT FROM AUTHOR]

It is common practice for assessment programs to organize qualifying sessions during which the raters (often known as "markers" or "judges") demonstrate their consistency before operational rating commences. Because of the high-stakes nature of many rating activities, the research community tends to continuously explore new methods to analyze rating data. We used simulated and empirical data from two high-stakes language assessments, to propose a new approach, based on social network analysis and exponential graph models, to evaluate the readiness of a group of raters for operational rating. The results of this innovative approach are compared with the results of a Rasch analysis, which is a well-established approach for the analysis of such data. We also demonstrate how the new approach can be practically used to investigate important research questions such as whether rater severity is stable across rating tasks. The merits of the new approach, and the consequences for practice are discussed.

Families have been getting more and more involved in their children's education. This paper presents findings of a study investigating family involvement in their members' undergraduate studies attending a state (non-fee-paying) and a private (fee-paying) university in Cyprus. The data presented in the paper were collected via online logs completed by administrative personnel in the two universities documenting incidents of family involvement. The data analysis demonstrates how similarly the phenomenon manifests in both universities and suggests that where the two universities differ is more in the intensity of the various modes of family involvement. It also projects students' desires and encouragement of their family's involvement and their families' intervention to protect or guide them. The findings of the study are discussed in view of cultural and societal norms and changes about parenthood and the relationship between families and their university age members in a marketized higher education context.

This paper examines reading achievement when the maternal/paternal language has become a de facto second language. The performance of a cohort of Greek-Australian high school students (N = 270) on a diagnostic Greek reading test was significantly below that of pupils in second to fourth grades in Greece. The mean item difficulty for Greek-Australian high school students was 0.35 compared with 0.51 for second grade, 0.69 for third grade and 0.80 for fourth grade pupils in Greece. The pattern of responding indicated that the Rasch model fitted the data. The effects of background factors were also examined. Students whose mother or father spoke Greek had statistically significant higher levels of ability than those who spoke Greek and English or English alone. It was also found that the length of the key words (correct responses) had a large effect on the difficulty of the questions; the longer the key word, the more difficult the question was. The implications of the results for assessment of reading in and the acquisition of an ethnic or second language are discussed. (Contains 3 tables and 4 figures.)

Established literature suggests that language problems lead to lower attainment levels in those subjects that are more language dependent. Also, language has been suggested as a main driver of ethnic minority attainment. We use an original dataset of 2,020 secondary school students to show that ethnic minority students in Cyprus underperform overall, but they do not perform less well in subjects considered more language dependent. We suggest that the main determinant for their low performance is not linguistic but cultural. This is the first study to investigate the performance of Georgians as an ethnic minority group in a European country. It is also one of the very few studies which were deliberately designed to compare the attainment of ethnic minority students on more and less language-dependent subjects. Methodologically, this is an instructive study using a repeated-measures design and combining teacher assessments with test results to alleviate floor and ceiling effects.

The purpose of this article is to explore crossing differential item functioning (DIF) in a test drawn from a national examination of mathematics for 11-year-old pupils in England. An empirical dataset was analyzed to explore DIF by gender in a mathematics assessment. A two-step process involving the logistic regression (LR) procedure for detecting uniform and nonuniform DIF was applied to identify crossing DIF. The results showed 36 uniform and 19 nonuniform statistically significant gender DIF items. Out of the 19 nonuniform DIF items, 10 items were crossing DIF. We explained nonuniform DIF using the crossing point in item characteristic curves and the LR-DIF coding scheme. This study showed that crossing DIF exists in empirical data and the findings from this study provide a potentially valuable contribution in understanding such items.

This document is an editorial note from the International Journal of Social Research Methodology, acknowledging and thanking all the individuals who have assessed manuscripts submitted for publication up until July 2023. The note lists the names and affiliations of the referees, who come from various universities and research institutions around the world. The purpose of this note is to recognize the contributions of these individuals in the peer review process. [Extracted from the article]

Researchers interested in exploring substantive group differences are increasingly attending to bundles of items (or testlets): the aim is to understand how gender differences, for instance, are explained by differential performances on different types or bundles of items, hence differential bundle functioning (DBF). Some previous work has modelled hierarchies in data in this context or considered item responses within persons, but here we model the bundles themselves as explanatory variables at the item level potentially explaining significant intra-class correlation due to gender differences in item difficulty, and thus explaining variation at the second item level. In this study, we analyse DBF using single- and two-level models (the latter modelling random item effects, which models responses at Level 1 and items at Level 2) in a high-stakes National Mathematics test. The models show comparable regression coefficients but the statistical significances of the two-level models are smaller due to the larger values of the estimated standard errors. We discuss the contrasting relevance of this effect for test developers and gender researchers. (Contains 4 tables and 3 figures.)

This article describes briefly the education system of Cyprus and elaborates on the recent changes in its large-scale assessment (LSA) programme. Until 2005, two independent LSA programmes existed: one for school-graduation purposes and one for gaining entrance to higher education. The introduction of a dual-purpose LSA program in 2006, due to unexpected external political and legal events, had unintended consequences, thus making Cyprus an instructive modern case-study. The article follows three threads based on the same story. The first thread stresses the problems that emerged because the same LSA programme is used for graduation (mostly criterion-referenced) and selection (purely norm-referenced) purposes. The second thread discusses the problems arising from the lack of comparability between examination subjects. The third thread stresses the vulnerability of the system, because Cyprus partly depends on other countries to offer access to higher education to its citizens. (Contains 2 notes.)

The population in Cyprus, a recent European Union member, has become much more heterogeneous during the past decade. Here, we examine the attainment patterns of minority and native students enrolled in six secondary schools from different cities in Cyprus, and identify factors responsible for these patterns. The combination of examined factors has not, to our knowledge, taken place in previous studies. Findings confirm that ethnic minority groups perform significantly lower than native students. In terms of aetiology, we show that ethnic background, gender, parental education, parental occupation, generation status, absenteeism and school minority concentration have a significant effect on student attainment. (Contains 6 figures and 2 tables.)

The study aimed to field-test a Greek version of the Wechsler Quicktest and to examine its psychometric properties. The Quicktest was individually administered to 208 students, aged 5-14 years, along with a reading test. Based on the Rasch analysis, data for the Quicktest subtests showed acceptable fit to the model. Also, correlations were found between the Quicktest subtests. Moreover, the Quicktest was significantly correlated to school grades, teachers' predictions, scores on the Learning Disability Evaluation Scale (LDES) and Trigka's Reading Test. A Spearman analysis indicated significant correlations between scores on the Quicktest subtests and the grade level of students. Also, a regression analysis indicated that scores on the Quicktest can predict the scores on the other measures. Results supported the Quicktest's criterion-related validity and construct validity and, also, its utility as part of a systematic diagnostic procedure for learning disabilities. (Contains 5 tables.)

The aims of this study are (a) to examine the sources of differential functioning by gender via differential bundle functioning (DBF) in mathematics assessment and (b) to use DBF to explore whether the differential functioning displayed is construct-relevant or construct-irrelevant. Three qualitatively different areas, namely curriculum domains, test modality, and problem presentation, were investigated using the Roussos and Stout (1996) multidimensionality-based differential item functioning analysis paradigm with an adaptation of the Logistic Regression approach to model DBF. An empirical dataset from a mathematics national examination of 11-year-old pupils in England was analyzed. We argue that DBF found in curriculum domains may be construct-relevant of the measure, whereas problem presentation and test modality include arguably construct-irrelevant and so may indicate gender bias. (Contains 4 tables and 1 figure.)

Accepting that school based assessment may have the potential to bring additional reliability to the assessment outcomes of an educational system, this research uses Generalizability Theory to address the question "why school based assessment is not a universal feature of high stakes assessment systems"? Three major issues are identified: (a) there is a conflict between the psychometric model and classroom assessment practice; (b) different schools are not equally effective; and, (c) teachers' judgments are frequently accused of being biased. The role of public examination boards is discussed in this context.

Heated discussions about the comparability of standards between examination subjects have kept Qualification Authorities, Testing Services, independent researchers and academics around the world busy for many years. As a result, many countries have adopted statistical techniques which aspire to make aggregated scores based on different subjects comparable. This paper presents eight international case studies and highlights the criticisms against the statistical comparability methods. The side-effects (problems) they may have sparked in various countries are discussed in length. The author maintains the view that the Gordian knot in this case is not statistical: it is one of perceived unfairness. The Qualification Authorities and the Testing Services should step forward and explain to the stakeholders what can be done and what cannot be done using statistical techniques. Once the stakeholders accept the limitations of any statistical technique, it will be easier for them to strike a balance between the desire for comparability and the need to democratically allow students to choose their own combination of subjects to study. The psychometricians should not be forced to apply statistical solutions to complex problems caused by political decisions; they are more likely to fail than to succeed. (Contains 1 figure.)

This study investigates the effect of reporting the unadjusted raw scores in a high-stakes language exam when raters differ significantly in severity and self-selected questions differ significantly in difficulty. More sophisticated models, introducing meaningful facets and parameters, are successively used to investigate the characteristics of the dataset. The application of the Rasch models to the data showed that examinees could benefit significantly from being marked by lenient raters and by responding to less demanding essay questions. It was also shown that the third rater failed to adjust the raw scores in a way similar to the statistical adjustment by the Rasch models. The study discusses the consequences of reporting unadjusted raw scores with particular emphasis on issues of fairness. (Contains 9 figures and 4 tables.)

Background: Athletics programmes for secondary schools include a variety of skills, knowledge and cognitive abilities, which are currently assessed through written, practical, oral and/or video-based tests. Skills are traditionally taught in practice-based sessions, while the knowledge aspect is often reinforced in class-based sessions with video/computer-assisted instruction. Currently, it is still not clear whether these teaching environments have an effect on written and video modes of assessment. This research attempted to investigate this lack of clarity for the content knowledge of techniques, planning tactics and rules in athletics. Purpose: To investigate the impact of written and video-based assessments on three teaching environments. The aim of this research was to assess: (i) which of the three teaching environments contributed most to learning; (ii) whether the written and video-based assessments communicate the same assessment information in relation to the teaching environment; and (iii) whether content knowledge, such as techniques, planning tactics and rules, taught in a practical performance teaching situation adequately prepares students for written and video-based assessments. Setting: A written paper and a video-based paper were used to measure the learning gains of the participants of three teaching environments: practice; practice and handouts; and video-based class sessions in the context of athletics. The practice-based sessions were similar to traditional performance-based physical education (PE) lessons. During the practice and handouts class, information sheets were used to consolidate the content. The class-based sessions included discussions, video clips, information sheets and written class work. Participants: There were 49 participants, (16 females and 33 males). Intact classes from different schools were randomly assigned to the experimental teaching environments; practice 12; practice and handouts 25; and class sessions 12. The participants of the study were aged 16 years, in their first term of post-secondary schooling in Malta. Intervention: Content knowledge of rules, planning tactics and techniques of sprints, relays and the shot put in athletics was taught in three experimental teaching units of six hours each. Two assessment instruments--a written paper (35 test items) and a video-based written paper (same test items with video clips)--were administered to each participant before and after the experimental teaching. Research design: The study was a "quasi" experimental design. Two experimental assessment instruments: a written assessment; and a video-based assessment were administered to each participant before and after the experimental teaching treatment of six hours. The three experimental teaching units represented three teaching environments: Athletics Practice; Athletics Practice and handouts; and Athletics class sessions. Data collection and analysis: Raw and logit scores for the three content domains were computed for the written assessment and the video-based assessment. A partial Credit Rasch Model analysis was used to measure the uni-dimensionality, reliability and the fit of the test items. ANOVA (analysis of variance), MANOVA (multiple analysis of variance repeated measures) and Scheffe Multiple Range test procedures were employed to investigate the effectiveness and any main effects and significant differences of the experimental teaching environments on the assessment of content domains. Findings: The subjects of the class-based teaching environment achieved statistically significant (p = 0.05) better scores on rules (M = 24.21; SD = 3.58; Range = 0 - 30), planning tactics (M = 19.00; SD = 6.37; Range = 0 - 30), and techniques (M = 20.67; SD = 16.18; Range = 0 - 40), when video-based assessments were used. Conclusions: The findings of this study suggest that a written assessment and a video-based assessment test different competencies on the same test items. The findings of the study indicate that students familiar with video analysis will cope satisfactorily with video-based assessments. Practice-based teaching environments without video-based analysis are sufficient for written assessments, yet not for video-based assessments in athletics. (Contains 1 figure, 7 tables and 1 note.)

The "Cultural Journeys in the Information Society" is a dynamic hypermedia environment, which proposes the "Electronic Roads" as a meta-form for exploring cultural information that can form "Cultural Journeys." The Electronic Roads meta-form facilitates travelers to explore the information space in a natural and continuous way similar to the exploration of physical roads. This meta-form is advantageous over existing cultural exploration approaches, which usually apply only to discrete information spaces, failing to provide users with continuous explorations and as a result users may lose their intended destinations. The proposed environment provides extendable cultural and historical content for three Mediterranean countries, i.e. Cyprus, Jordan and Egypt. Such a system could be ideal for teaching history and culture; therefore several "Learning Activities" have been developed in this context. The learning effectiveness of the environment was evaluated in primary schools in Cyprus, where pupils were asked to complete specific Learning Activities by either using the Cultural Journeys in the Information Society environment or a static hypermedia system. The results showed that the pupils found our environment to be easier to use, compared to the static system and that the performance of the pupils that used it was significantly higher.