Search

Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although numerous stopping rules have been proposed to guide users in this decision, they have yet to undergo comprehensive evaluation. In this study, we evaluated the performance of three stopping rules: the knee method, a data-driven heuristic, and a prevalence estimation technique. We measured performance via sensitivity, specificity, and screening cost and explored the influence of the prevalence of relevant studies and the choice of the learning algorithm. We curated a dataset of abstract collections from meta-analyses across five psychological research domains. Our findings revealed performance differences between stopping rules regarding all performance measures and variations in the performance of stopping rules across different prevalence ratios. Moreover, despite the relatively minor impact of the learning algorithm, we found that specific combinations of stopping rules and learning algorithms were most effective for certain prevalence ratios of relevant abstracts. Based on these results, we derived practical recommendations for users of AI-aided screening tools. Furthermore, we discuss possible implications and offer suggestions for future research.

Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% (SD = 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days (SD = 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity (M = 42%, SD = 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.

The digital gender divide is one of the most critical issues in education today. Digital gender gaps can exist in students' access to and use of ICT, attitudes toward technology, and digital knowledge and skills. However, previous research has primarily examined these divides in isolation and largely ignored their interdependencies. Using student data from the 2013 and 2018 International Computer and Information Literacy Study (ICILS), this study examines how these gaps are related. Specifically, we synthesize evidence on gender gaps in digital knowledge and skills, controlling for differences in ICT access and technology attitudes. In addition, we examine the role that technology attitudes play in the gender gap in digital knowledge and skills. Our findings suggest that (a) girls outperform boys in digital knowledge and skills ([beta] = -0.11 to -0.29); (b) gender differences in attitudes toward technology partially explain gender differences in digital knowledge and skills; (c) the variability within and between countries in the gender gap in students' digital skills is partly explained by the type of digital skill and the country's socio-economic development and gender inequality. Overall, this research shows a relationship between the gender gaps in students' digital knowledge and skills and the gender gaps in students' attitudes toward technology. We conclude that the first- and second-level gender digital divides are connected.