Search

As the empirical literature in educational technology continues to grow, meta-analyses are increasingly being used to synthesise research to inform practice. However, not all meta-analyses are equal. To examine their evolution over the past 30 years, this study systematically analysed the quality of 52 meta-analyses (1988–2017) on educational technology. Methodological and reporting quality is defined here as the completeness of the descriptive and methodological reporting features of meta-analyses. The study employed the Meta-Analysis Methodological Reporting Quality Guide (MMRQG), an instrument designed to assess 22 areas of reporting quality in meta-analyses. Overall, MMRQG scores were negatively related to average effect size (i.e., the higher the quality, the lower the effect size). Owing to the presence of poor-quality syntheses, the contribution of educational technologies to learning has been overestimated, potentially misleading researchers and practitioners. Nine MMRQG items discriminated between higher and lower average effect sizes. A publication date analysis revealed that older reviews (1988–2009) scored significantly lower on the MMRQG than more recent reviews (2010–2017). Although the increase in quality bodes well for the educational technology literature, many recent meta-analyses still show only moderate levels of quality. Identifying and using only best evidence-based research is thus imperative to avoid bias. Implications for practice or policy: Educational technology practitioners should make use of meta-analytical findings that systematically synthesise primary research. Academics, policymakers and practitioners should consider the methodological quality of meta-analyses as they vary in reliability. Academics, policymakers and practitioners could avoid misleading bias in research evidence by using the MMRQG to evaluate the quality of meta-analyses. Meta-analyses with lower MMRQG scores should be considered with caution as they seem to overestimate the effect of educational technology on learning.

Research on distance education, online learning, and blended learning are examined from the perspectives of quantitative (meta-analyses) and qualitative (meta-syntheses) systematic reviews. While not all of the reviews are outstanding examples of the best characteristics of systematic reviews, overall, they portray an interesting and informative perspective on the state of research, and its evolution over the past 15 or so years. Future directions for research and practice are presented and discussed.

Critical thinking (CT) is purposeful, self-regulatory judgment that results in interpretation, analysis, evaluation, and inference, as well as explanations of the considerations on which that judgment is based. This article summarizes the available empirical evidence on the impact of instruction on the development and enhancement of critical thinking skills and dispositions and student achievement. The review includes 341 effects sizes drawn from quasi- or true-experimental studies that used standardized measures of CT as outcome variables. The weighted random effects mean effect size ( g+) was 0.30 ( p < .001). The collection was heterogeneous ( p < .001). Results demonstrate that there are effective strategies for teaching CT skills, both generic and content specific, and CT dispositions, at all educational levels and across all disciplinary areas. Notably, the opportunity for dialogue, the exposure of students to authentic or situated problems and examples, and mentoring had positive effects on CT skills.

This article has two interrelated purposes. The first is to explain how various forms of bias, if introduced during any stage of a meta-analysis, can provide the consumer with a misimpression of the state of a research literature. Five of the most important bias-producing aspects of a meta-analysis are presented and discussed. Second, armed with this information, we examine 15 meta-analyses of the literatures of distance education (DE), online learning (OL), and blended learning (BL), conducted from 2000 to 2014, with the intention of assessing potential sources of bias in each. All of these meta-analyses address the question: “How do students taking courses through DE, OL, and BL compare to students engaged in pure classroom instruction in terms of learning achievement outcomes?” We argue that questions asked by primary researchers must change to reflect issues that will drive improvements in designing and implementing DE, OL, and BL courses.

This paper summarizes the results of a pan-Canadian online survey study that investigates the extent to which school practitioners (N = 1,153) use research to inform their practice. The self-reports indicate that the majority of the respondents used educational research, yet this engagement was infrequent. Although the respondents shared neutral attitudes about research, their comments add rather negative connotation to their perceptions. This study’s findings are relevant to school leadership organizations, teacher education institutions and research-generating bodies as they point to the necessity of increasing research relevance and accessibility, cultivating teaching as a research-based profession, and building school capacity to use research.

This systematic review draws from and builds upon the results of a meta-analysis of the achievement effects of three types of interaction treatments in distance education: student–student, student–teacher, and student–content (Bernard et al., Review of Educational Research, 79(3), 1243–1289, 2009). This follow-up study considers two forms of student–student interaction treatments, contextual interaction and designed interaction. Typical contextual interaction treatments contain the necessary conditions for student–student interaction to occur, but are not intentionally designed to create collaborative learning environments. By contrast, designed interaction treatments are intentionally implemented collaborative instructional conditions for increasing student learning. Our meta-analysis compared the effect of these two types of interaction treatments on student achievement outcomes. The results favored designed interaction treatments over contextual interaction treatments. Examples of designed interaction ...

The trend towards using research knowledge to improve policies and practices is on the rise. However, despite considerable effort and notable progress in recent years, it seems that school practitioners continue to make little use of research and it is not clear what conditions would facilitate or obstruct this use. This review focuses exclusively on the available empirical1 research about (a) the use of research by school practitioners and (b) the determinants of use, and identifies future directions for research.

A vexing problem for meta-analysts is how to account for the differential methodological quality of studies. We critically examine five approaches that deal with methodological quality. The first approach is to ignore the methodological quality of studies and aggregate all available research for which effect sizes (ES) can be calculated. The second approach is to use methodological quality as an inclusion criterion, thereby excluding all studies that fail to meet some predetermined level of quality. The third approach is to use methodological quality to classify included studies and report their findings separately. The fourth approach is to weight studies by methodological quality. And the fifth approach is to treat methodological quality as a predictor of ES by removing its influence from the collection of evidence. After discussing some of the limitations of the five approaches, we describe a method for treating methodological quality as a study feature, coding for it, and adjusting for its influence. ...

This research study employs a second-order meta-analysis procedure to summarize 40 years of research activity addressing the question, does computer technology use affect student achievement in formal face-to-face classrooms as compared to classrooms that do not use technology? A study-level meta-analytic validation was also conducted for purposes of comparison. An extensive literature search and a systematic review process resulted in the inclusion of 25 meta-analyses with minimal overlap in primary literature, encompassing 1,055 primary studies. The random effects mean effect size of 0.35 was significantly different from zero. The distribution was heterogeneous under the fixed effects model. To validate the second-order meta-analysis, 574 individual independent effect sizes were extracted from 13 out of the 25 meta-analyses. The mean effect size was 0.33 under the random effects model, and the distribution was heterogeneous. Insights about the state of the field, implications for technology use, and prospects for future research are discussed.

This paper describes a range of models and methods for synthesizing diverse forms of research evidence. Although this topic is applicable to many contexts (e.g. education, health care), the focus here is on the research traditions of distance education and online learning. Thirteen models and methods are described, and seven examples from distance education and online learning are used to illustrate them. The models are classified as systematic, purposeful, and mixed and are described and compared in terms of purpose, methodological aspects, and expected outcomes.

A brief review of evidence is limited in time and/or scope compared to a comprehensive review. However, brief reviews are important not only in meeting the needs of policy makers and practitioners, but also in providing students and researchers with an overview of the evidence. In this paper we summarise and evaluate alternative methods for brief reviews, including: using strict inclusion criteria; reviewing only a sample of evidence and eliminating or reducing steps in the review process. We examine a sample of brief reviews and found that the majority did not meet the methodological standards of comprehensive reviews. We conclude by recommending some methodological standards for brief reviews.

This meta-analysis of the experimental literature of distance education (DE) compares different types of interaction treatments (ITs) with other DE instructional treatments. ITs are the instructional and/or media conditions designed into DE courses, which are intended to facilitate student–student (SS), student–teacher (ST), or student–content (SC) interactions. Seventy-four DE versus DE studies that contained at least one IT are included in the meta-analysis, which yield 74 achievement effects. The effect size valences are structured so that the IT or the stronger IT (i.e., in the case of two ITs) serve as the experimental condition and the other treatment, the control condition. Effects are categorized as SS, ST, or SC. After adjustment for methodological quality, the overall weighted average effect size for achievement is 0.38 and is heterogeneous. Overall, the results support the importance of the three types of ITs and strength of ITs is found to be associated with increasing achievement outcomes. A strong association is found between strength and achievement for asynchronous DE courses compared to courses containing mediated synchronous or face-to-face interaction. The results are interpreted in terms of increased cognitive engagement that is presumed to be promoted by strengthening ITs in DE courses.

Critical thinking (CT), or the ability to engage in purposeful, self-regulatory judgment, is widely recognized as an important, even essential, skill. This article describes an ongoing meta-analysis that summarizes the available empirical evidence on the impact of instruction on the development and enhancement of critical thinking skills and dispositions. We found 117 studies based on 20,698 participants, which yielded 161 effects with an average effect size ( g+) of 0.341 and a standard deviation of 0.610. The distribution was highly heterogeneous ( QT= 1,767.86, p < .001). There was, however, little variation due to research design, so we neither separated studies according to their methodological quality nor used any statistical adjustment for the corresponding effect sizes. Type of CT intervention and pedagogical grounding were substantially related to fluctuations in CT effects sizes, together accounting for 32% of the variance. These findings make it clear that improvement in students’ CT skills and dispositions cannot be a matter of implicit expectation. As important as the development of CT skills is considered to be, educators must take steps to make CT objectives explicit in courses and also to include them in both preservice and in-service training and faculty development.

The purpose of this research is to investigate the relationship between computer technology's role and students' perceptions about course effectiveness. Students from two universities (one Canadian, n = 1465; one American, n = 831) completed a 71–item questionnaire addressing different aspects of their learning experience in a given course. Factor analysis revealed a 3–factor solution: “course-structure,” “active-learning and time-on-task,” and “computer-use.” Regression analysis indicated that the 3 variables are predictive of perceived course effectiveness at both sites, with the presence of an interaction between location and “computer-use” and “course-structure” on students' perceptions about course effectiveness. Findings reveal that student perceptions directly reflect the 14 APA learner-centered principles on which the instrument was based.

Although the overall research literature on the application of educational technologies to classroom instruction tends to favor their use over their non-use, these results vary considerably depending on what kind of technology is used, who it is used with and, more importantly, under what circumstances and for what instructional purposes it is used. Relatively recent, but well-developed and powerful methodology of systematic reviews, particularly quantitative syntheses (also known as meta-analyses) is especially suitable for addressing questions of that type by systematically summarizing research evidence in given areas of interest in social sciences.This meta-analysis summarizes data from 674 independent primary studies that compared higher degrees of technology use in the experimental condition with less technology in the control condition, in terms of their effects on student learning outcomes in postsecondary education. The result was an overall average weighted effect size of = 0.27 (k = 879, p < .01), indicating low but significant positive effect of technology integration on learning. The follow-up analyses revealed the influence of educational technology used for cognitive support and blended learning instructional settings designed interaction treatments, and technology integration in teacher training, especially when student-centered pedagogical frameworks are used. These findings are of potentially high interest and applied value for educational practitioners, including teachers and school administrators, as well as for instructional designers and developers of educational software.

English In this article we explore issues in developing an Argument Catalogue as a tool to compile evidence systematically on a topic of interest. A recent review is used to illustrate the steps in undertaking the development of an Argument Catalogue from multiple sources. An Argument Catalogue codebook, which provides categories and values that can be applied in order to classify conjectures, opinions, concerns and expert judgements, is also developed. Arguments are then compared and contrasted in two different ways: by quantifying the predictor and outcome variables described and by summarising qualitatively the major messages in each document.

This meta-analysis employs a theoretical framework in quantitatively synthesizing empirical studies that investigate the effects of distance education (DE) versus classroom instruction on undergraduate student achievement. Analyses of 218 findings from 103 studies were conducted according to how media were used to support DE pedagogy. The results indicate that the effect sizes far synchronous instructor-directed DE were consistent and not significantly different from zero; in asynchronous DE, media only supporting independent learning was generally less effective than media supporting collaborative discussion among students, although both subsets were significantly heterogeneous. Follow-up analysis of asynchronous DE findings was framed in terms of three patterns of interaction—student-content, student-instructor and student-student.

This article extends the issues and arguments raised in Bernard, Abrami, Lou, and Borokhovski (Distance Education, 25(2), 175–198, 2004) regarding the design of quantitative, particularly experimental research in distance education. A single experimental, study from the distance education literature is examined from six different perspectives to show the differences between preexperiments, true experiments, and quasi‐experiments in terms of their impact on interpretability and generalizability (i.e., internal and external validity). Arguments for and against experimentation are discussed and the article ends with a description of meta‐analysis, the quantitative synthesis of experimental research, and its potential for providing answers to questions that no single study can adequately address.

The effect size is a standardized, scale-free measure of the relative size of the effect of an interven- tion, and it has important and practical implications for clinicians in the speech and hearing field who are interested in estimating the effects of interventions. This article develops a conceptual interpretation of the effect size, makes explicit assumptions for its proper use in estimating the size of the effect of behavioral-based stuttering interventions, and explains how to compute the most commonly used effect sizes and their confidence intervals. The focus is on effect sizes for experimental studies on behavioral-based stuttering interventions that produce outcomes measured on a continuous scale. Also explained is how to synthesize these effect sizes across multiple studies to arrive at an average effect size and its confidence interval through a process called meta- analysis. Key assumptions that underlie the use of meta- analysis are explored, as are techniques for assessing whether the average effect size is representative of the multiple effect sizes from which it was derived. The article concludes with a summary of main points and enumerates additional resources for speech and hearing clinicians and practitioners to access and learn more about practical applications of effect sizes and their synthesis through meta-analysis.