Search

As the empirical literature in educational technology continues to grow, meta-analyses are increasingly being used to synthesise research to inform practice. However, not all meta-analyses are equal. To examine their evolution over the past 30 years, this study systematically analysed the quality of 52 meta-analyses (1988-2017) on educational technology. Methodological and reporting quality is defined here as the completeness of the descriptive and methodological reporting features of meta-analyses. The study employed the Meta-Analysis Methodological Reporting Quality Guide (MMRQG), an instrument designed to assess 22 areas of reporting quality in meta-analyses. Overall, MMRQG scores were negatively related to average effect size (i.e., the higher the quality, the lower the effect size). Owing to the presence of poor-quality syntheses, the contribution of educational technologies to learning has been overestimated, potentially misleading researchers and practitioners. Nine MMRQG items discriminated between higher and lower average effect sizes. A publication date analysis revealed that older reviews (1988-2009) scored significantly lower on the MMRQG than more recent reviews (2010-2017). Although the increase in quality bodes well for the educational technology literature, many recent meta-analyses still show only moderate levels of quality. Identifying and using only best evidence-based research is thus imperative to avoid bias.

This paper summarizes the results of a pan-Canadian online survey study that investigates the extent to which school practitioners (N = 1,153) use research to inform their practice. The self-reports indicate that the majority of the respondents used educational research, yet this engagement was infrequent. Although the respondents shared neutral attitudes about research, their comments add rather a negative connotation to their perceptions. This study's findings are relevant to school leadership organizations, teacher education institutions, and research-generating bodies as they point to the necessity of increasing research relevance and accessibility, cultivating teaching as a research-based profession, and building school capacity to use research.

This paper examines sources of potential bias in systematic reviews and meta-analyses which can distort their findings, leading to problems with interpretation and application by practitioners and policymakers. It follows from an article that was published in the "Canadian Journal of Communication" in 1990, "Integrating Research into Instructional Practice: The Use and Abuse of Meta-analysis," which introduced meta-analysis as a means for estimating population parameters and summarizing quantitative research around instructional research questions. This paper begins by examining two cases where multiple meta-analyses disagree. It then goes on to describe substantive and methodological aspects of meta-analysis where various kinds of bias can influence the outcomes and suggests measures that can be taken to avoid them. The intention is to improve the reliability and accuracy of reviews so that practitioners can trust the results and use them more effectively.

Teaching methods that individualize and adapt instructional conditions to K-12 learners' needs, abilities, and interests help improve learning achievement. The most important variables are the teacher's role in the classroom as a guide and mentor and the adaptability of learning activities and materials. This Campbell systematic review assesses the overall impact on student achievement of processes and methods that are more student-centered versus less student-centered. It also considers the strength of student-centered practices in four teaching domains: (1) Flexibility: Degree to which students can contribute to course design, selecting study materials, and stating learning objectives; (2) Pacing of instruction: Students can decide how fast to progress through course content and whether this progression is linear or iterative; (3) Teacher's role: Ranging from authority figure and sole source of information, to teacher as equal partner in the learning process; and (4) Adaptability: Degrees of manipulating learning environments, materials, and activities to make them more student-centered. This review presents evidence from 299 studies (covering 43,175 students in a formal school setting) yielding 365 estimates of the impact of teaching practices. The studies spanned the period 2000-2017 and were mostly carried out in the United States, Europe, and Australia. This review confirms previous research on the effectiveness of student-centered and active learning. It goes further in suggesting the teacher's role promotes effective student-centered learning, and excessive student control over pacing appears to inhibit it.

This systematic review builds upon the work of Abrami et al. (2006) and McGreal and Anderson (2007). It seeks to provide a synthesis and discussion of publicly available government policy documents with regard to e-learning in Canada. In total, 138 policy documents from Canadian provinces and territories and several federal agencies, dated from 2000 to 2010, were identified, retrieved and analyzed using prescriptive and emergent coding approaches with the purpose of uncovering and describing areas of commonality and inconsistency and, hence, determining where discussions about e-learning are lacking. The review confirmed that Canadian policy makers view technology as offering various potential benefits to learners, but also revealed a troubling lack of specific details, consistency and coordination in facilitating the development of e-learning to fulfill these optimistic expectations. (Contains 1 table and 6 figures.)

This study examines the achievement outcomes accompanying the implementation of a Grade 3 laptop or so-­called "ubiquitous computing" program in a Quebec school district. CAT­3 reading, language, and mathematics batteries were administered at the end of Grade 2 and again at the end of Grade 3, after the first year of computer implementation. Overall gain was found in all three content areas, but was differential when compared with the norms of the CAT­3. Additionally, some evidence suggested a differential gain for lower and middle-­level learners during the school year. Teachers were administered an instrument called the "Technology Implementation Questionnaire" (TIQ) that assessed the purposes and extent of technology integration. Negative correlations were found in reading achievement gain for items associated with the higher use of communicative, evaluative, and creative uses of computers. Open-­ended teacher responses indicated the need for more professional support for instructional implementations of computing.

Abrami et al. (this issue) provide a review of e-learning in Canada from 2000 onwards by synthesizing information drawn from multiple sources, not only primary research. In total, there were 726 documents included in our review: 235 views expressed in the public printed media (an expression of general public opinion); 131 views from trade/practitioner perspectives; 88 views of policymakers contained in public policy documents; 120 sources of evidence contained in reviews of research; and 152 sources of evidence contained in primary empirical studies. It is the authors' understanding that this is the first review of its kind to be as inclusive of sources. This is both the review's greatest strength, allowing the authors to determine whether consensus existed among myriad sources, and potentially its greatest weakness, as both the time for such a comprehensive review and the cost meant the authors sacrificed a degree of depth for an increase in breadth. In this rejoinder to the commentaries, they explore several issues: (1) the nature and importance of systematic reviews; (2) answering questions about what works and why; and (3) what is e-learning and what are its impacts.

This review provides a rough sketch of the evidence, gaps and promising directions in e-learning from 2000 onwards, with a particular focus on Canada. The authors searched a wide range of sources and document types to ensure that we represented, comprehensively, the arguments surrounding e-learning. Overall, there were 2,042 entries in their database, of which they reviewed 1,146, including all the Canadian primary research and all scholarly reviews of the literature. In total, there were 726 documents included in the review: 235--general public opinion; 131--trade/practitioners' opinion; 88--policy documents; 120--reviews; and 152--primary empirical research. The Argument Catalogue codebook included the following eleven classes of variables: (1) Document Source; (2) Areas/Themes of e-learning; (3) Value/Impact; (4) Type of evidence; (5) Research design; (6) Area of applicability; (7) Pedagogical implementation factors; 8) A-priori attitudes; (9) Types of learners; (10) Context; and (11) Technology Factors. The authors examined the data from a number of perspectives, including their quality as evidence. In the primary research literature, they examined the kinds of research designs that were used. They found that over half of the studies conducted in Canada are qualitative in nature, while the rest are split in half between surveys and quantitative studies (correlational and experimental). When they looked at the nature of the research designs, they found that 51% are qualitative case studies and 15.8% are experimental or quasi-experimental studies. The findings of this review showed that studies that can increase the general understanding of "what works" in e-learning settings are underrepresented in the Canadian research literature. This article discusses the results of the research as well as implications for K-12 practitioners, post-secondary education, and policymakers.

Research evidence concerning the effects of adaptive teaching and individualized learning remain relatively inconclusive, while there is obviously a need for better understanding how K-12 formal education may be more successful in addressing students' personal needs and interests, accounting for their diverse abilities with the main goal of advancing their learning. The proposed systematic review will not only rely on the most rigorous and comprehensive methodology of meta-analysis, but also should be conceptually sound (i.e., thoroughly exploring educational practices to find consistent links among a multitude of individual pedagogical approaches) and timely (i.e., account for the most recent developments in education). The main research question of the proposed meta-analysis is: Can more Student-Centred (SC) (i.e., more adaptive and individualized) approaches to K-12 instruction be distinguished from more Teacher-Centred (TC) approaches in terms of their effect on student achievements and what substantive and demographic factors moderate these effects? The review outcomes will inform education practitioners and research community of the best instructional practices, preconditions for their successful implementation and potential pitfalls, as well as of directions for further empirical research in the area.

Critical thinking (CT) is purposeful, self-regulatory judgment that results in interpretation, analysis, evaluation, and inference, as well as explanations of the considerations on which that judgment is based. This article summarizes the available empirical evidence on the impact of instruction on the development and enhancement of critical thinking skills and dispositions and student achievement. The review includes 341 effects sizes drawn from quasi- or true-experimental studies that used standardized measures of CT as outcome variables. The weighted random effects mean effect size (g+) was 0.30 (p < 0.001). The collection was heterogeneous (p < 0.001). Results demonstrate that there are effective strategies for teaching CT skills, both generic and content specific, and CT dispositions, at all educational levels and across all disciplinary areas. Notably, the opportunity for dialogue, the exposure of students to authentic or situated problems and examples, and mentoring had positive effects on CT skills.

This article contains a second-order meta-analysis and an exploration of bias in the technology integration literature in higher education. Thirteen meta-analyses, dated from 2000 to 2014 were selected to be included based on the questions asked and the presence of adequate statistical information to conduct a quantitative synthesis. The weighted random effects average was g [superscript ++] = 0.393, p <0.000. The article goes on to report an assessment of the methodological quality of the thirteen studies based on Cooper's ("Research synthesis and meta-analysis: a step-by-step approach." Sage, Thousand Oaks, 2010) seven stages in the development of a meta-analysis. Two meta-analyses were found to have five out of seven stages where methodological flaws could potentially create biased results. Five meta-analyses contained two flawed stages and one contained one flawed stage. Four of the stages where methodological flaws can create bias are described in detail. The final section attempts to determine how much influence the methodological flaws exerted on the results of the second-order meta-analysis.

This paper serves several purposes. First and foremost, it is devoted to developing a better understanding of the effectiveness of blended learning (BL) in higher education. This is achieved through a meta-analysis of a sub-collection of comparative studies of BL and classroom instruction (CI) from a larger systematic review of technology integration (Schmid et al. in "Comput Educ" 72:271-291, 2014). In addition, the methodology of meta-analysis is described and illustrated by examples from the current study. The paper begins with a summary of the experimental research on distance education (DE) and online learning (OL), encapsulated in meta-analyses that have been conducted since 1990. Then it introduces the Bernard et al. ("Rev Educ Res" 74(3):379-439, 2009) meta-analysis, which attempted to alter the DE research culture of always comparing DE/OL with CI by examining three forms of "interaction treatments" (i.e., student-student, student-teacher, student-content) within DE, using the theoretical framework of Moore ("Am J Distance Educ" 3(2):1-6, 1989) and Anderson ("Rev Res Open Distance Learn" 4(2):9-14, 2003). The rest of the paper revolves around the general steps and procedures (Cooper in "Research Synthesis and Meta-Analysis: A Step-by-Step Approach," 4th edn, SAGE, Los Angeles, CA, 2010) involved in conducting a meta-analysis. This section is included to provide researchers with an overview of precisely how meta-analyses can be used to respond to more nuanced questions that speak to underlying theory and inform practice--in other words, not just answers to the "big questions." In this instance, we know that technology has an overall positive impact on learning (g[superscript +] = +0.35, p < 0.01, Tamim et al. in "Rev Educ Res" 81(3):4-28, 2011), but the sub-questions addressed here concern BL interacting with technology in higher education. The results indicate that, in terms of achievement outcomes, BL conditions exceed CI conditions by about one-third of a standard deviation (g[superscript +] = 0.334, k = 117, p < 0.001) and that the kind of computer support used (i.e., cognitive support vs. content/presentational support) and the presence of one or more interaction treatments (e.g., student-student/-teacher/-content interaction) serve to enhance student achievement. We examine the empirical studies that yielded these outcomes, work through the methodology that enables evidence-based decision-making, and explore how this line of research can improve pedagogy and student achievement.

This article has two interrelated purposes. The first is to explain how various forms of bias, if introduced during any stage of a meta-analysis, can provide the consumer with a misimpression of the state of a research literature. Five of the most important bias-producing aspects of a meta-analysis are presented and discussed. Second, armed with this information, we examine 15 meta-analyses of the literatures of distance education (DE), online learning (OL), and blended learning (BL), conducted from 2000 to 2014, with the intention of assessing potential sources of bias in each. All of these meta-analyses address the question: "How do students taking courses through DE, OL, and BL compare to students engaged in pure classroom instruction in terms of learning achievement outcomes?" We argue that questions asked by primary researchers must change to reflect issues that will drive improvements in designing and implementing DE, OL, and BL courses.

The trend towards using research knowledge to improve policies and practices is on the rise. However, despite considerable effort and notable progress in recent years, it seems that school practitioners continue to make little use of research and it is not clear what conditions would facilitate or obstruct this use. This review focuses exclusively on the available empirical research about (a) the use of research by school practitioners and (b) the determinants of use, and identifies future directions for research.

This systematic review draws from and builds upon the results of a meta-analysis of the achievement effects of three types of interaction treatments in distance education: student-student, student-teacher, and student-content (Bernard et al., "Review of Educational Research," 79(3), 1243-1289, 2009). This follow-up study considers two forms of student-student interaction treatments, "contextual interaction" and "designed interaction". Typical contextual interaction treatments contain the necessary conditions for student-student interaction to occur, but are not intentionally designed to create collaborative learning environments. By contrast, "designed interaction treatments" are intentionally implemented collaborative instructional conditions for increasing student learning. Our meta-analysis compared the effect of these two types of interaction treatments on student achievement outcomes. The results favored designed interaction treatments over contextual interaction treatments. Examples of designed interaction treatments and a discussion of study results and their potential implications for research and instruction in distance education and online learning are presented. (Contains 3 tables.)

In a recent meta-analysis of distance and online learning, Bernard et al. (2009) quantitatively verified the importance of three types of interaction: among students, between the instructor and students, and between students and course content. In this paper we explore these findings further, discuss methodological issues in research and suggest how these results may foster instructional improvement. We highlight several evidence-based approaches that may be useful in the next generation of distance and online learning. These include principles and applications stemming from the theories of self-regulation and multimedia learning, research-based motivational principles and collaborative learning principles. We also discuss the pedagogical challenges inherent in distance and online learning that need to be considered in instructional design and software development.

This study investigated the change in the relationship between pedagogy, computer-use and students' perceptions about course-effectiveness over time. Students from a Canadian university completed a questionnaire in two different years (2003 = 1,834 participants and 2007 = 1,866 participants). Of greatest interest were characteristics of technology that interact with pedagogy to achieve positive learning outcomes. A factor analysis revealed a three-factor solution: "course-structure," "active-learning," and "computer-use." Multiple regression analysis showed that the three variables are predictive of perceived course effectiveness, with "course-structure" being most predictive in both years. "Computer-use" was least predictive with the 2003 sample while it was second in predictive power with the 2007 sample, most likely reflecting increased technology integration in post-secondary education. When comparing use of various applications in "Arts" versus "Science" courses, results indicated higher computer technology use in 2007 for all applications with arts courses while only web-based computer applications increased in use with the science courses. Separate regression analyses were conducted for each type of program while comparing the two different study years with results indicating that "course-structure" is the most stable predictor. Findings reveal that while pedagogy seems to be of highest importance to students, the relationship between computer use and perceived course effectiveness is changing over time. Implications are discussed and suggestions for future research are presented.

This research study employs a second-order meta-analysis procedure to summarize 40 years of research activity addressing the question, does computer technology use affect student achievement in formal face-to-face classrooms as compared to classrooms that do not use technology? A study-level meta-analytic validation was also conducted for purposes of comparison. An extensive literature search and a systematic review process resulted in the inclusion of 25 meta-analyses with minimal overlap in primary literature, encompassing 1,055 primary studies. The random effects mean effect size of 0.35 was significantly different from zero. The distribution was heterogeneous under the fixed effects model. To validate the second-order meta-analysis, 574 individual independent effect sizes were extracted from 13 out of the 25 meta-analyses. The mean effect size was 0.33 under the random effects model, and the distribution was heterogeneous. Insights about the state of the field, implications for technology use, and prospects for future research are discussed. (Contains 1 note and 3 tables.)

This paper describes a range of models and methods for synthesizing diverse forms of research evidence. Although this topic is applicable to many contexts (e.g. education, health care), the focus here is on the research traditions of distance education and online learning. Thirteen models and methods are described, and seven examples from distance education and online learning are used to illustrate them. The models are classified as systematic, purposeful, and mixed and are described and compared in terms of purpose, methodological aspects, and expected outcomes. (Contains 4 tables and 1 figure.)

This paper reports the findings of a Stage I meta-analysis exploring the achievement effects of computer-based technology use in higher education classrooms (non-distance education). An extensive literature search revealed more than 6,000 potentially relevant primary empirical studies. Analysis of a representative sample of 231 studies (k = 310) yielded a weighted average effect size of 0.28 surrounded by wide variability. A mixed effects model was adopted to explore coded moderators of effect size. Research design was found to be not significant across true, quasi- and pre-experimental designs, so the designs were combined. The variable "degree of technology use" (i.e., low, medium, and high) was found to be significant, with low and medium use performing significantly higher than high use. For the variable "type of use" (i.e., cognitive support tools, presentational tools, and multiple uses), cognitive support (g+ = 0.40) was greater than presentational and multiple uses.

This meta-analysis of the experimental literature of distance education (DE) compares different types of interaction treatments (ITs) with other DE instructional treatments. ITs are the instructional and/or media conditions designed into DE courses, which are intended to facilitate student-student (SS), student-teacher (ST), or student-content (SC) interactions. Seventy-four DE versus DE studies that contained at least one IT are included in the meta-analysis, which yield 74 achievement effects. The effect size valences are structured so that the IT or the stronger IT (i.e., in the case of two ITs) serve as the experimental condition and the other treatment, the control condition. Effects are categorized as SS, ST, or SC. After adjustment for methodological quality, the overall weighted average effect size for achievement is 0.38 and is heterogeneous. Overall, the results support the importance of the three types of ITs and strength of ITs is found to be associated with increasing achievement outcomes. A strong association is found between strength and achievement for asynchronous DE courses compared to courses containing mediated synchronous or face-to-face interaction. The results are interpreted in terms of increased cognitive engagement that is presumed to be promoted by strengthening ITs in DE courses. (Contains 11 tables.)

Critical thinking (CT), or the ability to engage in purposeful, self-regulatory judgment, is widely recognized as an important, even essential, skill. This article describes an ongoing meta-analysis that summarizes the available empirical evidence on the impact of instruction on the development and enhancement of critical thinking skills and dispositions. We found 117 studies based on 20,698 participants, which yielded 161 effects with an average effect size (g+) of 0.341 and a standard deviation of 0.610. The distribution was highly heterogeneous (Q[subscript T] = 1,767.86, p less than 0.001). There was, however, little variation due to research design, so we neither separated studies according to their methodological quality nor used any statistical adjustment for the corresponding effect sizes. Type of CT intervention and pedagogical grounding were substantially related to fluctuations in CT effects sizes, together accounting for 32% of the variance. These findings make it clear that improvement in students' CT skills and dispositions cannot be a matter of implicit expectation. As important as the development of CT skills is considered to be, educators must take steps to make CT objectives explicit in courses and also to include them in both preservice and in-service training and faculty development. (Contains 1 note, 17 tables, and 1 figure.)

Critical thinking (CT) has been of longstanding interest among scholars, educators, and others who are concerned with thinking skills. The Watson-Glaser Critical Thinking Appraisal (WGCTA) is the oldest and among the most widely used and studied CT measure. It was constructed around five subscales (or CT skills): inference, recognition of assumptions, deduction, interpretation, and evaluation of arguments. This paper describes a two part analysis of the psychometric properties of the WGCTA, based on 13 sets of subscale inter-correlations and 60 sets of subscale means retrieved from published studies. We performed a meta-analysis on the inter-correlations of the 10 combinations of subscales and found that all of the average correlations that resulted were significant, but that all but one was significantly heterogeneous. Subsequently, we conducted principal components analysis on 60 subscale means of two different versions of the WGCTA. Each produced a one-factor solution, accounting for 82.69% and 79.55% of the total variance, respectively. Together these two parts of this study suggest that the WGCTA should be viewed as a measure of general competency, and that the subscales should not be interpreted individually.

The purpose of this research is to investigate the relationship between computer technology's role and students' perceptions about course effectiveness. Students from two universities (one Canadian, n = 1465; one American, n = 831) completed a 71-item questionnaire addressing different aspects of their learning experience in a given course. Factor analysis revealed a 3-factor solution: "course-structure," "active-learning and time-on-task," and "computer-use." Regression analysis indicated that the 3 variables are predictive of perceived course effectiveness at both sites, with the presence of an interaction between location and "computer-use" and "course-structure" on students' perceptions about course effectiveness. Findings reveal that student perceptions directly reflect the 14 APA learner-centered principles on which the instrument was based. (Contains 5 tables.)

This article extends the issues and arguments raised in Bernard, Abrami, Lou, and Borokhovski ("Distance Education", 25(2), 175-198, 2004) regarding the design of quantitative, particularly experimental research in distance education. A single experimental, study from the distance education literature is examined from six different perspectives to show the differences between preexperiments, true experiments, and quasi-experiments in terms of their impact on interpretability and generalizability (i.e., internal and external validity). Arguments for and against experimentation are discussed and the article ends with a description of meta-analysis, the quantitative synthesis of experimental research, and its potential for providing answers to questions that no single study can adequately address. (Contains 4 tables and 1 figure.)

This meta-analysis employs a theoretical framework in quantitatively synthesizing empirical studies that investigate the effects of distance education (DE) versus classroom instruction on undergraduate student achievement. Analyses of 218 findings from 103 studies were conducted according to how media were used to support DE pedagogy. The results indicate that the effect sizes for synchronous instructor-directed DE were consistent and not significantly different from zero; in asynchronous DE, media only supporting independent learning was generally less effective than media supporting collaborative discussion among students, although both subsets were significantly heterogeneous. Follow-up analysis of asynchronous DE findings was framed in terms of three patterns of interaction-student-content, student-instructor and student-student. (Contains 4 tables.)

This article is about the quantitative research practices and methodologies that are used in distance education (DE). It begins with an analysis and assessment of a segment of the DE research literature, DE/classroom comparison studies, based on a recently completed meta-analysis of that literature from 1985 to 2002. Overall, the 232 studies reviewed were judged to be of poor methodological quality and severely lacking in critical information about research practices. Studies of synchronous and asynchronous DE are discussed separately and recommendations are made for improving designs and measures within these patterns. Suggestions for future quantitative research areas are provided. In discussing these findings, we recognize that high-quality research is being conducted in the field and that qualitative forms of research contribute greatly to the mosaic of evidence that is the base of available knowledge about DE. Finally, three organizations, the What Works Clearinghouse (USA), EPPI-Centre (UK), and the Campbell Collaboration (international), all devoted to improving the quality of research and research synthesis in education, are described briefly; suggestions are made as to how their philosophies and approaches for judging the worthiness of research evidence can be used to improve DE research.

The study reported here concerns the development and predictive validation of an instrument to assess the achievement outcomes of DE/online learning success. A 38-item questionnaire was developed and administered to 167 students who were about to embark on an online course. Factor analysis indicated a four-factor solution, interpreted as "general beliefs about DE," "confidence in prerequisite skills," "self-direction and initiative" and "desire for interaction." Using multiple regression we found that two of these factors predicted achievement performance (i.e., Cumulative Course Grade). Comparisons of pretest and posttest administrations of the questionnaire revealed that some changes in opinion occurred between the beginning and the end of the course. Also, categories of demographic characteristics were compared on the four factors. The overall results suggest that this instrument has some predictive validity in terms of achievement, but that Cumulative Grade Point Average (i.e., the university's record of overall achievement) is a much better predictor.

A meta-analysis of the comparative distance education (DE) literature between 1985 and 2002 was conducted. In total, 232 studies containing 688 independent achievement, attitude, and retention outcomes were analyzed. Overall results indicated effect sizes of essentially zero on all three measures and wide variability. This suggests that many applications of DE outperform their classroom counterparts and that many perform more poorly. Dividing achievement outcomes into synchronous and asynchronous forms of DE produced a somewhat different impression. In general, mean achievement effect sizes for synchronous applications favored classroom instruction, while effect sizes for asynchronous applications favored DE. However, significant heterogeneity remained in each subset.

Studied the effects of group formation strategy and levels of tutor intervention on online collaborative learning in a distance education class. Results with 45 undergraduates favor the random assignment of students to collaborative groups and the introduction of tutor intervention during the course of project work. (SLD)