Search

Background: In fall 2020, community colleges (CCs) served nearly five million students, representing 29% of U.S. undergraduates. Despite providing unprecedented access to postsecondary education, rates of degree attainment are low. Among first-time, full-time, degree/certificate-seeking students whose first postsecondary school is a CC, only 31% graduate within three years. To identify the principal causes of this problem, practitioners and scholars have identified multiple systemic issues, institutional practices, and student-level barriers that appear to create and sustain low graduation rates. (Baum, Kurose, & McPherson, 2013; Braxton, 2002; Calcagno, Bailey, Jenkins, Kienzl, & Leinbach, 2008). To address the challenges created by these impediments, many CC interventions have been implemented. The components that comprise these interventions vary, including, for example, financial supports, advising, and tutoring. Purpose: Rigorous evaluations have found that some CC interventions cause students to perform better academically; however, only a few interventions have had large impacts. In addition, there are only a handful of syntheses of the lessons learned from this growing body of rigorous research. Existing syntheses tend to focus on a specific intervention or class of interventions (e.g., learning communities) or specific student subpopulations (e.g., students referred to developmental education), or include studies using research designs requiring strong assumptions to draw causal conclusions. The present research looks across a broad array of CC interventions, including a variety of student populations, all of which have been rigorously evaluated via RCTs, to explore two main questions: (1) What relationships exist between the "comprehensiveness" of interventions and "intervention impacts" on students' academic progress?; and (2) What relationships exist between the intensity of specific intervention components and intervention impacts on students' academic progress? Setting: The setting of this research is around 45 (mostly) CCs throughout the United States. Population: The population includes around 60,000 (mostly) CC students. The interventions targeted varying student populations, like students from low-income backgrounds, students who were new to college, and students who were referred to remedial coursework. Interventions: Interventions varied from single component to multi-faceted. The most common intervention components (and thus those explored here) are: financial supports, enhanced advising, tutoring, learning communities, student success courses, promoting full-time or summer enrollment, and instructional reforms. Table 1 summarizes the prevalence of each intervention component and each intervention's comprehensiveness, or number of components, across the 39 interventions. Research Design: Findings were obtained from an analysis of individual-level data from 30 well-executed RCTs. These RCTs evaluated 39 interventions (some are multi-arm trials) based on data for a total sample of over 60,000 students from 45 (mostly) CCs throughout the U.S.. See Figure 1 for a Consort Diagram. Data Collection and Analysis: Data for the present study come from MDRC's The Higher Education Randomized Controlled Trials Restricted Access File (THE-RCT RAF) (Diamond et al., 2021). THE-RCT RAF is a well-documented student-level database that was created by MDRC and is housed at the University of Michigan's Inter-university Consortium for Political and Social Research (ICPSR) and is available to researchers. Each intervention was coded by the research team in terms of the intensity of each intervention component and the comprehensiveness of the intervention. Table 2 describes the codes. We use the fixed-intercept, random treatment coefficient (FIRC) model developed by Bloom, Raudenbush, Weiss & Porter (2017) for studying cross-site impact variation. We estimate predictive relationships between intervention features and intervention impacts. Findings: The results of this exploratory research consistently indicate that the impacts of CC interventions increase with: (1) the comprehensiveness of the intervention, as measured by the number of its components (see Figure 2), and (2) the promotion of full-time enrollment (during fall and spring) and summer enrollment (see Figure 3). Less consistent, but still promising evidence suggests the impacts of CC interventions also increase with the extent that they increase: (1) advising usage among students; (2) tutoring usage among students; and (3) financial supports for students (the evidence is least consistent for this component). Conclusions: The preceding five intervention features seem like a reasonable, evidence-based place to start when developing policy, designing a new intervention, or enhancing an existing intervention. To be clear -- the present analyses are hypothesis generating (exploratory) rather than hypothesis testing (confirmatory). This research provides suggestive evidence for decision-makers and hopefully the findings can be used for the creation and testing of new interventions -- so it may inform practice and research, in an exploratory way.

Background: Some education programs' early positive effects disappear over time. Other programs have unanticipated positive long-term effects. Foundations have warned of the dangers of putting too much weight on in-program effects, which "often fade over time." The U.S. Department of Education's Institute of Education Sciences (IES) even has a special funding category dedicated to continued follow-up to allow the exploration of such issues. Objective: This research begins to tackle the topic of post-program effects in an unexamined context: postsecondary education. Are in-program effects--that is, the effects observed while the intervention was active--maintained once the program ends? Do they grow and improve? Or do they fade out? Theorists from various education-related fields have hypothesized what may happen to effects after a program ends. The most prominent set of hypotheses comes from early education researchers, who outline three processes by which program effects may be sustained, grow, or fade over time: development of trifecta skills, "foot-in-the-door" programs, and sustaining environments. Though initially developed around a different transition point, these theories may also apply to postsecondary post-program effects. Trifecta skills are "malleable, fundamental, and would not have developed eventually in the absence of the program." In postsecondary education, trifecta skills may be fostered by something like a student success course that teaches study and time management skills at college entry. Foot-in-the-door programs leverage sensitive periods to "avoid imminent risks" or "seize emerging opportunities," as is intended by, for example, emergency financial aid. Finally, sustaining environments are experienced by students who, after a program ends, "move into high-quality environments that support their continued growth," such as a high-quality postsecondary institution after a summer bridge program. Setting/Participants/Intervention/Data: This investigation capitalizes on two decades of rigorous program evaluations conducted by MDRC, including 29 postsecondary interventions. The programs vary in terms of their features (like financial supports, advising, tutoring, learning communities, etc.), their duration (lasting from one semester to three years), the populations they served, and their contexts. Each evaluation was high quality, using a randomized controlled trial (RCT) design to estimate program effectiveness. The sample includes over 40,000 students who participated in 29 RCTs across over 35+ institutions. This work draws on THE-RCT dataset, which is a restricted access file (RAF) containing de-identified student-level data from 31 of MDRC's higher education randomized controlled trials (RCTs), involving 45 institutions and 67,400 students. Data include demographics (such as gender and race and ethnicity), outcomes (such as enrollment, credits earned, and credentials), and study-related variables. The programs range from "light touch" interventions to comprehensive interventions, with varying durations (one semester to three years), targeting various populations and contexts. Several of the studies had multiple intervention arms, meaning the RAF includes 41 unique programs. Results: Figure 1 (below) presents findings from each of 29 programs. The y-axis represents the estimated effect of the program on cumulative credits earned, an indicator of progress toward a degree. These effects refer to the average number of additional credits students earned because of the program--credits they would not have earned in the absence of the program. The x-axis represents time in years, centered around the final program semester. Negative time values (zero included) are "in-program" semesters. Positive time values are "post-program" semesters, the emphasis of this research. Since most studies have at least one year of post-program follow-up, that time frame is highlighted--that is, the time between the two vertical dashed lines. The results are striking: During the year after these programs ended, effects on academic progress (as measured by credit accumulation) are consistently maintained. Specifically, when pooled across all studies, the change in the effect on cumulative credits earned during the first post-program year is just +0.02 credits. This change is neither practically nor statistically significantly different from zero. Surprisingly, despite the many differences across the 29 RCTs, the evidence points to a consistent pattern of post-program maintenance of effects in every study. There is not clear evidence of post-program growth--that is, improved effects--or fade-out, for any of the programs examined. Conclusion: The finding that effects on credit accumulation are broadly maintained after postsecondary programs end ought to be encouraging to education reformers concerned about fade-out. While in-program effects are sometimes important on their own, benefits that are maintained into the future are especially powerful.