Search

BackgroundAlthough healthcare is delivered in inherently multilevel contexts, implementation science has no widely endorsed methodological standards defining the characteristics of rigorous, multilevel implementation research. We identify and describe eight characteristics of high-quality, multilevel implementation research to encourage discussion, spur debate, and guide decision-making around study design and methodological issues.RecommendationsImplementation researchers who conduct rigorous multilevel implementation research demonstrate the following eight characteristics. First, they map and operationalize the specific multilevel context for defined populations and settings. Second, they define and state the level of each construct under study. Third, they describe how constructs relate to each other within and across levels. Fourth, they specify the temporal scope of each phenomenon at each relevant level. Fifth, they align measurement choices and construction of analytic variables with the levels of theories selected (and hypotheses generated, if applicable). Sixth, they use a sampling strategy consistent with the selected theories or research objectives and sufficiently large and variable to examine relationships at requisite levels. Seventh, they align analytic approaches with the chosen theories (and hypotheses, if applicable), ensuring that they account for measurement dependencies and nested data structures. Eighth, they ensure inferences are made at the appropriate level. To guide implementation researchers and encourage debate, we present the rationale for each characteristic, actionable recommendations for operationalizing the characteristics in implementation research, a range of examples, and references to make the characteristics more usable. Our recommendations apply to all types of multilevel implementation study designs and approaches, including randomized trials, quantitative and qualitative observational studies, and mixed methods.ConclusionThese eight characteristics provide benchmarks for evaluating the quality and replicability of multilevel implementation research and promote a common language and reference points. This, in turn, facilitates knowledge generation across diverse multilevel settings and ensures that implementation research is consistent with (and appropriately leverages) what has already been learned in allied multilevel sciences. When a shared and integrated description of what constitutes rigor is defined and broadly communicated, implementation science is better positioned to innovate both methodologically and theoretically.

Schools play a major role in delivering behavioral health services to autistic youth. School culture and climate are strong predictors of the extent to which these services incorporate evidence-based practices; however, little is known about how school leaders shape culture and climate. Drawing on the concept of culture and climate embedding mechanisms, we conducted a qualitative study to understand the ways in which school principals embed cultures and climates that support effective implementation of evidence-based practices for youth with autism. Semi-structured interviews with 32 teachers in schools that implemented three closely related evidence-based practices for youth with autism (discrete trial training, pivotal response training, and visual schedules) explored teachers' experiences regarding (a) implementation of the three evidence-based practices, (b) perceptions of school culture and climate, and (c) principals' behaviors, practices, and decisions that supported or detracted from the aspects of culture and climate that supported successful implementation. Thematic analysis detailed seven mechanisms that principals used to embed cultures and climates that shaped evidence-based practice implementation. These mechanisms represent actionable targets for school leaders and inform strategies to improve the implementation of evidence-based practices for youth with autism in schools.

This study examines how the introduction of TeachTown:Basics, a computer-assisted intervention for students with autism spectrum disorder, influenced teachers' use of other evidence-based practices. In a randomized controlled trial that enrolled 73 teachers nested within 58 schools, we used three-level hierarchical linear models to evaluate changes in teachers' use of evidence-based practices across the school year for those who received TeachTown:Basics versus those assigned to control. Both groups received training and implementation support to deliver three well-established evidence-based practices for autism spectrum disorder. Qualitative interviews were conducted with 25 teachers who used TeachTown:Basics to better understand their experience. Compared with teachers in the control group, teachers in the TeachTown:Basics group reported significantly less growth over the 9-month period in their use of evidence-based practices that require one-to-one instruction (ps < 0.05), but no difference in their reported use of evidence-based practices that do not involve one-to-one instruction (p = 0.637). Qualitative interviews indicated that teachers viewed TeachTown:Basics as an effective substitute for one-to-one instruction because it was less burdensome, despite the lack of support for TeachTown:Basics' effectiveness. Before introducing new practices, education leaders should carefully consider both evidence of effectiveness and the potential impact on the use of other evidence-based practices.

Objective: The study: (1) provides the first assessment of the a "priori" measurement model and psychometric properties of the Organizational Social Context (OSC) measurement system in a US nationwide probability sample of child welfare systems; (2) illustrates the use of the OSC in constructing norm-based organizational culture and climate profiles for child welfare systems; and (3) estimates the association of child welfare system-level organizational culture and climate profiles with individual caseworker-level job satisfaction and organizational commitment. Methods: The study applies confirmatory factor analysis (CFA) and hierarchical linear models (HLM) analysis to a US nationwide sample of 1,740 caseworkers from 81 child welfare systems participating in the second National Survey of Child and Adolescent Wellbeing (NSCAW II). The participating child welfare systems were selected using a national probability procedure reflecting the number of children served by child welfare systems nationwide. Results: The a priori OSC measurement model is confirmed in this nationwide sample of child welfare systems. In addition, caseworker responses to the OSC scales generate acceptable to high scale reliabilities, moderate to high within-system agreement, and significant between-system differences. Caseworkers in the child welfare systems with the best organizational culture and climate profiles report higher levels of job satisfaction and organizational commitment. Organizational climates characterized by high engagement and functionality, and organizational cultures characterized by low rigidity are associated with the most positive work attitudes. Conclusions: The OSC is the first valid and reliable measure of organizational culture and climate with US national norms for child welfare systems. The OSC provides a useful measure of Organizational Social Context for child welfare service improvement and implementation research efforts which include a focus on child welfare system culture and climate. (Contains 6 tables and 2 figures.)