A Critical Analysis of an Instrument Used to Measure 21st Century Skills Attainment among High School Career and Technical Education Students

"To prepare our children for the world of tomorrow, we must enhance the learning environments of today" ("Partnership for 21st Century Skills," 2009, p. 24). In the first decade of the 21st Century, a common set of skills necessary for postsecondary success has emerged which includes creativity, critical thinking, problem solving, communication, and other applied skills. Acquisition of these skills is thought to give students the ability to continue learning and adjusting to change, effectively preparing them for postsecondary success in college, careers, and civic endeavors (ACTE, et al., 2009; Conley, 2007a; Kay, 2010; "National Survey of Student Engagement," 2006). This new demand for 21st Century skills has challenged educational leaders to examine the context in which these skills are best taught, as well as the tools available to provide evidence that these skills are being learned in high school. To date, little research has emerged regarding measurement procedures for these non-academic skills. The purpose of this study was to examine the reliability and validity of an instrument currently being used to measure student perceptions of 21 st Century skill acquisition in a Career and Technical Education (CTE) high school. The research is a quantitative study to determine empirical factors of the survey instrument, as well as determine reliabilities. The study utilized existing data collected by the school during a routine annual program evaluation process. Factor analysis was used to analyze 396 student responses to 55 survey items to identify underlying constructs (factors). Five factors, reflective of 31 contributing survey items, emerged from the preponderance of evidence as most appropriate. To determine internal consistency, Cronbach's alpha was conducted to analyze reliability among all items, among items related to each factor, and among factors. In all cases, the test for reliability (a > 0.70) was met. In order to determine correlation among factors, Pearson's correlation coefficient "r" were computed. A significant positive correlation was found among and between all five factors. Finally, a series of one-way multivariate analyses of variance (MANOVA) were used to determine whether differences existed among respondent demographic groups identified for gender, ethnicity, Socio-economic Status, high school zone of attendance, and program of study. When statistically significant differences occurred, an appropriate post hoc test was conducted. Results indicated no statistical difference in responses occurred in the demographic categories of ethnicity, gender, socio-economic status, or high school zone. Statistically significant differences were identified in the demographic category of Program of Study. Two key finding emerged from this study. The first suggests that the instrument could be revised to reflect the five empirical factors associated with 21st Century skill acquisition, along with the 31 contributing survey items, resulting in a psychometrically sound tool. This would provide a shorter version of the assessment resulting in a reduction of time and resources necessary for administration. The second key finding of this study, based on the limited response differences among study groups, suggests that the instrument may yield similar results when implemented across a range of students from varying demographic backgrounds. Though follow up research should be conducted to provide data regarding the stability of the revised instrument over time, these findings may be helpful in advancing the search for an appropriate, efficient tool that would allow high school to assess acquisition of 21 st Century skills. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]