Search

Professional standards related to outcomes assessment call for student affairs professionals to use research to inform programming. If professionals are to rely on research to build programs that positively impact student learning outcomes, the research should be credible. We examined the quality of program effectiveness research available for programming decisions. We reviewed five years of quantitative and mixed methods program effectiveness studies published in four student affairs journals. Despite frequent assertions of program effectiveness, the research designs and analyses did not often support such claims due to plausible threats to the validity of those claims. Articles claiming that programming is effective without credible evidence to support such a claim can result in professionals offering ineffective programming and engaging in inefficient assessment efforts. To address the credibility of effectiveness claims, we call for increased training in research methods, careful review of authors' claims by editors, and assistance from assessment practitioners.

Well-developed professional development opportunities are a crucial component in ensuring that faculty engaging in assessment are equipped to do this work well. Creating these opportunities requires clear expectations of the knowledge, skills, and attitudes necessary to conduct assessment work. Additionally, the knowledge and skill requirements of faculty conducting assessment are often different from those of professionally trained assessment practitioners. Although higher education student affairs organizations have developed frameworks for assessment skills, no formal framework of knowledge, skills, and attitudes exists to drive professional development in assessment within academic affairs. This article provides a framework of assessment-related knowledge, skills, and attitudes that are important to the professional development of faculty assessment practitioners, targeting three levels of complexity. This framework can be used to evaluate current professional development offerings and plan new, intentionally designed programs in accordance with backward-design principles.

In April 2017, a three-day Learning Improvement Summit, held in Washington, DC, brought together a broad array of national and international assessment stakeholders. The purpose of the summit was to empower a diverse group of faculty and assessment leaders to develop, refine, discuss, and address evidence-based learning improvement initiatives within higher education. In this short note, we provide a brief discussion of the need for the summit, the summit events, emerging themes, and lessons learned.

Propensity score matching techniques are becoming increasingly common as they afford applied practitioners the ability to account for systematic bias related to self-selection. However, "best practices" for implementing these techniques in applied settings is scattered throughout the literature. The current article aims to provide a brief overview of important considerations at each step of the propensity score matching process. Our hope is that this article will serve as a resource to assessment practitioners and augment previously published papers.

The Mapmark standard setting method was adapted to a higher education setting in which faculty leaders were highly involved. Eighteen university faculty members participated in a day-long standard setting for a general education communications test. In Round 1, faculty set initial cut-scores for each of four student learning objectives. In Rounds 2 and 3, participants used a Mapmark item map to consider information from four student learning objectives at one glance and to integrate this information into a single cut-score. Participants and faculty leaders reported that the process was intuitive, and there was support for a defensible cut-score from the majority of participants and faculty leaders. Practical suggestions and implications are discussed.

As assessment becomes an ever more critical facet of higher education, it is apparent that some faculty do not always value assessment (Crossley & Wang, 2010; Ebersole, 2009). Further, faculty may react with resistance, particularly when they perceive that assessment is being imposed upon them from external sources (Crossley & Wang, 2010; Marrs, 2009; Welsh & Metcalf, 2003). Motivation for faculty to participate in assessment is therefore a critical question. We conducted a qualitative study to explore faculty attitudes towards general education assessment, focusing particularly on faculty motivation for engaging in assessment. General education coordinators were interviewed about their perceptions of student learning outcomes assessment, using a semi-structured interview approach, and then coded by consensus according to Expectancy-Value Theory of motivation (Eccles et al., 1983; Wigfield & Eccles, 2000; Wentzel & Brophy, 2014). Implications for future assessment practice are also shared.

Often program administrators are interested in knowing how students benefit from participation in programs compared to students who do not participate. Such comparisons may be sullied by the fact that participants self-select into programs, resulting in differences between groups prior to programming. By controlling for researcher-identified--self-selection variables, propensity score matching enables researchers to create comparable matched groups. However, when employing propensity score matching, researchers are faced with a plethora of matching options. In the current study, we compared the quantity and quality of matches obtained when applying common matching techniques to real data. The methods produced matches of varying quantity and quality. Moreover, group comparisons on the outcome led to different conclusions depending on the matching method employed.

Claims of the (non-)sustainability of a given agricultural practice generally hinge on scientific evidence and the reliability of that evidence, or at least the perception of its reliability. Advocates of sustainable agriculture may dismiss science as purely subjective, or at the other extreme, may inappropriately elevate scientific findings to the status of pure objective truth. Thus, students of sustainable agriculture and natural resource management would benefit from gaining a better understanding of expert notions of the nature of science (NOS); students should learn that science is empirical, inferential, theory-laden, tentative, and creative; there is not one specific foolproof scientific method by which scientific knowledge is uncovered; and, probably most importantly in the context of sustainability, student should learn that science influences and is influenced by its wider social/cultural context. Thus, we have designed a course in sustainable agriculture that emphasizes NOS understanding. Course design elements include a rich real-world context, an inquiry setting (i.e., involving students in small-scale gardening research projects), explicit instruction in NOS concepts, and ample reflection on the intersection of NOS concepts with ideas of sustainability and farming. Student learning of NOS concepts was assessed before and after the course using a modified version of a questionnaire called Student Understanding of Science and Scientific Inquiry (SUSSI). Here we present our preliminary findings, which suggest that significant learning gains in the understanding of NOS occur during the semester, particularly in understanding the social/cultural aspects of science. (Contains 1 figure and 1 table.)

Herein we report the development of an analytic rubric bank to assess non-content learning, namely higher order cognitive skills, the understanding of the nature of science, and effective scientific communication skills in student research projects. Preliminary findings indicate that use of this tool enhances our students' learning in these areas, particularly in the area of communication, as well as eases and speeds the assessment of these skills. Our rubric bank is notable for its adaptability to a wide range of assignments, developmental levels, and science courses. (Contains 2 tables and 3 figures.)

Given the prevalence of low-stakes testing internationally (e.g., NAEP, TIMSS, PIRLS), it is crucial to try to better understand examinee motivation in these contexts. In the current study, mixture modeling results supported three different profiles of test-taking effort over the course of five tests. Classes 1 and 2 had varying levels of effort whereas Class 3 had steady levels of effort across the five tests. Examination of relationships between class membership and external variables suggested the classes could be differentiated by achievement goals, personality, and ability. Implications of these results for international assessment and directions for future research are discussed. (Contains 3 tables and 2 figures.)

The current research explored the theory of social goal orientation. More specifically, we conducted three studies utilizing six-independent university student samples to evaluate the construct validity of the Social Achievement Goal Orientation Scale (SAGOS; Ryan & Hopkins, 2003), a measure representing the construct of social goal orientation. The purpose of Study 1 was to: (1) compare the three-dimensional (mastery, performance-approach, and performance-avoidance) model of social goal orientation to three theoretically based competing models, (2) examine item functioning, and (3) assess generalizability of the factor structure. The fit of the proposed three-factor model was promising; however, areas of misfit and problematic items were identified. Stronger support for the three-factor structure of goal orientation was found using scores from an abbreviated 13-item SAGOS. In Study 2, item wording was altered slightly to evaluate a revised Social Achievement Goal Scale (SAGS), yet resulted in similar findings. Study 3 examined external validity evidence for the SAGS, garnering some support for the meaning of the scores. Although continued refinement of the SAGOS and SAGS is recommended, the findings help contribute to our general understanding and conceptualization of social goal theory and the role that social goals may play in academic contexts.

The current study evaluates existing and new validity evidence for the Academic Motivation Scale (AMS; Vallerand et al., 1992). We first provide a narrative review synthesizing past research, and then conduct a validity investigation of the scores from the measure. Data analysis using a sample of 1406 American college students provided construct validity evidence in the form of a well-fitting seven-factor model and adequate internal consistency of the item responses. Convergent and discriminant validity evidence provided insight into the distinctiveness of the seven subscales. However, support for the scale's simplex structure, which would represent the self-determination theory continuum, was not fully substantiated. Implications for theory and the scale's use in the current form are discussed.

Professional standards related to outcomes assessment call for student affairs professionals to use research to inform programming. If professionals are to rely on research to build programs that positively impact student learning outcomes, the research should be credible. We examined the quality of program effectiveness research available for programming decisions. We reviewed five years of quantitative and mixed methods program effectiveness studies published in four student affairs journals. Despite frequent assertions of program effectiveness, the research designs and analyses did not often support such claims due to plausible threats to the validity of those claims. Articles claiming that programming is effective without credible evidence to support such a claim can result in professionals offering ineffective programming and engaging in inefficient assessment efforts. To address the credibility of effectiveness claims, we call for increased training in research methods, careful review of authors' claims by editors, and assistance from assessment practitioners. [ABSTRACT FROM AUTHOR]

Given the pervasiveness of stress and anxiety in our culture it is important to develop and implement interventions that can be easily utilized by large numbers of people that are readily available, inexpensive and have minimal side effects. Two studies explored the effectiveness of a computer-based heart rate variability biofeedback program on reducing anxiety and negative mood in college students. A pilot project ( n = 9) of highly anxious students revealed sizable decreases in anxiety and negative mood following utilizing the program for 4 weeks. A second study ( n = 35) employing an immediate versus delayed treatment design replicated the results, although the magnitude of the impact was not quite as strong. Despite observing decreases in anxiety, the expected changes in psychophysiological coherence were not observed. [ABSTRACT FROM AUTHOR]