Search

This presentation, which was delivered to the Interagency Language Roundtable Testing Committee on 21 FEB 2014, provided a systematic approach to planning, developing, and implementing assessments in immersion or iso-immersion contexts. The presentation covered the following topics: developing an assessment strategy, designing an immersion assessment, and concluded with two examples of how Special Operations Forces (SOF) components have conducted iso-immersion to assess their students., Presented to the Interagency Language Roundtable Testing Committee, Foreign Service Institute, Arlington, VA, Feb 2014.

This presentation, which was delivered to the Interagency Language Roundtable (ILR) Testing Committee on 21 MAR 2014, described how a rigorous needs assessment process can provide a foundation for developing mission-focused, capability-based language curricula and assessments. The purpose of a training needs assessment is to guide the design of activities to close gaps in underlying knowledge, skills, abilities or competencies related to performance deficiencies. The presenter used models of performance (i.e., Borman, 1991; Campbell, 1999) and theories of human behavior to demonstrate that performance is determined by individual and environmental factors. The presenter described how conducting a needs assessment that fits the purpose and context, can produce information at the level of detailed needed to construct language curricula and assessments., Presented to the Interagency Language Roundtable Testing Committee, National Foreign Language Center, College Park, MD, Mar 2014.

This presentation, which was delivered to the Interagency Language Roundtable (ILR) Testing Committee on 22 MAR 2013, summarized four studies that provided evidence on the potential interchangeability of Defense Language Proficiency Test (DLPT) scores and Oral Proficiency Interview (OPI) ratings. The four studies addressed Briner and Rousseau s (2011) evidence-based approach to organizational decision-making. The first study was a meta-analytic review of the relationships among speaking, listening, and reading proficiency. Results indicated that, in general, listening and reading proficiency tests are not good proxies for speaking proficiency tests. The second and third studies explored whether DLPT listening (L) and reading (R) scores can be used as a proxy for determining OPI speaking proficiency ratings using two different samples. Findings indicated that neither the DLPT-L nor the DLPT-R are reliable or accurate predictors of OPI-S ratings. The fourth study explored Special Operations Forces (SOF) operators perceptions of the DLPT and the OPI. Specifically, the study explored whether SOF operators perceive that either test provides job-relevant and accurate assessments of language proficiency. Results indicated that SOF operators perceive that OPI is more related to job performance than the DLPT. Overall, these studies provide evidence that the OPI should remain the test of record for SOF., Presented to the Interagency Language Roundtable Testing Committee, National Foreign Language Center, College Park, MD, Mar 2013.

This presentation, delivered to the Interagency Language Roundtable (ILR) Testing Committee on 22 MAR 2014, summarized results from two organizational-level needs assessments conducted by SWA Consulting Inc. in 2004 and 2009 that captured SOF operators and leaders perceptions of mission-related language needs. Organizational-level needs assessments are used to identify gaps between current and desired states across an organization, as well as examine related factors that influence the gap. The information gleaned from an organizational-level needs assessment can be used by the organization to inform policy and strategy. In both studies, operators and leaders reported that participatory speaking and listening were the most frequently used communicative modalities on deployment. Further, operators and leaders indicated that the Oral Proficiency Interview (OPI), which measures participatory speaking and listening, is more related to their language-related job requirements than the Defense Language Proficiency Test (DLPT), which measures non-participatory listening and reading. In summary, both studies showed that there is a lack of alignment between what is required by the mission, what is tested, and what is incentivized., Presented to the Interagency Language Roundtable Testing Committee, National Foreign Language Center, College Park, MD, Mar 2013.

The Air Force Special Operations Command (AFSOC) conducted a comprehensive needs assessment to identify, specify, and analyze language, regional expertise, and culture (LREC) mission needs across specified units and occupations in AFSOC to (1) inform the development of LREC training requirements and specifications, (2) determine who needs LREC instruction, and (3) compare REC training needs with current REC instruction provided to AFSOC personnel. The presentation was delivered to the Interagency Language Roundtable (ILR) Culture Committee on 22 FEB 2013. The focus of this presentation was how the REC knowledge, skills, and abilities (KSAs) requirements were defined. The REC KSAs were informed by multiple resources, including the Air Force Culture and Language Center s LREC Force Development Levels, Defense Regional and Cultural Capabilities Assessment Working Group (RACCA WG), and the ILR Skill Level Descriptors for Competence in Intercultural Communication., Presented to the Interagency Language Roundtable Culture Committee, Foreign Service Institute, Arlington, VA, Feb 2013.

Research regarding the effectiveness of team teaching has been mixed. Integrating original empirical work (both quantitative and qualitative) from adult, military learners and instructors with published practices and research, keys for leveraging team teaching in various formats are explored. These two sources of information yield evidence-based practice recommendations., Presented at the ACTFL Annual Convention and World Languages Expo held in Orlando, FL in 2013.

Foreign language instructor supervisors are responsible for developing and formally or informally mentoring foreign language instructors through observation, assessment, and feedback activities. This session provided research-based recommendations for how to facilitate foreign language instructor development. Research presented in this session combined comprehensive literature reviews, interviews with foreign language program administrators and instructor supervisors, and analysis of archival data to identify the behaviors and skills required for effective supervision of foreign language instructors in an adult, military learning context. Findings from this study were also discussed in terms of how these best practices could be applied to other learning contexts, such as primary or secondary education programs. Synthesis of information from these sources helped attendees effectively develop and mentor foreign language instructors., Presented at the ACTFL Annual Convention and World Languages Expo held in Orlando, FL in 2013.

This report examines both the costs and potential benefits of the United States Special Operations Command's (USSOCOM) pilot program to pay U.S. Army Special Forces (SF) operators foreign language proficiency pay at lower proficiency levels than the Army's foreign language proficiency bonus program (FLPB). Currently, the Army pays soldiers when they obtain an Interagency Language Roundtable (ILR) proficiency Level of 2 or higher in reading and listening on the Defense Language Proficiency Test (DLPT). In contrast, the pilot program pays SF operators a Level 1 rate if they achieve at least a Level 1 in any two modalities. After examining the underlying logic of the incentive program, the need for an incentive, what benefits may accrue, and available pilot program data, the report recommends that the pilot program be extended, but that significant changes should be made to its incentive structure. Specifically, the report recommends paying different amounts for different proficiency levels (the original pilot program pays the same amount regardless of proficiency level), paying for improvement, and paying via a lump sum versus monthly installments. Paying differentially provides incentives to actually improve performance versus simply maintaining current levels. Paying all individuals the same amount does not encourage them to put forth extra effort to attain higher levels. Also, the current program does not require Soldiers to improve over time. Unfortunately, this communicates the acceptability of maintaining the status quo. Rather, it is desirable to communicate not only that improvement is desirable, but expected. Requiring improvement to continue to receive the bonus accomplishes this goal and saves money by refusing to reward those who do not increase their proficiency. Analyses suggests that implementing all the recommendations could provide a more effective incentive while also reducing costs per operator by up to 31.9 percent.

This report evaluates the importance of trainee characteristics to learning in Special Operations Forces (SOF) initial acquisition foreign language training (IAT). The report focuses on trainees' general cognitive ability and motivation to learn as potential predictors of achievement during IAT. Specifically, the report attempts to answer the following questions: (1) Do trainees' general cognitive ability and motivation to learn predict their achievement during IAT?, and (2) Do trainees' general cognitive ability and motivation to learn independently or jointly predict achievement during IAT? The sample consisted of 550 trainees who completed foreign language IAT at a U.S. Army base in the southeastern United States between 2005 and 2007. Trainees' general cognitive ability was found to predict achievement on assessments of both listening and reading skills. However, motivation to learn showed an inconsistent relationship with achievement. Trainees' cognitive ability and motivation to learn showed independent relationships with achievement. The authors conclude that trainee characteristics are predictive of learning performance during IAT. Specifically, trainees' cognitive ability, and to a lesser extent, motivation to learn are useful indicators of their performance on listening and reading skills assessments during training.

As part of an ongoing Special Operations Forces Tele-training System (SOFTS) training evaluation project, the Special Operations Forces Language Office (SOFLO) requested that SWA Consulting Inc. investigate the effectiveness of the Interagency Language Roundtable (ILR) Can Do Statements as a placement tool for SOFTS courses. Researchers asked the following questions: (1) Are the Can Do Statements measuring perceived language speaking proficiency consistently and accurately for all students?, and (2) Are the Can Do Statements related to similar constructs such as students' confidence in their ability to perform language tasks? The SOFTS Can Do Statements measure perceived speaking proficiency, which is an individual's perception of his or her language ability as opposed to actual speaking proficiency, as measured by validated instruments such as the Oral Proficiency Interview (OPI). A sample of 1,710 student responses to the Can Do Statements was provided by Progressive Expert Consulting (PEC) Inc. An analysis of course feedback open-ended items also was conducted. Although this study provides some support for the use of the ILR Can Do Statements as a placement tool for SOFTS courses, some limitations may restrict the usefulness of the findings. Most notably, this study used data that had already been collected before the research questions were formulated. A follow-up study could involve measuring actual proficiency with an OPI at the beginning of language training for a sub-sample of SOFTS students.