Search

The article discusses the highlights of the 31st Annual Symposium of the International Test and Evaluation Association (ITEA) held on October 6 to 9, 2014 in Arlington, Virginia. Topics covered at the symposium include test and evaluation (T&E) tools and methodologies to support testing of complex systems, reducing risk in 2020 testing and T&E practices for cyber systems.

Model-based systems engineering (MBSE) approaches are based on a paradigm shift from document-centric engineering to model-based engineering. Although MBSE methods are intended to apply across the entire system life cycle, one area that has not received much attention to date is the role of test and evaluation. Test and evaluation activities provide information that reduces the uncertainty about system performance, effectiveness, and suitability. This uncertainty reduction becomes particularly important within the context of defense systems, which can cost billions of dollars. This paper describes a methodology that uses an MBSE framework and Monte Carlo simulation to define uncertainty reduction goals for test planners to use in developing test strategies and detailed test designs for evaluating technical performance parameters. As tests are completed, physical models can be updated with test data and additional analyses conducted with combat models to determine if the system meets user requirements. The methodology is demonstrated through a simple case study involving a series of tests to predict the landing performance of an aircraft. ©2012 Wiley Periodicals, Inc. Syst Eng 16: [ABSTRACT FROM AUTHOR]

This paper describes a general methodology for developing optimal system test-and-evaluation strategies by using Monte Carlo simulation to evaluate different test-and-analysis approaches and comparing the amount of residual uncertainty provided by each approach. The methodology is applied to a notional test for measuring the false-alarm rates of a postulated fire-control radar-system upgrade. [ABSTRACT FROM AUTHOR]

Collecting data that can be transformed into information used to make decisions is one of the primary purposes of test and evaluation. Two key challenges facing test planners are (a) determining what data to collect (e.g., how much, under what conditions, to what degree of accuracy and precision, etc.) and (b) how to analyze the data once it has been collected. Of course, the data collection and analysis must be done in a resource constrained environment; there is never enough funding, time, or people to conduct the perfect test. [ABSTRACT FROM AUTHOR]

Since 2005 the Air Force Flight Test Center (AFFTC) has had an initiative to improve the use of statistics in planning tests and evaluating subsequent test results. Initial efforts focused on design of experiment techniques and an attempt to train all flight-test discipline engineers in the use of statistics. Over time the approach has evolved to the use of a statistical consulting group and a broadening of techniques to the use of a wide variety of statistical methods. This article summarizes the AFFTC experience, including an overview of requirements, policy, and organizational needs; the technical and programmatic challenges encountered in implementation; and some examples describing the general use of statistics at AFFTC. The article concludes with a brief discussion regarding future initiatives to institutionalize statistics at AFFTC. [ABSTRACT FROM AUTHOR]

The phrase "testing in a joint environment" refers to testing military weapons and supporting systems in the joint mission environments in which those weapons and systems are expected to operate. The Office of the Secretary of Defense chartered the Joint Test and Evaluation Methodology project to institutionalize testing in a joint environment by improving the ability to conduct tests, across the acquisition life cycle, in realistic joint mission environments. Specifically, the project was directed to develop methods and processes for using distributed live- virtual-constructive joint test environments to evaluate system performance and joint mission effectiveness. In 2007, the project completed a series of such tests to assess~ an initial set of methods and processes. Tests of network-enabled air-to-surface weapons and ground-launched surface-to-surface precision attack missiles were used to provide context for system performance evaluations. Joint mission effectiveness was evaluated by conducting Joint Fires and Joint Close Air Support with the above weapons and other supporting systems. These tests were accomplished as part of the 2007 INTEGRAL FIRE event sponsored by the Air Force Integrated Collaborative Environment program. This article describes results after methods and processes for testing in a joint environment were used by experienced testers to design and assemble an actual distributed joint test environment. Results identified improvements to the processes as well as recommendations for test organizations. To streamline routine test planning for distributed testing, we recommend test organizations consider procedures such that each acquisition program has a lead test organization designated for distributed testing. We also recommend that test organizations consider establishing formal relationships to manage the distributed test environment as a single facility. [ABSTRACT FROM AUTHOR]

The Office of the Secretary of Defense chartered the Joint Test and Evaluation Methodology project to institutionalize testing in a joint environment. The project has now finished most of its major activities. In this article we describe our key accomplishments, findings, conclusions, and recommendations. Testing in a Joint Environment refers to tests of military systems as participating elements in overarching joint systems of systems. The concept first appeared in Strategic Planning Guidance and was formally introduced as Department of Defense policy in a roadmap signed by the Deputy Secretary in 2004. Several working groups were formed to implement the roadmap. The Joint Test and Evaluation Methodology project was initiated in 2005 to continue efforts of the methods and processes working group. Throughout the past three years we have developed, tested, and evaluated a number of methods and processes for defining and using a distributed live, virtual, and constructive joint test environment to evaluate system performance and joint mission effectiveness. We briefly describe those processes, what we learned by testing them, and the extent to which they improve the ability to conduct tests, across the acquisition life cycle, in realistic joint mission environments. We also describe the results of two large-scale distributed tests-INTEGRAL FIRE 07 and Joint Battlespace Dynamic Deconfliction 08-which used mixes of live, virtual, and constructive elements to test a number of systems in joint environments. Several challenges remain, and we make recommendations to continue progress toward the goals of testing in a joint environment. The Department's long-term strategy calls for evaluations of joint system effectiveness throughout all phases of all weapon systems' development and deployment. [ABSTRACT FROM AUTHOR]