Search

In this research, we answer the following primary question: Would an advanced analytical model be a more effective metric to estimate total ownership cost (TOC) with life-cycle cost under uncertainty and risk than the current method of life-cycle cost estimates for surface electro-optical infrared (EO/IR) sensors? To accomplish this, we developed and analyzed a computational model for Total Ownership with Life-Cycle Cost Model Under Uncertainty for Surface Electro-Optical Infrared Sensors. During the development of the model, we identified the required data and examined the current Department of Defense (DoD) method for determining system life-cycle costs for defense systems and determined that the proposed model is a useful alternative to the current method of determining the life-cycle costs for EO/IR sensors on surface ships. Finally, we concluded that the developed model can be applied to cost estimating in other sectors of DoD cost projections.

This study is an analysis of workforce turnover behavior among two segments of the Department of Defense (DoD) civilian workforce. Its research objective is to identify turnover trends among the civil service medical healthcare workforce and the civilian component of the defense acquisition workforce. The study employs a logistic regression model to estimate the relationship between employee personnel characteristics and the likelihood of separating from the civil service. In both groups, the overall pattern of attrition was consistent with a life-cycle model of turnover behavior. The results highlight specific employment categories where improvement in recruiting a more diverse workforce is needed.

Awarding federal contracts is perceived as an excessively lengthy process. The purpose of this research is threefold: (1) to understand the drivers of procurement administrative lead time (PALT), (2) to identify opportunities to reduce PALT, and (3) to predict when specific requirements are likely to be awarded. These analyses will be performed using newly available, government-wide data for over 5 million federal contracts.

Topological data analysis (TDA) is an unconventional machine learning technique that is used to understand the underlying topology of data. The premise is that data has shape. The two methodologies used in TDA are persistent homology and the mapper algorithm. Traditional machine learning techniques include supervised unsupervised methods such as clustering, Bayesian networks, neural networks, support vector machines (SVM), and random forests. The goal of this study is to apply TDA methods in conjunction with traditional machine learning algorithms to Defense Acquisition Executive Summary (DAES) data to determine if TDA helps to improve prediction measures (accuracy, f-measure, sensitivity, and specificity) over using traditional methods only when predicting program manager ratings from Major Defense Acquisition Programs (MDAPs). We show that TDA when used in conjunction with traditional machine learning models at a local level of the DAES data improved the accuracy of predicting PM cost ratings of MDAPs at 80% of all nodes in training and testing as compared to implementing these models without TDA at the global level.

In this paper, we survey a small group of Air Force contracting personnel to understand their views on contracting in sole-source environments. Our findings suggest Air Force contracting personnel in this setting know that sellers in non-competitive relationship have more leverage and power than the buyer. Indeed, 90% of respondents feel they operate at a negotiating disadvantage in sole-source contracts. Arming them with certified cost and pricing data does not improve leverage according to majority of them. Rather, they identify two constraints in their qualitative responses. First, the sole-source environment itself contributes to the problem. Second, many respondents feel they operate at an informational disadvantage compared to their private counterparts. This suggests specific training on their contracts would be more valuable than any general training on business acumen.

The Air Force Materiel Command is undergoing a digital transformation to increase the speed of delivering new warfighter capabilities. This Digital Campaign consists of six Lines of Effort (LOEs) formed with diverse goals to transform the enterprise. This research investigated using the Zachman Framework and Systems Modeling Language to analyze this transformation. Extensive modeling captured the as-is Preliminary Design Review (PDR) process and mapped LOE goals as primary impacts to Zachman cells. This led to an identification of a to-be digital PDR process. Secondary impacts were also identified and traced using a relationship analysis. Four discoveries were made. (1) Enterprise modeling in Zachman is analogous to a system decomposition under typical systems engineering approaches. (2) As long as the transformation goals do not change, the Zachman cells, and those entities mapped into those cells, will be directly affected by the new digital enterprise. (3) Different from past process transformation efforts, the Digital Campaign has focused on technology upgrades to drive process change. (4) Lastly, model analysis revealed transformation gaps within certain cells that should be covered with new goals. This research provides a formal, model-based methodology for guiding enterprise-wide improvements in pursuit of Air Force digital transformation.

This paper focuses on numerous existing and recent initiatives and programs involving allied and partner support to the U.S. defense industrial base and explores how they are currently instantiated in DoD acquisition processes. Are DoD acquisition processes able to effectively utilize efforts such as the National Technology Industrial Base (NTIB), Reciprocal Defense Procurement Agreements, Defense Production Act (DPA) Title III, and others to foster programs with partners that build industrial resilience in the defense industrial base? Research and interviews found that most of these efforts have promising foundations, but DoD acquisition processes need adaption to effectively increase allied and partner involvement. The author makes a series of recommendations to address these findings about the programs analyzed and the acquisition system itself. The author concludes that reframing acquisition processes and programs to better include partners and allies is ultimately a win-win proposition for all parties involved. Pursuing this approach will help to provide a concrete foundation for the future of international industrial collaboration and will build the industrial resilience needed to face the national security challenges of today and tomorrow. The paper also points to areas for further research to further refine the recommendations and explore specific areas in greater depth.

The time required to complete a software development or upgrade, like any other project, depends on the content of the project, how it is managed, and its preexisting conditions. The factors limiting the speed of software acquisition and enhancement fall into these principal categories, in rough order of importance: 1. Required functionality—what you need the software to do (and not do) 2. Architecture—the organizing structure of the software and its operating environment 3. Technology maturity—to what extent the intended design uses novel solutions 4. Resources—the people, skills, funds, data, and infrastructure needed to do the work 5. Testing strategy—acquiring the information to fix defects early in development 6. Contract structure—the alignment of contractor incentives with DoD satisfaction 7. Change management—the processes for trading off performance, schedule, cost, and sustainability This paper considers this taxonomy and examines how each category affects the pace of development.

The application of the five production management levers addresses software projects as production systems, allowing for the application of well-established operations science techniques to improve throughput and predictability throughout the project life cycle. Traditional project management triple constraints of cost, schedule, and performance have long been the cornerstone of Department of Defense (DoD) software project management. Often, these constraints are managed independently, using past performance data to assess and predict future project performance, leading to highly variable outcomes. Traditional waterfall models for software project delivery exacerbate this variability and limit the ability to delivery capability at the speed of relevance. By treating projects as a production system, a deeper understanding of planning variables (product design, process design) and execution variables (capacity planning, limiting work in process, and variability) can be achieved and close the gap on project delivery performance. Further, the study connects industry best practices across software, manufacturing, and construction to improve DoD software project delivery. Operations science and production management principles and techniques are foundational to the agile and lean movement. The software industry has already adopted many of the operations science and production management techniques (such as limiting work in process, capacity-based sprint planning, software factories, etc.). A succinct process to apply, monitor, control, and report out on the five levers can encourage adoption and clarify execution of the Adaptive Acquisition Framework, DoD Instruction 5000.02, Software Acquisition Pathway.

Engineering involves designing solutions to meet the needs of markets or missions. Organizations would like to have the flexibility and agility to address both uncertain needs and uncertain technologies for meeting these needs. This article presents and illustrates a decision framework that enables flexibility and agility and provides guidance on when to pursue optimal, highly integrated solutions. We consider how uncertainties arise, contrasting the automotive and defense domains. We propose an approach to managing uncertainties. We consider how to represent alternative solutions and project the value of each alternative, including how market or mission requirements can be translated into system requirements. Possible use cases for our framework are discussed. A detailed case study of autonomous vehicles for enhancing the mobility of disabled and older adults is presented.

In 2011, the Department of Defense (DoD) set a goal of paying small business contractors within 15 days of invoice receipt rather than the standard 30 days. In 2012, other federal agencies also set a goal of accelerated payments to small businesses, and all agencies later expanded this goal to include all contractors regardless of size. We study whether small businesses benefited from these accelerated payment goals. Using a difference-in-difference design, we find that small business participation in government contracts rose following the setting of accelerated payments goals. Importantly, contracts for perishable foods and construction services, which were unaffected by the new accelerated payments policies, did not see an increase in small business participation. We also find that the benefits of accelerated payments are concentrated among small businesses with a backlog of ongoing projects. This is consistent with the hypothesis that accelerated payments alleviate the liquidity constraints that may be particularly acute for small businesses.

This research explores where Agile practices used for software development within the information technology domain, can be successfully applied in other domains. Private industry is already applying Agile principles in manufacturing, construction, health, research and development, and finance. These approaches are captured in this research in terms of best practices and lessons learned. This research explores where accepted Agile principles and practices can accommodate organizational agility and program management environments. This research identifies suggested guidance where Agile can be applied more broadly, especially when adopting the four Agile values. 1. Determine if Agile is a good fit. 2. Determine the scope of the Agile effort. 3. Consult with experts. 4. Tailor Agile values and principles to the situation. 5. Develop an approach. 6. Leverage lessons learned.

This study examines five advanced decision support methodologies—Lean Six Sigma (LSS), Balanced Score Card (BSC), Integrated Risk Management (IRM), Knowledge Value Added (KVA), and Earned Value Management (EVM)—in terms of how each can support the information technology (IT) acquisition process. In addition, the study provides guidance on when each methodology should be applied during the acquisition life cycle of IT projects. This research includes an in-depth review of each methodology in the context of the acquisition life cycle. All acquisition projects within the Department of Defense must go through the acquisition life cycle. While each acquisition project is unique, all must pass a series of common hurdles to succeed. Understanding how and when the methodologies can be applied to an IT acquisition is fundamental to its success. The study concludes with a set of recommendations for the use of each methodology in the acquisition life cycle of IT projects.

This research quantifies fragility within the U.S. Defense Industrial Base (DIB) and translates it into supplier risk. The proposed model identifies systemically critical suppliers, where critically is characterized in terms of the supplier either being highly coupled within the industrial base, operating in a limited competition space, or owning a disproportionately large market share within a specific commodity. Each of these properties is quantified using centrality and community detection methods. By correctly assessing critical suppliers in the defense base, it allows for a methodical approach to addressing standard failure modes that typically result in material disruptions in advance of realizing interruptions. Quantifying fragility in supply chains based on systemic centrality and communities is a novel effort. Direct application of this process within the DIB fundamentally approaches assessing and strengthening our supply base resiliency in a completely different manner.

The Department of Defense (DoD) envisions that digital engineering information exchange, system modeling, and data driven system engineering processes will become core to product and process development. As this transformation occurs, it will change the way Systems Engineering (SE) is measured and valued. Over the past 3 years, the Systems Engineering Research Center (SERC) has studied the Digital Engineering (DE) transformation processes and progress. This work has focused on DoD acquisition and program office activities but is applicable to all enterprises undergoing DE and Model-Based Systems Engineering (MBSE) transformations. A previous SERC research task created an Enterprise System-of-Systems Model for DE-enabled acquisition, conceptually modeling the potential future DoD acquisition enterprise. This research helped to understand the structure of future DoD/contractor program enterprises when the five goals of the DoD DE strategy were achieved, and the expected outcomes of that transition. That research cited the need for the community to standardize and implement measures that reflect success at the enterprise level. A second research task was completed to define metrics that represent value, benefits, and change progress in enterprise DE transformation. A third task is currently underway to design and implement measures that quantify DE benefits.

The DoN is undergoing a digital transformation that is set to address the needs of sustaining fleet assets for extended periods of time, while maintaining a superior lethality. Within the engineering domain, the DoN is starting to identify model-based systems engineering (MBSE) tools and concepts to streamline processes and enhance capability. The capstone looked to lay the foundation for a conceptual system model development process that utilizes SysML and OOSEM to produce system model data and artifacts derived from a single scenario. During the digital transformation, communication of system model data to stakeholders was identified as a need, and a SysML tool was used to generate model-based documentation from a formatted Microsoft Word document. With incoming digital product support capabilities from the MBPS program, communication from a MBSE environment is critical and requires XML-formatted data. Using the information collected in the completion of the scenario, it was discovered that SysML elements will lose their SE-specific stereotypes when converted directly into XML format. To counter this, the capstone developed UML instances derived from the S3000L UML class-based data model to be converted into XML format. The findings and developments of this capstone support the ability for organizations to standardize the way system modeling data is developed, collected, and communicated to other systems external to the engineering domain.

Requirements form the backbone of contracting in acquisition programs. Requirements define the problem boundaries within which contractors try to find acceptable solutions (design systems). At the same time, requirements are the criteria by which a customer measures the extent that their contract has been fulfilled by the supplier. Therefore, requirements are instrumental in the success of acquisition programs. In this context, the quality of a requirement set is determined by the level of contractual safety that it yields. From a technical perspective, contractual safety is driven by the accuracy, precision, and level of completeness of the requirement set. Unfortunately, textual requirements do not provide acceptable levels of contractual safety, as they remain a major source of problems in acquisition programs. This is partly caused by the inherent limitations of natural language to statically capture written statements with precision and accuracy. In addition, natural language is difficult (often impossible) to parse into consistent logical or mathematical statements, which limits the use of systematic and/or automated tools to explore completeness. Model-based requirements have been proposed as an alternative to textual requirements, with the promise of enabling higher accuracy, precision, and completeness when eliciting requirements. However, this promise has not been demonstrated yet. Therefore, research is needed to understand the contractual impacts of using model-based requirements instead of textual requirements before model-based requirements can be widely adopted to support acquisition programs. This paper presents preliminary results of a research project that measures the contractual safety yielded by model-based requirements. Specifically, the research addresses the main question of whether using model-based requirements improves the contractual safety of acquisition programs compared to using textual requirements. The accuracy, precision, and comple

This concept paper explores an innovative approach to detecting and managing software vulnerabilities in cyber-physical defense systems. Software-based vulnerabilities that hinder or preclude the maintainability and evolvability of combat systems are a pernicious form of technical debt that threaten all cyber-physical systems. The risks associated with technical debt across increasingly interdependent DoD cyber-physical systems will accelerate if left unchecked. Without changes in acquisition and maintenance practices, we can foresee cascading, potentially catastrophic cross-system failures. To illustrate the risk and possible solutions, we focus on the software embedded in combat systems that are subject to ongoing modernization efforts that extend their applicability to evolving operations. Our research revealed that software vulnerabilities in critical combat systems can threaten the reliability and readiness of those systems. These vulnerabilities provide an opportunity for the defense acquisition communities to create a new capability within their organizations, an Acquisition Technical Debt Team (ATDT) to help detect, manage, and mitigate technical debt. We explore risk classification by including interoperability into risk evaluation schemas. We then apply common distressed debt management models to suggest when and how the ATDT might help manage and mitigate technical debt to help rehabilitate an ailing system.

Current methods of measuring acquisition project performance are based upon generations of old cost models that compare budgets with expenditures. These methods provide little in the way of forecasting, leaving the project manager to react to past performance rather than making decisions of future potential. The principal means of measuring project performance in large complex projects is earned value management (EVM), which measures actual work performed to budgeted work performed. While this method provides valuable insight into how well a project is doing relative to a predetermined plan, new methods of forecasting when projects begin to become volatile are needed. Knowing when a project will enter a phase of volatility will provide greater transparency and flexibility to decision-makers before critical events have occurred. This paper describes the state of current project management theory and how it has driven the project management community to a point of stagnation. Additionally, this paper will outline a new and innovative approach to forecasting when a project will become volatile long before current EVM and risk management tools can today. This research approach proposed in the paper intends to extend current theory and practice in project management by examining several information systems programs and providing new, more effective decision-making tools for future project managers. The methods defined in this research will be known as advanced earned value management (AEVM).

The Major Defense Acquisition Program (MDAP) market is a monopsony facing oligopoly. In the last four decades, the Department of Defense has placed a great deal of emphasis in its acquisition reform efforts on the power of competition to help control cost overruns and cost growth. In this research, quantitative analyses were used to determine the effect of two reform measures—competitive prototyping and competitive contracting—on cost overruns and cost growth during the Engineering and Manufacturing Development (EMD) phase of the acquisition life cycle. We performed a case study of 63 hardware MDAP contracts from all services. The findings show that while competitive prototyping and competitive contracting lead to greater competition, as the defense acquisition community believes, they fail to control cost overruns and cost growth, just as microeconomic theory predicts.

This paper continues a research agenda started in 2016 with an aim of more realistic acquisition program scheduling estimates, especially for the development (SSD) phase. This, our third look at the scheduling problem, starts with a discussion of scheduling data, and how that data could be applied to help the DoD address this challenge. This section includes ideas on how to use acquisition data for the scheduling problem. Next, we present a case study of the result of field interviews with senior DoD leaders. Finally, we present a discussion on using the system performance as a metric.

This continuing project aims to examine weapons systems development schedules to identify ways to predict the duration of the defense acquisition schedule more accurately. This study aims to provide a more complete understanding of the myriad factors that drive the weapon system development timelines. In particular, we propose to investigate insights from contemporary data science methods to identify variables critical to predicting acquisition schedules through Schedule Estimating Relationships.

Sustainable public procurement plays an important role in addressing not only environmental but also economic and social issues through government acquisitions from technology-based small suppliers. In this context, the objective of this study is to better understand the holistic public procurement process by assessing the operational efficiency of technology-based small suppliers and associating the economic aspect of public procurement with the social aspect, such as women-owned businesses. To this end, we analyzed U.S. Department of Defense Small Busi-ness Innovation Research grantees by combining network data envelopment analysis with bootstrap truncated regression analysis. Drawing on the analysis results, we found that (1) there is heterogeneity in the performance of research and development, network building, and commercialization sub-processes, and (2) there is a positive relationship between the overall performance and women-owned small suppliers who excel particularly in network building. The former implies that small suppliers may have different expertise in the chain of public procurement; the latter suggests that woman entrepreneurs with a business network may be able to outperform their counterparts in the public procurement market.

Cybersecurity is a national priority, but the analysis required for acquisition personnel to objectively assess the integrity of the supply chain for cyber compromise is highly complex. This paper presents a process for supply chain data analytics for acquisition decision makers, addressing data collection, assessment, and reporting. The method includes workflows from initial purchase request through vendor selection and maintenance to audits across the lifecycle of an asset. Artificial intelligence can help acquisition decision makers automate the complexity of supply chain information assurance.

Executable programs run on computers and digital devices. These programs are pre-installed by the device vendor or are downloaded or copied from a storage media. It is useful to study file similarity between executable files to verify valid updates, identify potential copyright infringement, identify malware, and detect other abuse of purchased software. An alternative to relying on simplistic methods of file comparison, such as comparing their hash codes to see if they are identical, is to identify the "texture" of files and then assess its similarity between files. To test this idea, we experimented with a sample of 23 Windows executable file families and 1,386 files. We identify points of similarity between files by comparing sections of data in their standard deviations, means, modes, mode counts, and entropies. When vectors are sufficiently similar, we calculate the offsets (shifts) between the sections to get them to align. Using analysis on these shifts, we can measure file similarity efficiently. By plotting similarity vs. time, we track the progression of similarity between files.

The U.S. Air Force has traditionally acquired software through the military acquisition process. This acquisition process is used broadly for diverse types of purchases and requires considerable time. Concerned that this approach is not ideally suited for the acquisition of rapidly evolving, innovative software, the U.S Air Force re-assessed its previous make or buy decision regarding software and stood up Kessel Run. Kessel Run is an innovative, in-house software development organization, intended to assess the viability of "re-insourcing" software development. This research reports on a case analysis of Kessel Run. We explore why and how the Air Force developed Kessel Run and offer insights into its operation. Our results suggest that, while cost-reduction goals certainly facilitated Kessel Run's origin and cost reduction has remained a benefit, the primary driver and benefit of re-insourcing Kessel Run is capability development. This finding seems to support a resource-based logic to insourcing. Our results also show that the leaders of Kessel Run succeeded in establishing the organization by relying on commercial start-up concepts, which allowed the organization to attract talent, develop capabilities, and meet customer needs far faster than typical acquisition approaches. We identify lessons learned and implications for other re-insourcing opportunities.

Development of advanced Artificial Intelligence (AI)/Machine Learning (ML) system-enabled weapons and combat systems for deployment in the U.S. Navy has become a reality. This is also true for the other armed forces, as well as in homeland security and even the Coast Guard. From the Navy standpoint, the Naval Ordnance Safety and Security Activity (NOSSA) is attempting to get ahead of the acquisition cycle by focusing on the development of policies, guidelines, tools, and techniques to assess mishap risk in Safety Significant Functions (SSF) that are identified. NOSSA's efforts have the potential of influencing the acquisition community, including in requirements, development, and test and evaluation engineering. This paper makes recommendations for the Functional Hazard Analysis (FHA) and Subsystem Hazard Analysis (SSHA) analysis templates and focuses on ways to decrease autonomy within system operations and increase its correlated Software Control Category (SCC). The questions and discussions devised from this research aim to form guidance and offer best practices to address AI/ML system safety issues.

The objective of this research is to mathematically formulate and manage the relationship between the quantitative complexity level of an acquisition or engineering development program and its relationship to the increased technical and programmatic risk, respectively. This research builds upon the PIs previous research experience and grants (NPS BAA 14-002, NPS BAA 15-001, NPS FOA 16001). This research aims to discover and determine the relationship between the quantitative complexity value of an acquisition program (at various points in its lifecycle) as a measure of increased actual technical and programmatic risk respectively. The main goal is to improve the current inaccurate subjective practice of assessment of risk in different stages of a wide range of engineered system development programs as well as acquisition programs. Currently lifecycle risk assessment methodologies such as color-coded risk matrix are heavily subjective in their nature and therefore weak in the assessment of the actual risk. As a result, acquisition programs frequently are exposed to unforeseen technical and programmatic risks and failures; cost and schedule overruns that are due to inaccurate risk identification and assessments. This research proposal focuses on expanding and examining the novel set of new complexity measures that are recently created by our team (with the PIs previous NPS research grants) as pre-indicators of emergence of risks at different stages of a systems development process and lifecycle. The detailed set of created complexity measures, will be modified and categorized based on their application category in physical/hardware/software systems as well as DoD System of Systems level studies. The refinement and categorization of the complexity/risk measures will be applied to and examine several historical case studies of engineered systems success or failures. The focus of this part of research will be on discovering the suitability of each of the 12 complexity/ri

Navy ships are complex enterprises comprised of multiple organizations that must interact smoothly and interface externally without threats to efficiency and combat-readiness. As logistical challenges increase and technology pushes response times, it is critical to introduce state of the art computational methods for analyzing the interlocked systems and training for different events. To address these challenges in this context, we introduce a framework called LAILOW: learn, optimize, and wargame. LAILOW exploits data arising from multiple sources in a complex enterprise by offering data mining, machine learning, and predictive algorithms that can be used for analysis and discovery of patterns, rules, and anomalies. LAILOW's output can then be used to optimize business processes and course of actions. We show three use cases of using the of LAILOW framework. We show the whole LAILOW framework to search for vulnerability of a major Marine equipment's maintenance and supply system for difficult tests and evolve resilience and novel solutions accordingly. We show using of lexical link analysis (LLA) as part of LAILOW to improve the prediction accuracy of probability of failure of critical Navy Ship parts, related to C4I systems, for NAVWARSYSCOM's Predictive Risk Sparing Matrix (PRiSM) product. We also show the comparison of LLA prioritizing items in the Financially Restricted Work Que (FRWQ) with the baseline calculation.

The National Defense Authorization Act (NDAA) of 2020 directed the secretary of defense to implement a professional certification program for all members of the acquisition workforce that is based on standards developed by a third-party accredited program based on nationally or internationally recognized standards. In response to this NDAA (2019) requirement, the Department of Defense (DoD) senior procurement executives agreed to the establishment of a new contracting competency model and a single level of certification program based on the National Contract Management Association’s (NCMA) Contract Management Body of Knowledge (CMBOK; NCMA, 2019a) and American National Standards Institute–accredited Contract Management Standard (CMS; NCMA, 2019b). The purpose of this research is to develop a new competency assessment instrument based on the NCMA CMBOK and CMS to be used in assessing the DoD’s contracting workforce competency. This research will answer the following question: How can the CMBOK/CMS competency structure be used as the basis for developing a survey-based instrument for assessing the competencies of the DoD contracting workforce? An additional research question is: Based on the competency assessment results, in which contract management competencies is the workforce less proficient and less knowledgeable? We conduct this research by developing a survey-based assessment instrument for assessing the competencies of the DoD contracting workforce. We then deploy the assessment instrument to DoD contracting organizations and analyze the assessment results to identify contract management competencies that need additional training emphasis.

The acquisition of artificial intelligence (AI) systems is a relatively new challenge for the U.S. Department of Defense (DoD). Given the potential for high-risk failures of AI system acquisitions, it is critical for the acquisition community to examine new analytical and decision-making approaches to managing the acquisition of these systems in addition to the existing approaches (i.e., Earned Value Management, or EVM). In addition, many of these systems reside in small start-up or relatively immature system development companies, further clouding the acquisition process due to their unique business processes when compared to the large defense contractors. This can lead to limited access to data, information, and processes that are required in the standard DoD acquisition approach (i.e., the 5000 series). The well-known recurring problems in acquiring information technology automation within the DoD will likely be exacerbated in acquiring complex and risky AI systems. Therefore, more robust, agile, and analytically driven acquisition methodologies will be required to help avoid costly disasters in acquiring these kinds of systems. This research provides a set of analytical tools for acquiring organically developed AI systems through a comparison and contrast of the proposed methodologies that will demonstrate when and how each method can be applied to improve the acquisitions lifecycle for AI systems, as well as provide additional insights and examples of how some of these methods can be applied. This research identifies, reviews, and proposes advanced quantitative, analytically based methods within the integrated risk management (IRM)) and knowledge value added (KVA) methodologies to complement the current EVM approach. This research examines whether the various methodologies—EVM, KVA, and IRM—could be used within the Defense Acquisition System (DAS) to improve the acquisition of AI. While this paper does not recommend one of these methodologies over the other, ce

This technical report summarizes the work conducted by Massachusetts Institute of Technology under contract award HQ0034-20-1-0008 during the performance period May 22, 2020 – July 31, 2021. Digital engineering transformation changes the practice of systems engineering, and drives the need to re-examine how engineering effectiveness is measured and assessed. Early engineering metrics were primarily lagging measures. More recently leading indicators have emerged that draw on trend information to allow for more predictive analysis of technical and programmatic performance of the engineering effort. By analyzing trends (e.g., requirements volatility) in context of the program’s environment and known factors, predictions can be forecast on the outcomes of certain activities (e.g., probability of successfully passing a milestone review), thereby enabling preventative or corrective action during the program. Augmenting a companion research study under contract HQ0034-19-1-0002 on adapting and extending existing systems engineering leading indicators, this study takes a future orientation. This report discusses how base measures can be extracted from a digital system model and composed as leading indicators. An illustrative case is used to identify how the desired base measures could be obtained directly from a model-based toolset. The importance of visualization and interactivity for future leading indicators is discussed, especially the potential role of visual analytics and interactive dashboards. Applicability of leading edge technologies (automated collection, visual analytics, augmented intelligence, etc.) are considered as advanced mechanisms for collecting and synthesizing measurement data from digital artifacts. This research aims to provide insights for the art of the possible for future systems engineering leading indicators and their use in decision-making on model-centric programs. Several recommendations for future research are proposed extending from the stu

Additive manufacturing (AM) has the potential to fundamentally change how military expeditionary operations are conducted. By manufacturing spare parts in remote sites, rather than relying on lengthy and extensive supply chains or remaining tethered to an “iron mountain” of logistics support, the expeditionary units have the potential to be more agile, to maintain their readiness at high levels while deployed, and to extend their operational reach. AM has enjoyed success in a number of specialty fields. Potential benefits for expeditionary units include achieving higher readiness at lower cost, because deployed units can use AM to create replacement parts at or near the point of demand, rather than either relying on carrying large quantities of spare parts or dealing with long lead times for replacements. Another potential benefit is the ability to reduce wastage of the materials used in the three-dimensional (3D) printing process and subsequent post-treatments by only producing what is needed. Finally, if the same compounds can be used to manufacture a variety of parts, AM could help forward-deployed units maintain a high level of readiness while dramatically reducing their logistics footprint. To realize this potential, program managers have several decisions to make. They must determine how best to acquire AM capabilities, what classes of components are suitable for AM, whether the resulting structural and reliability are comparable for components made using AM and current methods, and how differences in reliability may affect the supply chain and readiness levels. If the suitability and reliability are not factored into the decision-making process, then AM may end up being a costly and largely redundant logistics system running in parallel with the current supply chain, rather than being a transformative capability.

This paper contributes to the process of building knowledge about what we term as behavioral acquisition, which explores defense acquisition from a behavioral standpoint, including the impact of psychology, organizational behavior, and politics. Behavioral acquisition studies the decisions acquisition professionals make in Department of Defense (DoD) acquisition programs. The paper focuses on one aspect of these decision processes in the defense acquisition environment: behavioral biases. In three defense acquisition programs studied, we find strong evidence that planning fallacy, difficulty in making trade-offs, over-optimism, and recency bias affected the management and decision-making within these programs. This research helps us better understand and predict how acquisition professionals and senior leaders think and make decisions about program strategy, managing resources, and leading people. A key element in this perspective is that important insights into these decisions derive from models in which agents are not fully rational. Behavioral acquisition is analogous to behavioral finance, which has successfully applied social science theories—especially from psychology—to improve the accuracy of predictions about the behavior of actors across the entire financial landscape.

The mission of the Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD A&S) is to quickly and cost effectively deliver and sustain secure and resilient capabilities to warfighters and international partners. There are urgent requirements to develop adaptive acquisition framework (AAF) to speed up software development and acquisition processes that strengthen the concepts of operations (CONOPS) such as distributed maritime operations (DMO). It is imperative for the Department of Defense (DoD) to shape the AAF using data-driven analysis linked to the National Defense Strategy and the nature of global threats, and scale new capabilities to counter new threats. The threat and capability coevolutionary matrix (TCCM) addresses the requirement. A threat is a problem a capability tries to deal with. A capability is the solution to the problem that represents a threat. Coevolutionary algorithms explore domains in which the quality of a capability or combination of capabilities is determined by its ability to successfully defeat a threat or combination of threats. TCCM has the potential to systematically optimize, recommend, and coevolve capabilities and threats in new and contested environments. We show a use case regarding helping a program executive office (PEO) to wargame capabilities and threats against a specific domain DMO using unclassified data compiled from open sources.

Program technical reviews are discrete points in time, within a system's life cycle, where the system is evaluated against a set of specific accomplishments, known as "entrance criteria." These criteria are used to track the technical progress, schedule, and program risks. Technical reviews serve as gates used to demonstrate that the program is on track, and should be allowed to proceed. Current technical reviews are based around evaluations of static, contractually obligated documents that are used to demonstrate successful completion of the entrance criteria. These documents represent "snapshots" of the systems and do not represent a total view of the system. As a result, systems are often viewed by the entrance criteria individually, which fail to account for the system from a holistic perspective. Department of Defense (DoD) organizations are migrating to Model-Based Systems Engineering (MBSE) environments, with a vision of modernizing, developing, delivering, operating, and sustaining systems. Model-based reviews allow for complexity to be managed more efficiently because data, not "systems engineering products," is the commodity used to evaluate the entrance criteria. The data-driven MBSE technical reviews will provide greater insights and details across a program's life cycle. This paper discusses the applicability of current technical reviews criteria to MBSE.

The federal government’s use of Other Transaction Authority (OTA) agreements has exploded in recent years, thanks in large part due to a surge in popularity within the Department of Defense (DoD). Neither a contract, grant, or cooperative agreements, OTAs are an acquisition approach that enable certain federal agencies to access goods and services outside of the traditional acquisition system. This research examines the trends in OTA usage across DoD to provide insights into what DoD is using OTAs for, how they are spending under an OTA, and to whom the majority OTA obligations go.

System-of-Systems (SoS) capability emerges from the collaboration of multiple systems, which are acquired from independent organizations. Even though the systems contribute to and benefit from the larger SoS, the data analytics and decision-making about the independent system is rarely shared across the SoS stakeholders. The objective of this work is to identify how the sharing of datasets and the corresponding analytics among SoS stakeholders can lead to an improved SoS capability. Our objective is to characterize how the sharing of connected data sets may lead to deployment of different predictive (predicting an outcome from data) and prescriptive (determining a preferred strategy) analytics and lead to better decision outcomes at the SoS level. We build and demonstrate a framework for this objective based on extensive literature review and generating appropriate predictive and prescriptive methodologies that can be used for SoS analysis: Additionally, we propose to utilize machine learning techniques to predict the SoS capability achievable by sharing pertinent datasets and to prescribe the information links between systems to enable this sharing. Two case studies demonstrate the use of the framework and prospects for meeting the objective. Highlights of our study are summarized next.

Major defense acquisition programs take about eight years to proceed from program initiation to an initial operational capability. This cycle time is longer than it takes adversaries to create new problems for operational military forces. Prior statutory changes have not significantly affected cycle times. Recent changes created middle tier acquisition programs intended to deliver capabilities and products in less than five years. These middle tier acquisition programs are rapid prototyping and fielding pathways with new governance, acquisition authorities and schedule duration tied to requirements approval dates. The Department of Defense acquisitions continue to evolve, and program offices must concurrently adapt to both emergent guidance and programmatic realities. Including innovations such as system modularity and agile system development methods into these new program types can create additional programmatic schedule risks and opportunities. These in-stride adaptations can affect the capability of a program office to deliver an effective system within promised cycle times. This research explored schedule growth risks associated with new acquisition pathways and process innovations. It used public data to identify schedule-related risk factors associated with middle tier acquisition and process innovations. We developed quantitative schedule models for middle tier acquisition programs to predict schedule durations and schedule risks associated with application of various innovations within rapid acquisition pathways. We identified and analyzed schedule growth risk mitigation strategies. This research contributes to the understanding of the risks and opportunities associated with recent acquisition process changes. The research results will be useful to program offices and acquisition leadership in executing current and future rapid acquisition programs.

This technical report summarizes the research conducted by Massachusetts Institute of Technology under contract award HQ0034-19-1-0002 during July 22, 2019 – August 31, 2021. Involved research team members include: Dr. Donna H. Rhodes, Principal Investigator; Dr. Eric Rebentisch, Research Associate; and Mr. Allen Moulton, Research Scientist. Systems engineering practice is evolving under the digital engineering paradigm, including use of model-based systems engineering and newer approaches such as agile. This drives a need to re-examine the existing use of metrics and leading indicators. Early engineering metrics were primarily lagging measures, whereas more recent leading indicators draw on trend information to provide more predictive analysis of technical and programmatic performance of the engineering effort. The existing systems engineering leading indicators were developed under the assumption of paper-based (traditional) systems engineering practice. This research investigates the model-based implications relevant to the existing leading indicators. It aims to support program leaders, transitioning to model-based engineering on their programs, in continued use of leading indicators. It provides guiding insights for how current leading indicators can be adapted for model-based engineering. The study elicited knowledge from subject matter experts and performed literature review in identifying these implications. An illustrative case was used to investigate how four leading indicators could be generated directly from a model-based toolset. Several recommendations for future research are proposed extending from the study. A companion research study (“phase 2”) under contract HQ0034-20-1-0008 provides insights for the art of the possible for future systems engineering leading indicators and their use in decision-making on model-centric programs. For completeness, selected background information and illustrative case are included in the technical reports in both stu

This report is an extension of the originally proposed sequence of three studies that developed a cutting-edge modeling and simulation tool for the Acquisition Workforce (AWF). The initial objective of that sequence was to build a Dynamic Retention Model (DRM) from the ground-up for the AWF to restore and maintain a capable and flexible acquisition workforce in support of the needs of the modern warfighter. The current report uses the previous model to analyze the phenomenal and unprecedented impact of COVID-19 pandemic on the U.S. civilian sector and its potential effects on the size and composition of the AWF in the coming years. After going steadily down for almost a decade and being at the historical low of 3.5% in February 2020, the U.S. unemployment rate spiked to almost 15% in April 2020. This event represented an unparalleled increase of more than 11% in just two months. As surprising as the initial increase was the sharp fall in the U.S. unemployment rate that followed. As of November 2021, just a year and a half after the peak, the unemployment rate is hovering around 4.6%, barely more than one percentage point above the previous historical low. While the impact of COVID-19 has been so far much harsher on the civilian sector employment than on the government sector (i.e., and the AWF), it is unclear how the latter will evolve in the mid- and long-run after the fast, ongoing recovery of the private sector. We take advantage of the DRM developed in the previous studies and extend it to explore the potential consequences of economy-wide shocks (such as COVID-19) on the AWF as the economy shows signs of strong recovery. We start analyzing the behavior of a representative AWF worker at the beginning of the pandemic, when the strength of the economic recovery was highly uncertain. We find that, under a number of different scenarios regarding the speed of recovery, it takes several years (in expectation) before the AWF employee returns to the pre-pandemic behavio

This analysis seeks to assess the role of Congress in the resourcing of defense acquisition programs. Via the appropriations process, Congress exercises its oversight authority on the executive branch’s defense policy and can choose to match, modify, or eliminate the Department of Defense’s (DoD) requested funding levels for acquisition programs primarily funded by procurement and research, development, test, and evaluation (RDT&E) accounts. Congress’ decisions in this process can have a significant impact on the executive branch’s defense plans by making adjustments to acquisition projects’ program of record. This in turn can force DoD program management teams to alter schedules and contracting actions and can have second-order effects on private sector partners in the acquisition process. To measure Congress’ impact on defense acquisition funding, this study compares the actual funding level for procurement and RDT&E accounts with the original level proposed in the administration’s budget request and identifies patterns in which accounts are regularly adjusted by Congress. It assesses procurement and RDT&E accounts between FY 2001 and FY 2020 and conducts data cuts of acquisition funding at the account, category, military department, and budget activity levels. This analysis ultimately aims to inform defense planners, acquisition officials and program managers, and industry partners of trends in congressional appropriations for defense so they can better anticipate Congress’ impact on defense acquisition funding.

Cyber attacks continuously target organizations, however, the mitigation actions taken for defense are not sufficiently effective. Ability to compute the cost of attacks is crucial to assess the effectiveness of countermeasure investments. In this study, we developed a framework to have a well-informed decision-making process in cybersecurity acquisition by evaluating the business impact caused by the operability losses of assets. We tested the developed framework using various attack and mitigation scenarios. The findings suggest that using a simulation approach to calculate the business impact of cyber attacks provides the ability to support decision-making process.

To support the modern warfighters tasked with increasing demands in a constantly changing global environment, it is imperative that the defense acquisition system continue to evolve to maintain its capability and flexibility. In this effort, growing a talented, experienced, and well-qualified civilian workforce will be vital. As part of this broad effort, the Section 809 Panel has recommended change to the DoD’s career management framework to grow and augment the workforce, and the DoD AWF Strategic Plan — FY 2016 – FY 2021 has emphasized efforts since 2010 to restore and restructure the AWF after a period of twenty years of shrinkage. This technical research report is the second in a proposed series of three linked studies to provide a cutting-edge modeling and simulation tool that leverages the increase in availability of AWF data and the large increases in computing power in the last decades. Building on the proof-of-concept model created as part of the first-year effort, we continue our development of a “Dynamic Retention Model (DRM)” designed from the ground-up for the AWF. Using a large personnel dataset of the acquisition workforce as well as a representative dataset of the civilian population from the Bureau of Labor Statistics, we estimate our DRM. DRM is a leading-edge technique that uses a powerful mathematical/econometric technique called dynamic programming. It takes a complex, multi-period problem (such as the lifetime labor market decisions of an acquisition worker) and breaks it down into simpler, one-period sub-problems in a recursive manner. Solving a single-period problem “nests” the future decisions that the worker will make, allowing the estimation and prediction of complex behavior in a surprisingly manageable framework. With estimates from the model, we simulate how various modifications in personnel policies, such as changes in salary structure and bonuses, would have affected the labor market decisions of the workforce. In particular, our mo

This paper is part of a research agenda outlined in Franck, Hildebrandt and Udis (2016) – directed toward improving the realism of defense acquisition schedules. Defense acquisition schedules have long been a difficult problem. In this particular effort, we consider primarily the case of the 737MAX – which has been a fortuitous example of the risks of scheduling-by-fiat. We analyze the 737MAX misadventure using Systems Dynamics and Root Cause Analysis methods.

In phase one of this research, (Gamble, 2020, Brook 2020), DoD areas on the GAO’s High-Risk List (HRL) were compared to similar longstanding high-risk non-Defense programs to determine if the DoD is a high-risk anomaly. Three attributes characterizing risk emerged: (1) the more technical programs have greater risk; (2) defense and national security areas have greater financial risk; (3) larger programs have greater, more prolonged risk. The study concluded DoD is a high-risk anomaly as the agency, and every area within, has two of the three attributes; but is not an anomaly in one as this attributes are present in non-Defense areas as well. From this analysis questions emerged regarding whether the DoD can ever get off the HRL and whether the DoD should prioritize getting off the HRL. Additionally, we perceived a theory of interaction between the DoD, the GAO and Congress that would influence the answers. To explore these questions, we examined the interaction theory and its practice, and interviewed over twenty current and former officials and staff experts in the GAO, the DoD and on Capitol Hill. The analysis amplifies the earlier findings and reveals mixed and inconclusive views on whether the DoD could ever get off or should even prioritize getting off the HRL in its management agenda.