Recent military operations around the world demonstrate the superiority of U.S. weapon systems developed by the Department of Defense (DoD). Furthermore, an ever-increasing percentage of the weapon systems' functionality is provided by software, which constantly becomes more sophisticated and complex. While the DoD has risen to the challenge, cost overruns and unsatisfactory performance have led the General Accounting Office (GAO) to designate the DoD systems development and modernization efforts a high-risk area [1].

Significant risk factors include the enormous size and complexity of the software within these systems and acquirers' inadequate, inefficient, or unexpected processes for managing software-intensive system acquisitions. As one congressional source said when describing the acquisition of U.S. weapon systems, "It's not about bending metal any more, it's about routing electrons."

Software enables a myriad of complex capabilities from massive data fusion across geographically disparate large-scale sensor systems, to decisional systems that automatically select the most appropriate weapon and platform to attack a given target, to autonomous systems that operate without human intervention to destroy incoming missiles. Software creates the network- centric operation — the cornerstone of the DoD's transformation.

Several root causes for the GAO's designation point to long-standing cultural issues (culture being defined as the collective patterns of behavior exhibited by the numerous participants in the acquisition process and the incentives for their behavior). These cultural issues were highlighted in 1992 GAO reports [2, 3]. Two of these still-relevant issues are the acquisition community's bias toward hardware, and the fact that the community addresses critical software issues too late in the acquisition process.

In a 1998 CrossTalk article [4], Capers Jones defined a major DoD system as having 12.5 million C Statements² (roughly the size of a major computer operating system of that day) and a development team that numbered in the hundreds. Typically, lack of process and intergroup communications was a problem; paperwork and software rework absorbed the bulk of development costs. Formal configuration control and change management were expensive and poorly implemented for projects that large. The probability of termination for one of those major software-intensive systems, Jones said, was 65 percent; he cited poor project management and inadequate quality control as primary factors.

Fast-forward five years to today's jointly developed system of systems. Take, for example, the Army's Future Combat Systems (FCS), a joint Army/Defense Advanced Research Projects Agency program. The Army's vision for the FCS is to create an integrated battlespace where networked information and communications systems provide a competitive edge to soldiers in the field and commanders in the control room. You would be hard pressed to even try to estimate the numbers of FCS developers as its extended team consists of one prime contractor, eight major subcontractors, and 55 other companies under contract [5].

According to congressional sources, "The FCS is estimated at 32 million total SLOC," or software source lines of code. The actual number, however, will likely be greater as past experience with software estimation has shown that we typically both underestimate size and add functionality as the development progresses.

Successful fielding of the FCS requires more mature acquisition, development, and testing approaches than used in the past for smaller systems. Previous approaches simply will not be adequate to guarantee that development cost, schedule, and performance baselines are met. Specifically, greater effort will have to be spent on managing changes to requirements and ensuring that information is shared among all stakeholders. What does all this mean to both the program offices and Congress? Mature processes must be used to ensure that the system functions as intended, and that major problems and errors are caught well in advance of operational tests.

Given that software-intensive projects are among the most expensive and risky undertakings of the 21st century, the investment in weapons from fiscal years 2003 through 2009 will exceed $1 trillion [6]. Furthermore, many of the DoD's most important technology projects will continue to deliver less than promised unless changes are made [7]. Improving how we acquire software-intensive systems is both long overdue and an imperative.

In the late 1980s, software developers began investing in process improvement by adopting best practices. Many public and private organizations based their improvement programs on the Software Engineering Institute's (SEI^SM) Capability Maturity Model® (CMM®) for Software (SW-CMM). While adoption was slow at first, by the mid-90s companies with improvement programs were showing results.

For example, SEI reported in 1995 [8] that a major defense contractor that implemented a process improvement program in 1988 had reduced its rework costs from about 40 percent to about 10 percent of total project cost, increased staff productivity by 170 percent, and reduced defects by about 75 percent over a seven-year period. According to a 1999 SEI report [9], a software development contractor reduced its average estimated schedule deviation from 112 percent to 5 percent between 1988 and 1996. During that same period, SEI reported that this same contractor reduced its average estimated cost deviation from 87 percent to minus 4 percent.

By 2001, software development units within the DoD were also showing results from their improvement programs. According to one GAO report [10], each DoD unit with a software process improvement (SPI) program reported positive effects on software/systems quality. The Defense Finance and Accounting Service, for example, reported that its SPI program had reduced its overall software delivery cost by about one-third less than organizations of similar size; one Navy software activity reported reduced costs, improved product quality, and a 7:1 return on its SPI investment; and an Army activity reported that it had almost doubled its productivity in writing software for new systems because of improvements made under its SPI program.

While many defense and civilian contractors developing software-intensive systems have made performance gains through SPI, those acquiring these same systems have lagged behind. In situations where acquirers with a low level of process maturity contract for software from developers with a high-level process maturity, problems occur. For example, acquirers may try to circumvent development and management processes because they feel that following the process impacts their ability to meet the goal. The result of this process avoidance by the acquirer can be rework, additional delays, or unexecutable cost and schedule quotes — exactly what the process was designed to avoid had it been followed.

Other problems can occur at the end of the development process. If cost and delivery schedules become more important to the acquirer than having the developer meet their exit criteria for delivering a quality product, then the result can be software delivered with avoidable defects. An acquirer with a low process maturity is at a greater risk of having its program meet schedules and costs, but fail to deliver required performance.

The GAO has been reviewing weapon systems investments for more than 20 years. What they have found are consistent problems — cost increases, schedule delays, and performance shortfalls — along with underlying causes such as pressure on program managers to promise more than they can deliver [6]. In recent years, several of those reports have included consistent recommendations to implement best practices for software-intensive systems acquisition, and to initiate broad improvement programs.

In a 2001 report to the Armed Services Committee, for example, the GAO recommended that DoD establish and implement a department-wide SPI program based on accepted best practices [10]. In response to GAO's recommendations, the DoD identified two existing groups within the Office of the Secretary of Defense (OSD)^{3, 4} as appropriate places for SPI to be addressed. The DoD also pointed to a revision of DoD Regulation 5000.2-R (since cancelled) as the needed policy guidance for improving software. The author believes that subsequent DoD inaction in response to GAO's recommendations played a pivotal role in Congress legislating software acquisition process improvement.

On Dec. 2, 2002, Section 804 of the National Defense Authorization Act for Fiscal Year 2003 [11] (or simply Section 804) was enacted. The Senate report accompanying its version of the National Defense Authorization Act for Fiscal Year 2003 [12] was clear on its intent and purpose. The report articulated the Senate's concerns with the negative impact of longstanding software problems on major defense acquisition programs. The Senate noted the recommendations from [10] and stated that the purpose of Section 804 was to implement the GAO's recommendations.

Section 804 mandates improvement of the DoD's software acquisition processes. This legislation directly instructs the secretaries of each military department and heads of selected defense agencies to establish software acquisition process improvement programs — an apparent message of frustration with the way software improvement had been handled in the past.

Software acquisition process improvement program requirements include the following:

• A documented process for software acquisition planning, requirements development and management, project management and oversight, and risk management.
• Efforts to develop appropriate metrics for performance measurement and continual process improvement.
• A process to ensure that key program personnel have an appropriate level of experience or training in software acquisition.
• A process to ensure that each military department and select defense agency implement and adhere to established processes and requirements relating to the software acquisition.

Section 804 also requires that the assistant secretary of defense for Command, Control, Communications, and Intelligence, in consultation with the undersecretary of defense for Acquisition, Technology, and Logistics do the following:

• Provide applicable improvement program administration and compliance guidance, and ensure that secretaries of the departments and selected agencies comply with that guidance.
• Assist the departments and agencies with their respective improvement programs by ensuring they use applicable source-selection criteria and have access to a clearinghouse for information regarding best practices in software development and acquisition in both the public and private sectors.

Norm Brown, founding director of the former Software Program Managers Network, and Navy department member of the 2000 Defense Science Board Task Force on Defense Software said:

Anyone looking at the past congressional actions and listening to the frustration expressed in congressional hearings will find the fundamental improvements mandated in Section 804 come as no surprise. The only surprise is that Congress has been as patient as they have been. Now, congressional patience seems to be turning to impatience; an impatience to see significant improvement in fixing our perennial problems with cost, schedule, and performance — and in addressing the underlying drivers that are causing these problems.

Congressional sources affirm that:

... [the] DoD is going to have to pay attention from the ground up, in other words, at the program manager level, or programs will continue to get tanked. Congress will remain interested and we're not going to let this go until [the] DoD significantly improves how it acquires softwareintensive systems. The only way it's going to get fixed is by people on the inside — it simply makes no sense on any level to continue to ignore it.

Another indication of Congressional intent is the GAO's tasking to monitor the DoD's compliance with Section 804. Initially, the GAO was tasked to evaluate the DoD's efforts to develop programs for improving software acquisition processes and to assess how those efforts compared with leading commercial companies' practices. This initial GAO report (GAO-04- 393) was scheduled for publication in March 2004. Subsequent GAO assessments will likely focus on compliance with specific Section 804 requirements.

On March 21, 2003, the DoD issued a memorandum to provide the uniform implementation guidance that Section 804 requires. This memorandum identified applicability, delineated organizational roles and responsibilities for overseeing implementation, and clarified initial expectations for improvement programs. It also instructed military departments and selected defense agencies to establish software acquisition process improvement programs. Requirements for these programs included defining and applying measures, following applicable methods based on some structured approach that included an appraisal method, and determining and reporting status of process adherence and performance effectiveness.

The DoD memorandum also gave the OSD Software Intensive Systems Steering Group the role of leading a DoD-wide effort to improve software acquisition processes. This role entailed providing program guidance; identified best practices; established a clearinghouse of information regarding best practices and lessons learned in software development and acquisition; and provided guidance for documenting, performing, and continuously improving a minimum of eight specific software acquisition processes (the original four processes called out in Section 804, plus four additional processes⁵).

The OSD's implementation guidance has not been prescriptive. Component and agency approaches to compliance vary widely. That variety is clearly illustrated by the list of best practice models selected as the basis for software acquisition improvement programs. Model selections range from the IDEAL^SM Model⁶, to the CMM Integration^SM (CMMI®)⁷, to the Software Acquisition CMM (SA-CMM®)⁸, to the Federal Aviation Administration (FAA) Integrated Capability Maturity Model (FAA-iCMM®)⁹, to hybrid models (i.e., combining elements of two or more different models), to no identified model at all. There is no one right answer, but instead a variety of approaches are being tested by the small but growing DoD-wide software acquisition process improvement community of practice¹⁰.

A new tool will soon be available to help those looking for acquisition best practices. The CMMI Steering Group, cochaired by the DoD and industry, has sponsored the development of a CMMIbased guide for acquisition programs. The CMMI Module for Acquisition¹¹ focuses on effective acquisition practices used by first-level acquisition projects (e.g., system project offices/program managers). It also provides guidance to acquisition organizations above the acquisition project level to support institutionalization of those acquisition practices. In addition to covering the 804 requirements, many of the acquisition practices and amplifications in the Module are drawn from existing sources of best practices including the SA-CMM, the CMMI, the FAA-iCMM, as well as additional coverage areas defined by experienced acquisition professionals.

As a key participant in the Naval Air Systems Command's (NAVAIR's) software acquisition process improvement program, the author is able to share with readers NAVAIR's approach as one data point. That approach is divided into three phases: 1) requirements determination, 2) gap analysis and planning, and 3) implementation, as explained below.

The requirements phase began by forming a small, command-endorsed team. That team selected relevant best practice models, mapped existing command policies to those practices, and developed and implemented a communications plan. The team chose a hybrid improvement model for mapping policies to practices. For precontract process areas¹², it selected the SACMM and for post-contract process areas, it identified the CMMI¹³. The team also added a ninth process area (to the eight provided by the OSD) — Measurement and Analysis from the CMMI — in order to emphasize the importance at NAVAIR of performance measurement.

The next phase entailed performing a policy gap analysis and developing a command- wide improvement plan. Policy owners identified changes to policy needed to comply with the selected best practices. A broader team was then formed — with representation from all executive program offices — to develop a NAVAIR software acquisition process improvement plan (SAPIP). In addition, an existing SPI enterprise team, the NAVAIR Software Resource Center (SRC), was tasked to build or identify the infrastructure to support the SAPIP through a network of strategic partners.

Phase three was simply stated but represents a significant, long-term commitment: program managers execute the SAPIP and comply with revised NAVAIR policies. During the ongoing implementation phase, the SRC will work with individual programs to help them select the best practice model(s) that best support their business goals and baseline their processes.

Section 804's mandate for the DoD software acquisition process improvement programs is here to stay. It is not a onetime legislation with little or no follow-up, but the result of a consistent, well documented, and growing need. Already, congressional sources are considering actively identifying certain key programs for greater scrutiny to see if they have adequately implemented the requirements of the legislation. According to GAO sources, "The outcome is what's important, not which best practice improvement model is used as a road map to achieve the mandated requirements."

Given that the GAO and Congress feel that the acquisition of systems with major software components needs to be improved, it is imperative that DoD program managers understand that their efforts will be measured against Section 804 requirements.

As members of the DoD community, Section 804 is our collective call to action. While some DoD components and agencies have already taken steps to improve their software acquisition processes, others have not. NAVAIR, for example, has been addressing software development process improvement issues well in advance of Section 804 through an existing framework of system/software leadership teams.With the signing of Section 804, NAVAIR emphasizes its strategic goal to improve its software acquisition performance, continue to focus resources on refining policy, communicate implementation guidance, and expand its SPI support infrastructure. To achieve its goal, NAVAIR understood that top management support and metrics to gauge implementation effectiveness were essential.

How will your organization satisfy this critical need to improve?

1. U.S. General Accounting Office. "High Risk Series: An Update." GAO/HR- 99-1. Washington, D.C.: GAO, Jan. 1999.
2. U.S. General Accounting Office. "Mission-Critical Systems: Defense Attempting to Address Major Software Problems." GAO/MTEC-93-3. Washington, D.C.: GAO, Dec. 1992.
3. U.S. General Accounting Office and National Security and Internal Affairs Division. "Weapons Acquisitions: A Rare Opportunity for Lasting Change." GAO/NSIAD-93-15. Washington, D.C.: GAO, Dec. 1992.
4. Jones, Capers. "Project Management Tools and Software Failures and Successes." CrossTalk July 1998: 13.
5. Caterinicchia, Dan. "Firms Added to Army FCS Mix." Federal Computer Week June 2002.
6. U.S. General Accounting Office. "Defense Acquisitions: Assessment of Major Weapon Programs." GAO-03- 476. Washington, D.C.: GAO, May 2003.
7. U.S. General Accounting Office and National Security and Internal Affairs Division. "Observations on the Department of Defense's Fiscal Year 1999 Performance Report and Fiscal Year 2001 Performance Plan." GAO/NSIAD-00-188r. Washington, D.C.: GAO, 30 June 2000.
8. Hayley, T., et al. "Raytheon Electronic Systems' Experience in Software Process Improvement." CMU/SEI-95- TR-017. Pittsburgh, PA: Software Engineering Institute, Nov. 1995.
9. Ferguson, Pat, et al. "Software Process Improvement Works!" CMU/SEI-99- TR-027. Pittsburgh, PA: Software Engineering Institute, Nov. 1999.
10. U.S. General Accounting Office. "DoD Information Technology: Software and Systems Process Improvement Programs Vary in Use of Best Practices." GAO-01-116. Washington, D.C.: GAO, Mar. 2001: 12.
11. Public Law PL-107-314.
12. Senate Report S.2514.

1. Defense Science Board Task Force on Defense Software, Nov. 2000.
2. The C Statement count is based on the average number of C Statements found within the typical function point.
3. The Independent Expert Program Review Working Group, established in Jan. 2001.
4. Software Intensive Systems Steering Group, chartered in Sept. 2000.
5. Additional process areas included configuration management, test and evaluation, integrated team management, and solicitation and source selection.
6. Information on the IDEAL Model can be found at www.sei.cmu.edu/ideal.
7. Information on the CMMI can be found at www.sei.cmu.edu/cmmi/cmmi.html.
8. Information on the SA-CMM can be found at www.sei.cmu.edu/arm/SA-CMM.html.
9. Information on the FAA-iCMM can be found at www1.faa.gov/aio/ProcessEngr/iCMM/index.htm.
10. See the OSD's Acquisition Community Connection Web site at www.acq.osd.mil.
11. At the time this article was printed, the CMMI Module for Acquisition was pending publication. Its document number will be CMU/SEI-2004-TR-001.
12. Acquisition planning, and solicitation and source selection.
13. Requirements development and management, configuration management, risk management, project management and oversight, test and evaluation, and integrated team management.

1. Defense Science Board reports can be found at www.acq.osd.mil/dsb/reports.htm.
2. General Accounting Office reports can be found at www.gao.gov.
3. Software Engineering Institute reports and other publications can be found at www.sei.cmu.edu/publications/search.html.
4. Back issues of CrossTalk can be found at www.stsc.hill.af.mil/crosstalk.
5. A search function and online archive for Federal Computer Week can be found at www.fcw.com/onlinearchive.asp.
6. Section 804 of Public Law PL 107-314 and other related documents can be found at the STSC Web site at www.stsc.hill.af.mil. Enter Section 804 into the search engine.
7. You can search the Congressional Record at www.senate.gov/pagelayout/legislative/d_three_sections_with_teasers/congrecord.htm.

Lisa Pracchia is a member of the Naval Air Systems Command's Software Resource Center. Her software background includes process improvement, business analysis, project management, product life-cycle management, and product marketing in a wide range of industries (discrete product manufacturing, international publishing, telecommunications, and defense). Pracchia has a master's degree in management from the University of Redlands.

Commander,NAWCWD
41K300D (L. Pracchia)
BLDG. 1494, STOP 6308
1 Administration CIR
China Lake, CA 93555
Phone: Phone: (760) 939-2188
Phone: DSN: 437-2188
Fax:
E-mail: lisa.pracchia@navy.mil