The Tri-Service Assessment Initiative was instituted to provide an independent, objective review and analysis of software processes, product development, and integration. Numerous cost overruns, schedule slippages and performance issues motivated the focus on software. The goal of the initiative is to provide direct assistance to program managers as well as gain visibility over recurring issues that impact software-intensive systems acquisition.

The initiative's primary objective is to provide assistance to DoD program managers (PMs). Independent assessment teams provide a comprehensive review of the programs, identify risks, and make recommendations for management and risk mitigation. Participation in the initiative is strictly voluntary; PMs must request an assessment, and the reports are limited to distribution only upon approval of the PM. The team and the PM jointly establish assessment scope and initial issue areas.

An equally important objective is to utilize what is learned from assessments to assist software intensive systems as a whole. Based upon generic, systemic issues found across the assessments, the initiative provides feedback to DoD and to senior acquisition managers, identifying issues in areas of policy, education, and training.

To accomplish these objectives, the initiative maximizes opportunities for leveraging and collaborating across DoD, government agencies, federally funded research and development centers (FFRDCs), and industry to utilize the widespread skills, experience, tools, and techniques. This is a key feature.

As a direct result of the assessments conducted to date (19 since inception), PMs are implementing relatively low-cost post-assessment recommendations and realizing high returns. Former Deputy PM of the Army's Crusader Self-Propelled Howitzer Program Kevin Fahey agreed that the initiative has been beneficial. As is the case with many large-scale programs, Crusader has been through several assessments, and Fahey stated that, "Tri-Service [Assessment Initiative] for the most part was probably the best because they focused on solutions instead of just the challenges. They did an outstanding job. They reconfirmed what we had to focus on. They gave us a lot of good ideas and places to look, people to use, tools to use."

In May 2000, Deputy Under Secretary of Defense, Science and Technology (S&T), Dr. Delores Etter, assumed sponsorship of the Tri-Service Assessment Initiative. This was the result of a Defense Science Board recommendation to the Under Secretary of Defense (AT&L) Dr. Jacques Gansler emphasizing the value of independent assessments performed for PMs in which they would retain the data. Thus results are used directly by the PM, rather than as a reporting mechanism.

Dr. Etter, seeing this as a key element of her mission to oversee and improve software-intensive systems acquisition, incorporated the Tri-Service Assessment Initiative into her Software Intensive Systems Directorate to fulfill this recommendation. Office of the Secretary of Defense (OSD) sponsorship of the initiative offers several critical advantages. First, it maintains a joint-service nature, promoting assessments as a service to any PM in DoD. Second, the initiative builds upon Dr. Etter's efforts to establish a collaborative environment of DoD software expertise. Thus, participation on the assessment teams leverages the expertise across the DoD community. Finally, OSD sponsorship promotes the initiative's systemic issue analyses, which provide valuable lessons learned to improve DoD policy, education and training.

The assessment initiative is service-executed and is overseen by a Tri-Service management board. The service executive agent is the U.S. Army's Tank Automotive-Armaments Command, Armament Research, Development and Engineering Center. Each service is represented on the management board, along with members from the Office of the Assistant Secretary of Defense for Command, Control, Communications and Intelligence, OUSD (AT&L) Systems Engineering and ODUSD (S&T) Software Intensive Systems. The totality of organizations involved as sponsors, executors and resource providers is shown in Figure 1.


Figure 1: Oversight structure
(Click on image above to show full-size version in pop-up window.)

The initiative manager (IM) manages day-to-day operations and implementation of the initiative, from strategic coordination and planning to tactical-level assessment execution and logistics. OSD sponsors promote the initiative by providing funding and endorsement; and gain insights into the emerging non-program attributed systemic lessons learned. The service sponsors advocate the use of the assessments within their respective services, and help the IM identify programs that can benefit from and participate in the initiative. The service sponsors also help set strategic direction and work to implement enterprise level recommendations.

The IM selects assessment team members on a program-by-program basis by carefully matching the program-unique issues to the appropriate skills and expertise available in a nationwide resource pool composed of organizations within DoD, other federal agencies, FFRDCs, and industry (see Figure 2).


Figure 2: Assessment team members from diverse orgs
(Click on image above to show full-size version in pop-up window.)

To ensure consistent application of the assessment process, the initiative developed an assessment architecture. It is called an architecture because it provides an integrated structure that ties existing software and systems assessment tools and techniques together into a comprehensive assessment methodology.

The assessment architecture has two components as shown in Figure 3: the Assessment Process Model and the Assessment Information Model. The process model describes tasks, products, and responsibilities for each of the seven activities of the assessment process. The three core program assessment activities are: initiate and plan assessment, perform assessment, and integrate and report assessment results.


Figure 3: The assessment initiative
(Click on image above to show full-size version in pop-up window.)

The assessment's span, on average, is a two- to three-month timeframe that begins by meeting with the PM to scope and strategize the assessment and understand his concerns. The selected team then meets to review program information and assessment strategy, and understand their individual roles. The conduct of the assessment will normally involve site visits with the program office, contractor (s), and other involved organizations as necessary, depending on the assessment scope. The assessment is focused to be as nondisruptive as possible to the program while still obtaining the needed information. The team meets with key individuals at each location to discuss the program and review available artifacts. Minimal preparation is required from organizations being assessed with the exception of assisting the assessment team with logistics, such as scheduling interviews, reserving conference rooms, etc.

The assessment team members sign nondisclosure agreements and stress open communications with the customer and their developer locations. Discussions held during the site visits are not attributed so that issues reported are not tied to individuals. At the end of the site visit, the assessment team conducts an exit briefing to discuss initial observations and provide an opportunity for feedback in the event that something was missed or misunderstood.

The team then reviews and analyzes the data, relates the findings within and across the issues, determines program strengths and risks, and formulates program recommendations. The PM receives and reviews a draft version of the report. Additionally, the initiative provides an independent peer reviewer. Final briefings are provided to the PM to ensure the findings and recommendations are understood. The IM or assessment leader delivers the report directly and solely to the PM.

The second volume of the architecture is the Assessment Information Model, which contains an issue-based menu that guides the team through an assessment of 10 different programmatic and technical issues. The model provides a consistent baseline for identifying, assessing, and correlating program issues and strengths. The issues deliberately encompass not only software issue areas, but also programmatic areas. The assessment teams found that software issues are, more often than not, traceable to larger, more systemic program issues such as a lack of engineering inspection, poor requirements management, inadequate functional requirements mapping, and shortcutting or eliminating software processes to accommodate unrealistic schedules. The 10 primary issue areas, described briefly below, help ensure that the assessments are performed consistently, are repeatable, and have considered all the varied aspects of a program. The assessment issue structure includes:
Environment What is the regulatory, labor, reform, and political environment in which the program operates?
Mission Requirements How complex is the development? Can it defeat the threat arrayed against it? Is the operational requirement reasonable? Does the developer have the expertise, plant, and equipment to successfully develop the required system? What are the critical dependencies between other systems?
Financial Is funding sufficient, timely, and stable? Is there enough flexibility in funds management to deal with program issues?
Resources Do the PM and the developer have the personnel, facilities, tools, and training to complete successful development?
Management Do the PM and the developer have the capability to plan, resource, control, and monitor the effort? This includes the acquisition strategy, project planning, contracting and subcontracting, and communication processes.
Technical Process Does the developer possess the capability to implement the technical processes needed to manage and conduct the development and ensure process conformance? Is that process appropriate? Is it applied to the program?
Technical Product How well do the products and services being produced conform to the requirement? This issue considers product lines, testing requirements, quality, human factors, and safety.
Schedule Is the schedule realistic? Does the schedule reflect resource usage and availability? Does the schedule track progress and dependencies?
User/Customer Is the end user or customer of the product appropriately sup-ported? This issue includes customer satisfaction and new equipment training or transition support.
Project Specific Are there any project-unique issues that cannot be mapped into one of the previous nine issue areas?

Although the initial focus of the assessments was software, the initiative is undergoing an expansion to encompass a total systems perspective. The issue areas lend themselves to this expansion. PMs have repeatedly asked that systems issues be investigated in the assessments.

For example, one program requested the team evaluate the human factors that technologies applied in their program. Another asked for a specific focus on simulation-based acquisition and guidance on its implementation. Yet another has requested review and assistance with communication and interoperability issues.

Interoperability and integration of multiple software systems into one platform is a common stumbling block for programs. The expansion of the initiative into the systems realm was also pursued to encompass these interoperability issues. The initiative is now used to perform assessments across programs such as within a program executive office or across a weapon system domain.

Rear Adm. Kathleen Paige, assistant secretary of the Navy (Research, Development and Acquisition), chief engineer, has overseen the implementation of the Tri-Service Assessment Initiative across several of the programs within the Navy. She says, "It is important to have specialist teams take a high-level systems look at some of our Naval programs and identify areas that need attention, especially in the area of software. Additionally, the Navy and the Marine Corps are taking advantage of this DoD capability to look not only at individual programs, but also at the issues that arise because of the critical interfaces between these programs."

As part of the lessons learned analysis, findings are mapped against each of the ten issue areas. The mapping is a "one-to-many" relationship since often the observed effects stem from multiple causes. For example, in a program that consistently misses milestones, an assessment team may find that the developer had difficulty adequately staffing the development, the program manager had trouble obtaining sustained and adequate funds, and the program team (both government and contractor) had no documented requirements management process. This "missed milestones" finding is related to the schedule, technical process, management, resources, and financial issue areas.

Figure 4 shows the collective distribution of findings for the 19 assessments conducted to date. The number to the right of the bar indicates the number of findings mapped to the issue area.


Figure 4: Distribution of findings
(Click on image above to show full-size version in pop-up window.)

Some of the technical process findings seen in almost every assessment are in the area of requirements management (traceability, definition, and volatility) and testing (inadequate functionality testing, incomplete test plans, and requirements that are not testable). Test findings also related to the technical product issue area as well. Without adequate testing, a product's goodness (e.g. completeness, supportability, reliability, quality) remains largely unknown.

In the management issue area, programs suffer from a lack of communication among the acquirer, developer management, and development engineers. Furthermore, decision-making roles, relationships, and responsibilities are not well understood by team members.

The findings-issues mapping is the foundation for analyzing and understanding the systemic issues that impact program success across the services. The initiative will continue to analyze the systemic issues to better understand cause-and-effect relationships between development and acquisition issues. By doing so, the initiative can continue to more precisely formulate program-level and enterprise-level recommendations that positively impact program success as well as acquisition policy, education, and training.

What is the difference between these assessments and software process assessments like the Capability Maturity Model® (CMM)? Whereas CMM evaluations focus more on the capability of an organization to establish and follow processes, the goal of an initiative assessment is to determine the actual performance and health of a program, and to make specific recommendations that will allow the program to achieve its planned goals and milestones. While initiative assessment recommendations may fall in the area of software process improvement, they can also focus on systems integration, acquisition strategy, resource management, and technology. Assessment teams can use the existing process assessment tools like those related to the CMM to assess a program's process capability if that is the focus.

Another difference is in the timing and the focus of the assessments on the specific program needs. Col. Patrick O'Reilly, PM of Theater High Altitude Area Defense (THAAD) commented, "Not only was the software assessment team's assessment critical to augmenting the IV&V contractor's and project office's risk assessment, the timing of the [assessment] was critical. The assessment provided insight and advice that was incorporated into the THAAD engineering, manufacturing, and development phase request for proposals resulting in a much lower risk contract proposal."

In consonance with Dr. Etter's goal to establish a collaborative environment, the initiative is continually strengthening its partnership with organizations and agencies such as the Software Technology Support Center (STSC) and the Defense Systems Management College (DSMC). These and similar organizations can benefit from the initiative as well as provide support, expertise and a venue to convey systemic lessons learned. The STSC is providing support through expertise on the program assessment teams, and assistance in the lessons learned analysis. The initiative is partnering with DSMC to insert future lessons learned into DoD-level acquisition education. DSMC also plans to support the initiative by making tools available to assessment teams and by participating on assessment teams. To provide the initiative a way to continually update its assessment model, the initiative plans to use DSMC's Executive Program Managers Course as a vehicle to identify problems and issues encountered in the acquisition community.

Similarly, collaboration with industry has expanded; one area is based upon prime contractors expressing interest in participating on assessment teams. Assessment teams normally consist of a team leader and members with appropriate skill mixes for the particular program, including domain experience and a PM representative. The initiative will experiment with augmenting the assessment teams with a member of the prime contractor team(s). Contractor representatives would be selected not from the specific development site, but from a different site or corporate element. These important and experienced representatives have a vested interest in program success and may spearhead implementation of team recommendations. They can also provide another perspective to the assessment process that helps to balance the information being obtained and evaluated.

If you are interested in requesting an assessment or providing your expertise on the teams, please contact us.

1. The project specific issue area does not appear in the figure because all assessment findings to date have been adequately covered by the other nine issue areas. Clearly technical process, technical product, and management are the three leading issues, with resources and schedule following closely behind.

Kristen J. Baldwin manages the Tri-Service Assessment Initiative within the office of the Deputy Under Secretary of Defense for Science and Technology, Directorate for Software Intensive Systems. Prior to this assignment, she spent two years as a science and technology advisor for the Army's Office of the Deputy Chief of Staff for Operations and Plans. Baldwin began her government career as a program engineer for the Army's Armament Research, Development and Engineering Center, Picatinny Arsenal, N.J. She has a bachelor's degree in mechanical engineering from Virginia Tech University and a master's degree in systems management from Florida Tech University.

1931 Jefferson Davis Highway
Crystal Mall 3, Suite 104
Arlington,Va. 22202
Phone: 703-602-0851
Fax: 703-602-3560
E-mail: baldwikj@acq.osd.mil

Laura Dwinnell supports the initiative's strategic and tactical implementation goals. She analyzes systemic issues, maintains the lessons learned database, and is a contributing author to the Assessment Process and Information Models. Dwinnell, a Litton PRC employee since 1989, received a master's degree in operations research and management science and a bachelor's degree in mathematics from George Mason University.

PRC Inc.
1500 PRC Drive, MS 2N3
McLean,Va. 22102-5050
Phone: 703-883-8707
Fax: 703-883-8700
E-mail: dwinnell_laura@prc.com

® The Capability Maturity Model is a registered trademark of the Software Engineering Institute and Carnegie Mellon University.