This article provides a methodology for writing an IV&V plan based on the Institute of Electrical and Electronics Engineers (IEEE) standards and updated as a result of three U.S. Navy project iterations. The plan's ideas and format are the result of these iterations, and they incorporate CMM concepts and reduced document maintenance costs. The comprehensive plan covers the full spectrum of IV&V project involvement but can easily be tailored to any level of IV&V effort.

In the current software development environment with automated tools, reduced budgets, and CMM, the task of writing an effective IV&V plan requires a well planned and open-minded approach. The current guidance and documentation for the preparation of IV&V plans, such as [1] and [2], are vague and should incorporate current topics such as CMM.

Today, companies bidding on software development contracts use terms such as CMM Level 3, 4 and 5 in their proposals. Sometimes several organizations within the corporate structure have not achieved a Level 3 or higher, but the company bids the corporate or division CMM level. Since a company bids itself as a CMM Level 3 or higher, the group letting the contract may assume all organizations bid within the company are at the proposal's level and thus no IV&V effort is needed. In some cases this assumption is true but many projects have fallen into this pitfall.

Instances where the assumption is false create a situation where the "children are watching themselves," and there is a need for an IV&V effort to watchdog the software developer. Sometimes organizations are downsized to a level where they cannot oversee the software development and may require IV&V support. The decision whether to have IV&V support should be made at or as close to project start-up as possible. Generally the IV&V support is begun at the requirements definition phase.

When the decision is to perform IV&V, a plan is required. The IV&V organization may not have attained a CMM Level 2 or higher rating. Level 2 is defined as "repeatable," which indicates there are some processes that are followed for each effort. Level 3 is defined as "defined," indicating the team has standard processes to follow and a documentation library. The IV&V effort may be tasked to monitor the software developer's compliance with their standard software development processes.

This article provides tips, ideas, and a framework developed while producing master IV&V plans on three U.S. Navy projects. The basic structure provided by the IEEE standards [1] and [4] for verification and validation plans was modified and updated to produce the framework for this IV&V plan.

Many IV&V plans omit an important statement: "This plan is a living document." IV&V plans are dynamic and must be updated as the software project matures. Thus any IV&V plan must be flexible and provide means for implementing changes.

Another myth many have fallen into is that IV&V is synonymous with testing. Although testing is a very important activity within IV&V, it is not necessarily restricted to this activity. IV&V methodology is also important to requirements and design phases of the software development life cycle and software maintenence.

First several definitions from IEEE [3] are needed:

IV&V is defined as "performed by an organization that is technically, managerially, and financially independent of the development organization." The organization is usually external to the software development company.

Verification and validation is defined as "the process of determining whether the requirements for a system or component are complete and correct, the products of each development phase fulfill the requirements or conditions imposed by the previous phase, and the final system or component complies with specified requirements."

Verification is defined as "the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase, providing formal proof of program correctness."

Validation is defined as "the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements."

With these definitions in mind, we will now approach the structure of the IV&V plan. An assumption is made that plan development will be performed on a PC in a Windows environment. All files associated with the document and appendices are stored in the same Windows folder or contain internet/intranet access to all external documentation.

This section contains four general topics. Topics 1 and 4 are the same as Section 1 in IEEE standards [1] and [4]. Topics 2 and 3 have been added to ensure everyone understands the objectives, goals, and approach. Topics 2 and 3 must be completed before any subsequent parts of the plan can be written. These two topics drive the development of the plan.
1. Purpose-This topic answers the question, "Why is this plan being written?" In some documents it is referred to as Scope.
2. IV&V Objectives and Goals-This topic lists the main objectives and goals for the IV&V effort. These form the basis for performing the IV&V.
3. IV&V Approach-This topic describes the high-level methodology and how it fits in with the development cycle. This material is dependent on the CMM level of both the software developer and the IV&V agent. This approach is further decomposed in Section 4.
4. System or Project Background-This topic provides a general understanding of the project under study.

This section is the same as Section 2 in IEEE standards [1] and [4]. Within this section are references to all associated documents, processes, and plans that provide information to complement this plan. Whenever possible, hyperlinks to the document locations should be included. If the documents are stored in a CMM Level 3 program library this is a simple effort. Using hyperlinks, documentation revisions and dates will not need to be maintained in the plan. Maintenance cost savings will be realized since the information is not maintained as part of the IV&V plan, thus changes in the external documentation remain at the external documentation level, i.e. program library.

This section discusses the project organization, schedules, resource allocation and tools, techniques, and methodologies. Generally, this section follows Section 4 in IEEE standards [1] and [4], except for the Work Breakdown Structure (WBS) addition to Topic 1 and the use of the appendices.
1. Organization-This topic describes the project organization providing the roles and responsibilities of all participants. A WBS or similar organizational breakdown is included. We recommend placing the WBS in an appendix as described later. In addition, this topic describes the use of integrated product teams or other means to coordinate the workload.
2. Schedule-This topic discusses the project management plan and the IV&V master schedule, which must be strongly coupled to the project plan. Since all schedules are constantly changing, we recommend placing the actual schedules in an appendix as described later.
3. Resources Summary-This topic summarizes the resources required to perform the IV&V tasks, including the personnel comprising the IV&V team and their technical experience requirements, test support equipment descriptions, and test site facilities. It is recommended that the specific personnel resources allocations be placed in the appendices as described later to allow changes to be made independent of the IV&V plan.
4. Tools, Techniques and Methodologies -This topic discusses access to data and facilities and the automated tools, techniques, and methodologies that will be required to perform the IV&V tasks.

Section 5 in IEEE standards [1] and [4] describes the IV&V effort in a chronological order. The government is moving toward the Earned Value Management System (EVMS) on new Department of Defense projects and contracts as its method of tracking contract progress and cost. To prepare for this, we have found a WBS-like organizational approach to be the easiest to implement. EVMS practically demands this approach. For this reason, this section describes the five integrated IV&V activities that perform the IV&V tasks: management, requirements analysis, design analysis, independent testing and analysis, and process improvement. Figure 1 shows how these activities are integrated.


Figure 1: IV & V activities
(Click on image above to show full-size version in pop-up window.)

Each activity contains paragraphs for inputs, descriptions of the activity's specific tasks, metrics, outputs, and resources needed. Metrics are included for each activity and are reported to the IV&V management activity, which evaluates the metrics and generates progress and cost reports to the project manager. Some examples of the IV&V metrics can be found in Annex E of reference [4].
1. IV&V Management-This activity is the focal point for all inputs and outputs to and from the IV&V team. Issues identified in the course of analysis activities are reported out via this activity. All documentation and project data are received by this activity and flow to the appropriate analysis activity for processing. IV&V management can be broken down into planning tasks that include generation and maintenance of this plan, directing tasks, reporting tasks, and controlling tasks.
2. Requirements Analysis-This activity verifies that system, software performance, and interface requirements have been prepared in accordance to a standard set of criteria. If the software developer is a CMM Level 3 or higher, this activity also verifies that requirements are prepared in accordance to the standardized processes. The criteria set should be defined in this plan unless a lower level requirements analysis plan will be written. There are three main task areas in this activity: documentation analysis, interface requirements analysis, and requirements traceability analysis. System and software performance requirements are analyzed in the first area.
3. Design Analysis-This activity verifies that the software design and implementation phases of the software life cycle are performed in accordance with the software development plan. The task areas comprising this activity include design analysis planning, design products analysis, interface design analysis, code analysis, and design traceability analysis. The interface design analysis may be combined with the interface requirements analysis if a single interface design specification is provided as an input to IV&V. The design traceability focuses on the traceability of the performance and interface requirements derived during the requirements analysis activity to the design documentation and code implementation.
4. Independent Testing and Analysis- This activity reviews the test artifacts, develops the IV&V test plan, evaluates the testing process, and integrates the independent testing tasks. The basic task areas are independent testing and analysis planning, which includes the test plan generation; independent testing and analysis, which includes general descriptions of the types of IV&V tests to be performed as described in the IEEE standards; and independent testing and analysis reporting, which describes the processes to report testing results. This activity also coordinates with the project test and evaluation (T&E) team to reduce test redundancy and performs the validation tests assigned by the T&E manager.
5. IV&V Process Improvement-This activity is not included in the IEEE standards and supports CMM Level 3. It receives recommendations from the other activities for new analysis processes or changes to the current IV&V methodologies. Included are descriptions of how new tools, ideas, and concepts will be evaluated for incorporation into the IV&V processes. Key task areas are concept assessment, tool qualification, and reporting.

This section is similar to Sections 6 and 8 of the IEEE standards [1] and [4] and describes the various reports and artifacts produced during the IV&V effort. Outputs from the activities described in Section 4 should be categorized as either required or optional reports. Figure 2 shows the flow-down of the products.


Figure 2: IV & V Products flow
(Click on image above to show full-size version in pop-up window.)

This section tracks closely with Section 7 of the IEEE standards [1] and [4]. It discusses the general administrative topics such as anomaly reporting and resolution, task iteration policy, process deviation policy, and control procedures. The first two topics cover procedures for reporting issues, defects, etc. found during the analysis activities, along with tracking the issues through either fix implementation or deferred or cancel status. Process deviation policy describes the procedure to be followed to deviate from the IV&V master plan. One statement that must be in the process deviation policy is, "Any such deviations will be documented and approved before they are allowed to occur." The control procedures include the IV&V products and data configuration management, quality assurance, protection and security, and storage procedures.

Since the IV&V plan is dynamic, those references that will undergo continuous modification and result in potential risk areas should be placed in the appendices using hyperlinks to the actual referenced data, preferably located in the same Windows folder.

Examples of these references are the project master schedule, IV&V schedule, and work breakdown structure and personnel resource allocations.

This concept is new to the IEEE standards and is used for risk mitigation during development and maintenance of the IV&V plan, reducing maintenance costs.

Risk events that can cause major changes to the IV&V plan are development schedule slippages, test equipment and simulation software development delays, and test and delivery site equipment deliveries and/or checkout delays.

Another major risk mitigated by this concept is the loss of key personnel. By hyperlinking to the associated data, the IV&V plan remains current without additional maintenance when a project change occurs, and without the need to modify the IV&V plan each time a change occurs.

Appendix A contains the glossary with all acronyms and major terms defined. The IEEE standards include this as Section 3 but since this list is always changing, a better location is Appendix A. This way a list of the project's acronyms and major terms may be updated and a new Appendix A distributed without a new release of the document.

While IEEE Std 1012-1998 provides a considerable amount of detail about what is involved in verification and validation, this article presents tips and ideas implemented with success during the development of three unique U. S. Navy IV&V efforts: TOMAHAWK, National Missile Defense, and Cooperative Engagement Capability (CEC). Each iteration provided new ideas culminating in the plan's structure described above.

The IV&V Plan for CEC added the IV&V process improvement activity and the concept of hyperlinked Section 2 and appendices. CEC schedules change several times each quarter, sometimes weekly. By hyperlinking to a project schedule file within the same folder, only the project schedule file is updated, and a new revision of the document is not required.

This plan encompasses a full spectrum of involvement in a software development life cycle from requirements definition through maintenance. Any IV&V project team can easily tailor this plan to meet their specific objectives and goals. If the IV&V effort begins at the acquisition phase, this plan can be adapted easily.

There are four very important lessons to be learned from this article:

1. If EVMS or a similar cost tool is required, follow a WBS-like approach.
2. An IV&V process improvement activity is needed to address the CMM level requirements. This activity is at the same level of importance as the other IV&V activities.
3. All items that will change continuously over the life cycle, should be placed in appendices that are hyperlinked to the project master files if possible, or copies in the same Windows folder as the IV&V plan.
4. Hyperlink all document references in Section 2 to project master files if possible to reduce redundant maintenance efforts.

1. IEEE Std 1012-1986, IEEE Standard for Software Verification and Validation Plans.
2. IEEE Std 1059-1993, IEEE Guide for Software Verification and Validation Plans, December 2, 1993.
3. IEEE Std 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology.
4. IEEE Std 1012-1998, IEEE Standard for Software Verification and Validation, March 9, 1998.
5. CMU/SEI-93-TR-25, CMM Practices.

The author thanks Robert Lewis and Rodney Tuten of the Naval Surface Warfare Center (Dahlgren, Va.), and John W. Guthrie, Thomas Blackwell, and Mark Sullivan of EG&G Technical Services.

Dan F. Walters is a senior systems engineer with EG&G Technical Services in Dahlgren, Va. He has a bachelor's degree in physics from SUNY at Brockport, NY, and is seeking a master's in software systems engineering at George Mason University in Fairfax, Va. He has worked on the AEGIS Display System, Shipboard Gridlock System, Vertical Launching System, and is currently working on the Cooperative Engagement Capability projects for the U.S. Navy.

EG&G Technical Services
16156 Dahlgren Road, P.O. Box 552
Dahlgren,Va. 22448-0552
Phone: 540-663-9475
Fax: 540-663-0332
E-mail: waltersd@egginc.com