Wouldn't it make sense to have a way for government program offices to determine the maturity of a contractor's software development process without incurring the cost and time to conduct a total software capability evaluation? Wouldn't it be efficient to have a way to eliminate redundant reviews of contractor software development processes by different government offices?

Well, now there is a way to obtain this data. Just simply ask. While not fully operational until next year, the capability to provide such data is currently in place at almost half of the field locations within the Defense Contract Management Agency (DCMA).

As the on-site government representatives at contractor facilities, DCMA provides assistance to all branches of the military. Its scope of effort is defined within the Federal Acquisition Regulation (FAR). A complete description of DCMA capabilities was previously described in CrossTalk [1]. They include the evaluation and surveillance of contractor management systems such as the processes used in software development [2]. For this, the agency has adopted use of the Software Engineering Institute's (SEI) Capability Maturity Model^® for Software (SW-CMM).

The SW-CMM is the language we needed to speak, and speak fluently, to communicate with the broad range of customers across the Department of Defense (DoD). It is the language spoken by government program offices when conducting software capability evaluations for source selections or lesser reviews. It is the language selected by the DoD to reduce risk on acquisitions [3]. It is the language employed by contractors when conducting a CMM-based appraisal for internal process improvement (CBA IPI).

Our initiative to speak this common language, what we call CMM-based insight, is simple in concept. Taking advantage of DCMA's in-plant presence, we will primarily organize daily observations into findings based on the CMM. Observations undergo an internal peer review for conformity to the CMM, then data is freely shared with the applicable contractor and passed to program offices. Findings will be used to concentrate DCMA effort based on risk. Details concerning the process, responsibilities, and outcomes are captured in the Method Description Document (MDD), which is available on line at www.dcma.mil/onebook/4.0/4.3/initiatives.htm.

The goals (see Table 1) directly benefit program offices, contractors, and the DoD. Regardless of DCMA location, program offices will have consistent data concerning a contractor's software process maturity for programs within DCMA cognizance. Since data is freely shared with the contractor, concern or disagreement on high-risk areas can be resolved at the working level, or elevated as necessary to the DCMA/Contractor/Program Office Management Council [4]. The data can be used in future process reviews to reduce or eliminate redundant areas. The results from this continuous review could also be used as a vehicle to ensure that contractors have maintained a process capability level per DoD policy [5] or in support of independent expert program reviews of software intensive systems [6].


Table 1: CMM-Based Insight Goals
(Click on image above to show full-size version in pop-up window.)

CMM-based insight is not a software capability evaluation or a CBA IPI (see Table 2). While data could be used to substantiate another evaluation, DCMA will never rate a particular company through CMM-based insight. The initiative is focused on identifying areas of concern on individual programs (i.e., higher risk process areas) and allocating the appropriate level of resources commensurate with that risk.


Table 2: Comparison of Evaluation
(Click on image above to show full-size version in pop-up window.)

As previously discussed, the initiative is simple in concept. But like the process of teaching an adult to speak (and think) in a new language, making this transition has involved a culture change in DCMA software surveillance activities. As such, incremental phases (see Table 3) were designed to assist the transition.


Table 3: CMM Based Insight Implementation
(Click on image above to show full-size version in pop-up window.)

Phase I validated the approach at the home locations of our agency Software Engineering Institute (SEI) affiliates. Phase II verified that approach for suitability and effectiveness in a typical field environment. Phase III will verify the capture and transmission of data before the initiative is implemented agency wide.

The primary purpose of Phase II was to verify the approach. Due to the shear number of inputs -- necessary for the correlation of observations to the applicable key practices, internal peer reviews, identification, and subsequent action on high risk areas -- the need for an adequate support tool was recognized early. (This situation will be resolved in Phase III when data collection is incorporated into the common tool supporting the entire DCMA Risk Assessment Management Program.) Despite this burdensome data collection, currently 45 percent of our field locations have volunteered as pilot locations and converted operations. Why? It is because of the benefits realized. These are perhaps best illustrated with actual examples.

A program office concerned with a history of poor software quality wanted the contractor to operate at CMM Level 3. The contracting company's upper management believed the company was well within these parameters and retained an outside consultant to verify this position. Initial results indicated the contractor was operating at CMM Level 3. The DCMA field office disagreed, however, based upon observations and findings per the CMM.

Working with the program office, the findings were questioned and the issue elevated to upper management. The program office held that if the review revealed a significantly different result than that observed in day-to-day operations, the government would sponsor an independent software capability evaluation. If the government evaluation revealed the contractor was more interested in paper ratings than software quality improvement, the government would consider developing a second source for the procurement.

What was the end result? The final evaluation revealed operations at CMM Level 1. The contractor was well on the way to Level 2, but far from the desired Level 3 target profile. Was this a typical contractor/government confrontation? Quite the opposite, it fostered a spirit of process improvement. For the first time, there was an accurate and understood baseline. The contractor developed a roadmap for process maturity and during the course of two years, achieved the desired Level 3 profile. DCMA, the government on-site representative, participated in the mini reviews and was a team member on the final contractor-conducted CBA IPI.

correlation between capability (maturity level) and actual performance (cost, schedule, and technical) [7] is accepted, it would seem reasonable to assume that there is less government surveillance of higher maturity operations than those with lower maturity. In the absence of data, however, people often focus on those areas where they are comfortable. Consequently, a low-risk area might get as much attention as a high-risk area.

This is not so with the CMM-based insight methodology because it is based on data and focuses expended effort in proportion to risk. This is the case at one of our pilot locations where the contractor has achieved CMM Level 5. The CMM-based insight data will be used to ensure that DCMA effort and resources are allocated to the areas of highest risk.

CMM rating truly representative of all programs at a given facility? As the name states, the model measures a capability. It would seem logical to assume that if a capability has been demonstrated on one program, that it has been applied to all. With mandated levels, though, there are other pressures that come into play. At one pilot location the contractor had conducted a CBA IPI that resulted in a finding of CMM Level 3. The contractor had selected programs across the business base and then hung a banner over the building entrance saying "CMM Level 3 Certified." What was wrong with that?

First, the rating Level 3 certified is confusing and misleading. Certified by whom? Secondly, the review did not include the largest program, one which had been experiencing problems at the international level. While the CBA IPI shows a company's capability to operate at a given level, it is not necessarily true for all programs. It should be, and seems to be in most cases, especially when the focus is on process improvement. However in this particular case, it was not. With the DCMA data, the banner was removed and the applicable program office understood that operations on their program were not at CMM Level 3, and why.

Duplicative Reviews: Concerned about software quality, a joint program office planned a review of the contractor's software development processes. The DCMA pilot location, a front-runner for this initiative, already had the data in the common language of the CMM. It clearly identified strengths and weaknesses. The review was cancelled and the DCMA data was used in follow-on actions with the contractor. This is only one example, but the dollar savings across the department quickly add up. The cost to conduct a software capability evaluation has been estimated at $50,000 for both the government and contractor [8].

A complete process is defined as having 1) procedures and methods for defining the relationship of tasks, 2) tools and equipment, and 3) people with skills, training, and motivation [9]. The first two elements have already been addressed. Concerning people, the agency has more than 400 personnel supporting software quality assurance. To assure this workforce is properly prepared to deliver consistently first-rate assessments, we have instituted a multi-phase development program:

• Basic Training: The agency's formal training is called the DCMA Software Professional Development Program. Individuals proceed through two training levels. Level one requires completing 72 hours of computer-based training, 40 hours of classroom instruction, and a formal mentoring program focused on practical application of course material. Level 2 requires an additional 97 hours of computer-based training, 120 hours of classroom instruction, and further mentoring. The SEI's CMM is integrated into the computer-based training, classroom instruction, and mentoring. Currently 78 percent of agency software personnel have obtained Level 2 status. To maintain this level, individuals must complete a minimum of 12 hours of software-related training each year.
• Application Training: As each field location begins operating under the CMM-based insight initiative, all personnel undergo an additional 20 hours of specific application training conducted on site by the DCMA Software Center. Applicable contractors and government program offices have been welcomed into this training. It is specifically focused on the initiative's implementation and daily operation. The material was developed by the DCMA Software Center and is currently being reviewed by the SEI.
• On Call Assistance: DCMA personnel have direct access to the six-person DCMA Software Center. In addition, one-eighth of the total field workforce has completed the SEI's Software Capability Evaluation training. Additional assistance is available to any of our evaluators from highly qualified agency personnel who are SEI-certified lead assessors.
• Implementation Measurement: Training provides a foundation for conducting business per the CMM, but it does not directly correlate to experience, which can only come with time. Progress in implementing this initiative has been promising. For instance, more and more of our personnel have been requested as team members by companies when conducting CBA IPIs. However, to gauge implementation progress for this initiative across the entire agency and to make necessary adjustments, the agency is employing a top-level metric based on key process area coverage. Progress will be reviewed by the agency director, his senior leadership team, and DCMA field commanders.

The baseline for our efforts is the SW-CMM. We fully expect, and are making preparations, to switch over to the Capability Maturity Model Integrated^SM (CMMI^SM) at a later date. The agency is part of the SEI-led CMMI Steering Group responsible for developing the SW-CMM/CMMI turnover within the DoD [10]. For CMM-based insight, the transition should incur little breakage moving to the integrated model. The biggest challenge in using either model is the discipline and knowledge of application -- both of which we are gaining with our current effort and are fully transferable. Field sites coming aboard in each phase are shown in Table 4.


Table 4: Pilot Locations
(Click on image above to show full-size version in pop-up window.)

A past issue of CrossTalk raised the point, "A Level 3 development effort coupled with a Level 1 acquiring effort often equates to a Level 1 delivery capability; yet the Level 3 developer is often blamed, and the Software (SW) CMM is cited as inadequate[11]." I saw this firsthand, with disastrous results, as a junior officer. So how does DCMA measure up?

To answer that question, we took the sister capability maturity model, the Software Acquisition CMM, and tailored it for DCMA use. We pilot tested and made adjustments as applicable. We then went agency wide, conducting reviews from November 1999 until April 2000. Eight equally qualified teams were used to maintain consistency. What were the results? There were a few organizations operating at the defined level but predominately the field offices within the agency operate at the performed level (Level 1).

More importantly, we established a solid baseline and each location has a detailed roadmap for improvement per the model structure. Field locations have been working improvements and the first round of follow-on appraisals began in the spring of 2001. The original evaluation team members constitute the personnel pool to support independent evaluation of improvements similar to the industry approach with a CBA IPI.

DCMA was always required and continues to conduct evaluations of contractor's software development processes per the FAR. The agency is now deploying a standard methodology via continuous process evaluations that is organized in the CMM, the common DoD language, and is based on the day-to-day observations of the inplant DCMA personnel. Findings are peer-reviewed, and all data is freely shared with the applicable contractor and is available to government program offices.

While full agency implementation will not occur until spring 2002, the approach has been developed with SEI affiliates and is currently in pilot testing at 45 percent of DCMA field locations. Program offices, the contractors, and the DoD are already realizing benefits. So, how much does the agency believe in using this approach to gauge contractor operations? Enough so that we are walking the walk and measuring our operations to the same framework.

1. Holt, Kevin E., Software Acquisition Support in the Defense Contract Management Command, CrossTalk, March 1997.
2. Federal Acquisition Regulation, Part 42, Section 302(a)(41).
3. USD(AT&L) Memorandum, Software Evaluations for ACAT I Programs, Oct. 26, 1999.
4. Malishenko, Timothy, Management Councils Emerge as Valuable Asset in the Program Manager's Tool Kit, Program Manager Magazine, March/April 1999.
5. USD(AT&L) Memorandum, Software Evaluations for ACAT I Programs, Oct. 26, 1999.
6. USD(AT&L) Memorandum, Independent Expert Program Reviews of Software Intensive System Acquisition, Dec. 21, 2000.
7. Lawlis, Dr. Patricia K.; Flowe, Capt. Robert M.; Thordahl, Capt. James B.; A Correlation Study of the CMM and Software Development Performance, CrossTalk, September 1995.
8. SCE Reuse: Ending Redundant Reviews, Acquisition Reform Today, Vol 3, No.1, January/February 1998.
9. Software Engineering Institute, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley Publishing Company, 1994, pg. 9.
10. Deputy Under Secretary of Defense (S&T) letter dated Dec. 11, 2000, Use of CMMI Evaluations by the Department of Defense.
11. Jarzombek, Lt. Col. Joe, Integrating Acquisition with Software and Systems Engineering, CrossTalk, August 1999.

Lt. Col. Robert Lang currently serves as the director of the Defense Contract Management Agency Software Center. He has 20 years of active duty service in the U.S. Air Force in various acquisition specialties. His previous assignment was program manager for the Iceland Air Defense System. He has a bachelor's degree in engineering technology from Norwich University, a master's degree in engineering management from Western New England College, and is a graduate of the Defense Systems Management College, Advanced Program Management Course.

DCMAC-G
495 Summer Street
Boston, MA 02210-2184
Phone: (617) 753-3739
E-mail: rlang@dcmde.dcma.mil

®The Capability Maturity Model and CMM are registered in the U.S. Patent and Trademark Office.