The Special Operations Forces Systems Program Office (SOF SPO) at Warner Robins Air Logistics Center (WR-ALC), Robins Air Force Base, Ga., provides combat weapon systems, equipment, and agile combat support for special operations and Air Force helicopter forces. We deliver best value sustainment and contingency response through world-class cradle-to-grave leadership and management.

The SOF SPO has primary responsibility for systems engineering and technical services to the SOF Fleets and Combat Search and Rescue, consisting of AC-130H/U gunships, MC-130E/H/P Combat Talon/Shadow aircraft, MH-53 J/M Pave Low helicopters, H-1/H-60 helicopters, and CV-22 Osprey tilt-rotor aircraft.

We recognized that in order to make improvements, we needed a model on which to pattern processes. We settled on the Software Acquisition-Capability Maturity Model^® (CMM^®) (SA-CMM) as a model that would provide the ability to assess our processes. The SA-CMM was structured, contained specific goals, established levels of competence, and provided the framework needed to facilitate improvement. Using the SA-CMM, the SOF SPO established an improvement infrastructure, developed a lifecycle checklist of its directorate's processes, and started improvement efforts.

The SOF SPO improvement program, known as the LU Acquisition and Sustainment Process Improvement/Reengineering Effort or ASPIRE, made many improvements during the next three years. Our successes were of such a nature that other directorates at Robins AFB have adopted our process guides for their own in-house processes. There was, however, one tiny barrier to complete satisfaction with the program. It was the use of the SA-CMM as a model for improvement.

Software development and management in relationship to acquisition and sustainment is a very small part of what SOF SPO does. Since the SA-CMM referred to primarily software acquisition and development, it was initially viewed as a turn-off to the LU work-force. They wanted something more related to what they did. In the course of due time, it even dropped all mention of software from its process improvement activities.

The work-force simply did not fit the model, or more accurately, the model did not fit it. So it continued to search for a model on which to base its program, but did not abandon the SA-CMM in the meantime.

In the fall of 2000, Dr. Thomas Christian, chief engineer, and Greg Stanley, deputy director, attended a conference held in Washington, D.C. During the conference, SEI presented, in draft form, a new model: Capability Maturity Model Integration^SM/Systems Engineering/Software Engineering/Integrated Product and Process Development/Acquisition (hereafter referred to as CMMI-A). This model dealt with processes within an acquisition organization. Now here was something the SOF SPO could sink its teeth into. It maintained the proven structure of earlier CMM models, but also contained process areas (PAs) on acquisition. There were, of course, several questions that needed to be answered about the model.

When the CMMI-A was unveiled, not only did the model address acquisition but it also came in two versions: staged and continuous. This was different! The staged version provides a framework to identify improvement opportunities and a set of goals to guide process improvement. The continuous model groups processes into categories and designates capability levels for each process.

In other words, in the staged representation, the whole organization is appraised and receives a maturity level based on an appraisal of all of the PAs contained in the staged representation. In the continuous model each PA is appraised and assigned a capability level. There is no overall level assigned to the organization. What is the most significant factor here? The organization examines the continuous representation, selects the PAs that either apply to the organization or are PAs the organization wants to improve upon, and uses only the selected PAs during appraisals.

So here we have the old SA-CMM model and the new CMMI-A model with two different representations -- staged and continuous. What do we do?

Dr. Christian and Stanley returned to Robins to brief me and my staff on the new model. We discussed at length our partnership with SEI, the advantages and disadvantages of the SA-CMM and the CMMI-A, and the LU workforce's desire to find a model more suited to what we do. Many questions and answers were posed: "Has it been tested?" "No;" "How do we know it will work?" "We don't;" and "Has anyone been appraised using the model?" "No."

In the end, we decided that even with these negative responses, the CMMI-A with continuous representation was worth our time and effort as a model for process improvement. Finally, we had a model that fit us; we did not have to try to fit the model.

During their Washington, D.C. trip, Dr. Christian and Stanley discussed the new model with Dr. Jack Ferguson, deputy Acquisition Resources and Analysis director for Software Intensive Systems, Office of the Secretary of Defense, regarding adopting and implementing the model. They pointed out that it had not been tested or implemented anywhere else. Dr. Ferguson asked them to consider using LU as a pilot appraisal organization to test the new model. He would secure the necessary funding to conduct the appraisal. We agreed, and dates were established. We were now on the road to the CMMI-A implementation in the SOF SPO.

We developed three immediate objectives concerning implementing the CMMI-A in LU. First, we wanted training on the new model. Second, we needed to determine the specific PAs on which to be appraised. Third, we wanted a balanced team of appraisers.

The first objective, training, was easy enough. We contracted with SEI to provide training for 30 LU personnel in February 2001. Participating in the training were most of the personnel selected to be on the appraisal team. Training covered the two model representations and the process areas within each; however, the three-day session was only able to present the basics of the model. Training feedback primarily concerned the depth of training they were able to present in three days. While the instructors were very professional and knowledgeable, it was very difficult to come to a level of familiarization with the model in only three days. All agreed that training needed to be longer and more in-depth.

We devoted a lot of effort to ensuring the composition of the appraisal team was as balanced as we could make it. Our considerations in putting the appraisal team together centered on three central thoughts. First, we needed people experienced in conducting appraisals and using previous CMM models. Second, we wanted to make sure the team consisted of people experienced in acquisition and sustainment activities. The latter proved invaluable in interpreting the wording of the model into LU activities. Finally, we needed to make sure we had a team composed of personnel from both external and internal sources.

We were able to achieve all three of these objectives. The team was composed of five LU personnel, three other WRALC personnel who had appraisal experience, one representative from SEI, and an experienced appraiser from the Software Technology Support Center, Hill Air Force Base, Utah. We felt this personnel combination provided the expertise and objectivity needed to conduct a thorough appraisal. After establishing the team, our lead appraiser began a series of training sessions designed to teach our functional experts how to conduct an appraisal.

In examining the CMMI-A with continuous representation, we needed to decide which PAs to appraise. In order to accomplish our objectives and the need to "test" the model, we settled on 17 different PAs, see Table 1. These included all the PAs we used in our initial appraisal back in 1997 and added several new PAs. The groundwork for conducting the appraisal was now completed.


Table 1: Process Areas

Conducting the appraisal required a great deal of time and effort. The preliminary step was to survey the work-force. The team then gathered all pertinent data and documentation from the "projects" being appraised. The team then briefed us on how they would conduct the appraisal and developed a series of questions to use in the personnel interviews. Interviews were conducted over a series of days and provided the bulk of information used in determining our capability levels. Their final step was to correlate all the available information and assign a capability rating to each PA.

The team interviewed 47 people, reviewed more than 112 documents, and worked 130 hours during two weeks. The team examined every detail of the selected process areas. Using a team decision process, the results for LU were extremely gratifying. Sixteen of the 17 PAs received a capability rating of Level 2 "A Managed Process." One PA, Organizational Process Focus, was rated Level 3 "Defined Process." This was quite a remarkable achievement for an organization that four years earlier had no documented processes at all and had not used the model to develop its technical activities of systems engineering and configuration management.

Especially remarkable was the Level 2 rating given three Integrated Product and Process Development (IPPD) areas: Integrated Teaming, Integrated Project Planning, and Integrated Supplier Management. We had not been able to plan and implement procedures and processes for the IPPDs due to the fact they were not fully defined by SEI until shortly before the appraisal. These ratings demonstrated we had intuitively recognized and were already in conformance with the standards established for the IPPDs at the time they were being developed by SEI.

Based on these results, we developed an action plan to address weaknesses in our processes. Our primary focus will be to establish a measurement and analysis program in order to move us to the next capability level for our selected PAs. Our commitment to process improvement has only been strengthened by our conversion to the CMMI-A.

Every decision we make, regardless of the subject, teaches us something. It either reinforces our thought process as being correct, or it shows us where we didn't think things through. Converting to the CMMI-A was the correct decision, but we did learn a few things along the way: some about our program, some about the model, and some about the appraisal process.

We learned that training on the model should be spread out and more in-depth. Our personnel on the appraisal team and the work-force could have used additional training, which would have eased our transition. Feedback from members of the appraisal team leads us to believe had they had a better understanding of the model, we would have received higher capability ratings in several more PAs.

We learned that reviewing organizational documentation should begin as soon as possible. We waited until the last minute to send out our survey. It was quite extensive and required time to respond. Unfortunately, our delay caused recipients to feel pressured to respond quickly, so most ignored it. It also gave us very little time to analyze the results.

We also realized the scope of the appraisal was much too great for the amount of time and number of interviews needed to cover the PAs. The team reviewed more than 112 documents. With all the other things the team needed to accomplish, this was too much to review at a moment's notice. We found examination of 17 different PAs too much to accomplish within a two-week period. We recommend you restrict yourselves to the more important PAs (probably around 10) or make a decision to appraise some now, some later. Careful selection of the PAs would result in a more reasonable effort.

Lastly, while the appraisal methodology calls for the appraisers to construct questions that solicit data about your processes without being direct, our interviewees found this to be both frustrating and confusing. If you decide to follow the current interview methodology, you will need to brief the interviewees that you intend to ask general questions and will be looking for specifics based on their answers. Constructing questions concerning "how, why, and where" may be a more efficient method.

The SOF SPO has a long and productive relationship with process improvement. From senior leadership down to individual members of our Process Action Teams, we have developed an appreciation of making things better. Our association with the CMM has only enhanced this appreciation. The CMMI-A is our guide to the future. We are convinced our involvement with the CMMI-A is a key factor in why we have developed a "passion for excellence."

Donald R. Michels, GS-15, is the director of Special Operations Forces Systems Program Office at Warner Robins Air Logistics Center, Robins Air Force Base, Ga. He is the U.S. Air Force's single manager for fleet management of Air Force Special Operations Command fixed wing aircraft and all U.S. Air Force helicopters. He directs more than 420 personnel at Robins Air Force Base, Wright-Patterson Air Force Base, Ohio, and other operating locations. He also directs oversight of more than 20 contractor support agencies. Michels has a bachelor's degree in management from Metropolitan State College, Denver, Co., a master's degree in logistics systems from Georgia College and State University, Milledgeville, Ga., and is a graduate of the Air War College, Maxwell AFB, Ala. He is also a graduate of the Defense Systems Management College's Program Management Course and holds Acquisition Professional Development Program Level 3 Certificates in both Program Management and Acquisition Logistics.

Bonnie Bollinger is a former GS-12, software engineer, and was assigned to the Avionics Directorate at Robins Air Force Base, Ga., prior to her retirement from federal service in 2001. The Software Engineering Institute, Carnegie Mellon College, Pittsburgh, Penn., has certified her as a lead assessor. Bollinger is currently a private consultant to SEI and government agencies.

Point of Contact: Stephen D. Acuff
226 Cochran Street,
Robins AFB, GA 31098
Phone: (478)926-6160
Fax: (478)926-4911
E-mail: stephen.acuff@robins.af.mil