The Software Engineering Institute (SEI) initially developed the Capability Maturity Model® for Software (SW-CMM®) [1] with the initial purpose of providing a map for improving software processes. The SW-CMM also provides a basis for assessing the maturity of an organization's software processes. Because of its success, other capability maturity models were developed. These include the following:

• A Software Acquisition CMM by the SEI [2].
• A testing capability maturity model [3].
• Several systems engineering capability maturity models [4, 5, 6].

Due to the growing variety of capability maturity models, the SEI developed a consolidated approach called the CMM Integration^SM (CMMI®) [7]. Capability maturity models have been a topic of many articles in CrossTalk. In this article, capability maturity model is used generically. When a specific capability maturity model is intended, it is identified explicitly.

Lean Six Sigma^TM (LSS) is a systemsengineering approach to defining, measuring, analyzing, and improving processes. LSS was initially developed for manufacturing, but has been successfully applied to all types of processes - including transactional processes, services, and software. A brief introduction to this topic is given in [8, 9].

It is assumed here that the reader has a reasonable familiarity with capability maturity models and has at least an introductory knowledge of LSS. The purpose of this article is to compare capability maturity models and LSS¹.

In the first section, key features of these two are contrasted. Having looked at their differences, the next section will focus on success factors. Lastly, two examples are presented in which industry has used Six Sigma in conjunction with capability maturity models. This article ends with some conclusions and recommendations.

The following sections compare various attributes of capability maturity models and LSS. These attributes include institutionalization, assessment approaches, focus, and measurement. The primary differences between capability maturity models and LSS derive from the fact that capability maturity models are models, whereas, LSS is a method.

Capability maturity models are models; they focus on what. The SEI's CMM specifies that policies, procedures, and guidelines be explicitly defined, including Key Process Areas (KPAs), goals for each KPA, and practices associated with each KPA. The CMM defines maturity in terms of whether or not management and engineering processes have been defined, implemented, and consistently used throughout the organization. The CMM has an underlying assumption that defined processes are good. It does not provide a procedure for defining or evaluating processes. Statistical methods are not explicitly specified by the CMM. Experience has shown that the CMM influences management's behavior, but engineers seem to perform the same way regardless of the capability level of the organizations [10].

LSS is a methodology; its focus is on how. In a sense, LSS is simply codified good systems engineering. One of the foundations of LSS is statistical quality control; LSS defines process performance in terms of its mean and variance. A concept that permeates the method is reducing the cost of poor quality. This concept is viewed at the broadest possible level. LSS does not explicitly provide a list of procedures and policies needed by an organization.

Both the SEI's CMM and LSS recognize that institutionalizing processes is a key to success, but their approaches are different. The CMM requires institutionalization by specifying the following:

• Written organizational policies that exist regarding the use of engineering and management processes.
• Adequate resources are provided for implementing processes.
• Appropriate oversight is provided (which could have the form of either taking certain measurements or management reviews).

LSS does not ask the question of whether a process is institutionalized. It is successful only when LSS itself is institutionalized. Specifically, LSS requires an extensive training program. All lead managers and engineers are expected to become experts in LSS, and are frequently referred to as Six Sigma black belts. This status requires taking a four- to six-week course over four months while applying what is learned in the course to a specific process improvement task. Following this training, the trainee is required to lead two more tasks and then take a test to be certified.

With LSS, everyone in the organization is trained to a level that is jobdependent. Training could be a oneweek course with application to a specific task for Six Sigma green belt status. Other training is at the executive level in which people take one- to three-day courses to obtain a basic understanding of the process.

Institutionalization is obtained by training and application throughout the organization. Institutionalization is impossible to obtain for either LSS or the CMM unless there is a long-term, substantial corporate commitment.

The CMM has the advantage of being controlled by the SEI, which has developed a substantial body of material for use in conducting capability maturity assessments, and has conducted many of these assessments. The SEI provides training courses in this area, and people can be certified as software capability evaluators. Generally, an external auditor assesses the CMM level by inspecting several projects across an organization. The organization provides all requested documentation for review and access to key people for interviews.

In the case of government procurements, the CMM assessments provide an indication of an organization's maturity based on other projects, but do not guarantee that the same processes and approach will be used for the system being procured. To remedy this problem, some acquisitions require a periodic CMM assessment of the contractor's effort during system design and development. The results of these assessments are (theoretically) tied to award fees.

LSS has no organization that is considered either the governing body or the standard bearer. Consequently, every Six Sigma organization defines it somewhat differently. This situation exists because the development and use of LSS has been driven by industry - in contrast to the CMM whose development was funded by the government and implemented by a federally funded research and development center. No external body exists to declare whether or not an organization is LSS. Nonetheless, there are recognized best practices associated with Six Sigma.

To complicate matters, Six Sigma organizations do their own certification. Thus, certification from one organization might not be accepted by another organization. In practice, anybody who is certified by one company is generally recognized as a Six Sigma expert by other organizations. However, if a certified expert (a Six Sigma black belt) changes organizations, he or she still needs to take Six Sigma training at the new organization to assure that he or she would be applying the methodology consistently with other people in the new organization.

If an organization claims to be a LSS organization in a proposal, assessing the veracity of this claim is relatively straightforward. The buyer could conduct a review of (1) the organization's training and certification program, (2) the certification of people committed to the program, and (3) the process documentation and performance data (for all processes to be used in the proposed acquisition).

The CMM is introspective. This focus is due to the nature of the model. Assessments determine whether measurements are being taken, policy exists, resources are applied, people are trained in the process, and products are reviewed internally. When the model looks outward, as it does in the Subcontract Management KPA, it is from the perspective of whether the internal management processes and policies exist to handle subcontracts.

LSS is inherently focused outward. The primary criterion used in assessing whether a process is lean is to determine whether each activity in the process adds value - i.e., it provides something the customer is willing to pay for. The Six Sigma part of LSS looks at the cost of poor quality. This criterion is directly tied to customer satisfaction and the supply chain (including subcontractors). Many companies also tie this criterion to their business plans and strategic goals.

The process improvement approach of Six Sigma is partitioned into five phases: define, measure, analyze, improve, and control. Having defined an existing process in the first phase, the next phase is to measure its performance. Performance measurements of throughput and quality are taken. Throughput is the number of items produced, services rendered, etc. Wait time and cues are also measured. Quality is expressed statistically as the process mean and variation. The cost of each step of the process is measured in terms of currency, time, and resources. The physical layout between process stages is measured to determine wait time and cost, or transportation expense between stages.

Various analyses are then performed, which include defect analysis (for example, cause and effect or fishbone charts) and analysis of variance. Simulations based on experiments' design are performed to determine candidate improvements. During the improvement phase, a prototype or initial improvement is made and measured. The results are compared with the simulation results to validate the improvement before it is implemented for the process. The improvement is implemented as an operational change in a controlled manner while measurements are taken to validate the prototype results. Measurement is a way of life in Six Sigma.

For measurement in capability maturity models, this discussion shall focus on the CMMI because it is the most recent and comprehensive model [11]. Measurement permeates throughout the CMMI. In the staged model, Level 4 is Quantitatively Managed. The purpose of this level is to obtain the data needed for the organization to effectively optimize its processes. Level 5 is Optimization. It is clear from thinking about the purposes of Levels 4 and 5 that at their core the CMM and Six Sigma have a great deal in common.

Unlike other capability maturity models, the CMMI has a process area (PA) Measurement and Analysis. This PA [12] specifies that a measurement capability be established to support management needs. The Measurement and Analysis PA is oriented toward systematically collecting typical program data (defect density, activity logs, peer review coverage, and so on). Measuring process capability, as such, could be included in this PA, but it is not a core purpose.

In the CMM, common features that contain key practices organize each KPA. The common features are ability to perform, activities performed, measurement and analysis, and verifying implementation. The CMMI slightly modifies the common features by replacing directing implementation with measurement and analysis. The CMM documentation is good at indicating the types of items that might be measured for each process, but does not explicitly say what to measure. The CMM documentation indicates that analysis of the data is necessary, however, neither type of analysis nor analytical procedures are explicitly discussed.

Both LSS and CMM are based on institutionalizing defined processes, performing quantitative measurement of the processes, and improving the processes based on these measurements. Both approaches address the systemic problems that have existed in our approach to software and systems engineering. Neither approach will be successful unless a substantial corporate commitment is made. This commitment includes the following:

• No-nonsense leadership from the top.
• Training (to various levels) of everybody in the organization.
• An up-front financial investment to get the process started.
• Organizational recognition of the importance of a capability maturity model or LSS.
• Rewarding people who are successfully implementing capability maturity models or LSS.

Organizational recognition does not mean that a big bureaucracy is needed. For example, Dow Chemicals had 2001 sales of $27.8 billion; they have more than 50,000 employees distributed over more than 40 countries. Six Sigma is implemented throughout the company with training materials in 13 languages. More than 90 percent of Dow employees will be involved with Six Sigma in some way by 2003 [13]. Their corporate staff for Six Sigma is about five people. There are also a few staff-level people in each of their operating businesses.

Rewards are critical because employees pay attention to a leader's actions more than his or her words. When rewards are primarily given to people for being a hero - working a large number of problems to save a program in trouble - that is what people believe is expected. Rewards need to be given primarily to people who did the job right in the first place, i.e., within budget and schedule.

Both approaches have been used successfully. The SW-CMM Level 5 organizations have the data to prove that they can deliver projects on time and within budget. It has been reported that variation between the actual cost and schedule to the estimated cost and schedule for projects performed by these organizations is usually within 3 percent [14]. Even Level 3 organizations have benefited dramatically from SWCMM. For example, John Vu of The Boeing Company has provided statistics that demonstrated variation of labor hours went from historical figures (Levels 1 and 2) of +20 percent to -145 percent to a Level 3 variance of +20 percent to -20 percent [15]. He also provided data to show that simply implementing a formal review and inspection procedure caused an increase of design effort by four percent and a decrease of rework by 31 percent. That change represents a cost benefit ratio of 1:7.75 - almost an order of magnitude.

Corporate presidents have discussed the benefits of LSS in terms of profit added to the bottom line. For example, at the 1999 Annual Meeting of General Electric, Jack Welch said that the Six Sigma effort at GE had already saved $3.5 billion beyond their investment of $1 billion, and they were just at the knee of the curve [16].

These two approaches to process improvement have the same goal. In fact, if an organization is truly a CMM Level 5 organization, it is also in spirit, if not in fact, a Six Sigma organization. Conversely, a true Six Sigma organization is in spirit, if not in fact, a CMM Level 5 organization. In each case, processes must be defined, data must be collected, and data used quantitatively to improve the processes. Some organizations do not begin integrating LSS with CMM until Level 3 has been attained (so processes have been defined), whereas others use LSS techniques to help define processes during the lower levels of maturity.

Examples of companies that have integrated Six Sigma with the CMM are Motorola, Tata Consultancy Services (TCS), Honeywell, and PS&J Software Six Sigma.

Motorola Labs used multivariate analysis techniques of Six Sigma to determine the causes of delays in closure of corrective action reports, and to improve their audit process. How to apply multivariate techniques to software processes is included in the Motorola University I-Cubed Presentation Series [17]. Motorola has several facilities evaluated at CMM Level 5, and is the founder of Six Sigma.

TCS also combined Six Sigma with the CMM. They specifically applied Six Sigma to their software review process and to decisions on program metrics [18]. This work was done for their Chennai, India, engineering center for General Electric. This TCS center has been evaluated as a CMM Level 5 organization.

Honeywell and PS&J Software Six Sigma introduced Six Sigma techniques into the Personal Software Process as defined by Watts Humphrey at the Software Engineering Institute [19].

The SEI's CMM and LSS have independently changed the way many major corporations think about their processes by addressing systemic problems in a constructive manner. These approaches are complementary. They both apply to the acquisition and development of complex systems. Their successful application depends on committed leaders, training, institutionalization, demonstrating a positive return on investment, and continuous reinforcement and reward.

1. Humphrey, Watts. Managing the Software Process. Reading, MA:Addison-Wesley, 1990.
2. Cooper, Jack, and Matthew Fisher, Eds. Software Acquisition Capability Maturity Model® (SA-CMM®). CMU/ SEI-2002-TR-010. Pittsburgh, PA: Software Engineering Institute, Mar. 2002.
3. Burnstein, Ilene, Taratip Suwannasart, and C. R. Carlson. "Developing a Testing Maturity Model." CrossTalk 9.8, 9.9 (Aug., Sept. 1996).
4. Shere, Kenneth D., and Mark J. Versel. Extension of the SEI Software Capability Model to Systems. Proc. of the 18th Annual International Computer Software and Applications Conference, Los Alamitos, CA, 1994. New York: IEEE Computer Society Press, 1994: 195-200.
5. Software Productivity Consortium. A Systems Engineering Capability Maturity Model Ver. 1.0. SPC-95007-CMC. Herndon, VA: Software Productivity Consortium, May 1995.
6. Arunski, Karl, et. al. Systems Engineering Capability Model. EIA/ IS 731. Arlington, VA: Electronic Industries Alliance, 17 Jan. 1999 www.geia.org/sstc/G47/731dwnld.htm.
7. Software Engineering Institute. Capability Maturity Model Integration (CMMI®), Ver. 1.1: CMMI for Systems Engineering, Software Engineering, Integrated Product and Process Development, and Supplier Sourcing V1.1 CMMI-SE/SW/IPPD/SS. Pittsburgh, PA: Software Engineering Institute, Mar. 2002.
8. Shere, Kenneth D. "Lean Six Sigma - How Does It Affect the Government?" CrossTalk 16.3 (Mar. 2003): 8-11.
9. Siviy, Jeannine. Six Sigma: Software Technology Review. Pittsburgh, PA: Software Engineering Institute, 1 May 2001 www.sei.cmu.edu/str/descriptions/sigma6_body.html.
10. Humphrey, Watts S. "What Is Excellence?" International Conference on Software Process Improvement. College Park, MD, Nov. 2002 www.software-process-institute.com.
11. Software Engineering Institute. Capability Maturity Model Integration (CMMI®), Version 1:1: CMMI for Systems Engineering, Software Engineering, Integrated Product and Process Development, and Supplier Sourcing V1.1 CMMI-SE/SW/ IPPD/ SS. Pittsburgh, PA: Software Engineering Institute, Mar. 2002.
12. Ibid: 163-180.
13. Parker, Mike. "Special Commemorative Issue." Around Dow. Midland, MI: Dow Chemical Company, 2000: 31, 44 www.dow.com/webapps/lit/litorder.asp?objid=09002f13800ba251&filepath=/noreg .
14. Johnston, Margaret. "Integrators Aim High on Software Methodology." Federal Computer Week 25 Jan. 1999 www.fcw.com/fcw/articles/1999/FCW_ 012599_ 53.asp.
15. Vu, John. "What Justifies a Rating of CMM Level 5?" Software Engineering Process Group Conference. New Orleans, LA, Mar. 2001.
16. Welch, Jack. "Presentation of the Chairman of the Board." Annual Meeting. Fairfield, CT: General Electric, 1999.
17. McCarty, Tom. Private Communication. Motorola, 2003.
18. Moorthy, Vinay. Private Communication. Tata Consultancy Services, 2003.
19. George, Ellen, and Steve Janiszewski. "SPC Is the Perfect Tool for PSP Post- Mortem Data Analysis." PS&J Software Six Sigma, 2001 www.SoftwareSixSigma.com.

1. For another comparison of Six Sigma to the Capability Maturity Model, cf. Card, David. "Sorting Out Six Sigma and the CMM." IEEE Software. May/June 2000: 11-13.

Kenneth D. Shere, Ph.D., is a senior engineering specialist at The Aerospace Corporation where he provides systems and software engineering, acquisition, and strategic leadership support to various government organizations. He is certified as a Lean Six Sigma green belt and a Software Engineering Institute Software Capability Evaluator. Shere has published 18 articles and two books. He has a bachelor's of science degree in aeronautical and astronautical engineering, a master's of science degree in mathematics, and a doctorate in applied mathematics, all from the University of Illinois.

The Aerospace Corporation
15049 Conference Center Drive
Chantilly, VA 20151
Phone: (703) 633-5331
Fax: (703) 633-5006
E-mail: kenneth.d.shere@aero.org

® Capability Maturity Model, CMM, and CMMI are registered in the U.S. Patent and Trademark Office.

^SM CMM Integration is a service mark of Carnegie Mellon University.

^TM Six Sigma is a trademark of Motorola, Inc.