The Software Testing Maturity Model^SM (SW-TMM^SM) is an exciting tool that can help generate significant changes in an organization's testing process. The SW-TMM is a testing process improvement tool that can either be used in conjunction with the Capability Maturity Model® for Software (SW-CMM®) or as a stand-alone tool. While the SW-CMM is an excellent software development tool that recognizes that reviews and testing are activities intended to enhance quality, it does not provide sufficient depth of testing coverage; the SW-TMM fills that void. The SW-TMM is not a tool to be used in addition to SW-CMM but is designed to be used in conjunction with the SW-CMM.

Testing maturity models are not new. Available documentation shows that most of them were developed around 1996, but they have never found much acceptance. One of the main reasons for this is the fact that there is very little documentation on the models. The articles, books, and Web sites on testing maturity models are written in a very theoretical style. Most people read them and say something like, "That's an interesting concept. When I have the time, I'll look into it." Of course, they never find the time.

I have taken the time to study the various maturity models and have discovered that the SW-TMM can be easily implemented and provides significant improvements in the testing process.

Terry Weatherill with ImagoQA Ltd. undertook a comparison of the testing maturity models currently available. His article "In the Testing Maturity Model Maze" [1] documents the results of his comparison. He studied the following six testing maturity models:

• Testability Maturity Model [2].
• Software Testing Maturity Model.
• Test Process Improvement (TPI) [3].
• Test Organization Maturity^TM [4].
• Testing Assessment Program [5].
• Proposed Evaluation and Test SW-CMM Key Process Areas (SW-CMM KPA) [6].

Weatherill concluded there were only two testing maturity models that were useable in their current format: SW-TMM and TPI. I read his article with extreme interest since I had already been searching for a testing maturity model to help my clients improve their testing processes. I judged the acceptability of a testing maturity model on the following:

• The ease of understanding and use.
• Allowing organizations to perform their own assessments.
• The ability to provide a baseline of the current testing function and a road map for improvement.
• The capability of being used for telecommunications, Web-based, and information technology testing applications.
• The ability to be used in conjunction with SW-CMM.

I had already been researching the SW-TMM and, after reading Weatherill's article, I decided to further research the TPI. I came back to the SW-TMM as the best fit for my requirements.

Dr. Ilene Burnstein of the Illinois Institute of Technology and her associates designed the SW-TMM to be a companion to SW-CMM. Since 1996, she and her associates have published several articles in professional magazines, including the following:

• "Developing a Testing Maturity Model: Part I" [7].
• "Developing a Testing Maturity Model: Part II" [8].
• "A Model to Assess Testing Process Maturity" [9].

The major weakness with SW-TMM is that there is no single book on the subject. I have corresponded with Burnstein, and the institute plans to release a book on the SW-TMM in 2002.

One of the biggest problems I have encountered while working with clients on their testing process is, that many times, there is no consistency within their organization as to the health and professionalism of the testing process. If you were to ask individuals at various levels of the organization their opinion of the current status of the software testing process, you will be surprised at the diverse answers you get. The answers given range from, "We have an excellent process in place, and don't need to change it," to "We have a horrible testing process.We should scrap it and start all over." The true level is usually somewhere in between these two extremes.

An assessment of the testing process using a testing maturity model will not only document the current level, but will also highlight the variances between the imagined level and the actual level. Only when the current status is known can significant improvements be made. Using the SW-TMM will not only help document the current level, but will also provide a road map for making the necessary process improvements.

As previously stated, one significant advantage of the SW-TMM is its compatibility with the SW-CMM. It contains a set of five maturity levels, like the SW-CMM, with goals and sub-goals at each level that can be used as building blocks for improvement.

Level No. 1 is where most organizations start. Testing is a chaotic process. It is ill defined and not distinguished from debugging. The tests are developed ad hoc after coding is complete. The objective of testing at this level is to show that the system and software works. It usually lacks a trained professional testing staff and testing tools.

Most organizations will quickly recognize the need to develop a more organized and professional testing process. Many do not realize there is a structured process, like SW-TMM, available to make these improvements. Instead of using a structured process, they try to implement random improvement measures without a clear documented plan of approach.

When a testing process reaches Level No. 2, it identifies testing as a separate function from debugging. This is also the level where testing becomes a defined phase following coding. When an organization reaches this level, their testing goal is to show that the system and software meets specifications. They have standardized their process to the point where basic testing techniques and methods are in place. Table 1 shows a comparison of the SW-TMM goals and the SW-CMM key process areas (KPAs) at this level.

Table 1: A Level 2 Comparison
(Click on image above to show full-size version in pop-up window.)

By the time the testing program reaches Level No. 3, the testing is integrated into the entire life cycle. The test objectives are now based on the system requirements. A formal testing organization is in existence. It establishes formal testing technical training, controls and monitors the testing process, and begins to consider using automated test tools. One of the major milestones reached at this level is that management recognizes testing as a professional activity. Table 2 shows a comparison at this level of the SW-TMM goals and SW-CMM KPAs.

Table 2: A Level 3 Comparison
(Click on image above to show full-size version in pop-up window.)

This is the first level where testing is mentioned in activities five, six, seven, and nine under KPA Software Product Engineering. An organization should not wait until they arrive at this level to assess their testing processes for improvement. It is best to incorporate SW-TMM as a companion tool in the software process assessment (SPA) performed at Level 1 so that improvement can begin at Level 2. If an organization does decide to wait until reaching this level, two things may occur:

1. The costs associated with the improvements can significantly increase because of the sheer volume of necessary improvements.
2. Because of the magnitude of the process improvements and the time constraints, they may never get accomplished.

At Level No. 4, the testing is a measured and quantified process. The development products are now also tested for quality attributes such as reliability, usability, and maintainability. The test cases are collected and recorded in a test database for reuse and regression testing. Defects found during testing are now logged, given a severity level, and are assigned a priority for correction. Table 3 shows a comparison of the SW-TMM goals and SW-CMM KPAs at this level.

Table 3: A Level 4 Comparison
(Click on image above to show full-size version in pop-up window.)

When an organization reaches the highest maturity level, CMM Level 5, the testing is institutionalized within the organization. It is well defined and managed, and testing costs and effectiveness are monitored. At this level, automated tools are a primary part of the testing process and there is an established procedure for selecting and evaluating testing tools. Table 4 shows a comparison of the SW-TMM goals and SW-CMM KPAs at this level. It should be noted that they are both concerned with defect prevention at this level.

Table 4: A Level 5 Comparison
(Click on image above to show full-size version in pop-up window.)

Now that we have discussed the five levels of the SW-TMM, the question on everyone's mind is: "The theory is nice, but why do I need to use SW-TMM?" If your organization is using the SW-CMM, the answer is obvious: SW-TMM is an excellent companion tool. The next question will probably be, "What makes SW-TMM an excellent companion tool?" The answer is that SW-TMM was designed to be a companion to SW-CMM. SW-TMM performs the following:

• Provides a methodology to baseline the current test process maturity.
• Is designed to guide organizations in selecting process improvement strategies and identifying the few issues most critical to software test process maturity.
• Is designed as an evolutionary path that increases an organization's software testing process maturity in stages.
• Provides a road map for continuous test process improvement.
• Provides a method for measuring progress.
• Helps an organization satisfy activities five, six, seven, and nine in Level 3 under KPA Software Product Engineering.

The first five bullets are virtually identical to the SW-CMM objectives. Organizations have to satisfy the activities listed in the last bullet in order to achieve Level 3. The SW-TMM provides a process, which can be incorporated into their SW-CMM structure, for accomplishing those activities. Furthermore, the figures above illustrate that the goals of the SW-TMM complement the KPAs of the SW-CMM at every level. It is also easy to understand and use. Thus, I believe that SW-TMM fulfills the design objective of being an excellent companion to SW-CMM.

If an organization is not using the SW-CMM, they can still use the SW-TMM as a stand-alone tool to help improve their test processes. We know the following about this versatile tool:

• We know the basic elements of the SW-TMM.
• We know that the SW-TMM was designed to be a companion to the SW-CMM.
• We know that many organizations do not know their true testing maturity level.
• We know that performing an assessment using the SW-TMM will baseline an organization's current testing maturity level.
• The SW-TMM will help an organization map incremental improvements.

The next step in the process is to determine your organization's current testing maturity level. The only way to document the true level of testing maturity is to perform an assessment. If your organization is using the SW-CMM, then the SW-TMM can easily be incorporated into the SPA. It becomes just another assessment tool. As previously stated, the SW-TMM is not a tool to be used in addition to, but it is designed to be used in conjunction with the SW-CMM.

If your organization is not using the SW-CMM, then management will not approve making improvements to the testing process unless you can prove to them that it truly needs improving. They will not spend the money just because the SW-TMM is a really neat tool. Here are a few selling points that might be used with management: 1) the SW-TMM will provide an unbiased assessment of the current testing process, 2) the SW-TMM will provide a road map for incremental improvements, and 3) as the testing process moves up the maturity levels, there are usually some cost savings. Do not push for a commitment to implement the SW-TMM. Instead, consider it a win if you can get management approval to perform an independent testing process assessment using the SW-TMM.

The answer is "Yes." (Remember that one of my criteria when I evaluated the various maturity models was "allowing organizations to perform their own assessments.") In fact, an organization must perform their own assessment to feel ownership and have confidence in the results.

It is usually advantageous to hire a consultant to lead you through the process the first time. A consultant has performed the process before and can help reduce the learning curve. They also offer an unbiased perspective when analyzing the results and developing an action plan. The following suggested process works well either incorporated into the SW-CMM SPA or used as a stand-alone assessment.

The logical first step in assessing testing maturity is the assessment preparation. Now is the time to choose a team leader and team members. This team should develop the assessment plan and prepare the tools they will be using. Do not limit the assessment to just the testing organization. Include individuals - from senior management to the non-technical developer - from across the entire organization. These individuals should be either directly or indirectly involved with the testing process. You want to sample as many different and varied opinions as possible.

One of the evaluation tools that will be used is a questionnaire. I have modified the questionnaire for the TMM developed by the Illinois Institute of Technology that I make available to my clients. The questionnaire provides structure and consistency to the process and makes it easier to identify the current level of maturity. During this phase, it is essential to conduct all training and management briefings. The training and briefings educate everyone on the objectives and evaluation process to be used. Periodic management status briefings should also be scheduled at this time.

Once all of the preparations have been completed, it is time to conduct the assessment. The first step is to collect and record information. Here are some of the methods that can be used:

• Request the organization being evaluated to prepare a presentation and briefing for the team. This gives them an opportunity to present any information they feel is important from their perspective. It also demonstrates to them that they are an integral part of the assessment.
• Conduct interviews with all individuals on the assessment list. During the interview, the team members should complete the questionnaire.
• Review and photocopy all testing documentation and procedures to determine the actual testing process currently being used.

One of the most important assessment activities is to document the findings. By compiling all of the information collected by the team, they should be able to do the following:

• Document the organizations' current testing process based on the records and documentation review.
• Compile and summarize the questionnaire data using either a spread sheet or database program.
• Document the interview information. It is best if more than one person has conducted the interview. The interviewers will compare notes and document all agreed information.

While the documentation process is under way, it always becomes apparent that the team has missed some essential information or needs clarification of information. Now is the time to secure that information or clarification.

The assessment report should include a section containing the analysis of findings. The analysis should document the current maturity level and any areas of disagreement highlighted during the evaluation. It should also identify areas for improvement and a prioritized list of recommended improvement goals. The recommendations should include anticipated benefits resulting from implementation.

Usually the team will discover testing processes that are excellent, but are not utilized throughout the entire organization. I like to call these the best-of-breed processes. The team should include as many of the best-of-breed processes as possible in the improvement plan. There are several reasons for this:

• There will be better acceptance of the improvement plan if the team recommends building on current processes.
• It will accelerate the implementation and improvement process.
• People enjoy the feeling of pride that accompanies having one of their processes adopted organization-wide

It should be emphasized that it is important for an organization not to try to progress from Level 1 to Level 5 in one giant step. That will result in almost certain failure. The recommendations should be a road map of how to reach only the next level of maturity.

The assessment team should develop an action plan for implementing the recommendations. The plan should describe each specific action, the resource requirements, and a recommended schedule for implementation. A cost/benefit analysis is considered helpful supporting documentation. The action plan should be an integral part of the final report.

While a written final report is essential, a management briefing of the findings and recommendations should also be scheduled. It is usually easier to secure management approval of the recommendations after a management briefing. The written report should be provided as supporting documentation for the briefing.

After securing management approval to implement the improvement plan, it is time to implement the improvements. It is usually best, if possible, to implement the improvements either in a pilot project or in phases. This allows the organization to track progress and achievements prior to expanding organization wide.

Implementing in a limited application also makes it easier to fine-tune the new process prior to expanded implementation. Since the SW-TMM assessment process is repeatable, improvements can easily be tracked by repeating the assessment six months to a year after implementation.

The SW-TMM was designed as a companion to the SW-CMM to evaluate an organization's current testing maturity and to plan test process improvements. It accomplishes that goal. To recap, the use of the SW-TMM will provide the following:

• Baseline the current testing process level of maturity.
• Identify areas that can be improved.
• Identify best-of-breed testing processes that can be adopted organizationwide.
• Provide a road map for implementing the improvements.
• Provide a method for measuring improvement results.
• Provide a companion tool to be used in conjunction with the SW-CMM.

Clients who are using the SW-CMM that I have exposed to the SW-TMM can immediately recognize that it is an excellent companion tool. It can be easily incorporated into their SPA, thus helping them map the test process improvements necessary to reach the next level of maturity. Organizations not using the SW-CMM will also find the SW-TMM an excellent tool to realize their goal of improving their testing process.

1. Weatherill, Terry. "In the Testing Maturity Model Maze." Journal of Software Testing Professionals Mar. 2001: 8-13.
2. Gelperin, David, and Hayashi Gelperin. "How to Support Better Software Testing." Application Trends May 1996.
3. Kooman, Tim, Martin Pol, Henk W. Broeders, and Hans Voorthuyzen. Test Process Improvement. Addison-Wesley, May 1999.
4. www.evolutif.co.uk.
5. Software Futures Ltd.
6. Weatherill, Terry. "In the Testing Maturity Model Maze." Journal of Software Testing Professionals Mar. 2001. www.softdim.comliist/journal.htm.
7. Burnstein, Ilene, Taratip Suwannasart, and C.R. Carlson. "Developing a Testing Maturity Model: Part I." CrossTalk Aug. 1996.
8. Burnstein, Ilene, Taratip Suwannasart, and C.R. Carlson. "Developing a Testing Maturity Model: Part II." CrossTalk Sept. 1996.
9. Burnstein, Ilene, Ariya Homyen, Robert Grom, and C.R. Carlson. "A Model to Assess Testing Process Maturity." CrossTalk Nov. 1998.

Thomas C. Staab owns an independent consulting firm, Wind Ridge International, which helps clients improve their software quality assurance and testing processes. He has more than 35 years experience in information technology and quality assurance. Staab has a master's of science degree in quality systems and is listed in the "International Who's Who of Information Technology." He has currently published more than 25 articles and presented more than 50 speeches and tutorials at regional, national, and world conferences.

Wind Ridge International
11321 E.Folsom Point Lane
Franktown, CO 80116-9105
Phone: (303) 660-3451
Fax: (303) 660-2057
E-mail: tcstaab@windridgeinternational.com

SM Testing Maturity Model and TMM are service marks of the Illinois Institute of Technology.