Ilene Burnstein, Ariya Homyen, Robert Grom, C.R. Carlson
Illinois Institute of Technology
This article describes a test process assessment model based on the Testing Maturity
ModelSM (TMM) we have reported on in this publication. We discuss the test
process assessment procedure, assessment inputs and outputs, the assessment questionnaire,
and team selection and training criteria associated with the TMM Assessment Model
(TMM-AM). Forms and tools to support test process assessment are also described, and we
report on preliminary experiments with the TMM questionnaire.
OFTWARE SYSTEMS CONTINUE to have an increasingly
strong impact on vital operations such as military, medical, and telecommunication
systems. For this reason, it is imperative that we address quality issues that relate to
both the software development process and to the software product. Our research focuses on
process. We are developing a TMM designed to help software development organizations
evaluate and improve their testing processes [1, 2]. Testing is applied in its broadest
sense to encompass all software quality-related activities. We believe that improving the
testing process thorough application of the TMM maturity criteria will have a highly
positive impact on software quality, software engineering productivity, and cycle time
reduction efforts.
In previous Crosstalk articles (August 1996, p. 21;
September 1996, p. 19), we have reported on our approach to building Version 1.0 of the
TMM [1, 2]. We have also described the internal structure of the TMM, including its
maturity levels, associated maturity goals, subgoals, activities, tasks, and
responsibilities. In this article, we describe the TMM-AM, which is designed as a tool
with which organizations may assess, evaluate, and improve their software testing
processes.
An Overview of the TMM
Development of the initial version of the TMM, as we have described in previous articles,
was guided by the work done on the Software Capability Maturity Model, a process
improvement model that has received widespread support from the U.S. software industry
[3]. TMM, Version 1.0 has two major components [1, 2], which are discussed below.
Set of Levels
The characteristics of each level are described in terms of organizational goals and
testing capability. Each level, with the exception of Level 1, has a structure that
consists of
- A set of maturity goals - these identify testing improvement goals that must be
addressed and satisfied to achieve maturity at that level (Figure 1).
- Supporting maturity subgoals - these define the scope, boundaries, and needed
accomplishments for a particular level.
- Activities, tasks, and responsibilities (ATR) - these address implementation and
organizational adaptation issues at a specific level. Activities and tasks are defined in
terms of actions that must be performed at a given level to improve testing capability;
they are linked to organizational commitments. Responsibilities are assigned for these
activities and tasks to three groups that represent the key participants in the testing
process: managers, developers and testers, and users and clients.
Figure 1. The 5-level structure of the TMM.
The Assessment Model
The TMM-AM can help organizations assess and improve their testing processes. The TMM
(levels, maturity goals, subgoals, and ATRs) serves as its reference model. The outputs of
a TMM assessment allow an organization to
- Determine its level of testing maturity (on a scale from 1 to 5).
- Identify its testing process strengths and weaknesses.
- Develop action plans for test process improvement.
- Identify mature testing subprocesses that are candidates for reuse.
The remainder of this article discusses the TMM-AM in greater
detail.
The TMM-AM: Development Approach
The TMM-AM has the following research objectives.
- Provide a framework, based on a set of principles in which software engineering
practitioners could assess and evaluate their software testing processes.
- Provide a foundation for test process improvement through data analysis and action
planning.
- Contribute to the growing body of knowledge in software process engineering.
We have used the Capability Maturity Model (CMM) and Software
Process Improvement and Capability Determination (SPICE) assessment models to guide
development of the TMM-AM [3-6]. We wanted the resulting TMM-AM to be CMM Appraisal
Framework (CAF) compliant [5] and integratable with the CMM assessment model so that
organizations could one day perform parallel assessments in multiple process areas. A set
of 16 principles has been developed to support TMM-AM design. For example, a testing
process assessment model should
- Be based on a testing maturity model as its reference model.
- Support test process improvement so that an organization can achieve software product
and process quality goals.
- Provide a profile of an organization's testing process capability.
- Help an organization make decisions about where to improve its testing process in order
to achieve testing process maturity.
- Be integratable with other assessment models.
- Provide high-quality data and repeatable, reliable results.
- Provide visibility to the testing process.
The TMM-AM Components
Based on the 16 principles, the CMM assessment model, SPICE, and the CAF [3-6], we have
identified a set of inputs and outputs and have developed a set of three components for
the TMM-AM:
- The assessment instrument (a questionnaire).
- The assessment procedure.
- Team training and selection criteria.
A set of inputs and outputs is also prescribed for the TMM-AM.
The relationship between these items is shown in Figure 2. A discussion of the components
follows.
Figure 2. The TMM assessment process: components, inputs, and outputs.
The Assessment Procedure
The TMM-AM assessment procedure consists of a series of steps that guide an assessment
team in carrying out a testing process self-assessment. The principle goals are to
- Ensure the assessment is executed with efficient utilization of the organization's
resources.
- Guide the assessment team as to who to interview and how to collect, organize, and
analyze assessment data.
- Support the development of a test process profile and the determination of a TMM level.
- Guide the assessors in developing action plans for test process improvement.
A brief summary of the steps in the assessment procedure
follows:
Preparation
This includes selecting and training the assessment team, choosing the team leader(s),
developing the assessment plan, selecting the projects, and preparing the organizational
units that are participating in the assessment. A statement of assessment purpose, scope,
and constraints is also prepared to guide the development of the assessment plan.
Conducting the Assessment
The team collects and records assessment information from interviews, presentations,
questionnaires, and relevant documents. All collected information must be protected by a
confidentiality agreement. The TMM level of the organization is determined by analysis of
the collected data and use of a ranking algorithm.
Our TMM-AM ranking algorithm is similar to the algorithm
described by S. Masters, et al., in their work on the CAF [5]. First, it requires a rating
of the maturity subgoals, then the maturity goals, and finally the maturity level [7]. Our
"degree of satisfaction" measure with respect to the maturity subgoals and goals
is more fine-grained than the corresponding measure in the Masters model. Our purpose was
to provide more detailed information to identify test process strengths and weaknesses. We
also provide guidance for prioritization of goal areas needed for test process
improvement.
Reporting the Assessment Outputs
The TMM-AM outputs include a process profile, a TMM level, a statement of test process
strengths and weaknesses, and the assessment record. The assessment team prepares the
process profile, which gives an overall summary of the state of the organization's testing
process. The profile is based on analysis of the assessment data and results of the
ranking process. The profile can be presented as a graphical display or in the form of a
matrix that indicates maturity goals and subgoals that are satisfied, not satisfied, not
applicable, or not rated. The profile also includes the TMM level, a summary of test
process strengths and weaknesses, and recommendations for improvements.
The assessment record is also completed in this step. This
written account includes
- Names of assessment team members.
- Assessment inputs and outputs.
- Actual schedules and costs.
- Tasks performed.
- Task durations.
- People responsible.
- Data collected.
- Problems that occurred.
The assessment outputs can be delivered as a presentation or a
written report (the final assessment report) or both.
Analyzing the Assessment Outputs
The assessment team uses the assessment outputs to identify and prioritize goals for
improvement. An approach to prioritization is described in [7]. Quantitative test process
improvement targets need to be established in this phase so they can support the action
plans developed in the next step.
Action Planning
An action planning team develops action plans that focus on improvements in the
high-priority areas identified in the previous step. The action planning team can include
assessors, Software Engineering Process Group members, software quality assurance staff,
or opinion leaders chosen from among assessment participants [8]. Inputs to action
planning include the final assessment report, the process profile, and prioritized areas
for improvement.
The action plan describes specific actions needed to improve
existing practices (and to support the addition of missing practices) so the organization
can move to the next TMM level. The action plan, like all other software engineering
project plans, should include measurable goals, tasks, responsibilities, resources
required, risks and benefits, and reporting and tracking mechanisms. Action planning can
be accomplished through the convening of a workshop directed by the action planning team.
The result should be a draft of an action plan. The workshop members should also identify
pilot projects that will implement the new process.
Implementing Improvement
Developed and approved action plans can be applied to selected pilot projects, which are
monitored and tracked to ensure task progress and achievement of target goals. Favorable
results set the stage for organizational adaptation of the new process.
The TMM Assessment Questionnaire
Assessment instruments are needed to help collect and record assessment information,
maintain a record of results, and provide information for assessment post-mortem analysis.
We use the questionnaire as our assessment instrument because it
- Supports CAF compliance.
- Facilitates integration with other process assessment instruments.
- Ensures assessment coverage of all activities, tasks, and responsibilities identified in
each maturity goal for each level of the TMM.
- Provides a solid framework in which to collect and store assessment data.
Our choice was also influenced by the success of the CMM
questionnaire as an assessment instrument [3]. The TMM questionnaire consists of eight
parts:
- Instructions for use.
- Respondent background.
- Organizational background.
- Maturity goal and subgoal questions.
- Testing tool use questions.
- Testing trends questions.
- Recommendations for questionnaire improvement.
- A glossary of testing terms [4, 7].
Components 2 and 3 of the questionnaire gather information
about the respondent, the organization, and the projects that will be involved in the TMM
assessment. Maturity goal and subgoal questions in component 3 are organized by TMM
Version 1.0 levels, and include a developer or a tester, a manger, and a client or a user
view. The questions determine to what extent the organization has in place mechanisms to
achieve the maturity goals and resolve maturity issues at each TMM level. The responses
are input to the ranking algorithm that determines a TMM level.
The testing tool component records information about type and
frequency of tool use. This information can help the action planning team make
recommendations for future tool usage. We added the testing trends section to provide a
perspective on how the testing process in the organization has evolved over the last
several years. This information helps the assessment team prepare the assessment profile
and assessment record.
The recommendations component allows each respondent to give the
TMM-AM developers feedback on the clarity, completeness, and usability of the
questionnaire. A complete TMM questionnaire is found in [7]. The questionnaire can also be
found on the Web site noted in the "Forms and Tools for Assessment Support"
section of this article.
Assessment Training and Team Selection
Criteria
Self-assessment of your organization's testing process requires a trained assessment team,
the members of which are selected from within the organization [7]. Team members should be
selected in a manner that ensures that they understand assessment goals, have the proper
knowledge experience and skills, have strong communication skills, and are committed to
improving the testing process. Assessment team size should be appropriate for the purpose
and scope of the assessment.
We have adapted SPICE guidelines to select and prepare an
effective assessment team [6, 7]. Preparation is conducted by the assessment team leader
who is experienced in TMM assessments. Preparation includes topics such as an overview of
the TMM, process management concepts, interviewing techniques, data collection, and data
analysis. Training activities include team-building exercises, a walk-through of the
assessment process, filling out a sample questionnaire, performing data analysis, and
learning to prepare final reports.
Forms and Tools for Assessment
Support
We have developed several forms and templates and a tool that implements a distributed
version of the TMM questionnaire to support a TMM assessment team [7, 9]. These tools are
important to ensure the assessments are performed in a consistent, repeatable manner, to
reduce assessor subjectivity, and to ensure the validity, usability, and comparability of
the assessment results. Tools and forms also help to collect, formalize, process, store,
and retrieve assessment information. The tools and forms we have developed include the
Process Profile and Assessment Record forms, which have been described in previous
sections of this article, and also include
- Team Training Data Recording Template - This allows the team leader to record and
validate team training data. This data can be used in future assessments to make any
needed improvements to the assessment training process.
- Traceability Matrix - This matrix is filled in as assessment data is collected,
allows the assessors to identify sources of data, resolve data related issues, and ensure
the integrity of the data.
- Web-Based Questionnaire - A complete version of the TMM-AM questionnaire is at http://www.csam.iit.edu/~tmm. The Web-based
questionnaire was designed so that assessment data could easily be collected from
distributed sites and organized and stored in a central data repository that could be
parsed for later analysis [9]. Developed using an HTML-based development tool, it runs on
multiple operating systems, allowing data collection from users around the world, thus
providing support for test process assessment to local and global organizations. A
detailed description of tool development is given [9]. The Web-based questionnaire and
links to supporting information related to the TMM is found at the above Web site. We
welcome comments and recommendations.
Preliminary Results on
Questionnaire Usage
Two software engineers from different development organizations have evaluated the TMM
questionnaire and have applied it to three development groups in their organizations (one
engineer evaluated two groups). Their feedback helped revise and reorganize some TMM
questions, experiment with our ranking algorithm using actual industrial data, generate
sample action plans, and study problems of testing process improvement in real-world
environments.
Obtaining and analyzing this industrial data, although on a small
scale, has been useful to our research team. One interesting result was that all three
groups were evaluated to be at TMM Level 1, but strengths and weaknesses of each group
were significantly different. Two groups satisfied several maturity goals at the higher
levels of the TMM. Given the quality of the existing processes for the latter two groups,
they should be able to reach TMM Level 2 in a relatively short time. More experimental
data is needed to further test the usefulness and effectiveness of the TMM and the
Assessment Model for test process assessment and improvement.
Future Plans
Our future plans include research on formal integration of TMM and CMM components so that
organizations can carry out parallel assessments in several process areas. We also are
planning the development of more intelligent tools to aid the assessors. Wider industrial
application of the TMM-AM is planned to help us evaluate its usefulness and effectiveness
for test process improvement. 
About the Authors
Ilene Burnstein holds a doctorate from
Illinois Institute of Technology, where she is an associate professor of computer science,
teaching undergraduate and graduate courses in software engineering. Her research
interests include software process engineering, testing techniques and methods, automated
program recognition and debugging, and software engineering education.
Computer Science Department
Illinois Institute of Technology
10 West 31st Street
Chicago, IL 60616
Voice: 312-567-5155
Fax: 312-567-5067
E-mail: csburnstein@minna.iit.edu
Ariya Homyen is a doctoral candidate in
computer science at Illinois Institute of Technology. She has a master's degree in
computer science from the University of New Haven and a bachelor's degree from
Chulalongkorn University in Thailand. Upon completing her doctorate, she will return to
her position at the Ministry of Science, Technology, and Energy in Thailand. Her research
interests include test process improvement, test management, and process reuse.
Computer Science Department
Illinois Institute of Technology
10 West 31st Street
Chicago, IL 60616
Voice: 312-567-5150
Fax: 312-567-5067
E-mail: homyari@minna.iit.edu
Robert Grom has a bachelor's degree from
Southern Illinois University and a master's degree in computer science from Illinois
Institute of Technology. He has worked as a hardware engineer and is currently manager of
data collection software for SAFCO Technologies, where he develops software for cellular
test equipment. His research interests are software testing, test process improvement, and
data communications.
SAFCO Technologies, Inc.
6060 N. Northwest Highway
Chicago, IL 60631
Voice: 773-467-2673
Fax: 773-594-2618
E-mail: rag@safco.com
C.R. Carlson holds a doctorate from the
University of Iowa. He is a professor in the computer science department at Illinois
Institute of Technology. He has published extensively in the fields of database design,
information architecture, and software engineering. His research interests include
object-oriented modeling, design and query languages, and software process issues.
Computer Science Department
Illinois Institute of Technology
10 West 31st Street
Chicago, IL 60616
Voice: 312-567-5152
Fax: 312-567-5067
References
- Burnstein, I., T. Suwanassart, and C.R. Carlson, "The Development of a Testing
Maturity Model," Proceedings of the Ninth International Quality Week Conference,
San Francisco, May 21-24, 1996.
- Burnstein, I., T. Suwanassart, and C.R. Carlson, "Developing a Testing Maturity
Model," Crosstalk, Software Technology Support Center, Hill Air Force Base,
Utah; Part I: August 1996, pp. 21-24; Part II: September 1996, pp. 19-26.
- Paulk, M., C. Weber, B. Curtis, and M. Chrissis, The Capability Maturity Model:
Guideline for Improving the Software Process, Addison-Wesley, Reading, Mass., 1995.
- Zubrow, D., W. Hayes, J. Siegel, and D. Goldenson, "Maturity Questionnaire,"
Technical Report, Software Engineering Institute, CMU/SEI-94-SR-7, June 1994.
- Masters, S. and C. Bothwell, "A CMM Appraisal Framework, Version 1.0,"
Technical Report, Software Engineering Institute, CMU/SEI-95-TR-001, February 1995.
- ISO/IECJTC1/WG10, "SPICE Products," Technical Report, Type 2, June 1995.
- Homyen, A., "An Assessment Model to Determine Test Process Maturity," Diss.,
Illinois Institute of Technology, July 1998.
- Puffer, J. and A. Litter, "Action Planning," IEEE Software Engineering
Technical Council Newsletter, Vol. 15, No. 2, pp. 7-10.
- Grom, R., "Report on a TMM Assessment Support Tool," Technical Report,
Illinois Institute of Technology, April 1998.
Testing Maturity Model and TMM are service marks of the Illinois Institute of
Technology.