Figure 1. Cleanroom software engineering process.
Cleanroom is a set of software engineering techniques that support the development of reliable software. Recently, the Systems Group (SG) at Texas Instruments, Software Engineering Institute (SEI) Capability Maturity Model (CMM) Level 3 organization, adopted cleanroom technology for a large-scale real-time system project. The project is a pilot SEI Level 4-5 program. This article outlines results of experiences in introducing cleanroom within an organization with an established process for software development.
Before delving into a discussion of cleanroom technology and its relation to the CMM, a review of CMM Levels 4 and 5 is warranted.
Background
SEI Level 4, "Managed," implements fact-based
management techniques during software development. Metrics that
define thresholds are used for decision making and status
assessment. Performance is measured, and appropriate action is
taken when limits are exceeded. The team executes a plan and
quantitatively assesses the impact of process change.
SEI Level 5, "Optimizing," is
addressed by implementing widespread continuous improvement.
Areas of focus at this Level include
This article describes an SEI Level 4-5 pilot program. Many of the Level 4 and 5 key process area (KPA) activities are described as part of the program's software development process.
Introduction
Cleanroom software development is a method to develop software
under statistical quality control. It is a theory-based,
team-oriented engineering process that attempts to develop the
highest quality software through error prevention and sound
engineering discipline rather than through error detection and
removal. Cleanroom software engineering is based on structured
programming ideas, focused thinking, and the methodical
application of mathematical reasoning to software.
Cleanroom software engineering attempts
to replace software "crafting," the reliance on human
creativity and experience to design processes, practices, and
solutions. The cleanroom approach proposes crafting be replaced
or supplemented by engineering practices that are derived from an
appropriate scientific or mathematical base. Cleanroom provides a
scientific base for software engineering.
The cleanroom development process is a
set of stepwise refinements or transformations, moving from the
requirements to the code, where each transformation is verified
to the previous level of refinement to minimize errors. Errors
are found earlier in the software development cycle, which
minimizes rework and speeds time to market.
Historical data shows that the
application of software cleanroom practices improves delivered
software in the following ways:
Improved Quality. The cleanroom
process increases quality, which reduces the overall cost. The
goal is to prevent rather than accommodate errors.
Increased Productivity. The
improved quality of the software reduces the time spent in
debugging and rework, which increase productivity and reduces
cycle time.
Improved Software Maintainability.
Software developed using cleanroom techniques has clear,
well-defined specifications and a less complicated design, which
allows maintainers to learn and modify the software faster and
with fewer errors.
Cleanroom Technology
Components
Figure 1 shows the cleanroom software engineering process
described below. The cleanroom process uses a three-team approach
to perform the cleanroom activities in an incremental fashion.
Figure 1. Cleanroom software engineering process.
Software Specification
During the specification phase, the software engineers use a
strict stepwise refinement and verification process using a
box-structured approach that allows precise definition of
required user function and system object architecture. This
approach scales to support large systems development.
Development
The main goal of the development team is to take a set of
software specifications, including hardware and human behavior
components, and design and verify those behaviors using data and
processes that implement the given specifications [2]. Box
structures, representing defined system behavior, are used in
development. The goal is to develop software that can be verified
for correctness against its specification using structured
programming correctness proofs.
Correctness Verification
During development, the software is verified using strict
correctness verification methods that prove the software meets
the specification. Verification reviews are held by the team to
formally or informally verify the software using a set of
correctness proofs that are standard in a structured programming
environment. This method allows the verification of programs of
any size by reducing the verification process to a manageable
number of independent verification checks. Proof of software
correctness is mostly done by direct assertion. Correctness
verification is done before the software is executed, so the
developers avoid a "debugging" mode of operation.
Certification
Cleanroom software engineering focuses on certifying the product
meets requirements and determining the reliability of the product
rather than attempting to test quality into the product. A
separate certification team tests the product using usage
model-based statistical testing. Usage data is used to develop
Markov models of the system that allow significant analysis
capability both pretest and post-test and straightforward
test-case generation. This testing approach provides
statistically valid quantitative measures to test progress and
software reliability. The quantitative measures provided by the
usage-based statistical testing approach provide mechanisms to
help management determine when testing can and should be stopped.
Incremental Development
The software product is developed in a series of functional
increments that sum to the final deliverable product. These
increments represent operational user functions. The integration
of these increments is done top down. Incremental development
allows early assessment of product quality and gives continuous
feedback to management as to the progress of development. When
the final increment is integrated and tested, the software
product is complete.
Although cleanroom technology holds the
promise of improved software quality and reliability, the
introduction of new software development methods poses risk.
Before deciding to adopt cleanroom technology within a highly
visible systems group project at Texas Instruments, there were
several key issues that needed to be addressed:
This article addresses these questions and reviews how the program incorporated cleanroom methods into the overall program structure.
Cleanroom Process Evolution
The road map we used to decide to incorporate cleanroom into our
existing process is shown in Figure 2. Several software
development techniques and methods were reviewed for potential
incorporation into the program's development process. The driving
criterion was the need to develop high-quality software with few
escaping defects. This would eliminate rework during later
integration and test phases and thus reduce cycle time. The
elimination of rework and shortened cycle time would in turn
reduce cost. Combined, these factors enabled the setting of
aggressive cost and schedule goals.
Figure 2. Road map for methods insertion.
After identifying cleanroom
techniques that had strong potential to meet our needs, an
introductory overview course was scheduled, and key software
engineers and systems engineers attended the training. The
consensus among these engineers was to pursue incorporation of
cleanroom. Formal training and a lab exercise was then held for a
pilot group of engineers that included software developers,
systems engineers, software technologists, and software process
engineers. Throughout this process, management was briefed and
kept aware of the activities.
Following the formal training, the
decision to incorporate cleanroom techniques was made. The
decision included all stakeholders: management, software
engineering, and systems engineering. The customer was included
in this process; liasons received additional training, which
helped secure concurrence with the chosen methods on the program.
Additional Steps Required
To evolve the program software process to include cleanroom
techniques required the following steps:
Updating the Software
Development Plan (SDP)
The SDP defines the software development process to be used by
the engineers throughout the entire development lifecycle. The
cleanroom techniques of increment planning, usage-based
statistical testing, and box-structured design were included in
the SDP, and the peer review philosophy was modified to align
with cleanroom practices.
Creating Procedures for the
Software Standards and Procedures Manual (SSPM)
The SSPM defines how the software will be developed and contains
the detailed procedures and guidelines used to develop the
software. Procedures that describe box-structured decomposition
(black-box implementation, stimulus and response enumeration,
mapping rules, state box, and clear-box implementation) and
usage-based statistical testing (test planning, usage model
generation, statistical testing, and certification) were
generated and included in the SSPM.
Updating the Risk Management
Plan
Risk management is included at all levels of the program. To
address customer concerns, the introduction of cleanroom was
included as a program risk. Metrics were identified to monitor
the success of the cleanroom introduction and were tracked as
part of our standard metric process. The metrics consisted of
cost and schedule performance metrics and a software defect
containment metric.
Updating the Software
Quality Assurance and Configuration Management Plan
Quality assurance and configuration management plans were updated
to reflect the cleanroom process, identifying required audits,
artifact capture, and baselining activities.
Tools Development
A strategic software tool set is in place for the systems group.
The focus for our program was to use cleanroom tool support for a
few high-impact areas. We also developed a stimulus and response
enumeration tool and automated testing support tools.
Process Modification
Cleanroom is built upon a set of team-based practices such as
team building, responsibility, and process adherence [7]. These
practices are required to implement core cleanroom principles
such as the existence of separate development and certification
teams and the use of formal and informal correctness
verification. Cleanroom fit naturally into our
already-established Integrated Product Team structure, which
emphasized some of the same team-building concepts that lead to
improved quality, ease of coordination, flexibility, and
team-based decision-making authority.
To evolve our process to incorporate
cleanroom, we replaced portions of our existing process with
appropriate cleanroom techniques. During the software
requirements analysis (SRA) phase, structured analysis techniques
were replaced with box-structured decomposition techniques.
Figure 3 shows a representation of our software requirements
analysis phase activities and associated phase inputs and
outputs. Requirements are flowed and allocated down from the top
level "A-spec" to the subsystems at the
"B-spec" level, then to each box in the subsystems.
Requirements analysis performed by each computer software
configuration item (CSCI) flows the allocated software
requirements to the CSCI.
Figure 3. Requirements generation using cleanroom.
The cleanroom SRA process defines the
CSCI boundaries, enumerates the stimuli and responses, and maps
those stimuli and responses to functionality in black-box mapping
tables. The cleanroom SRA process provides feedback to earlier
requirement allocation steps, identifying ambiguous, missing, and
incorrect requirements. CSCI requirements are derived from the
black-box mapping tables.
Not all requirements are developed from
the cleanroom process. Some environmental, performance, and
detailed algorithmic requirements are identified via other SRA
mechanisms. The total set of CSCI requirements are incorporated
into the Software Requirements Specification as shown in Figure
4. The arrow from algorithmic requirements to cleanroom
requirements in Figure 4 indicates that the algorithmic
requirements will ultimately appear in the cleanroom process at
lower levels of decomposition.
Figure 4. Flow of requirements into the SRS.
Adopting Usage-Based
Statistical Testing
The certification teams test software increments produced by the
development team using usage-based statistical testing. This
process replaces hand-crafted tests used in CSCI integration and
testing. The certification team develops the usage model, test
cases, and test data and performs the testing. A classic
qualification test is still performed on the final software
product to verify requirements. This qualification test uses a
combination of test cases generated during the certification
phases, along with any required handcrafted test cases.
Usage-based statistical testing is
performed in a CSCI software test environment that uses target
hardware supported by special test equipment (STE). STE provides
the simulated environment and interfaces to the various CSCIs
under test. Figure 5 shows the testing environment. An initial
usage model is developed based on historical prototype data,
domain expertise, etc. Based on the initial usage model, test
sequences are statistically generated with the support of a
commercial certification tool. The test sequences are turned into
scripts for the STE and data for the oracle.
Figure 5. Testing environment using cleanroom.
STE controls the sequencing of the
script during the test. STE also formats the output data and
responses. An oracle is developed to compare the expected outputs
to the actual outputs for verification purposes. Most of the tool
development associated with the incorporation of cleanroom was to
support the testing environment shown in Figure 5. This required
a concerted effort among multiple teams, including software
developers, STE developers, and systems engineers.
The cleanroom testing approach is a
departure from the traditional CSCI testing approach of unit,
CSC, and CSCI level testing. Concern over the lack of explicit
code coverage testing was raised by several parties. This concern
was addressed by initially executing test cases with the
increment instrumented for code coverage. The set of test cases
executed is a minimal covering set for the usage model, e.g., all
nodes and arcs of the usage model are exercised. The amount of
the increment's software exercised by the "covering"
test cases is then checked. Other organizations' experience with
this technique indicates that high percentages of code coverage
should be achieved. This approach also serves as a check on the
usage model for completeness.
Aligning with
Customer-Directed Deliverables, Milestones, and Schedules
A significant concern shared by management and the customer was
how to incorporate a cleanroom development schedule into the
program schedule while maintaining key milestone dates and
overall schedule integrity. In our case, the top-level software
schedule for each CSCI changed from an essentially traditional
waterfall lifecycle model to a cleanroom-based incremental
lifecycle model. As shown in Figure 6, the Preliminary Design
Review (PDR) is held earlier (in the middle of Increment 1) and
addresses the overall CSCI architecture. This is followed by a
succession of Critical Design Reviews (CDR), one for each
increment. The last CDR is held later than the CDR in the
traditional waterfall model, but by this time there are already
one or more increments of certified software developed.
Figure 6. Waterfall schedule vs. cleanroom incremental
schedule.
After certification of each increment, an informal Qualification Test (QT) is run, and a final, formal QT is held after the last increment is certified. Despite the incremental development approach, the CSCI schedule duration was effectively unchanged. To develop software in multiple increments results in tested software functionality earlier than the traditional waterfall approach. Table 1 summarizes the major milestones as they relate to cleanroom vs. our historical approach.
| Table 1. Major milestones-cleanroom vs. historical. | ||
| Milestone | Historical | Cleanroom |
| Software Specification Review |
Addresses requirements and external interfaces for the CSCI. |
Remains the same with increment
plan-mapping requirements to increment. |
| Preliminary Design Review |
Top-level architecture of the CSCI. |
Top-level architecture of the CSCI.
Updated as needed in later increments. |
| Critical Design Review | Detailed Design of CSCI. | Detailed Design of functionality for
particular increment. |
| Qualification Test (QT) | Verify CSCI requirements. | Verify CSCI requirements. Performed on
final CSCI increment. Earlier increments have informal QT. |
Summary
Based on our experience to date, we believe cleanroom is
consistent with and supportive of SEI Levels 4 and 5 software
processes. The evolution of our existing process and careful
tailoring of cleanroom practices resulted in a process that
addressed stakeholder concerns. Table 2 summarizes the
compatibilities, changes, and tailoring required to arrive at
this process. Table 3 summarizes steps used to incorporate the
new cleanroom techniques in our program environment [3].
| Features of cleanroom that were consistent with our SEI Level 4 and 5 processes. |
| Inchstone-based progress
management supported by cleanroom. Multiple increments supports causal analysis and process improvement. Metrics support quantitative decision making. Disciplined software development approach. |
| Features that caused us to change or modify our SEI Level 4 and 5 processes. |
| Separate development and
certification teams. Incremental development (more and smaller increments). Peer reviews (more frequent with less up front preparation). |
| Areas where cleanroom was tailored. |
| Algorithms and signal processing
software. Unit testing is optional. Development team responsible for a clean compile. People may transition between development and certification teams. |
| Table 2. Comparison of cleanroom and SEI Level 4 and 5 processes. |
| Table 3. Cleanroom practices and implementation at the introductory level. | |
| Cleanroom Practice | Program Implementation |
Management and Team Operations
|
|
Customer Interaction
|
|
Incremental Development
|
|
System Specification and Design
|
|
Correctness Verification
|
|
Statistical Testing and Reliability Certification
|
|
About the Authors
 |
David P. Kelly is a member of the group technical staff in the systems group of Texas Instruments, Inc. He is a software systems engineer in the electronic systems division. Previous assignments have included the full lifecycle of software development and serving as a site software process engineer. Kelly has a bachelor's degree in electrical engineering from Worcester Polytechnic Institute and a master's degree in computer science from the University of Texas at Dallas. |
Texas Instruments, Inc.
P.O. Box 655474 M/S
Dallas, TX 75206
Voice: 972-995-8439
Fax: 972-995-2308
E-mail: dkelly@ti.com
 |
Robert S. Oshana is a software team lead in the systems group of Texas Instruments, Inc. He has over 14 years experience in software development of real-time embedded systems using formal software development techniques and has experience in all phases of software development. Oshana has a master's degree in electrical engineering from the University of Texas at Arlington, in computer science from Southern Methodist University, and in business administration from the University of Dallas. |
Texas Instruments, Inc.
P.O. Box 655474 M/S
Dallas, TX 75206
Voice: 972-995-8317
Fax: 972-995-2308
E-mail: oshana@ti.com
References