Given the competitive importance of organizational performance and fact-based decision making, measurement programs have become increasingly important within the software and systems engineering communities. Measurement can no longer be implemented as a check-the-box process that is rolled out to satisfy a scheduled review or process improvement assessment. Today, measurement must provide real information to support critical project and organizational business and technical decisions, and the measurement results must be effectively communicated and used across the entire corporate entity.

The Navy provides us with two excellent examples of the impacts of using, and not using, measurement correctly. The F/A-18E/F program implemented a real-time management information system that communicated critical measurement-derived performance information to all project participants. This was instrumental in helping the F/A-18E/F program to be one of the most successful in recent memory. Conversely, on the Navy's A-12 program, objective information about project status was not available to the managers who were continuously required to deal with critical program constraints and performance shortfalls. Without accurate information, these managers were not able to properly make the difficult decisions required of them, and the program was ultimately cancelled.

Ten years ago, only a few organizations actually implemented measurement as an integral part of their technical and business processes. These forward-looking organizations established what worked well with respect to measurement, and even more importantly, what did not work. Under the umbrella of the Practical Software and Systems Measurement (PSM) Initiative, many software and systems measurement professionals that had successfully implemented measurement joined together in a government-industry-academia team to help put measurement into widespread practice. The goal of this PSM team was, and remains today, to help organizations meet a wide range of fact-based technical and business information needs by defining and helping them to implement a practical, information-driven measurement process.

We all know how complex weapons, communications, and information systems are becoming. At the center of this growing complexity is the need to deal with continuous technical and business change. We need to make better and more timely decisions that will result in the success of our projects, systems, and organizations.

In today's environment, our measurement processes must provide objective information to support critical decision making in an ever-changing environment. As a simple example, the defect status indicator shown in Figure 1 was developed to help make decisions concerning test completion by evaluating product quality. In this example, the program manager needs to assess whether the system will be ready for a user acceptance test by April 2002.

Figure 1: Quantitative Information Supports Critical Decision Making
(Click on image above to show full-size version in pop-up window.)

As an entrance criterion for the user acceptance test, all open defects must be closed prior to the start of this test. An indicator was generated that included a graph of the number of defects written and closed, along with a calculation of the number of open defects remaining. The analysis indicated that the closure rate of defects has remained relatively constant, while the number of new defects appears to be slowing. This has led to a decreasing number of open defects, and provides a positive indicator that the product will be ready for user acceptance testing.

Obviously, this is a very simple indicator. More elaborate indicators would usually be generated that might include indicators with defect data by severity (generally all high priority defects must be closed, but some number of low priority defects may be allowed), by status (to allow an assessment of progress), by component (to identify defect-prone components that may require additional inspection or testing), or by age (to identify those that have been open a long time).

For those organizations that are just starting to measure their software and systems engineering processes and products, producing usable measurement results like this example can be challenging. Certainly the thought of starting up a measurement program has some considerable implications, especially when you need information immediately. The good news is that most successful measurement programs are based on a few manageable concepts. Together these basics provide the foundation for an effective measurement program even in the most complex environments, and they provide for a flexible, cost-effective approach to meeting defined information needs.

There are three recurring lessons learned from successful measurement programs. These are the basic building blocks of any successful measurement program:

• Measurement is a consistent but flexible process that must be tailored to the unique information needs and characteristics of a particular project or organization. Measurement needs to change as the environment changes around it. Changing information needs drive the measurement process.
• Decision makers must understand what is being measured. Key decision makers, including both the technical and business manager, must deliver value-added objective results that can be trusted on the day-to-day issues that these managers face.
• Measurement must be used to be effective. The measurement program must play a role in helping decision makers understand project and organization issues and to evaluate and make key trade offs to optimize overall performance.

Although these three basic measurement concepts seem like common sense, it is amazing how many organizations ignore them when they establish their measurement programs. Even with all of the change that they must deal with, some organizations still try to build their measurement programs around the 10 best measures or try to measure so much that their measurement programs collapse under their own overhead burden. A large number of measurement programs fail early in their inception, usually because they do not provide information relevant to user information needs.

PSM has focused on delivering to both new and experienced measurement users guidance, tools, and other measurement products built around the foundational measurement concepts. A measure of PSM's success has been the adoption of its overall measurement approach by both the Capability Maturity Model® Integration^SM (CMMI^SM) [3] and by the international software and systems engineering community, as embodied in the new commercial software engineering standard ISO/IEC15939 Software Engineering- Software Measurement Process [4]. An important aspect of the three initiatives is the consistent treatment of measurement by PSM, CMMI, and ISO/IEC 15939. The measurement practitioner now has an integrated guidance set based on real measurement experience that has proven successful in actual applications.

Since measurement success is so closely tied to the three basic measurement concepts, let us take a look at each of them more closely.

An underlying concept of measurement is that it should be flexible and tailorable based on the unique information needs and characteristics of each project or organization. Measurement must be iterative to support necessary changes that result from changing information needs and improvements in the measurement process itself.

Figure 2: Four Key Activities Are Characteristic of Successful Measurement Programs
(Click on image above to show full-size version in pop-up window.)

The PSM process, shown in Figure 2, describes four activities that are part of a successful measurement program:

• Plan measurement: In this activity, measures are defined to provide insight into a project or organization's information needs. This includes identifying what the decision makers need to know, relating these information needs to those entities that can be measured, and then selecting and specifying prospective measures based on project and organizational processes. In the defect example in Figure 1, a comparison of the number of defects written and the number closed was used to address the question: "When will the system be ready for use acceptance test?"
• Perform measurement: This activity involves collecting measurement data, performing measurement analysis, and presenting the results so that the information can be used to make decisions. Analysis can include estimation, feasibility analysis of plans, and performance analysis of actual data against plans. For the defect example, the performance analysis included evaluating the trends of written and closed defects, and calculating test readiness.
• Evaluate measurement: In this activity, both the measurement process and the specific measures should be periodically evaluated and improved as necessary. For example, if the defect indicator does not provide enough information to adequately determine readiness for user acceptance testing, additional indicators may be added. The user may add an indicator of defect data by severity (generally all high priority defects must be closed, but some number of low priority defects may be allowed).
• Establish and sustain commitment: This activity involves establishing the resources, training, and tools to implement a measurement program effectively, and most importantly, ensuring that there is management commitment to use the information that is produced. In the defect example, if the measurement information is not used to develop plans for when user acceptance testing can begin, there is little need for collecting the data.

A measurement process that is flexible and tailored to project and organizational processes ensures that measurement is cost effective. Data should not be collected or reports distributed that are not needed or are not used. In addition, data collection and reporting should be automated whenever possible to provide an automatic by-product of normal project activity.

The process shown in Figure 2 provides a foundation for measurement for many disciplines, including software engineering, systems engineering, and process improvement measurement. An important thing to remember is that the same basic measurement process can support a wide variety of distinct and changing information needs in each of these areas. For additional information on the measurement process, see [3] and [4].

A second basic concept in a successful measurement program is the communication of meaningful information to the decision makers. It is important that the people who use the measurement information understand what is being measured, and how it is to be interpreted.

PSM does this by incorporating a measurement information model that links the entities that are measured to the associated measures and ultimately to the identified information need. The measurement information model provides a structure for specifying how a particular information need will be addressed within the measurement process. This allows the measures to be clearly and consistently defined.

In the defect example, it is important for the decision makers to know exactly what the data represents in order to ensure that objective decisions are made. For the sample defect indicator presented, it is important for managers to understand that only identified defects are included, and that some number of latent defects will remain in the product. It is also important to ensure that planned and actual data are quantified using the same methodology so that the two sets of data are comparable.

The measurement information model provides a mechanism for linking information needs to what can be measured. A measurement structure, called a measurement construct, describes how the relevant attributes of products and processes are quantified and turned into indicators that provide a basis for decision making. The measurement construct may involve three levels of measures: base measures, derived measures, and indicators as illustrated in Figure 3. In our defect example, base measures include number of defects written and number of defects closed. The derived measure of open defects was then calculated as the difference between the two. The indicator presented is a graph of the two base measures.

Figure 3: A Measurement Construct Relates What Is Being Measured to an Information Need

For each of the base measures, derived measures, and indicators, additional information also needs to be specified about how the various measures are calculated. Figure 4 illustrates the structure of a complete measurement specification, including the measurement method, measurement function, analysis model, and decision criteria. These, along with the procedures for data collection, data analysis, and reporting provide an operational definition of a measure, addressing a specific information need. Figure 5 contains a high-level description of a complete measurement specification for the defect example described previously. For a description of each of these terms in more detail, see [3] and [4].

Figure 4: A Measurement Specification Provides a Common Understanding of What Is Being Measured
(Click on image above to show full-size version in pop-up window.)

Figure 5: A Measurement Specification for the Defect Example
(Click on image above to show full-size version in pop-up window.)

Defining the measurement terms to this level of detail provides everyone working with the data a common understanding of what is being measured, and how this relates to the information needs. This formalizes what is important. By detailing the base measures to be used, this also helps to highlight common measures that can be reused to address multiple information needs. This assists in prioritizing the measures to be implemented. Sample measurement specifications for many common measures can be found in the PSM guidance.

In the past, measurement terms were often defined in unique and inconsistent ways from organization to organization. This led to confusion and difficulty in widespread measurement implementation. In many cases, decision makers were unsure of what the measurement results actually represented. One of the advantages of the measurement information model is that it defines a consistent terminology. This allows projects and organizations to document their information needs and selected measures to allow a common understanding of what is being measured. This is especially important as information needs change.

A third concept of a successful measurement program is that the measurement process is an integral part of the way business is conducted. In a successful organization, the measurement results are regularly used to make decisions. If the members of a project or organization are not able or willing to use measurement data to make decisions, the measurement program is of little use. In the defect example, if this measurement information is not used to develop plans for when user acceptance testing can begin, there is little need for collecting the information.

To support the use of measurement, information must be obtained early enough to allow managers to take the necessary actions to reduce risks or correct problems. Management decisions cannot wait for a complete set of perfect data to support management decisions, but should be derived from the minimum amount of data, complemented by real-time events and qualitative insight. Measurement should provide information on real-time events in a project, and facilitate communication.

The risk management and measurement processes should always be closely aligned. Risk management identifies the information needs that can impact project and organizational performance - information needs that should be objectively explained with the measurement results. The measurement data help to quantify risks, and subsequently provide information about whether risks have been successfully mitigated.

While measurement begins with a project-level focus, there are legitimate needs for information at higher levels of management. The information needs at different levels in an organization are related, as depicted in Figure 6. The project manager is concerned with the time and effort it takes to implement required product functionality and quality. At higher levels, managers are responsible for organizational performance and for improving organizational processes. At the enterprise level, managers are responsible for making investment decisions and ensuring performance is satisfactory. Since most organizations are composed of a portfolio of distinct projects, all of these uses of measurement rely on having good project-level information, aggregated to appropriate levels of the organization. The project level is where the processes and products are actually measured. As a result, a viable measurement program satisfies the needs of many levels of decision makers.

Figure 6: All Levels of an Organization Have Measurement Information Needs
(Click on image above to show full-size version in pop-up window.)

The PSM project is a Department of Defense and U.S. Army sponsored initiative. The PSM project comprises a fully integrated approach of products and services. Products are developed and improved incrementally by a joint government, industry, and commercial technical working group based on actual implementation experiences. Products are updated based on the technical consensus of best practices and are freely provided. The project is supported by a number of transition organizations that are qualified to teach PSM concepts and transition the PSM measurement guidance. PSM's products include the following:

• "Practical Software Measurement" (Addison-Wesley).
• "Practical Software and Systems Measurement Guidebook" (PSM version 4.0).
• PSM Insight Tool (a PC-based measurement tool that allows tailoring to an individual project's needs).
• "PSM for Process Management and Improvement Guidebook" (PSM: MPM) .
• PSM technology papers (measurement with object-oriented development, spiral/evolutionary development, interoperability, and product-line architectures).
• Training and workshop materials.
• Supporting materials (descriptions of products and services).
• A qualified technical team to provide direct project support.

The guidebook and papers are available from the PSM Web site at no charge www.psmsc.com. We invite your participation in the PSM project, and your input into future work products. Please see the Web site for more information about how you can get involved.

A successful measurement process becomes a way of doing business. Measurement is embedded in the organization, and performance improves because people are making fact-based decisions. Three lessons learned from successful measurement programs include the following:

1. Measurement is a consistent but flexible process.
2. Decision makers must understand what is being measured.
3. Measurement must be used to be effective.

These measurement concepts described here are relatively easy to implement. There are many available resources to support their implementation.

PSM's Relationship to ISO Standards and CMMI

Practical Software Measurement (PSM), a product of a Department of Defense measurement initiative, served as the base document for the new international standard on measurement ISO/IEC 15939 Software Engineering-Software Measurement Process. The international standard describes the measurement process in terms of the purpose and outcomes of a compliant process, along with associated activities and tasks. The standard also defines the measurement information model and associated terminology. PSM provides additional details on the activities and tasks presented in ISO/IEC 15939, and provides detailed steps to successfully meet these tasks. In addition, PSM provides detailed how-to guidance, including sample measures, lessons learned, case studies, and implementation guidance. PSM provides a set of sample measures using the measurement information model terminology. Both products are coordinated to provide users with a consistent framework for implementing a measurement program. In addition, the purpose and outcomes of the measurement process from ISO/IEC 15939 have been added to the revision to ISO/IEC 12207 Software Life-Cycle Processes, within a new supporting process entitled Measurement. Measurement concepts have also been added to ISO/IEC 15288 System Life-Cycle Processes. The new measurement terminology has also been coordinated with the revisions to ISO/IEC 9126 Software Product Quality and ISO/IEC 14598 Evaluation of Software Products, so that all these standards will use a common set of measurement terminology once the revisions are complete. In addition, the purpose and outcomes of the measurement process have been added to ISO 9000-3: Application of ISO 9001:2000 to Software. The draft international standard ISO/IEC 15939 in turn was used as an input to the measurement and analysis (MA) process area of the Capability Maturity Model® IntegrationSM (CMMISM) [4]. The MA process area provides a methodology for assessing whether a project's measurement program is compliant with the international standard, in addition to providing relevant information on CMMI-based process improvement activities. Overall, the CMMI helps organizations to institutionalize their measurement and analysis activities, rather than addressing measurement as a secondary function. In the MA process area, the activities of plan measurement and perform measurement are detailed in two specific goals that must be implemented and eight specific practices that are considered important in achieving the associated specific goals. The activities of evaluate measurement and establish and sustain commitment are considered through the generic goals, with elaborations specific to the MA process area. The coordination of these documents means that the software and systems engineering communities have a consistent set of information-driven standards and guidance for implementing project and process measurement. PSM, 15939, and CMMI Measurement and Analysis are Coordinated

1. Department of Defense. Practical Software and Systems Measurement Guidebook. v.4.0 Oct. 2000 www.psmsc.com.
2. McGarry, John, David Card, Cheryl Jones, Beth Layman, Elizabeth Clark, Joseph Dean, and Fred Hall. Practical Software Measurement -- Objective Information for Decision Makers. Addison-Wesley, 2002.
3. Software Engineering Institute. "Capability Maturity Model® Integrated^SM (CMMI^SM) for Systems Engineering, Software Engineering, Integrated Product and Process Development, and Supplier Sourcing Version 1.1 Staged Representation." Pittsburgh: Carnegie Mellon University, Mar. 2002.
4. ISO/IEC15939. Software Engineering - Software Measurement Process. 2002.

Cheryl Jones is a software engineer for the U.S. Army Tactical Army Command, Armament, Research, Development and Engineering Center, responsible for measurement and analysis, risk management, and estimation. Jones is also the project manager of Practical Software and Systems Measurement and one of the authors of "Practical Software Measurement: Objective Information for Decision Makers."

U.S.Army TACOM-ARDEC
Bldg. 62
AMSTA-AR-QAT-S
Picatinny Arsenal, NJ 07806
Phone: (973) 724-2644
Fax: (973) 724-2382
E-mail: cljones@pica.army.mil

® Capability Maturity Model and CMM are registered in the U.S. Patent and Trademark Office.

^SM CMMI and CMM Integration are service marks of Carnegie Mellon University.