STC logo Wednesday, 30 April 2003 Track 8: 3:50 - 4:30 Room 251 D-F

As organizations move toward the Capability Maturity Model® (CMM®) IntegrationSM requiring the integration of technical and management processes across functional disciplines, the tool suites used to plan, manage, and monitor these integrated processes must also evolve to support them. One example of this is an integrated engineering metrics set to reinforce process deployment, provide effective management oversight, and ensure alignment with organizational business goals. Harris Corporation used a Goal-Question-Metric approach to develop an integrated metrics set for quantitative management of performance, progress, cost, schedule, and resources across systems, software, and hardware engineering disciplines. Institutionalization of this metrics approach resulted in achieving CMM for Software Level 4.

At the Government Communications Systems Division (GCSD) of Harris Corporation, Melbourne, Fla., integrated metrics is a key element of successful quantitative management of every program and engineering discipline. Harris Corporation achieved the Software Engineering Institute's (SEI^SM) Capability Maturity Model® (CMM®) for Software (SW-CMM®) [1] Level 4 and is advancing to CMM Integration^SM (CMMI®) [2] Level 4 using integrated metrics across engineering disciplines. A SEI authorized lead appraiser performed the Level 4 SW-CMM appraisal of Harris GCSD in June 2002.

Integrated engineering metrics focus on quality, productivity, and predictability providing support data for estimating future jobs, tracking ongoing jobs, and identifying and evaluating process improvements.

Harris is recognized in the industry for developing and delivering quality products; however, to advance itself in a competitive industry the company has to continually improve its overall program performance. The reason many companies in the industry are advancing their capabilities by measuring engineering processes, products, and resources is to accomplish the following:

• Characterize - to gain understanding of processes, products, resources, and environments, and to establish baselines for comparisons with future assessments.
• Evaluate - to determine status with respect to plans. Measures are indicators of when projects and processes are drifting off-track so they can be brought back under control. Evaluations also assess achievement of quality goals and the impacts of technology and process improvements on products and processes.
• Predict - to gain an understanding of relationships among processes and products so the values observed could be used to predict others. This is done to establish achievable goals for cost, schedule, and quality so appropriate resources can be applied. Predictive measures are also the basis for trending so estimates for cost, time, and quality can be updated based on current evidence.
• Improve - to identify roadblocks, root causes, inefficiencies, and other opportunities for improving product quality and process performance. Measures of current performance give us baselines to compare whether or not improvement actions are working as intended, and what the side effects may be. Good measures also help communicate goals and convey reasons for improving. This helps engage and focus the support of those working within processes to make them successful.

Using the Goal-Question-Metric (GQM) [3] approach, integrated engineering metrics was derived from strategic business goals and best practices of our organization, the industry, and government. The main objective for integrated engineering metrics is to objectively measure the program health and status in relation to the following organization's goals:

• Project Management. Planning, estimating, monitoring, and controlling a project's costs, schedules, and quality.
• Process Improvement. Providing baseline data and measuring trends, tracking root causes of problems and defects, and identifying and implementing changes for process improvement.
• Organizational Vision. Effectively applying unified end-to-end engineering processes and methods encompassing proven and emerging standards/ approaches for the purpose of delivering high-quality, cost-competitive system solutions to our customers.

An action team (composed of systems engineers, software engineers, program managers, and assessment experts) focused on defining the Harris GCSD's goals and ensuring the metrics needed to measure the achievement of those goals were being captured at all stakeholder levels of the organization. Structured interviews were conducted with individuals representing the following four levels (from the highest to lowest) of stakeholders:

• Division management.
• Business area leadership.
• Project management and technical leadership teams.
• Functional owners of division processes.

The protocols for the interviews (individual and group) at each level of the organization were based on the results of the interviews from the previous level of the organization. Division management was asked to rate the importance of the goals in the division Strategic Guide Plan (and the goals supporting those in the plan). The business area leaders then were interviewed and asked to identify the subclass, questions, and metrics that they used, or would like to use, to achieve the goals identified by division management. The project leadership interviewees were asked to identify the questions and metrics that they used, or would like to use, to achieve the division goals and business area subgoals. The process owners were then asked to identify the questions and metrics that they would use to measure the process goals identified in the prior interviews as well as to achieve the improvement goals that the process owners identified.

The interviews were structured to correspond to the GQM [3] methodology, where issues, problems, and objectives led to the identification of measures. The interviewees were also asked to prioritize both the reasons for desiring the measurement information and the importance of the specific measures they recommended.

The Functional Analysis System Technique (FAST) [4] was used to graphically depict the linkage of each higherlevel goal to lower-level goals. FAST provided a mechanism for obtaining importance ratings, by interviews, on more than 100 goals without losing the goals' context. The team analyzed the importance rating and selected the highest-level goals that spawned a set of lower-level goals with a 90 percent or greater coverage. This generated a set of top-level goals that were briefed to division management and used as the foundation for organizational metrics.

The analysis identified division goals and sub-goals: Based on the metrics currently used in the division, metrics from the industry literature, and key practices from the SW-CMM [1] and CMMI [2], metrics were identified to measure the success in achieving the goals and subgoals. GQM [3] concepts were used to validate the results of the metrics derived from the other sources and to identify any metrics that might have been overlooked. It should also be noted that several existing division metrics were dropped, as they were not directly attributable to the defined division business goals. An example of a division goal mapped to metrics using GQM follows:

• Goal: Project Management, i.e., plan, estimate, monitor, and control project quality.
• Sub-Goal: Improve customer satisfaction by reducing defects.
• Question: Where are defects introduced and removed?
• Metric: Defects detected in peer reviews and testing.

A red team consisting of six project teams reviewed the resulting metrics. Each project team was composed of the project's program manager, chief system engineer, chief software engineer, chief hardware engineer, and quality assurance engineers. A structured evaluation technique was used against each metric using the following criteria:

• Utility to the customer.
• Utility to the project leadership.
• Utility to division management.
• Difficulty to collect.

The metrics definition effort identified metrics covering all aspects of project management and engineering performance across systems engineering, software engineering, and hardware engineering. The metrics were grouped into sets that represented a theme or view of performance familiar to each of the four levels of the organization.

The metric groupings took the form of the currently used division control panels containing up to nine metrics each (3 x 3). The control panels represented the integrated project engineering metrics distributed across systems engineering, software engineering, hardware engineering, and project engineering resources. The derived metrics differed from the pre-existing division metrics in two major areas:

• More emphasis on product quality via defect measurement and tracking.
• Additional measurement of the personnel resources' training, development, and tool support.

The metrics set supported the SW-CMM [1] and CMMI [2] Level 4 objectives of defined measurement standards. Each metric has a specified value that represents an enterprise performance goal. As data are collected, the goals are converted to control limits. The top six metrics in the example of Integrated Engineering Cost and Schedule Control Panel, shown in Figure 1, address the GQM as follows:

• Goal: Project Management, i.e., plan, estimate, monitor, and control project cost and schedule.
• Sub-Goal: Perform within planned cost and schedule.
• Question: How effective is the process execution versus the plan?
• Metric: Cost performance index (CPI), schedule performance index (SPI), and to-complete performance index (TCPI).

Additional information provided in the footer of these metrics is cost variance (CV), schedule variance (SV), and variance at completion (VAC).

Figure 1: Integrated Engineering Cost and Schedule Control Panel
(Click on image above to show full-size version in pop-up window.)

Integrated engineering metrics is used to gauge a project's progress and to alert program management of any potential risks to its quality, cost, and schedule. Each metric provides insight into systems/ software/hardware engineering development products and processes and process improvement and/or organizational improvement through one of the following four major indicator categories:

• Progress. The achievement or completion of goals or commitments.
• Resources. The availability or capability of organizational assets.
• Quality. The problems and/or defects with a product or process.
• Stability. The degree of change, completeness, or effectiveness.

Everyone who uses engineering processes and/or develops engineering products utilizes engineering metrics. Program team members are responsible for collecting and analyzing individual metrics. Project team leaders are responsible for collecting, analyzing, and reporting metrics to the program team and division management. Division management ensures the collecting and reporting of metrics, and the engineering process group conducts metrics analysis and trending. The integrated engineering metrics process has four steps: planning, collecting, analyzing, and reporting.

Planning is the first step in the integrated engineering metrics process. The collectiing, analyzing, and reporting of metrics are integrated into the project plans identifying the following:

• Metrics used to support quantitative management.
• Planned and/or expected performance in the metrics, including any required goals and/or control limits.
• Variance implication and corrective action for metrics falling outside their control limits.
• Source and collection mechanism of the measurement data.
• Responsible persons for collecting measurement data, analyzing of metrics, reporting the results, and managing the engineering metrics process.

Division control limits are statistically based upon historical data. Projects use the division control limits or statistically determine their own.

Collecting measurement data is the second and continuing step in the integrated engineering metrics process. The collection occurs at periodic intervals defined in the project plans and is monitored for completeness, integrity, and accuracy. The primary source for planning data is in the project plans. The primary source for actual data is in the accounting systems used to manage the project (e.g., financial management, configuration management, change management, and risk management) and is input into the division standard metric tool each period.

Analyzing metrics and making objective quantitative management decisions is the true benefit step in the integrated engineering metrics process.

Metrics are most often communicated graphically conveying a clear and easily understood message. It is better to have many graphs than it is to have many messages on one graph.

Metrics are indicators that give warnings of problems associated with issues. An issue may be tracked with several metrics that may be based on different measures. Insight into an issue typically requires statistical analysis of metrics over time and is trend-based or limit-based as follows:

• Trend-based metrics are used when expected or planned values change regularly over time. The analysis of a trend-based metric involves determining whether the performance implied in the trend is achievable.
• Limit-based metrics are used when the expected or planned values remain relatively constant over time. The analysis of a limit-based metric requires determining whether the performance crosses its established bounds. Limits can represent norms, expected values, or constraints.

Detecting a difference, limit or trend, between planned and actual recognizes problems. If the difference exceeds the threshold of acceptable risk, then the situation is investigated and corrected.

Reporting integrated engineering metrics is the final step in making quantitative management decisions and communicating to project team members, management, and customers. Reporting and reviewing metrics are integrated into the management process and occurs as soon as possible after analysis has been completed to assure that there is time for corrective action. Any metric falling outside the control limits is reviewed for variance, and corrective actions are recorded and tracked to closure. Meeting minutes are kept that record the variance explanations.

Integrated engineering metrics are collected, analyzed, and reported via the divisionstandard metric tool (Web client/database server) for consistency in application across the division. A required set of integrated engineering metrics is used by all projects to advance the engineering process maturity of the division.

Projects utilize additional metrics such as customer-required metrics, to complement the division-standard metric tool. A detailed definition of each engineering metric is built into the metric tool, including description, audience, purpose, method, measures, metrics, control limits, formulas, range of values, graphic information, and references. Control panels are the most common method for communicating an integrated view of engineering metric frames. A subset of the standard division metrics is presented at all program reviews.

Lessons learned from implementing a metrics program and tool within an integrated discipline work force are as follows:

• One metric does not tell the whole story. You need integrated, and many times, orthogonal views of metrics to get a complete picture; trending is key.
• Project planning is key, and data collection is the hardest.
• Having an organizational standard tool is a must for consistency; it should be user friendly with easy access.
• Cultural change is hard, so train everyone about the organizational metrics program and tool to increase acceptance and buy-in.

Integrated engineering metrics are required to provide effective management oversight and to ensure alignment with organizational business goals. As organizations move toward the CMMI [2] requiring the integration of technical and management processes across functional disciplines, the tool suites used to plan, manage, and monitor these integrated processes must also evolve to support them.

1. Paulk, Mark C., Charles V. Weber, Suzanne M. Garcia, Mary Beth Chrissis, and Marilyn W. Bush. Key Practices of the Capability Maturity Model®. Ver. 1.1. CMU/SEI-93-TR-25. Pittsburg, PA: Software Engineering Institute, Feb. 1993.
2. Carnegie Mellon University. CMMI^SM for Systems Engineering/Software Engineering, Ver. 1.1, Staged Representation. CMU/SEI-2002-TR-002. Pittsburg, PA: Carnegie Mellon University, Dec. 2001.
3. Park, Robert E., Wolfhart B. Goethert, and William A. Florac, Goal-Driven Software Measurement - A Guidebook. CMU/SEI-96-HB-002. Pittsburg, PA: Software Engineering Institute, Aug. 1996.
4. Kaufman, J. Jerry. Value Engineering for the Practitioner. Raleigh, NC: North Carolina State University, 1990.

Gary Natwick is the metrics leader for the engineering process group responsible for Harris Corporation achieving the Software Engineering Institute's (SEI^SM) Capability Maturity Model® for Software (SW-CMM®) Level 4 and advancing to CMM Integration^SM Level 4. Previously, he led the software engineering process group responsible for Harris Corporation achieving SW-CMM Level 3. Natwick has 30 years of software and systems engineering experience (management, development, and process improvement) with the U.S. Air Force (USAF) and Harris Corporation. He received the USAF Commendation Medal for Meritorious Service, and the Engineering Process and Golden Quill awards for advancing Harris Corporation. Natwick has a Bachelor of Science in Electrical Engineering from the University of Miami, Fla. He is a SEI Authorized Lead Appraiser (SCAMPI^SM and CBA IPI methods), and "Introduction to CMMI®" course instructor.

Harris Corporation
P.O. Box 37
Melbourne, FL 32902-0037
Phone: (321) 729-3970
E-mail: gary.natwick@harris.com

^SM SEI, CMM Integration, and SCAMPI Lead Assessor are service marks of Carnegie Mellon University.

® Capability Maturity Model, CMM, and CMMI are registered in the U.S. Patent and Trademark Office.