Since 1999, The MITRE Corporation and others have developed a number of information security standards that are increasingly being adopted by vendors and form the basis for security management and measurement activities across wide groups of industry and government. This article explores how these standards are facilitating the use of automation to assess, manage, and improve the security posture of enterprise security information infrastructures while also fostering resiliency and effective security process coordination across the adopting organizations.

The basic premise of the MSM effort is that for any enterprise to measure and manage the security of their cyber assets, they are going to have to employ automation. For an enterprise of any reasonable size, the automation will have to come from multiple sources. To make the finding and reporting issues consistent and composable across different tools, there has to be a set of standard definitions of the things that are being examined, reported, and managed by those different tools. That standardization is what comprises the core of the MSM efforts.

Information security measurement and management—as currently practiced—is complex, expensive, and fraught with unique activities and tailored approaches. Solving the variety of challenges currently facing enterprises with regards to incident and threat management, patching, application security, and compliance management requires fundamental changes in the way vendor technologies are adopted and integrated. These changes include the way enterprises organize and train to utilize these capabilities. Likewise, to support organizational discipline and accountability objectives while enabling innovation and flexibility, the security industry needs to move to a vendor-neutral security management and measurement strategy. The strategy must be neutral to the specific solution providers while also being flexible enough to work with several different solutions simultaneously. Finally, the new approach should enable the elimination of duplicative and manual activities as well as improve both the resiliency and organizational ability to leverage outside resources and collaborate with other organizations facing the same threats and risks.

These objectives can be met by bringing architecturally driven standardization to the scoping and organization of the information security activities that our enterprises practice. By acknowledging the natural groupings of activities or domains that all information security organizations address—independent of the tools and techniques they use—a framework can be established within which organizations can organize their work independent of their current technology choices and flexible enough to adapt to future offerings. Likewise, by examining these domain groupings and the types of practices of coordination and cooperation that persist across and between them, it is possible to improve the interoperability and independence of these groups by standardizing common concepts in the information that flows across and between them. These shared concepts are sometimes referred to as boundary objects and are a phenomenon known to those who study inter-community communications¹, but have not been leveraged explicitly for information security standardization.

By leveraging the practices of systems engineering [1], an organization can recast current cybersecurity solutions into a launching point for standard functional decomposition-based security architectures. These architectures will provide a flexible, logical, and expandable approach to building and operating cybersecurity solutions for the enterprise—one that improves resiliency and is more supportive of security measurement, management, and sharing goals.

In this article, I will examine the collection of cybersecurity-related activities that most enterprises practice including: inventorying assets; analysis of system configurations; analysis of systems for vulnerabilities; analysis of threats; study of intrusions; reporting and responding to incidents; change management; systems development assessment; integration and sustainment activities; and certification and accreditation of systems being deployed into the enterprise².

I will also examine the different types of information that have been identified to support these activities. Finally, I will identify the key activities and information that need to be sharable and unambiguous in and amongst the different functions of today’s cybersecurity environment. Identifying and collecting these functional components as standard reusable concepts illustrates one of the major benefits that architecture brings to the study of security in the enterprise information technology landscape.

We can lay the foundation for architecting measurable security by looking at security measurement and management as an architecture issue and using a systems engineering approach to functionally decompose it, identifying the basic functions and activities that need to be done, and then getting the appropriate technology to support the functions and activities.

Through the development and adoption of standard enumerations, the establishment of languages and interface standards for conveying information amongst tools and organizations, and by the sharing guidance and measurement goals with others by encoding them into these standard languages and concepts, organizations around the world can dramatically change the options available to address the enterprise’s cyber environment security.

Both the U.S. government and commercial enterprises are already starting to deploy new approaches to security measurement and management that leverage interoperability standards and enable enterprise-wide security measurement and policy compliance efforts. These security architecture-driven measurement and management standards [2] are already providing ways for these organizations to create test rules about their minimum secure configurations, mandatory patches, and/or unacceptable coding practices that can be assessed, reported, and any subsequent remediation steps planned, executed, and confirmed using commercial tools. At the same time, these standards also provide a basis for repeatable, trainable processes and sharing along with enabling automation-based testing methods for deployment validation and regression testing throughout the operational lifetime of the systems.

Maybe more importantly, the establishment of architectural methods within the cybersecurity community will help open the doors to more resilient, faster, and better-coordinated approaches to dealing with the next set of security problems. There is little doubt that each of the current solutions being implemented to fight today’s threats will be attacked in-turn by advances in how systems and enterprises are attacked. But with a more consistent basis for considering these new threats and methods, solutions can be leveraged faster and applied in more predictable timeframes and with more understanding for the risks that remain.

I believe there are four basic building blocks for architecting measurable security:

• Standardized enumerations of the common concepts that need to be shared.
• Languages for encoding high-fidelity³ information about how to find the common concepts and communicating that information from one human to another human, from a human to a tool, from one tool to another tool, and from a tool to a human.
• Sharing the information through content repositories⁴ in languages for use in broad communities or individual organizations in a way that minimizes loss of meaning when content is being exchanged between tools, people, or both.
• Uniformity of adoption achieved through branding and vetting programs to encourage the tools, interactions, and content remain standardized and conformant.

The following sections discuss these building blocks in more detail.

Enumerations catalog the fundamental entities and concepts in information assurance, cybersecurity, and software assurance that need to be shared across the different disciplines and functions of these practices. The June 2007 National Academies report on the state of cybersecurity and cybersecurity research, “Towards a Safer and More Secure Cyberspace” [3], highlighted that metrics and measurements particularly rely on enumerations. As an example, the report cited the Common Vulnerabilities and Exposures (CVE) [4] list—run by MITRE under funding from the National Cyber-Security Division of the Department of Homeland Security—as an enumeration that enables all kinds of measurement by providing unique identifiers for publicly known vulnerabilities in software. There are a number of enumerations in the information assurance, cybersecurity, and software assurance space. Some examples are shown in Table 1.

Table 1: Enumerations

Standardized languages and formats allow uniform encoding of the enumerated concepts and other high-fidelity information for communication from human to human, human to tool, tool to tool, and tool to human. For example, a configuration benchmark document written in the XML Configuration Checklist Data Format (XCCDF) and Open Vulnerability and Assessment Language (OVAL) languages [5, 6] would be readable by a human and it would be consumable by an assessment tool, in that the tool would be able to directly import the tests and checks that are expressed in the document. As with the enumerations, there are a number of information assurance, cybersecurity, software assurance measurement, and management-oriented languages and formats. Some examples are shown in Table 2.

Table 2: Languages

Repositories allow common, standardized content to be used and shared, whether across broad communities or within individual organizations. The sharing of content has been done for some time but doing so in standard machine-consumable languages and formats using standard enumerated concepts is fairly recent. Most of the listed repositories are in the midst of converting their content into machine-consumable form. Examples are shown in Table 3.

Table 3: Repositories

These are all examples of very public repositories with a variety of types of content that will be recast into standardized machine-consumable form using some of the languages identified in Table 2 and the enumerations in Table 1. However, there are also closed repositories where, for instance, a company may write a tailored set of policies about what they want to do to comply with the Sarbanes-Oxley Act or something similar. They don’t necessarily want to share this with the world, but they do want to be standard across all of the different elements of their company and they want their policies available for their auditors and possibly their partners.

Uniform adoption of standards by the community is best achieved through branding/vetting programs that can help the tools, interactions, and content remain conformant with the accepted standards.

MITRE’s CVE project employs a highly successful CVE Compatibility Program that has vetted numerous information security products and services to ensure they are CVE Compatible; that is, they can interoperate with other compatible products that each have correctly mapped their capabilities concept of a particular vulnerability to the correct CVE Identifier for that vulnerability. Similarly, OVAL employs an OVAL Compatibility Program and CWE has begun a CWE Compatibility Program. The National Institute of Standards and Technology (NIST) has also initiated a Security Automation Validation Program (SCAP) for those vendors that currently provide (or intend to provide) SCAP-validated tools.

All of these programs—and others that may be developed in the future—will help ensure consistency within the security community regarding the use and implementation of the standards. They also assure users that the tools, services, and information from those organizations adopting the standards are doing so correctly and that there is a high confidence that they will work correctly when the tools and services are used together.

The building blocks of architecting for measurable security are already in use in the enterprise security areas of configuration compliance assessment, vulnerability assessment, system assessment, and threat assessment.

An Office of Management and Budget (OMB) memorandum [7] references the content in NIST’s National Vulnerability Database (NVD). This guidance is also referred to as part of the Federal Desktop Core Configuration (FDCC) [8] and is intended to bring consistency in the specific secure system software configuration of Microsoft Windows XP and Vista in use by the federal government. The part of the memo that is directed at Vista directly points to a set of content that uses the XCCDF and OVAL languages along with the CPE and CCE enumerations [9, 10]. This is a fairly public example of benchmark documents in a repository using standard languages and enumerations.

Figure 1 shows how an organization can utilize a tool-consumable benchmark document from a knowledge repository for configuration guidance. The benchmark provides the checking logic for a commercial tool that is used by the organization to conduct their configuration guidance analysis for assessing the configuration compliance of the organization’s computer systems. OMB’s Vista Guidance from the NVD is an example of this.

Figure 1: Assessment of Configuration Compliance Using Standards Vulnerability Assessment
(Click on image above to show full-size version in pop-up window.)

As shown in Figure 1, the results of the benchmark examination are also provided in standard language and enumeration terms as it is fed to the enterprise’s IT change management and central reporting processes. Figure 1 also shows how security measurement and management activities can be abstracted through a systems engineering analysis view to establish the security activities of configuration guidance analysis, enterprise IT change management, and centralized reporting as functional areas that can be managed.

Vulnerability alerts (e.g., those referenced in the NVD) are another case in point. Sometimes these are standardized already, depending which source they come from. Figure 2 shows how an organization can utilize a tool-consumable vulnerability assessment document from a knowledge repository: It will provide the checking logic for a commercial tool that is used by the organization to conduct their vulnerability analysis for assessing the vulnerability remediation compliance status of the organization’s computer systems. One example is errata from Red Hat, Inc., which are regularly posted with CVEs, OVAL definitions, and CVSS scores. As shown in Figure 2, the results of the vulnerability assessments are fed to the enterprise’s IT change management and central reporting processes.

Figure 2: Assessment of Vulmerability Remediation Status Using Standards
(Click on image above to show full-size version in pop-up window.)

Figure 2 also shows how vulnerability assessment and analysis can be abstracted through a systems engineering analysis view as a functional area that can be managed.

System assessments and certifications are not currently standardized. This is an area where standardization is being pursued through the development of efforts like CWE and CAPEC to address the developed components of a system along with the vulnerability and configuration assessment illustrated in Figures 1 and 2.

Figure 3 shows how an organization could utilize a tool-consumable body of certification requirements from a knowledge repository for system certification guidance in order to capture the criteria for assessing the status of an organization’s computer systems. One example is the Enterprise Mission Assurance Support Service effort being developed within the DoD. As shown in Figure 3, the results of the certification and accreditation examination is fed to the enterprise’s IT change management and central reporting processes.

Figure 3: System Certification and Accreditation Using Standards
(Click on image above to show full-size version in pop-up window.)

Figure 3 also shows how certification activities can be abstracted through a systems engineering analysis view as a functional area that can be managed.

Threat alerts and assessment is another area that has not yet been fully standardized. Imagine how an organization could utilize tool-consumable information from a knowledge source (about new and existing threats) that provided an efficient way of comparing threat information such as targeted platforms, vulnerabilities, or weakness against the enterprise’s information about their assets and their status. One example is the commercial threat reports that several security service providers offer. Imagine that results of analyzing new threat information can be fed to the enterprise’s IT change management and central reporting processes. In this vision, threat analysis would be abstracted to a vendor and tool-neutral activity through a systems engineering analysis view.

This same process of abstraction can be used to identify and define the other security measurement and management activities that an organization conducts. Figure 4 contains our current cut at abstracting and decomposing the overall security management and measurement activities of an enterprise (as described so far in this article), along with the other enterprise security management processes of an inventory asset activity, studying incidents, assessment of systems development, integration, and sustainment activities.

Figure 4: Decomposition and the Repositories Feeding Standard Measurement and Management Activities
(Click on image above to show full-size version in pop-up window.)

Furthermore, Figure 4 illustrates how the different security measurement and management activities are tied together through standards-based data interfaces that utilize the standard enumerations and standard languages discussed earlier. By utilizing these abstracted activities and enforcing the use of the standards-based interactions between them, an organization can bring commercially available technologies and tools to bear on their security problems while still keeping control of the processes and activities⁵.

Standard repositories of governance and guidance can help drive the business value of these standard measurement and management activities. As shown in the OMB guidance example, the information about how systems should be configured is captured by OVAL, XCCDF, CCE, and CPE.

The configuration guidance analysis, enterprise IT change management, and centralized reporting activities depicted in Figures 1 through 3 are several of the security measurement and management activities abstracted by taking a systems engineering analysis view of some of the different security activities of an organization.

Similarly, as shown on the left side of Figure 4, these same standards can be used to capture how an organization has configured and set up a new system when it has been approved for use in an enterprise. By using these standards, this information can go right into operational network management so that an organization can make sure the new system continues to be configured in the way that it was approved. Standard guidance can also be included about what weaknesses from CWE [11] should be reviewed in an organization’s or supplier’s development activities. In addition, the common attack patterns from CAPEC [12] can be used to define and document the types of penetration testing and attack scenarios a development team thought about defending against when they were doing their development and penetration testing.

For asset inventory, standards-based information utilizing CPE and OVAL will let an organization know exactly what assets they have in a manner that is tool-independent and usable in the other standard activities (such as configuration analysis). Similarly, if an organization knows exactly how their assets are configured, it is much easier to perform vulnerability analysis based on CVE, CWE, OVAL, and CVSS. Likewise, if an organization knows what they have, how it is configured, and what it is vulnerable to, that will change the context and framework of how the threat analysis is done.

As mentioned earlier, vulnerability alerts are sometimes standardized already, depending which source they come from. Red Hat errata, for example, are regularly posted with CVEs, OVAL definitions, and CVSS scores. In this area particularly, the standards have already been adopted by industry.

Since threat alerts are not as of yet standardized, this is an area where standardization could happen, and efforts like MAEC are aimed at enabling that. Similarly, there are a lot of different ideas in incident reporting regarding what should be standardized and to what extent those areas should be standardized.

There are many aspects of usage that are still evolving, including the correct approach to managing changes, updates, or new content for shared repositories. The question of whether the repositories should be enabled as services, as static collections, or both is also open. Similarly, as new insights are made with respect to vulnerabilities, weaknesses, threats, and attacks, there certainly will be changes needed in how the different aspects of these types of information are woven together and used. By bringing the various aspects of cybersecurity, information assurance, and software assurance into a consistent security architecture framework, there will be many new opportunities and much faster responses to new threats and new information. A compelling use of the enumerations, languages, and repositories can be found in the new “Consensus Audit Guidelines” [13], offered by the Center for Strategic and International Studies to advance key recommendations from the report on Cybersecurity for the current 44th Presidency [14]. The guidelines incorporate many of the items described in this article as an approach to clearly and concisely communicate what needs to be done and what needs to be audited.

Measurable security and automation can be achieved by having government and public efforts:

• Address information security during the creation, adoption, operation, and sustainment—in a holistic manner.
• Use common, standardized concepts.
• Communicate this information in standardized languages.
• Share the information in standardized ways.
• Adopt tools that adhere to the standards.

Much has already been done to transform the way security measurement and management is conducted, but there is still plenty of work that needs to be addressed. The use of architecture and systems engineering principles has been shown to be effective and enabling. Ongoing efforts to address and evolve all of the activities in this arena will greatly benefit from the continued application of this methodology. Like most architecture efforts today, the true value of architecture is not apparent or appreciated until its enabling properties start to manifest themselves. This article has outlined the changes in security practices and technologies and has shown specific and measurable changes that are directly related to the use of architectural methods on security of information technologies in government and private industry. This article also showed the benefits in sharing that standardized information.

By creating and evolving these types of standards and new approaches to security measurement and management, each of us will need to step away from the traditional focus on local and enterprise issues. We must realize that much more powerful and productive solutions to these issues can be fostered through an emphasis on community-wide examinations of each of the technical areas where a multitude of concerns and needs are balanced and considered. The increased insights, resiliency, and ability to leverage the collective knowledge and first-hand experience of what vulnerabilities and attacks affect us are valuable benefits to trading off local versus community-wide concerns.

To further the goal of making security measurable and encouraging the participation and adoption of the different aspects of this work, MITRE has established a public MSM Web site <http://makingsecuritymeasurable.mitre.org> that informally collects all of the efforts listed in this article, as well as others that are known about, which together are helping or will help in making security more measurable.

1. Chestnut, Harold. Systems Engineering Tools. New York: John Wiley & Sons, 1965.
2. Martin, Robert A. “Transformational Vulnerability Management Through Standards.”May 2005 <www.stsc.hill.af.mil/crosstalk/2005/05/0505Martin.html>.
3. Goodman, Seymour E., and Herbert S. Lin. Toward a Safer and More Secure Cyberspace. Washington, D.C.: National Academies Press, 2007.
4. “Common Vulnerabilities and Exposures.” The MITRE Corporation. 25 July 2009 <http://cve.mitre.org>.
5. Ziring, Neal, and Stephen D. Quinn. “The Specification for the Extensible Configuration Checklist Description Format Vers. 1.1.4.” National Institute of Standards and Technology. Jan. 2008 <http://csrc.nist.gov/publications/nistir/ir7275r3/NISTIR-7275r3.pdf>.
6. “The Open Vulnerability and Assessment Language.” The MITRE Corporation. 25 July 2009 <http://oval.mitre.org>.
7. Evans, Karen S. “Ensuring New Acquisitions Include Common Security Configurations.” Memorandum for Chief Information Officers and Chief Acquisition Officers. 1 June 2007 <www.whitehouse.gov/omb/memoranda/fy2007/m07-18.pdf>.
8. “The Federal Desktop Core Configuration – FDCC.” NIST <http://nvd.nist.gov/fdcc>.
9. “Common Platform Enumeration.” The MITRE Corporation. 25 July 2009 <http://cpe.mitre.org>.
10. “Common Configuration Enumeration.” The MITRE Corporation. 25 July 2009 <http://cce.mitre.org>.
11. “Common Weakness Enumeration.” The MITRE Corporation. 25 July 2009 <http://cwe.mitre.org>.
12. “Common Attack Pattern Enumeration and Classification.” The MITRE Corporation. 25 July 2009 <http://capec.mitre.org>.
13. “Twenty Critical Controls for Effective Cyber Defense: Consensus Audit Guidelines.” SANS Institute. 9 May 2009 <www.sans.org/cag/print.php>.
14. “Commission on Cybersecurity for the 44th Presidency.” Center for Strategic and International Studies Corporation. 2009 <www.csis.org/tech/cyber>.

1. To learn more about inter-community communications, see “Sorting Things Out: Classification and Its Consequences” by Geoffrey C. Bowker and Susan Leigh Star, MIT Press, 1999.
2. This is an integrated list that includes activities tied to the operation of systems in the enterprise as well as those they create, deploy, and update.
3. High fidelity refers to the level of detail of the information encoded in a language that is sufficient to convey the understanding and knowledge of the one encoding the information to the one who decodes the information. If a person writes a test for how to check a configuration setting in a language, then that language needs to be able to convey the specifics of the test so that another person or a tool reading the check as written in the language understands enough about the check to actually perform the test that was intended by the original author. If a language cannot retain the fidelity of the information to support this, then it is not of sufficient fidelity.
4. Content repositories are currently envisioned to be collections of tests to verify settings, patches, and installed software on systems to comply with organizational policies regarding their information technology systems and processes. Repositories are typically meant to be understandable by humans but are used by tools to automate checking for compliance with the tests in the repository. Many different organizations are hosting public and private repositories already and this is anticipated to continue and expand as the need to share grows.
5. The unwanted alternative is ending up with activities that are defined by the scope of the tools being used and that are coupled together by proprietary mechanisms.

Software Defense Application The security, integrity, and resiliency of cyber systems is critical within the DoD and is essential for its mission and support capabilities. This article describes and defines how the use of standard knowledge representations, enumerations, exchange formats and languages, and a sharing of standard approaches is helping transform key compliance and conformance mandates for the DoD, such as the Information Assurance Vulnerability Management process, the Security Technical Implementation Guidelines, and systems development. By adopting standards and segregating the interactions amongst their operational, development, and sustainment tools and processes, the DoD is and will gain greater freedom in selecting technologies, solutions, and vendors while also obtaining deeper insights into the current operational security and integrity of mission systems. These MSM initiatives answer today’s increased process demands without artificially constraining the solution options of the DoD.

Robert A. Martin is a principal engineer in MITRE’s Information and Computing Technologies Division. For the past nine years, his efforts have been focused on the interplay of enterprise risk management, cybersecurity standardization, critical infrastructure protection, and the use of software-based technologies and services. Martin is a member of the Association for Computing Machinery, Armed Forces Communications and Electronics Association, IEEE, and the IEEE Computer Society. He has bachelor’s and master’s degrees in electrical engineering from Rensselaer Polytechnic Institute, and an MBA from Babson College.

The MITRE Corporation
202 Burlington RD
Bedford, MA 01730-1420
Phone: (781) 271-3001
Fax: (781) 271-8500
E-mail: ramartin@mitre.org