Figure 1. A sample GLUE screen.
This article describes an experiment using DomainSpecific Software Architecture (DSSA) Avionics Domain Application Generation Environment (ADAGE). This study was intended to provide an empirical basis to characterize and evaluate the usage, effectiveness, and impact of ADAGE by generating integrated avionics systems from a parameterized, layered avionics domain specific software architecture.
A major goal of
the DSSA ADAGE project is to concurrently improve programmer
productivity and reduce defects by an order of magnitude in the
avionics software domain. To evaluate the effectiveness of the
ADAGE architecture based software development approach, we
conducted an empirical experiment to quantify benefits and
limitations. This experiment involved real production code for
the navigation subsystem of the avionics domain and actual
avionics system programmers who used a subset of the ADAGE
environment: the configuration and composition tools GLUE
(Graphical Layout User Environment) and MEGEN (Module Expression
GENerator).
The study used a 24 factor experimental
design that simultaneously evaluated the impact of
Each of these factors was evaluated according to several software production criteria categories, including
The study showed improvement in
productivity by roughly a factor of 14, and a 8:1 reduction in
errors.
This article presents the results of
this experiment, first describing the application domain, the
layered "reference" architecture, its parameterized
components, and the graphical configuration and composition tools
used to generate the Ada source code.
The Domain
The highlevel model for a typical avionics system is a closed
loop system that includes the aircraft pilot. The pilot interacts
with displays to monitor the state of the aircraft and to be
apprised of current mission objectives, such as which flight plan
to follow, to which navigation reference point to fly, and to
which channel to set the radio. By manipulating the flight
controls, the pilot sends control signals to the plane that, in
turn, affect the aircraft sensors.
The core requirements of any advanced
avionics systems include the capability to navigate, provide
guidance, and provide flight direction. Navigation is the ability
to estimate the aircraft state relative to reference frames
relevant to aircraft operations, e.g., geodetic or other
coordinates. Guidance contains algorithms that convert temporal
and spatial mission objectives into desired lateral, vertical, or
speed profiles based on aircraft performance characteristics and
selected optimization criteria. The estimated aircraft state is
then compared with these profiles to produce heading, altitude,
or speed error signals. Flight direction accepts the error
signals and recommends control inputs to the crew (or aircraft
automatic flight control system for coupled flight) that nulls
the error signals, thus directing the aircraft on the desired
path. This experiment focused on the navigation subsystem.
The Avionics Navigation
Reference Architecture
DSSA-ADAGE is based on the premise that for many application
domains, much of the software is well understood and has been
built. For new systems, several features may require new and
innovative software, but much of the new system can be built by
combining and adapting existing software components. Analysis of
the software domain can identify the components and constraints
on their composition. This analysis produces a reference
architecture, which serves as a blueprint to automatically
generate application software for a particular domain.
The reference architecture defines a
hierarchy of abstract virtual machines. Each virtual machine may
be implemented by one or more components. The set of components
that implement the same virtual machine is called a realm. A
realm is,in effect, a library of plugcompatible and
interchangeable software components. One can also view realms as
a class of objects, whereas components are instances of objects
from these classes.
The Software Components
Once the reference architecture was established for ADAGE, the
software components were written to populate the domain's realms
and a set of software component transformation rules were defined
to specify required modifications to the components for a
particular software configuration. By convention, these
transformations divided the software components code into three
categories:
Realm code corresponds directly to
the software components in a realm. Whenever a corresponding
component is to be included in the design, a configured instance
of the code that corresponds to that software component must be
included in the generated source code. For example, in the
navigation domain, when the user selects a sensor to be included
in the system, code must be generated to deal with the sensor.
This is done through the use of "exemplars," which can
be viewed as templates for generating the configured code-similar
to an Ada generic package, except much more flexible. A copy of
the exemplar is made, and selected statements are tailored to the
desired software configuration.
Integration code can be viewed as code
that binds the realm code together. Once realm code has been
generated for all the realm components, integration code is
generated to link together (initiate the execution of) the realm
code. This integration code determines the execution model of the
system; it can create separate tasks to manage the realm
components, or it can instead sequentially invoke each realm
component in turn.
The final category of code is support
code, which in Ada provides the integration and realm code with
additional (static) services. This code includes definition
packages, global data, and common utilities used by the other
types of code. The only modification required for support code is
to change the values of selected constants.
In the case of the avionics navigation
domain, approximately 700 source lines of code (SLOC) must be
modified for a typical configuration after instantiating the
necessary exemplars. These modifications include the definition
of several large tables that are implemented as constant arrays.
The necessary modifications are straightforward but tedious, and
are easily automated.
ADAGE Tool Suite
The ADAGE tool suite consists of the following tools and
capabilities:
As stated earlier, for our experiment we only used the configuration and composition capabilities of the full ADAGE tool suite (a modified version of the programs GLUE and MEGEN were tailored to support the avionics navigation domain). This included instrumenting these tools using Amadeus [5] to facilitate metrics gathering. More information on the ADAGE tool suite, including a mock demo, is at http://www.owego.com/dssa/.
Configuration and
Composition Tool
GLUE is the tool used to configure and compose reusable software
components in the avionics navigation domain. MEGEN is the tool
used to automatically generate Ada source code once the software
has been configured. GLUE provides a GUI that allows the user to
specify a particular application's configuration in the form of a
design tree.
Figure 1 shows the main screen of GLUE.
It is divided into two major areas: the realm and component
selection area, and the component composition area. The realm and
component selection area in the top portion of the screen,
contains several list boxes that allow the user to select realms,
components, and component instances. The component composition
area in the bottom portion lets the user define the design tree,
depicting the dependencies between the software components.
Figure 1. A sample GLUE screen.
A design tree consists of component
instances. A software component may be used multiple times in a
design, and each use of a component is referred to as a component
instance. The branches in the tree depict dependencies between
software components. For unsatisfied dependencies, a hole labeled
with the name of the realm appears at the end of a branch. The
user must select a component from the specified realm to satisfy
the dependency. A hole appears as a rectangular inset area on the
screen.
A small black circle in a design tree
indicates a group dependency. A regular dependency is satisfied
by a single instance from a realm; group dependencies are
satisfied with one or more instances from a realm.
Throughout entry and editing of the
design tree, GLUE uses a C Language Integrated Production System
(CLIPS) rule-based expert system to verify that all constraints
on the software domain are satisfied. Once the software
configuration has been defined and all constraints satisfied,
GLUE invokes MEGEN to automatically generate Ada source code by
modifying the software components in the navigation domain for
the specified configuration.
Evaluation Approach
When conducting empirical studies, there is usually a trade-off
between obtaining fewer data points (commonly the result of
observations using larger projects) and more data points (the
result of using scenarios on larger projects or from using
smaller projects).
The "brute force" approach
involves redeveloping large projects completely-which is
expensive and results in relatively fewer data points. A hybrid,
selective approach examines only the most interesting portions or
activities of large software projects-the approach in which a set
of scenarios is conducted. The benefit in the scenario approach
is that more questions that regard technology effectiveness can
be addressed simultaneously. The corresponding risk is that
meaningful scenarios must be selected to provide a basis for
evaluation, based on intended benefits of the underlying
technology. In this case, the benefits are the ADAGE environment
and its ability to represent the software project lifecycle.
The scenario-based approach was adopted
to study ADAGE because it is not cost effective to shadow a whole
development project from start to finish. The empirical study was
driven by an experiment strategy using goals, scenarios, and
metrics; the intended benefits of ADAGE drive the goals and
scenarios to be examined. Multiple scenarios and multiple metrics
capture information about each potential benefit.
Experimental Participants
The experimental participants were four experienced software
developers from LockheedMartin in Owego, New York. Each had
extensive experience building applications in the avionics
domain. Two developers were team leaders on software projects;
the other two were team members on software projects.
Experimental Tasks or
Scenarios
We defined eight different experimental tasks or scenarios
according to the possible combinations of the following factors:
The experiment considered two types
of tasks: a maintenance task, in which an existing configuration
is modified, and a creation task in which the first version of a
new system is created. For the creation tasks, each software
developer was given a completed Software Requirements
Specification (SRS) and asked to generate the corresponding
software. For the maintenance tasks, the developers received a
marked up copy of the SRS with the changes indicated.
Two versions of each task type accounted
for different types of aircraft requirements. Due to time
constraints, the number of configured state variables and sensors
were much fewer than in a typical configuration.
Each developer was asked to
automatically generate the code using GLUE or to manually modify
the code. Again, due to time constraints, the developers were
asked to not manually modify the code, but to provide their best
estimates of the time required to make the necessary
modifications. Note that each developer was familiar with the
code design-the amount of time required to understand the
software design was not factored into the estimates.
Evaluation Criteria and
Metrics
During each of the experimental tasks, we collected the following
categories of data:
Some of this data was gathered by pre and post task interviews with the participants, and some was gathered by automated instrumentation of the GLUE environment.
Experimental Results
A summary of the results follows, each of which is statistically
significant. The factor of using GLUE vs. not using GLUE was
significant for all five evaluation criteria.
Summary
Overall, this experimental design used representative developers
on realistic tasks for the avionics domain. The empirical results
(a factor of 14 improvement in productivity and a 8:1 reduction
in errors), are encouraging for the use of Ada on
architecture-based software development approaches. These
improvements are not from reuse in the traditional sense because
the experiment contrasted manual and ADAGE assisted configuration
and the composition of parameterized components. The payoff
likely came because the configuration process (selecting
components and parameter values) had become "reusable"
and repeatable and because the composition process (substituting
parameter values and instantiating selected components) was
automated.
The existence of the decision tree not
only structured the configuration process and formalized the
instantiation process, but also because of the underlying
constraint checker and automatic generation, it eliminated two of
the major sources of error-selecting incorrect components or
parameter values and incorrectly (manually) updating the source
code. Additional time was saved because these kinds of errors
could only be found after the entire system was configured,
compiled, and taken to the laboratory for execution and test. But
gathering this type of data was beyond the scope of this
experiment.
About the Authors
Will Tracz is a senior programmer for Lockheed Martin
Federal Systems, and was a principle investigator on the DARPA
DSSA-ADAGE project. He is a member of numerous reuse councils,
boards, and steering committees, and has over 75 publications on
software engineering, microprogramming, and software reuse. He is
author of Confessions of a Used Program Salesman:
Institutionalizing Software Reuse (1995).
Lockheed-Martin Federal Systems
Mail Drop 0210
1801 State Route 17C
Owego, NY 13827-3998
Voice: 607-751-2169
Fax: 607-751-6025
E-Mail: Will.Tracz@lmco.com
Internet: http://www.owego.com/~tracz/
Lawrence P. Holmquist is an advisory programmer in the Advanced Technology Department at Lockheed Martin Federal SystemsOwego. His research interests include formal specifications and formal methods. He received a bachelor's degree in physics and mathematics from Macalester College, and a master's and doctorate degree in electrical engineering from the University of Minnesota.
(See Tracz address and fax)
Voice: 607-7517765
E-mail: larry.holmquist@lmco.com
Richard W. Selby is an associate professor of information and computer science at the University of California, Irvine. His research interests include software measurement, process, environments, and empirically based analysis techniques for largescale software. He received a bachelor's degree in mathematics from Saint Olaf College, and a master's degree and doctorate in computer science from the University of Maryland, College Park.
Department of Information and
Computer Science
University of California, Irvine, CA 92697
E-mail: selby@ics.uci.edu
References
This effort was sponsored by the U.S. Department of Defense Advanced Research Projects Agency in cooperation with the U.S. Air Force Wright Laboratory Avionics Directorate under contract F3361591C1788.