Two Air Force missile programs are bringing in a new era of acquisition reform that is providing the United States and allied war fighters with better and lower cost weapons. The heart of these programs is a more commercial business arrangement with the government and a single prime contractor in each case.

The first is the Advanced Medium Range Air-to-Air Missile (AMRAAM) program. It will yield an estimated life-cycle cost savings of more than $590 million from streamlined business practices, and optimized and significantly reduced government and contractor manpower, according to Judy A. Stokley, member of the Senior Executive Service, director of the Air-to-Air Joint Systems Program Office (SPO) Air Armament Center at Eglin Air Force Base in Florida.


AIM-120 advanced air to air missile

Prior to 1997, AMRAAM was a "Super SPO" manned with 325 government and support contractor people. Business was conducted the traditional way with intensive oversight of two competing contractors producing AMRAAM missiles and associated support equipment. Then in 1998 a buyout resulted in a single prime contractor, Raytheon in Boston, Mass., accepting Total System Performance Responsibilities (TSPR) with the government defining its enabling roles.

Another pioneer in the Department of Defense's (DoD) acquisition reform and streamlining efforts is Terry Little, program manager for the Joint Air-to-Surface Standoff Missile (JASSM) program. Little has incorporated numerous reforms in the JASSM development effort, and with its prime contractor, Lockheed Martin Missiles and Fire Control, Orlando, Fla., is dramatically reducing cost and acquisition schedules. He says current projections peg the unit cost of the missile near $300,000, compared with an original threshold cost of $700,000, and $1.6 million for its predecessor, the Tri-Service Standoff Attack Missile (TSSAM).

Little notes that the government will save more than $960 million (in 1995 dollars) in fixed costs over JASSM's production period. This is possible he says, because instead of mandating countless military specifications, the JASSM program has just three key performance parameters: range, missile mission effectiveness, and carrier operability. All other requirements are tradable to keep costs down, he says.

Both government programs also boast of acquiring a bumper-to-bumper lifetime repair warranty on the missiles.

To understand more completely the acquisition reform measures Stokley and Little have initiated, CrossTalk recently interviewed these two pioneers at Eglin Air Force Base. Following is a condensed version of their comments. For full comments see www.stsc.hill.af.mil

CrossTalk: How has acquisition reform shaped the acquisition strategy on your program? What impact has it had on product cost, schedule, and performance?

Stokley: When AMRAMM was established in the late 1970s, we brought on two competitive producers, then Hughes in Tucson, Ariz. and Raytheon in Boston, Mass. At the time we were in the cold war, and we planned a procurement strategy that was based on the Air Force and Navy buying 24,000 missiles in 10 years. When I came back to the program in 1997, the plan was then for them to buy a little over 10,000 missiles in 21 years. So the acquisition strategy needed to change completely because we had two full-up producers and factories, the government controlled all of the more than 370 specifications, and we mandated a build-to-print package to each of those contractors. They were in head-to-head price competition each year to build the missiles.

We began to discuss with the two contractors how to split the work to increase efficiency. Coincidentally, Raytheon decided to buy Hughes. When they did that, we had the real opportunity to set up a team business structure and partnership, and a long-term pricing agreement with Raytheon. Once one producer was responsible for the product, we were able to divest a lot of the cross checking and government control, and allow Raytheon to assume total system performance responsibility (TSPR).

We wanted to save significant dollars out of every unit's procurement cost... and simultaneously shift more of our appropriated dollars to buying missiles as opposed to buying overhead. And we wanted to significantly reduce the size of the government workforce required to execute the program.

In a year and a half we accomplished those three initial goals, and established a business framework that would last for the life-cycle of the program. We wanted to save 25 percent of average unit cost. We actually saved 30 percent. Then we rolled some of that back into investments, one of which was software modernization. We were also able to reduce our workforce by two-thirds.

Little: It [JASSM] has shaped the entire acquisition reform program. We have found it necessary to use entirely different than normal processes in order to achieve our goals. First of all we picked contractors based on past performance. In our case, past performance was equal to price and missile performance combined. We did not ask the contractor to provide us any description of his processes. We merely looked at his performance on recent relevant activities that were similar to ours in those specific areas that we felt were important to our program.

Next, we essentially redefined the government's role: to establish the requirements, select the contractor, and work interfaces that are outside of the contractor's control. There is no function that we have assumed for oversight, other than what I exercise and the procurement contracting officer. Their [government employees] responsibility is to make the contractor successful as a player.

The other thing we did is we have no processes required in our contract. It is strictly a performance specification. So we do not care how they do what they do, as long as they meet the performance requirements.

The end result is we are right now projecting it will be under the objective of $400,000 per unit. Our schedule is going to be about seven years, which is about 60 percent of what it has taken historically to do this, and the performance is equal to and some respects better than the predecessor program.

CrossTalk: What impact have the changes had on the way your software acquisition organization does business?

Stokley: We simultaneously looked at this program in two ways: the near-term problem, which was having two vendors, and too much infrastructure cost. We also wanted to set up the program for long-term performance gains for war fighters. One of the things we decided needed to be done was to modernize the processors used in the missile and to modernize its software.

The software was still in Hughes-specific assembly code, and the processors were Hughes-specific. Long term we felt it was important to move to a commercial processor so that there would be common processor architectures and people who could work with those architectures across the site. We received approval to reprogram part of our savings back into the program to convert to a Motorola 750. Raytheon is converting part [not all] of the software to C++ that has to be changed when we upgrade the system for electronic countermeasures.

The cost to upgrade the AMRAAM processor and re-host the current software in a commercial high order language is $20 million with a projected payback of $62 million dollars. This payback comes from two sources: lower Preplanned Product Improvement (P3I) phase 3 costs ($12 million), and lower software development costs ($50 million) achieved over the 15-year life cycle of the missile.

Little: Our contractor has TSPR. We have a warranty. Our role is to do the things the contractor cannot do, or that we can do better. Essentially it is [our job] to help him succeed.

In general, what the government engineer brings to the table is a broader experience than what your typical contractor has. In our case, it may come from working on other weapons systems programs where there are some lessons learned. In the case of JASSM, we have some people who worked on its predecessor program Tri-Service Standoff Attack Missile. In the software area, the processes and lessons learned are pretty codified. The government does not particularly have anything to add, and I have no dedicated software people in my organization. That seems to be working since Lockheed is doing quite well in the software area.

CrossTalk: Describe the business arrangement between government and contractor. How are various platform interfaces managed?

Stokley: First of all both [original] contractors already gave the government a 10 year bumper-to-bumper warranty. In key business shifts, we got away from annual pricing and established a long-term pricing agreement with Raytheon.

Now, what did the contractor get? We gave him control of the hardware below the missile performance specification. Previously the government controlled some 370 specifications that required a fairly lengthy government review approval cycle to change. He [contractor] is now free to make those decisions as long as he meets performance, continues to give us this warranty, and gives us the product at this price. The contractor can build off common hardware or common vendors with other products, which allows him to get some economic buys.

A second thing we gave him is self-oversight. He does his own verification testing and signs a compliance form when he sells the product to us. We do not have government quality inspectors on the floor. He does his own quality inspections and uses his own processes to ensure the quality of the product.

Lastly, we have a full and open partnership. We work with him to look at his books and financial health. We try to ensure we make smart business decisions together so he stays healthy. He looks at our government financial reporting, obligations, and expenditures to ensure that we keep our [contractor's] billing up and do not lose money. We formed an international business team with him so Raytheon gets to be part of our government team working with international companies to sell missiles.

Little: We have a contract, of course, but the contract does not define the relationship. It is kind of a constitution or a charter. The details of the relationship are defined by how you behave day to day. It is a very, very close team relationship characterized by collaboration, trust, full openness, quick illumination of disruptive influences, and open transmission of information. There is nothing I know, or that I think I know, or suspect that Lockheed does not know. That kind of open, honest communication has gone a long way in helping us work through problems that inevitably occur in any development program.

One of the things I have done with the government team is I have tried to destroy individuals' focus on their functional stovepipe. For example, it is unacceptable to me for a contracts person to say, 'The contract is my problem and as long as the contract's all right, I have done my job.' That is not the way teams operate. The team must operate to an overall goal-a goal that in our case includes schedule, system performance, life cycle costs, profit for Lockheed and its suppliers, as well as being a pioneering acquisition reform program.

Regarding interfaces, I have an entire integration team. Essentially it is one person per aircraft whose job it is to work the interface between that aircraft and our missile. Lockheed has a parallel arrangement with the aircraft and subcontractor. In the end the team produces an interface control document that everyone signs and adheres to.

CrossTalk: What other ways has acquisition reform enabled a reduction in the cost per product?

Stokley: There were three areas that allowed us to save a very large number of government work force. One, we eliminated cross checking and duplication where we used to do independent analyses to check the contractor. We got rid of all official data managers and all official configuration control managers. Because we do not control those 370 specs, we do not have all the data and reports. Instead, the contractor does this, and always had to anyway. If they are at the plant, my enablers will sit in on his configuration control boards as a part of his work activities. My folks are not there to check the contractor. They are there as co-workers and facilitators.

Little: The first [JASSM] systems off the production lines will be under $400,000-unlike the old theory that if we produce enough of these, costs will finally go down. Our results are not only due to up-front planning but also up-front effort. During the program's preliminary design and risk reduction phase we spent as much time and more money on manufacturing risk reduction as we did on performance risk reduction. Second is price-based acquisition. The contractor offered us a very attractive price for the first five production lots. He has the ability to make any change that he wants at any time without the government's OK-so long as it does not affect performance. He has the right to put in something that may lower his cost to produce the missile, but he must pay for it. We have no provisions for value engineering change proposals.

Furthermore ... when we decide to negotiate a fair and reasonable price for additional lots, we are not going to look at what his costs are. We are going to look at how his price compares to the price for similar missiles in the world marketplace. As long as we get a price equal to or better than similar missiles, we are happy.

CrossTalk: Do you feel there is a business case for Software Process Improvement?

Stokley: Not as a stand-alone item. First of all we do not buy software. We buy a well nurtured missile system that has software in it and includes: support equipment in the field, analyses of flight tests that fold back into production and repair lines, a warranty, and high reliability. So I think of buying a missile system. I do not think of buying software or hardware.

I think of process improvement more as an attitude that we motivate industry to take to keep this a healthy viable product that meets its requirements and is affordable. I have great worry and trepidation about singling out any one element of a program and doing process improvement on it. I think the parts of the program are so interdependent that it is easy to optimize one thing at the expense of something else, often unintentionally.

Little: No, I am concerned with product, not processes. I am also no fan of government or third party process or capability evaluation as a way of predicting future performance. During the JASSM source selection we evaluated the offerers' past performance in software development as we did on my previous program, the Joint Direct Attack Munition. In both cases Lockheed got very good grades from their customers on the timeliness and quality of their software developments. They also performed admirably in the work we gave them. I think, in retrospect, that their great performance was due to superb execution of processes that were only moderately mature. I have no problem with that. We have no software processes required in our contract. Lockheed has to meet a performance specification and a schedule; I do not care how they do it.

CrossTalk: What are the most significant lessons that have been learned under acquisition reform?

Stokley: Historically what the government has always measured for contractors who are doing the repair work is turn time. How long does it take the contractor to repair each unit? When we asked the contractor, given that we want 90 percent or better availability of all the missiles we bought worldwide, 'how can we get there?,' he said, 'Well, quit measuring turn time-grade me on availability. Let me decide how I run these units through the repair line.' So if 90 percent of them can go out in three days, and they need two months on one of them, we do not penalize them for the one that needs longer. It also allows them to take over sparing. So we changed contracts to manage availability and have been at 92 percent to 95 percent availability since.

Second... getting a certified price package normally takes about three to six months. We knew we needed to get the contract awarded on time, so we actually briefed the auditors and asked them to come do this in parallel with us. Instead of the contractor preparing the package and handing it over to government, everybody went and lived at Raytheon. We turned around the whole thing in 30 days and awarded the contract on schedule. It was the first big thing the team did together. It really showed them that they could overcome barriers, and the people bonded quite well during that.

Third... the program had suffered a budget cut in development; it was considerably behind in expenditures. So I went to my counterpart at Raytheon and told him it [budget cut] was because he was behind in his expenditures. He said he never knew what those were used for. After all those years of working with the government, our AMRAAM team did not know how the Planning Programming and Budgeting System (PPBS) works. So I asked my financial manager to prepare a PPBS tutorial and go out and brief the company [Raytheon] at all levels. Since then we have been green in every appropriation from 1998 through 2000. It is the first time in AMRAAM's history that we have not lost a single dollar to budget cuts.

Little: When I came on to the program... we had essentially done no preparations for milestones; the mandate was to award a contract in seven months. The team at the time said there was no way to do that in under a year. I told my folks we have one objective, and that is to award a contract in seven months, or we are all out of it. About three hours later people returned and said there was no way it could be done. I told them [on Tuesday] that on Friday, I wanted them to tell me what they were going to do- and they did. They said they were not going to do the traditional process. They wanted to do oral proposals, use past performance instead of lengthy process descriptions, and focus on things really important to the program. We did it and awarded a contract in six months. A lot of the things in acquisition reform will come when people perceive there is urgency.

Second, when we utilized past performance, a lot of people argued that we would have a protest; that it would be too subjective. In fact, we did have a protest, and it was a lot of work. It went to the General Accounting Office, but it was not sustained.

CrossTalk: Is there room for further acquisition improvement?

Stokley: Yes. Historically we have acquired a very large amount of government-furnished equipment over the two plus decades of the program. We are trying to whittle away at that and decide if we still need it, because some of it is very old. And of course someone is tracking it and paying for storing it somewhere.

Second, for government-furnished equipment that is still useful to the program, how do we streamline its management and accountabilities? We would like to transfer as much of it as possible to Raytheon's control since most of it is used in their testing and analysis processes.

Third we are still interested in doing price-based vs. cost-based procurement when we let our next long-term pricing agreement for 2002 through 2007 buys. We are exploring opportunities to use the war fighters' price requirement as documented in the Operational Requirements Document, as well as our factory-price model and our data from our cost as an independent variable exercise on Phase 3 P3I to justify price-based procurement.

We would still like to see a lot of streamlining in the international sales market and more use of direct commercial sales. I have a foreign military sales office here ... I contract with Raytheon on behalf of the [foreign] countries. One of my rules of life is "very seldom is the middle man a good thing," because every place you send money through, they will get some small share. But thus far Raytheon has not been allowed to conduct direct sales to many foreign countries.

Lastly, I would really like to see expansion of real acquisition reform across all services and all programs. It is very difficult for our industry to really grow and prosper as a result of acquisition reform if it is not implemented across a plant site.

Little: First, more widespread use of price-based acquisition. Second, more use of past performance in lieu of process descriptions. I know companies typically use proposal writers who know all the buzz words to put in a proposal about software processes, but these words say nothing about the company's ability to actually do software development or control software schedule. Past performance, while not perfect, is the best indicator we have for future performance. Third, we need to break down the barriers that preclude predominantly commercial companies from doing defense software.

The biggest problem we have in software is not a software process maturity process; it is the ability of the predominant defense companies to find or retain highly qualified software engineering people. As a solution to this problem, I would like to see our defense prime contractors look at subcontracting some or all of their software development to commercial companies with a proven track record like Computer Associates, Microsoft, or Oracle. Right now this would be difficult because we have barriers that would preclude defense contractors from going to commercial companies for development-barriers like cost accounting systems and others associated with the way we do business.

Judy A. Stokley is the program director of the Air-to-Air Joint Systems Program Office, Air Armament Center (Air Force Materiel Command), Eglin Air Force Base, Fla. She leads a large extended team of civilian and military personnel, as well as several major defense companies, in delivery of critical systems to the warfighters. Stokley began her Air Force career in 1979 as a mathematician in the Armament Laboratory and, subsequently, worked in development planning and system program offices. She has a bachelor's (University of Alabama) and a master's (University of West Florida) degree in mathematics. In 1991 she graduated from the Defense Systems Management College at Fort Belvoir, Va., and in 1993 was approved for membership in the Air Force Acquisition Corps. In 1999 she was promoted into the Senior Executive Service.

Terry R. Little has been the program director for the Joint Air-to-Surface Standoff Missile Joint Program Office since 1996 and has more than 15 years of experience as a system program director for major programs. Previously he was the program director for the Joint Direct Attack Munition Program Office. Little entered the Air Force in 1967 and was a distinguished graduate of Officers Training School. Before becoming a civilian employee in 1975, he served on active duty as an aircraft maintenance officer, a computer systems design engineer, and an acquisition program manager. As a civilian employee he has been an operations research analyst, a program director for a classified project, a deputy program director and a weapons development planning manager. Little has a bachelor's degree in mathematics (University of Texas), a bachelor's degree in English (Southern Methodist University), a master's degree in systems analysis (Air Force Institute of Technology School of Engineering), and a master's degree in business administration (University of West Florida).

During her interview with CrossTalk, Judy Stokley, AMRAAM director, also addressed the following questions.

CrossTalk: What were your three main goals of Vision 2000? How successful were they?

Stokley: We wanted to save significant dollars from every procurement cost. At the time, we had a lot of infrastructure cost in the program so the unit cost was quite high. This was because the quantities had decreased significantly, as had happened with all systems after the fall of the Berlin wall, and the Department of Defense budget changed. We wanted to drop that [cost] down and, second, simultaneously shift more of our appropriated dollars to buying missiles as opposed to buying overhead. Third, we wanted to significantly reduce the size of the government workforce as required to execute the program. In about a year and a half we accomplished those three initial goals. We also tried to establish a business framework that would last for the lifecycle of the program. We wanted to save 25 percent of average unit cost. We actually saved 30 percent. Then we rolled some of that back into some investments, one of which was in the software area. And within one year we were also able to reduce our workforce by two-thirds. So we met our initial goals very quickly, and set up this long-term business strategy that is working quite well.

CrossTalk: How do you check contractor quality, or is it a complete trust relationship?

Stokley: It's both. First of all we use the insight of government engineers who live with the contractor working with his engineering force who are doing the design and upgrade work. The second way we track quality is through testing. There are more than 100 AMRAAMS fired each year by various test agencies, primarily field operators who shoot the missile down with what is called the Weapons System Evaluation Program at Tindle. As far as quality in manufacturing on the floor, although we're out there with the contractor all the time, we don't do any government cross-checking of his product.

CrossTalk: Do you see contractors using more off-the-shelf software?

Stokley: I think he does it, but I don't think he invests a lot of his own money to go out and search for commercial applications. The one thing, since our system undergoes such significant environmental challenges on airplanes, including vibrations, acoustics, heat, and cold, it requires a significant search for a lot of change to commercial parts. So unless it just turns out that there is something that comes along, I don't think that there's a big shift. After all, we're not in original development where you go out searching for the parts, we already have them along with established vendors.

CrossTalk: How do you determine what the contractor's financial health should be?

Stokley: That's a bigger picture question. How did we come up with all these things to do under the Vision 2000 business structure so there is benefit to the contractor and benefit to us? There are four tenants in Vision 2000. First, it is a win-win business strategy. We look at things and acknowledge what is the right thing to do. Then, how do we make it work for both the government and the contractor. The third tenant is teamwork and trust. Fourth is that the contractor has full system performance responsibility. He has control of capability for his product.

Now how did we come up with all this vision and business structure that make this thing work together? It is really quite interdependent, and yet part of the puzzle. We realized that when we began analyzing the program in 1996 that cost had gotten out of control. We had way too many government people and were not getting enough product. Then when Raytheon purchased Hughs, we went through a somewhat traditional approach to identifying as a team with Raytheon our vision, our objectives, and what would enable us to achieve these objectives. Things like self oversight and open financial books came out in this environment. Raytheon trusted us enough to explain how they are graded within their corporation regarding profit margins. I opened up to them what my budget process is and what briefings I take to congressional staffers. So we really opened up our worlds to each other. We defined how we would reach this Vision 2000.

We have continued our off-sights together, using one facilitator now for three years. We do two off-sights a year. One is to establish that year's specific goals and objectives. A second evaluates where we are that year and identifies any problems or barriers, and also identify places to improve next year when we set our goals. It is a continual ongoing process that requires quite a bit of nurturing. You don't just set a business relationship, walk away and let it execute itself. It is not static. We're always working together identifying how to make this work better and what has changed in our environment.

For example, when we established Vision 2000 we were only authorized to sell the AIM-120 B missiles to our international customers. A big change this year is that we were approved to sell the AIM-120 C a later block configuration. So that changes the business mix of our unit. Now we want to go out and visit all these countries and offer them the missiles. So one of the thing we did this year was form a new international team to go out and brief 19 countries around the world on this new missile product.

CrossTalk: Did these additional international sales contribute to decreasing per copy cost of AMRAAM along with the other measures you mentioned earlier?

Stokley: The way we did that is our savings were calculated on a base FMS quantity. As you know, FMS quantities can be quite volatile year to year. We set a pricing model around that so we're able to price the product plus or minus certain quantities. Depending on how many we sell we get a better price or more expensive price. We were quite fortunate since we had very good pricing model for changes in quantity based on historical data. We sized the program on U.S. production then modified costs up or down as our FMS goes up or down.

CrossTalk: Did the maintenance concept and warranty cut back significantly on deliverables that you typically see in a government contract such as software test reports and thus contribute to the savings?

Stokley: The contractor had always been the repair agency for the missile. We never had an organic depot. And we have had the 10-year warranty for a number of years. We did do some streamlining in sustainment, but I don't want you to think that's the only reason we have all those no CETRALS (?) We have almost no data items on this product. We don't get any reports, other than a safety report. We have an electronic data repository that both the contractor and we can access, which we use for all reporting and monitoring processes.

CrossTalk: Is the jury still out on ACT reform? As systems get into maintenance and sustainment phase, do we know that it will be successful?

Stokley: We used JDAM very successfully in Kosovo, which had been developed following acquisition reform business practices. So I think we have seen in a few cases that you can certainly deliver a good product that works under this regime. We haven't had any of those business practices in place for 20 years, so we can't certainly say they are proven for that time. My own view is that what we know about human nature in business motivations is that the less you fragment work and clearly identify who is accountable in name and cost, then the better results you'll get.

When we get companies under ACT reform to truly be accountable for long-term price and warranty, it is difficult for me to see how that can be less than a good thing. I think it is much better than the way we used to do business, which was to fragment the work so much that it was hard to determine who was accountable for the end product. I think you'll see things improve in the areas where we have fully implemented it.

CrossTalk: Does a higher level of process maturity allow the contractor to provide a better product and instill confidence?

Stokley: In my experience I have managed several major activities that were very software intensive. I think you're talking about the Software Engineering Institute's (SEI) levels of certification. And I've been through all that process. I believe that the SEI level certification is a very poor measurement of how well they [contractors] perform. It's an exercise that makes everybody feel good. But I've not seen that it is very indicative of how different teams within the company really work.

You can go into a company that is certified at Level 3, and on your particular program they can have a terrible situation for software development. And you can walk across the hall to another program that will be on track with very good metrics, structure and flow-down requirements, verification processes, and be writing test cases ahead of time. Why? The company is Level 3; the same functional office approves both groups.

So I think this certification thing is at best some really aggregate level indicator. But I would never rely on it as an indicator of how my team is going to perform. What's important to me is not what level the company has achieved, but how my team's performing. What I look for is a good flow down of requirements, a good structure to identify the work and that it is being accomplished, and a clear verification process to determine that the work is correct. I wouldn't choose or turn down a contractor based on whether he was Level 2, Level 3 or Sigma 6.

CrossTalk: How do you choose a contractor to open your entire books and life to knowing it will work?

Stokley: I have worked with basically every major company in this country: Boeing, Lockheed, Martin Marietta, Raytheon, and numerous small companies including Marvin, Alliance, Chamberlain. So far I have not ever been treated unfairly or unethically. And I've never felt that I was treated in any way dishonestly. I always go in believing that companies are in the defense business to make a profit; and also, because they have some views about patriotism and doing what's right for our country. So far I have not been disappointed. I have always gone into my work with other government agencies and industry believing that if I'm honest and straightforward with them, they will be with me. And whatever happens, I'll get better results than if I had tried the opposite approach. If someone isn't going to work well with you they're sure not going to work better because you're hiding your motivations and your data.

That isn't to say it's trouble free, that me and my counterparts don't debate sometimes. Debate is healthy. Sometimes I have to give, and sometimes they have to give. Chuck Anderson is my counterpart at Raytheon now. He and I communicate this to our team by saying debate is OK, disagreement's OK. We want to talk about it and figure out the right approach. We always ask first, 'What is the right thing to do for the war fighters and the tax payers?' Once we define that, we'll get to who's going to pay for it.

The very last thing we ever do is figure out if it's under a contract somewhere. I never look at my contracts, and I don't expect my industry partner to be off reading his. We should be deciding what's the right thing to do, and who can afford to pay for it. We'll pool our resources however necessary to get the right thing. If we can't do the right thing for the war fighter and tax payer, and it's just way too expensive in this case, then we're going to put our story together, go up the system, and say, 'Here's some things we can do and some things we can't do.' So far that's worked. So I just don't have that fear. And it troubles me that so many people seem to have this fear.

CrossTalk: How do you use your government people?

Stokley: Government people handle several really important things that I don't think we can expect industry to take on. One is working with combat pilots to establish their operational requirements. Then they translating those into missile performance specifications. We serve as the bridge between industry and our war fighters. The government technicians have to get enough understanding from the contractor to know what's possible with technology. Then obtain enough understanding of the war fighter's capability to build this bridge that flows from requirements to specifications.

Their second job ... is we facilitate and manage the interfaces with a variety of government agencies, one being aircraft program offices and aircraft contractors. We try as much as possible to get major fighter manufacturers to work together on interface management. Remember the weapon and the two airplanes have different budget line items. So we're constantly trying to ensure that with our different budgets and requirements the weapons are going to be properly integrated and fielded so they work properly when they hit the field together.

Third we handle a lot of field activities including safety for field, product and test ranges, and environmental requirements at different bases where the missile is stored and used including other countries. Interface management that requires a technical knowledge on both sides is a lot of what my folks work on.

Lastly, since we do launch a large number of missiles annually-well over 100-a lot of coordination is required to get all the data from those missile shots to Raytheon. There are classification issues, depending on airplane and test programs, and questions about who controls the data. We need to understand what the scenario was and what the war fighter was trying to achieve then get the data to Raytheon so they can analyze the shot. An important part of Raytheon's job is to monitor the performance of fielded missiles and ensure that if any corrections need to be made, that gets folded back into the development and repair line.

CrossTalk: What are the most important factors to look at when beginning a project that indicate which contractor will provide the most opportunity to succeed?

Stokley: The first thing to look at is past performance. Under ACT reform we made our past performance evaluations as program directors mean something. Today we have this neat line that says, 'If I were going to award a contract in the future, I would, probably would, probably would not, would not, award again to this contractor.' The only person allowed to check one and put their name on it is the program director. It means something when program directors write it down. We take how we do past performance evaluations a lot more seriously now so that they have become more useful.

Second is to get a grip on evaluating the contractor's motives. 'What is the corporations real commitment to this product?' 'Is this one of many this year, or is this a really important thing to them?' 'Are they going to go the extra mile?' 'What is their motivation?'

CrossTalk: How do you measure contractor performance and stay alert to foreseeable problems?

Little: We incorporate incremental testing throughout the program, including ground testing, hardware-in-the-loop testing, captive testing, and ultimately flight testing. Plus, we are committed to having an early production representative system. We are able to look at component deliveries and assess where we are.

One of the dramatic things we did differently is that our program is front-end loaded in terms of funding and effort. We have spent 70 percent of our total development budget, and we have yet to have our first flight test. We focused on maturing the system, so that when we actually got to flight testing, we would not be testing a prototype that would still have to undergo manufacturing development and continued refinement.

It's called "concurrent engineering," which is taught in software engineering schools and as a program management course. It's just that nobody ever does it because the pressure on schedule and money causes everything to be pushed out except those things directly related to testing. The end result is that a lot of programs are back-end loaded causing a lot of changes to the system. This makes for a very long schedule, and a very difficult transition to production. In our case, factory people using factory processes, and the same for our supplier parts, produce our very first development units in the factory.

By the way, we have had absolutely no problems with software. We are ahead of schedule in a fairly formidable program. It involves not just the missile's operational flight program, but the seeker algorithms. With these the seeker finds its own target and is able to compare what it sees to what it thinks it ought to see.

CrossTalk: Do you attribute your lack of software problems to choosing contractors based on their past performance?

Little: We begin with good software requirement definitions up front based on missile performance requirements. Then we have a posture here that after defining the initial requirements, there are no changes until or unless we go through a very bureaucratic requirements control working group. I'm not talking about user requirements necessarily, but the kind of program requirements that come from some ones interpretation of the contract or what some engineer decides might be a better thing to do or not to do. We don't change anything unless we know its impact to the program and we are willing to accept that impact. So we maintain a very, very stringent control of the requirements.

When legitimate requirement changes arise, such as upgrades in the software area or additional capabilities, we use a block change approach. That means we'll do the change and we're willing to pay for it, but it's not going in the first systems; and it's not part of our development. There is a very deliberate evolutionary approach, but we're not trying to do everything at once to satisfy everyone.

My own experience is that a lot of upgrades and changes come from people who are not really day-to-day users, but who look at what might be possible then theorize a use situation. When you do that without any financial accountability, you end up with continuing requirements coming to the top. My own view is that once the system gets out in the field, the real impetus to change, upgrade, or alter the performance will come from the day-to-day users.

CrossTalk: Has part of managing software development been to get your requirements process under control, whether you've followed CMMI or not?

Little: Yes, but that doesn't have anything to do with the contractor per say though. We've gotten it under control because we've tied a financial accountability to changes. We have a situation in our business where users establish and change requirements and for the most part don't assume financial accountability for that. What we've done with this requirements control working group is that we've essentially created a situation where people have to confront the financial and schedule implications of changes. Your willingness to change when you have to pay for it is always going to be different than it is when you don't have to pay for it.

What the contractor has from us is performance requirements for the overall system. He's the one that allocates the software, hardware, and mission planning. There is absolutely no government involvement in allocation.

CrossTalk: Is the government team nonexistent in the software model?

Little: Yes. We do have government people who are part of interface control working groups. These groups create an interface document that consists of the mechanical, electrical, and logical interface for each aircraft that uses the missile. Once that document is signed, it is a commitment on both our parts, and the aircrafts parts, to develop to that specified interface.

CrossTalk: How do you measure contractors' past performance? Are you looking at the contractor's past work or talking to his clients?

Little: Initially we ask each of the bidders to provide recent, relevant performance in several different areas, including cost schedule, aircraft integration, software development, and production support. In each area contractors provide three recent contracts that they think are most related to the job we're bidding. Then we negotiated with them on their choices because they have a natural tendency to want to pick and choose ones where they thought they would have good performance. Then we defined what we wanted to look at specifically. In software development it was functionality and scheduling. We asked contractors to assess these, then we went to their customers for each of these. You essentially have to develop an algorithm where you can compare contractor performance not in a general case, but in software performance that is most analogous to what you're doing. It worked great for us.

CrossTalk: Who conducted the analysis, government people?

Little: Yes. But you don't need software government people to do this. It's a matter of, 'Did you do what you said you would do?' They were all technical people. The lead of the entire past performance evaluation was a software person, my chief engineer. But that was not planned.

CrossTalk: What parameters in software development were used to rate past performance?

Little: Essentially functionality, that is, 'Did it do what you said it was going to do at the start-a promise made a promise kept? And did you meet the schedule that was laid out?' For comparison purposes, we also had to incorporate implicit weighting to balance context. The other thing we did, after we picked the two contractors based on price and missile performance was to have each contractor work for a period of time to evaluate production, price, and performance. 'What did you say you were going to do here vs. what you actually did?'

CrossTalk: Do you feel the jury is still out on acquisition reform?

Little: There are a lot of people in the department who want to say the jury is still out, or even that it's failed. Maybe most people. I don't believe the jury is out. We have three good examples of acquisition reform programs. The results are going to continue to be dramatically different from previous programs. I believe the primary reason our systems take so long and cost so much has to do with how the government does business. It is not what the systems are, but how we have chosen to buy them.