(Editor's Note: One curmudgeon per issue is generally enough for anyone. However, there was so much interest in Tom DeMarco's slightly irreverent article that we are reprinting it here. Copyright (c) 1993 IEEE. Reprinted with permission from IEEE Software, Vol. 10, No. 2, pages 89-90, March 1993.)
In the absence of meaningful standards, a new industry like ours comes to depend instead on folklore. As the industry matures, the first order of business is to recognize and question the folklore. For example, how you ask and how you answer the familiar question, "Why does software cost so much?" tells much about what folklore you've grown up with.
Author and consultant Jerry Weinberg claims to have encountered this question more than any other in his long career [1]. The correct answer, he says, is "Compared to what?" There is a likable logic to that: Most of the things we do with software in the '90s are barely conceivable without software, so there is no valid basis of comparison.
Yet Jerry's answer, charming as it is, won't do you much good. At best, it will just annoy the questioner. No answer is satisfactory because people don't ask that question to get an answer. "Why does software cost so much?" is not a question; it's an assertion. The assertion is that software costs too much.
The person who poses this nonquestion may seem to be motivated by intellectual curiosity: "Gee, I've always wondered, just why is it that software costs so much?" The real motivation, however, has nothing to do with curiosity and everything to do with getting the brutal assertion on the table.
It's a negotiating position: You are being put on notice that software costs are unconscionable, and no budget or schedule you ask for will be considered reasonable. Your boss or user may agree to your budget, but only under extreme duress. Since the amount budgeted is already terribly, terribly excessive, any slip or overrun is virtually a crime against nature.
In a recent interview [2], Cadre Technologies founder, Lou Mazzucchelli, observed that "software consumers are not satisfied with either the quantity or the quality of our output." Right on target. Software consumers in vast numbers are telling us that our efforts don't begin to measure up to their expectations. Software is much too expensive, takes too long to build, isn't solid enough, isn't easy enough to use, isn't good enough in any way.
I have a very grumpy question for those who complain that the software- development community hasn't measured up to their expectations: Where in hell did those expectations come from?
A Little Diatribe
You and I, and others like us, built the software industry from scratch over the last 30 years. Out of thin air, we made a $300 billion-a-year business [3]. In all of economic history, there has never been a more staggering accomplishment. $300 billion a year! Think of it. In the time it will take you to read this article, the software industry will generate something more than $12 million.
What has it taken to build this huge new industry so quickly? Hint: It wasn't just getting some programmers together and teaching them to sling code. It required the active participation of a marketplace. Somebody had to toady up huge quantities of money to buy all the software we built. And they did. Not only did they buy all we could produce at the cost we charged, they complained about not being able to buy even more, about the so-called backlog.
This growth was not the result of poor quality and poor productivity. The only conceivable explanation for the phenomenal success of the software industry is that it regularly delivered a quality and productivity far beyond the market's real expectations. But all the time that our buyers (our managers and our users) were lining up to cash in on the bargain, they were complaining. This behavior is not a recent phenomenon: They didn't congratulate us for years and years and then become upset only during the downturn in the '90s. No, they complained all the way from zero to $300 billion.
I am a bit peeved by this. (Perhaps you can tell.) I feel like we have accomplished wonders and have been yelled at the whole time. Through their actions and their words, our buyers have sent diametrically opposed messages.
Imagine how the Wright brothers would have felt had they had a similar reception. You are Orville, it's Dec. 17, 1903, at Kitty Hawk, N.C., 7:30 a.m., and you're climbing up into Flyer One. "Let'er rip," you say, or words to that effect. The engine coughs to life. You rev it up, and there is movement. There is not just movement, there is speed. Speed and bumps and wind ... by God, you're up! You've done it. You've pulled off a miracle, and the world will never be the same. You are so elated you're barely even afraid. Cool as a cucumber, you bring the flyer down.
Just as you are coming to a stop, you notice a guy in a business suit, looking sourly at his watch. He says, "Orville, I'm really disappointed in this project. I had great expectations, and you've let me down. Here it is nearly 8 a.m., and I have to be in LA for a dinner meeting tonight. And you guys are nowhere! You haven't invented the jet engine, or the flight attendant, or the airport, or those cocktails in tiny bottles. You've let me down completely!" This is the guy Mazzucchelli was talking about.
Where Expectations Come From
No one expected the software industry to achieve what it has achieved. Not a single futurist predicted the extraordinary productivity and quality we have accomplished. How is it possible that the industry is performing beyond our wildest expectations while every individual project is underperforming?
It's not. Those projects aren't underperforming at all. And their consumers know it. Pay attention to what they do, not what they say. The real message our consumers are telling us is that software is the best bargain they ever heard of. They complain to us because they know we work harder when they do complain. We have trained them to do this. When they complained in the past, we worked harder. We gave them more for their money (even more than the extraordinary bargain they would have gotten anyway) because they pretended to be discontent. Boy, are we dumb.
In a recent article, Albert L. Lederer and Jayesh Prasad set down some guidelines for better software cost estimating [4]. Their method involved a survey of some 400 professional software managers. What interested me was a tiny nugget tucked away in the commentary: The great majority of respondents reported that their software estimates were dismal--only about one in four projects come in at a cost "reasonably close" to the estimate. However, 43 percent said their current estimating method was "very" or "moderately" satisfactory (the two highest ratings).
What's going on here? Estimates bear little or no resemblance to reality but managers aren't dissatisfied? I think I'm beginning to understand. Maybe the purpose of the estimating process is not to come up with a realistic answer, but to come up with an unrealistic answer. Maybe the estimating process is not supposed to guide the manager as to what to expect. Rather, it's supposed to guide the manager as to what to pretend to expect.
This is the kind of process that tells your boss, for example, to set September 1994 as the expected delivery date. This "right" schedule is neither ridiculous nor feasible. That is, after all, the object of the exercise. The "right" schedule is one that is utterly impossible, but not obviously impossible.
How Do Managers Know?
It is a great tribute to quality software management that managers know how to set this "right" schedule. It isn't easy. In the abstract, at least, it is just as hard to predict a date that is just short of possible as to pick one that is safe and reasonable. Both require a prediction of product size as well as a correct assessment of team capability. How do our managers learn to do this? Again, we have trained them. They watch our faces when they set schedules. If we look relieved, they know they haven't turned the screws enough. If we just giggle, they know they've gone too far.
The assertion that software costs too much is part of a cost-containment ploy. The cynical notion that a "right" schedule is one that no one has a prayer of achieving is another part of that ploy. The constant refrain that software developers just are not productive enough is a goad. It appears to work because software developers are sincere and professional and a little dopey. The problem is that our industry is overgoaded. Our work is largely incompressible. When you're under pressure, your first response is to cut out extraneous activities: chats and bull sessions.
That may indeed be productive, but that's the end of it. As the pressure continues, there is nothing more you can do. You can't work faster; that just isn't possible. You might stay later, but that has a long-term cost: Overtime applied over months and months gives only an illusion of progress that is wiped out by compensatory "undertime," burnout, disillusionment, waste, and employee turnover.
The Moral
As time pressure increases, the only real option is to pay for speed by reducing quality. I sometimes think, rather bitterly, that reduced quality is a conscious goal of those who pressure projects. They are saying, "Loosen up, folks--learn to rush the product out the door without worrying so much about quality." This comes at the very moment that companies are paying lip service to quality as never before.
In the short run, paying for speed by reducing quality makes sense. In the long run, it will take us where it took the U.S. automobile industry. We might win a few battles, but not the war.
If you ask the wrong question, you'll never get the right answer. Instead of asking, "Why does software cost so much?" we must begin to ask, "What have we done to make it possible for present-day software to cost so little?" The answer to that question will help us continue the extraordinary level of achievement that has always distinguished the software industry.
Tom DeMarco
Atlantic Systems Guild, Inc.
P.O. Box 160
Camden, ME 04843
Voice: 207-236-4735
Fax: 207-236-8432
Notes