(Editor's Note: Booch presented this address at the Software Technology Conference in Salt Lake City, Utah April 10, 1995.)
Combat Coding
Shortly after the Gulf conflict, a friend of mine called me and introduced me to a new phrase that I will share with you, because I think it was really profound. This gentlemen was a reservist and was called to duty. He was a programmer by trade, and that was his military specialty as well. It seemed that the mission he faced was fairly straightforward. Many of the protocols for communication in that arena were such that there was a diversity of standards. So his mission was to produce a simple software program that would do a translation from one set of packets to another.
So here he was, brought in to a plane on the East Coast. He was armed with a laptop and object-oriented programming language suite and given about 24 hours--from the time he took off till he landed--to have this piece of software operational. It's a phrase he used, that his mission was combat coding. And I love that phrase, because it really points out that sometimes there is a need for heroic programming efforts.
The other side of that story is, though, that heroic programming is not a sustainable business. If I look at an application, a system of the magnitude of BSY-II, the Canadian Air Traffic Control System, or any enterprisewide modeling for any enterprise, the reality is that software of that magnitude is inherently complex and requires practices that are fundamentally different than what I find in small systems. The lesson here is, to quote a great American statesman, "Large software projects are like a box of chocolates; you never know what you're going to find." And so it is.
Good News ...
What I'm going to share with you today are some of my perceptions of the practical aspects of software engineering. There is some good news and some bad news in this regard. The good news from the perspective of the software developer is that software is absolutely pervasive. There is no end to the desire for software. As Sen. Orrin Hatch pointed out, it is pervasive throughout all kinds of industries. This means that there is an insatiable desire and an insatiable demand for software. As a software developer, that's great because it means I'll certainly be busy over the next several years, if not decades.
And Bad
The bad news is--and it's not difficult to predict this--all future software is going to be evolutionarily more complex, and I think there are two trends that push us in this direction. The first of these is that we have increased connectivity of distributed computing systems. That same technology leads us to the opportunity to build more complex systems. The second is a more social aspect, which is we find end users--who are becoming more aware of the power of computing--are in a sense demanding greater access to and visualization of data. Those two forces lead us to the natural evolution of more and more complex software systems.
Why Some Software Projects Fail
It's not difficult to explain why software projects fail. I will spend a little time on this, because I'd rather be more proactive and tell you what seems to work in terms of the successful projects we encounter and what makes them succeed.
One of the ways in which I have been graced over the last several years is the opportunity to work with a multitude of software applications in both the defense aerospace world as well as in purely the commercial world. These three lessons I find applicable across all of these domains. The three major reasons for failure across any kind of software project tend to be
Let's look on a more positive note. Rather than to figure out why projects fail, let's take a look at why projects succeed, especially the larger ones. We begin this effort by taking a look at what drives most projects.
The Focus and Culture of Each Project
One of my colleagues calls this the essential minimal characteristics of any application. We find things like time-to-market, completeness, performance, and quality. Reality is that every system tends to have a number of drivers like this. However, it is absolutely impossible to optimize every one of those. If I'm in a shrink-wrap market, I'm more focused on time-to- market issues. If I'm building something for the PC for example, then the other elements will tend to be less important to me. Certainly, quality is still important. Certainly, performance is still important, but driving time- to-market is going to be the dominant factor.
|
| Table 1: Establishing a Project's Focus. |
|---|
Whereas on the other hand, if I have a system that human lives depend upon, time-to-market may be a critical thing--in terms of its availability--but it's going to be secondary in terms of some of the requirements I have to meet in order to field that thing. Again, reality is I cannot optimize every one of those characteristics at once. And as Dave Parnas has pointed out, the net result of that is, if I begin with a system and a concept, there is no rational process that leads me from that vision to its implementation, because we really do have such competition among these various factors.
This has led me to realize that for every project I have encountered, there tends to be an insidious different culture in that organization. I'm not saying any one of these is better or worse than the others; they are just different. Let me point out some examples.
Calendar-Driven
Many projects in the commercial sector that I encounter tend to be calendar-driven. This is not to say that schedules are a bad thing. But it is this obsessive focus upon the calendar, setting arbitrary dates that have no essential meaning--other than just being a date--which leads you to a focus in which you optimize for the short term rather than for the long term.
Requirements-Driven
Many of the applications I run across in the defense aerospace sector tend to be requirements-driven. Schedule is important, but it's somewhat secondary relative to the kinds of requirements I must meet. Let me pause here for a moment and suggest that requirements-driven projects inherently tend to lead you to architectures that look like this (see Figure 1). This is actually a bad thing. This is what some call stovepipe architectures, and it's kind of inherent in requirements-driven processes where I look at a requirement, and I map it to a piece of code. I generally end up with this structure because you have a small infrastructure at the bottom--generally some piece of data and a bit of networking--but the requirements get satisfied in these tall stovepipes. That's a bad thing, because these kinds of architectures tend to be resistant to change. There is a lot of high inertia in these kinds of architectures.
By the way, this kind of architecture is not just one we find in a lot of requirements-driven projects in the defense aerospace world. We find this in commercial organizations that tended to have grown by acquisition over the last several years. The classic case is a large telephone company that I encountered that basically acquired more and more pieces of the organization by buying other companies, and they ended up with a system that had no enterprisewide vision to it, but it was defined in a lot of these little fiefdoms in a stovepipe architecture, but resistant to change.
Documentation-Driven
A degenerate case of this culture is what I call documentation-driven; it's similar to a requirements-driven culture but even worse, because here it focuses not so much on generating software but generating documents to satisfy some end user who specifies them. What you find happening in these kinds of cultures is the development team goes off and writes these documents, spends a lot of time doing it, forgets the vision of what they're building in the first place, once it's delivered, and nobody bothers to read it. Then, they have to spend all this time catching up in the software domain. So, it's really the degenerate side of it.
Quality-Driven
In some human-rated systems--the classic case is where I might find in the commercial sector of pacemakers and medical instrumentation or in the defense aerospace sector of avionics systems for an aircraft--quality is the fundamental driver. Schedule and requirements are important, but since human lives depend upon it, I'm going to subtly change the culture of how I build the system, because human lives do indeed depend upon my success or failure. In the applications I've encountered, I run across every one of these cultures.
Architecture-Driven
But, I can honestly say that among the successful projects I've encountered across the world, they tend to be embodied by a culture that I call architecture-driven. I will spend a little time on this, because it may look fundamentally different to you. Contrast this (Figure 2) with the stovepipe architectures we saw earlier. This is what I find to be the canonical architecture of most well-structured object-oriented systems. I notice one fundamental difference to it. The tops of these systems tend to be small. As one of my colleagues, Bertram Myers, says, "Real systems have no top," because what you have below this is a large infrastructure. The exciting thing is, we are beginning to see a lot of off-the-shelf components for a variety of these domains. I will go from the bottom up.
In this infrastructure, we see things such as just base operating systems. Reality is, most of the world's next generation commercial operating systems are in one way or another going be object-oriented both in terms of their presentation to the user as well as their underlying architecture. The good news is, this means we have a lot more entry points to those kinds of base- level operating systems.
There is a similar revolution going on in the areas of dispersed data storage, the realm of relational databases, and the like. We find on the horizon objectivism database technology, which is caught upon a different kind of model. We find even among the Sybases and Oracles of the world the evolution of a standard called SQL-3, which brings object-oriented extensions to that kind of domain. The new result is that the storage of data becomes something that is a piece of technology that I buy. I have this concept, this piece of data. I touch it in a special way, and it sticks. That's the notion of persistent data stores. And it's coming off-the-shelf.
Moving up the food chain, remember one of the things I said earlier was about the pervasiveness of distributive computing. Again, we see codification of some standards in this area. Things like UNAS, things like CORBA, and the common object request broker all represent notions of distributed object computing, which says I've got this piece of data; I really don't care where it lives; touch it in a certain way, and no matter where it might be in the world, it comes to me. Again, a piece of technology that is becoming available off- the-shelf.
Moving even higher up the food chain, we see things like domain-specific architectures. I'll come back to this point later, because it represents a different kind of reuse than we have seen in the past, a reuse of a whole enterprise, and that's a good thing, because it's a higher leverage reuse than we've seen. I'll come back to that one in a bit.
Mitigating Software Failure
If there is one thing you must understand from my talk this morning, it's that no one element of technology is going to help us mitigate these software risks we have in front of us. I would claim that there are at least these four issues that every successful organization must cope with: languages, tools, methods, and processes.
I remember talking a few years ago with Dr. Larry Druffell, who then was at the Ada Joint Program Office. Around that time, they were dealing with Steelman Requirements for Ada; they were taking a look at a thing called Method Man, which is a method to use the Ada programming language. Larry commented to me that in a way, we have gone about this backwards. We defined the language first, then the environment, then the method. What we should have done is define the method, then the environment, then the language. No matter which direction we apply it, reality is, there must be a balance among all those technologies.
Languages
Let's start with languages. What I show here (Figure 3) is a genealogy of languages, and it points out that over the last 10 years or so, there has been a fundamental shift in language technology. Before, we've primarily had nonobject-oriented or procedural programming languages, best typified by FORTRAN, JOVIAL, C, and the ilk like that. It was actually in the 1970s that we saw languages like Smalltalk pop up on the horizon. Languages from the artificial intelligence (AI) community, like Flavors and LOOPS, took a different view of the world.
So you saw this dichotomy of languages. There was certainly work in traditional procedural languages, COBOL, C, and its variants. But we also saw an evolution of things like Smalltalk and its variants, a language called CLOS, which recently has been a standardized AI language, and even Ada. Ada was to a large degree influenced by the things going on in this technology. And what particularly excites me is that we see now with Ada 95, you have reflection of what we have learned from the past several years from a whole variety of object-oriented programming languages.
On the horizon you also see C++, which is a topic of great discussion among many of you. Let me give you my observation in this regard. There are a number of merits to Ada 95. First, it's a standard; it's stable; it's safe. It scares me to see certain applications that are being built with languages like C++. I've seen more bad code in C++ than any other language, because inherently, it's a difficult language to build things precisely, correctly, and safely. And furthermore, Ada 95 is object-oriented.
Let me suggest the following, which may be a controversial statement, but represents a fair view of the technology. My view is that where architectural issues dominate, Ada 95 has clear advantages. Where programming issues dominate, C++ has some advantages. The net result of this is that there is probably room for both.
Tools
Let's look at tools. Again, remember I'm saying that not any one of these technologies is something you should focus on. It must be a balance of all of these. I claim that a mark of the immaturity of a development organization is it tends to have great arguments about language or tools or process in isolation, because those things cannot be dealt with in isolation.
In the tool world, we see the evaluation of integrated development environments. Things such as rational's Apex and what I-CASE is trying to do. In the PC world, we see a bevy of these kinds of things. What excites me in the tool domain is not so much the programming language tools, but rather, the frameworks that are emerging. This brings us back to the point of domain- specific kinds of frameworks.
To a large degree, I believe that reuse is overrated, and that we have been focusing upon high degrees of code reuse. But, I think that is the wrong thing to focus on. Rather, it should be a higher focus upon reusing expertise, knowledge, idioms, and patterns in that domain. And that's reflected in the kinds of frameworks we see here. Furthermore, it's something that is far easier for us to express in object-oriented programming languages. Things such as the Ship System 2000, which some of you may have encountered, built by CelsiusTech and basically a framework--in Ada--which codifies most of the interesting aspects of shipboard control systems. They were sold to the Swedes, the Finns, the Danes, and the Australians and basically built one framework to do that. Taligent in the commercial sector is trying to do the same kind of thing--codifying cross-platform kinds of frameworks. UNAS is an example of the same kind of thing capturing the essence of distributed object computing. And even in the commercial world where we find things like fin++, which is a library of C++ and Smalltalk, that codifies all the essences of what one does in trading systems.
What you find in every one of these cases is it provides tremendous leverage. It's like, let's build a framework, add bits of software to tailor it to our particular domain, and with little additional effort, I can now apply this to a new kind of system. We are finding, for example, fin++ used in Japan, throughout Europe, throughout the United States, and in a number of banks. That's where the leverage is.
Methods
Let's talk about methods for a moment. There's been a similar evolution in methods over the last several years. In the past decade or so, we have seen structured analysis and design techniques. With the evolution of object- oriented (OO) programming languages, there has been a need for object-oriented methods as well--methods for analysis and design. This is what my early work in Ada focused upon. It was clear in the early days of Ada that the traditional techniques were fundamentally wrong to use with Ada. If I'd used it in a functional way, I would have led myself to architectures that were brittle and resistant to change.
Taking ideas from Mary Shaw and others of abstract Ada-type techniques led us to simple object-oriented concepts, which match well to Ada. But over the past several years, other ideas have emerged that began to fold in which we believe will become somewhat of a de facto industry standard. There's my early work. There is work from General Electric called the Object Modeling Technique coming solely from the commercial sector. There is work from Ivar Jakobson in Europe, on a technique called Objectory, coming from large, complex telephone systems. In this domain, you've got all the classic cases of complexity: it's real time, it's space-constrained, and it's large. And it must be resilient to change. These seem to be the three dominant OO methods. And the nice thing is, we've collectively been working together to bring this to one industry standard.
____________________________________________________________________________ | | | Figure 4: Methods. | | | | Process maturity Booch '91 Metrics and QA | | * SEI CMM | | * ISO 9000 \ | / | | \ | / | | ___________________ | | | | | | CMT ----> | Booch Method | <---- Patterns | | |___________________| | | | | / | \ | | / | \ | | Other Methods Objectory Formalism | | * CRC cards | | * Responsibility-driven design | | * OBA | | * Harel | |____________________________________________________________________________|
Processes
The final element we spoke of in this list of things in which I must have balances is the notion of processes. There is a lot of talk on process these days, and frankly, it is a good thing. But beware, because just focusing on process will lead us to the same kinds of things we end up with if we just focus on languages. These things in isolation are not enough.
What do I mean by process? Well, there are a number of things going on. One of the elements is--and this is a bit controversial, but I think it is worth saying--the Software Engineering Institute Capability Maturity Model (SEI CMM) is a good thing. No doubt about it. And the successful projects I encounter tend to be those that will rate high on the SEI CMM. But the reverse is not necessarily true. Just because an organization has been given a high rating doesn't necessarily mean it's going to be successful. So it's a bit of a dichotomy. That means, don't become envious of the number rating you might have, because in and of itself, it's just a number.
In Europe, we run across a lot of interest in ISO 9000, so the CMM is not the only game in town there as well. We're finding many organizations that are impacted by the quality standards that ISO 9000 brings.
The nice thing is, that if I look between the two of those, both approaches are focused upon high-quality processes, being able to measure what you're doing and tuning the process. The notion of gathering metrics and tuning is fundamental.
Commercial vs. DoD Practices
Again, being graced with being in the position of having seen lots of different kinds of projects, I will give you some observations of the commercial world vs. the DoD world in terms of software development. One thing I would urge you to say is, if you take a look around among commercial practices, it's very easy to get intoxicated by some of the things you see. You see people quickly packing together marvelous applications in things like Visual BASIC and Power Builder, or Gupta SQL Windows, technologies that have made an impact in the commercial sector. But, beware. Those kinds of things-- and I'm not saying they're bad; they are indeed good--do not necessarily scale. One must always be wary about taking simple technology solutions and making sure they map up to the larger kinds of things. Because many of them codify simple domains but do not necessarily scale up.
One thing I can say about both of these domains is that chaos reigns everywhere. Software problems are universal. I find them in every country, in every organization. So we are not alone by any means. Yet, both camps have a lot to learn from one another.
The Five Habits of a Successful Project
To that end, I will reflect upon some of the habits I find common among the successful projects I encounter. I've seen a lot of successful projects, and I've seen a lot of failures. And these five lessons strike me as being the most common among those projects that succeed. In their absence, they tend to be good predictors of failure. If I'm missing one of these elements, my project is probably going to go south.
|
| Table 2: The Five Habits of a Successful Project. |
|---|
The first two of these have nothing to do with any kind of OO technology. They are just sound business management practices. Many of the decisions I have to make in successfully fielding or deploying an application are business decisions to be made. We often miss those kinds of things.
The first element I find common among successful projects is there tends to be ruthless focus upon those essential minimal characteristics I spoke of. This doesn't mean I have managers who are Attila the Hun types, but rather, they understand what the essential minimal characteristics of that project are and make the proper business decisions as the project unfolds.
The second element goes almost without saying, that there must be a spirit of communication among the entire team. There is a little OO bent in this, which excites me. And that is, in the successful projects I've encountered, object-oriented technology tends to be a good way to facilitate that kind of communication. Because, leading to the next point, this point, OO provides for me a good way to capture a systems architecture, and that is fundamentally something that I must communicate. So these three elements wrap up the OO paradigm. First, we find a good use of OO modeling, a focus upon architecture. And last--and this is a difficult one for some organizations to swallow--the presence of an iterative and incremental process.
Capers Jones--who will speak to you later--has pointed out that every software project tends to have a number of artifacts associated with it, and here are a handful of them (Table 3). Every one of these artifacts represents something that is reusable from the project. Architectures are reusable; designs are reusable. We tend to focus upon code reuse, which is one element of it, but clearly there are larger higher-leverage elements that are reusable. Every one of these things represents an element that can come from a project and be reused again and again.
|
| Table 3: The Artifacts of a Software Project. |
|---|
Architectural Modeling
I will talk about this notion of architecture one more time, because it is so fundamental. I ran across a project once, and I won't mention the name of the company, but it was the leader in design automation tools. They decided they wanted to continue in their growth, so they said, "We're going to adopt C++. We're going to adopt this new OO stuff, and we're going to move forward and leapfrog anyone else in this marketplace."
Well, they did so. They trained their managers; they trained all the developers; they tried to put a process in place. And they failed miserably. The last place you want to have your software meltdowns described is in Business Week, and unfortunately, this is where this company meltdown was described about two summers ago. Interviews with their managers pointed out that there was one fundamental reason for failure. They had done all the right things in terms of programming languages--they had a process in place; they had documentation in place, but they lost sight of the architecture for the system.
What do I mean by architecture ? This slide (Figure 5) comes from a colleague of mine named Philippe Kruchten who is the architect for the Canadian Air Traffic Control System. This is a large, complex, commercial Ada system. It's roughly 1 or 2 million lines of code, and clearly has some real-time aspects to it. Philippe points out that every archi-tecture tends to have these five elements. There is some logic model associated with it, which basically represents a description of the enterprise.
____________________________________________________________ | | | Figure 5: Architectural Modeling. | | _____________________ _____________________ | | | | | | | | | Logical View __|__________|__ Development View | | | |__________________| |__________________| | | __________________| Scenarios |__________________ | | | |________________| | | | | Process View | | Platform View | | | |_____________________| |_____________________| | |____________________________________________________________|
What are the things in the vocabulary of that problem domain? There is a development view, which says I'm building things that developers must manipulate, so I've got things like files and so on that are a part of my architecture. I have a process view. This is a distributed application; it's not just a hunk of code sitting here, so I must also architect the presence of processors in processes and how they are distributed around my system. And finally, there is something like a platform view that shows what processors and devices I have and what topology they take.
Unfortunately, most organizations tend to focus just upon this logical view. And this turns out to be a small element of what we must build. The successful organizations I encounter tend to take a view of all four of those domains. And they basically place someone or some artifact responsible for that particular area.
What unifies this is an emerging idea coming from Ivar Jakobson--the notion of scenarios, which says let's take a look at our world. Let's find some functions that walk through it and those functions that often represent system function or business processes. If we're dealing with a logistics system, it would be how would my people interact with this system? Those represent scenarios that thread themselves through all four of those pieces and provide a wonderful, unifying model. Again, all the successful projects I've encountered tend to have a focus on every one of those elements.
Discovery, Invention, and Implementation
That in and of itself looks fairly straightforward, but it's not. Here is where the real downside is. If you take a look at the activity of complex software development, you'll discover that there are some interweaving cycles. And this is what leads us to processes that must be iterative and incremental. This slide (Figure 6) was done by a friend of mind named Bran Selic who does a lot of work in hard real-time systems. He noticed that as he looked across at what his developers would do--here we have time across this, and the Y-axis you see here is sort of where the emphasis is--he saw that there are overlapping cycles.
The first cycle is early in the process. There is a period of discovery in which I'm looking at the world and saying, "Yea, verily, these are the things that I know I must have." There is a later overlapping phase, which says, this is a period of invention in which, knowing some things about my application, I'm going to go off and build some of the artifacts necessary to carry out the behaviors that I discovered in the earlier phase. And lastly, there is this lower curve, this slope of implementation.
What fascinates me about this is that this is the curve that I find in every successful project, and it tends to be overlapping, and it tends to be nondiscrete. Whereas I'm spending some time in discovery, I'm also spending some time in invention. I'm also spending some time in implementation as well. And it points out that successful software development is inherently iterative, because I do have those overlapping cycles.
That scares some managers, because they will say, "I have no control over a process like that." It leads me to discover a different axis of cultures. It's what I call high ceremony and low ceremony. In some of the software companies in northern California and in Oregon, you'll find this culture of low-ceremony projects in which the metric of success is pizza boxes. Look outside your programmers' cubicles and measure how quickly pizza boxes are accumulating. That will give you some idea of the progress they are making. That's low ceremony. I think there is some corollary to pizza topping and speed of development, so I'll do extensive research and report back to you next year on that one.
There are also high-ceremony projects--which we tend to find in large aerospace organizations, large insurance companies and banks--which say programmers are important, but we're going to define a process that is rigid.
Neither of those extremes leads to success, because on the one hand, a low- ceremony organization will tend to rely upon the best efforts of its heroic programmers. But that's not sustainable, as I pointed out earlier. If I'm building a shrink-wrap application, if I'm building something for a trading company that I've got to get out the next day, then it demands heroic programming. But if I'm building something that my entire enterprise works upon, I can't rely upon the efforts of heroic programmers. It's simply not sustainable. On the other hand, you do not want to go too much to the extreme of high-ceremony organizations, because they tend to stifle all creativity. They tend to lead you to discrete phases of discovery, invention, and implementation, which basically leads you to build systems that are not resilient to change.
So what do we look for in a well-defined process? I think every kind of process, be it low ceremony, high ceremony, or some place in between, does a number of these things for you. It guides what the team's activities are going to be; it specifies what artifacts are going to be created; it directs the tasks of individual developers. And what's more, it offers criteria of how I monitor the activities. Low-ceremony projects use the pizza box metaphor, for example.
The Macro Process
My observation of these overlapping cycles has led me to realize that every successful organization has multiple processes. It is what I distinguish by the micro process and the macro process.
The macro process is similar to the waterfall approach. Waterfall is not bad. It has some inherently not good things about it, but in and of itself, it's not an evil thing. I will redefine it a bit. Whereas we spoke before of traditional waterfall as analysis, design, and implementation, we're going to recognize that in successful projects, we do have those three overlapping cycles. So the focus among the successful projects I have encountered is this macro process, which is months if not years of time frame. It tends to focus upon the successive refinement of architectures. That's a subtle and fundamental shift. Rather than worrying about artifacts such as what are my requirements, nail them down; what's my design, nail it down; go off and implement it, we are saying what we're doing is defining a vertical slice of the whole system, iterating upon it, and refining it. This allows us to discover things later on in the lifecycle that we could not have discovered if we had a priori to make those decisions.
To put it this way, let's do it graphically (Figure 7). This is what I find to be the macro process for successful projects. There is a period of conceptualization. There is a period of discovery, which we call analysis. There is a period of design, which is basically invention of the architecture. And there is a period of evolution in which I successively refine that architecture.
____________________________________________________________________________ | | | Figure 7: The Macro Process. | | | | _______________________________ __________________________________ | | | | | | | | | Establish core requirements | | Develop a model of the desired | | | | | ==> | behavior | | | | (conceptualization) | | (analysis) | | | |_______________________________| |__________________________________| | | | | || | | || | | \/ | | __________________________________ | | | | | | | Create an architecture | | | | | | | | (design) | | | |__________________________________| | | | | || | | || | | \/ | | _______________________________ __________________________________ | | | | | | | | | Manage postdelivery evolution | | Evolve the implementation | | | | | <== | | | | | (maintenance) | | (evolution) | | | |_______________________________| |__________________________________| | |____________________________________________________________________________|
In many projects, it loops back around--unless I go off and field that system and say I'm never going to change it again--in which case it enters a period of maintenance. Software has a tendency to never die. You know you can't get rid of it sometimes. Sometimes, you'd like to shoot it; sometimes you can't. But the successful projects I've encountered that do have this focus on architecture allow them to iterate again and again.
The Rhythm of the Macro Process
The net result of this is this slide (Figure 8) that takes a bit of explanation. Every successful project I've encountered tends to have a rhythm associated with it. And as I go into a project, I'm often brought in to help projects that are in crisis. You've all seen projects in crisis. All of you may know one or two in your lifetime. It's where the managers are screaming into the night. That kind of thing. As I go into a project and try to capture its pulse, I discover this rhythm.
Across this cycle, across the top X-axis, I see the basic phases of this macro process: conceptualization, analysis, design, and evolution. What I see down this axis are some of the artifacts that Capers describes--the reusable artifacts of requirements and so on. The dark bands reflect where I'm spending more emphasis; the light bands reflect where there is not a lot of activity. What you find in these kinds of projects is fascinating.
Because early on, you see this flow here of during analysis, I tend to focus upon requirements, but certainly not put them in concrete. I spend time in design working on my architecture. I spend time on evolution working on the code, and you see this sort of natural waterfall. But during evolution, the time that I'm successively refining that architecture, you get the rhythm that comes into place. It says I spend a little time reanalyzing, redesigning, reimplementing. What you've done in this process is basically change where the risk lies in these kinds of applications. In most projects, traditionally, you would find a long integration phase. In this kind of process, we find that there is not a single phase of integration. But rather, it tends to be one of constant integration of the system, and that is where you see that kind of rhythm.
In organizations like Microsoft, you find that the rhythm is beat about once a day. There was a fascinating survey done in Ed Yourdon's journal, American Programmer, in which he pointed out that, if you look at Microsoft's developments practices, they do a rebuild of all of their software--their major shrink-wrap software--every night. And that's what drives the rhythm. In some of our projects, we tend to have the rhythm driven about once a week. Internally, we'll do a new build every week. That becomes a tremendous forcing function that forces the organization to close based upon the right business decisions, rather than deferring those decisions until later when it's too late to make any difference.
I will spend just a bit of time on some of those phases. I will talk about some of the interesting aspects of what I again find among successful projects that they tend to do in these various phases. The notion of having complete, consistent, nonchanging requirements is utterly a myth. It simply does not exist among any system of a certain size and complexity. As a result, I must always be on the lookout for what are really the behaviors of my system desires. Yes, I can put it in place and capture many of the early decisions early on, but I must have room in my process to allow for change and discovery.
This is what's really exciting about OO technology. It seems to bring us to focus upon what is often called scenario planning. Take a look from the top of my system. The f word is not a bad word to use here. What are the functions of my system? And even the most complex applications I've encountered will still have only a few dozen fundamental functions. Lots of variations on them, but a few fundamental functions. And those become part of the essential minimal characteristics that I must drive toward.
Who does this work? Well, this is a bit of a shift. What I find in successful projects I encounter is that analysis is best done not just by a set of people in the problem domain, but also allied with people who are going to be responsible for building this system. Because again, there must be business trade-offs. If I'm sitting there as an end user, there is no end to the things that I will demand of you, because it's a simple matter of programming as far as I'm concerned. And yet, I must temper that with people who understand the limits of this technology. And there are indeed limits. So what I find best to do, if I can orchestrate it, is to put my architect and end users together to hammer out what the essential minimal characteristics are.
How do I know when I'm done? What are my major milestones? It's basically capturing those major scenarios and having them reasonably stable and simple.
The notion of stability as a metric is one you're going to see over and over again, because I find that among the complex applications I encounter, it's a measure that we often ignore. We tend to look at microscopic things such as how many lines of code, and what's my percent of completion. Those are interesting, but they are misleading.
What is design? This is absolutely fundamental, because it is the hallmark of the successful project. It says that during design, focus on building an architecture along those four dimensions that we saw earlier. The interesting thing about doing this is it tends to be a risk mitigator, because it says let's put my architecture in place before I deploy all of my resources necessary to fill out that architecture. Building the architecture is perhaps the most difficult thing to do, and I want to do that with a small, well- qualified team. What's my main product? It tends to be an executable architecture, something that's not on paper, but something that absolutely runs. Why do I do this? Because it drives me to a discovery of the risks in my system earlier rather than later. As Tom Gilb has pointed out, if you don't actively attack your risks, they will actively attack you.
So, what's my major metric? It tends to be one of simplicity. I want to build simple architectures. I want to build architectures that have a balanced distribution of responsibilities: not too tall, not too fat; I don't want to see a stovepipe kind of architecture. And once I have it in place, it is reasonable to begin to commit the resources to evolve that architecture.
What I see in evolution is basically taking that architecture and refining it but giving me the opportunity to revisit what the real requirements of my system are and to make the right business trade-offs, because I know I can't optimize for everything.
The Micro Process
Let's talk about the micro process for a moment, because sometimes we forget our individual developers as well. Just because I have a wonderful process in place, just because I've chosen a wonderful language, doesn't mean that I'm necessarily going to succeed. Don't have managers that allow your project to go on autopilot. Because in the presence of a good process, in the presence of a good language, there's conflict that has to be resolved, and that must be done actively. It's also the case from the perspective of the individual developer. These people are not replaceable parts. And what you find on the developers is a different process going on. Successful organizations tend to recognize that there is that different kind of process.
It is what I call a micro process. A few years ago, Bill Curtis did a fascinating study in which he videotaped some developers at work. The manager said, "Well, obviously, they are going to analyze first, and design and implement next." That was not the case. In the review of the videotapes, he realized that the developers were sitting there programming for a while, sitting there for a while just thinking, and maybe going back and doing design. It was very much a stochastic process. Successful projects tend to recognize that there must be degrees, room for freedom, and room for creativity among the individual developers, but in the context of the macro process, which gives it control.
So, the micro process says what the individual developer does: I discover, I invent, I implement in an almost random fashion, but it's done in the context of the macro process, which gives it control.
If you look at the successful project, you'll tend to find that the team is organized in a common way. This was done by a study of a number of telephony projects--again, complex, real-time systems. A colleague of mine by the name of Jim Coplien discovered that in the hyperproductive projects he found, there was a common pattern of the organization. And it's centered around these four activities (Figure 9). The boxes indicate roles; the lines indicate lines of communication among them. In these projects, he found an architect; you found abstractionists, who work with the architect to take those architectural ideas and turn them into reality. Application engineers, who are your coders, and your project managers are in close alliance. It is interesting to point out that you also see patrons show up on the roll as well. Patrons are like the Renaissance era concept of someone who basically pays for the project. This person should not be isolated; there must be a connection. But it is these four in the middle that are in the best position to make the best kinds of business trade-offs. That is a different kind of organization than what you tend to find in nonobject-oriented projects.
I will finish up with a few observations on the software world, then I will wrap it up.
General Trends
I will remark upon some general trends I see in a variety of domains. The first of which is just general object-oriented stuff. One thing I can conclude is the language war seems to be over. C++, Smalltalk, Ada 95, and a few others seem to be the survivors. I will come back to Ada 95 in a moment. Every one of those has a role to play. There is no doubt in the commercial sector that C++ seems to be dominant. Smalltalk also has some dominance. But in those kinds of applications which I'm going to depend upon, the safety of that system on which human lives will depend, I'd much rather see that done in Ada 95 than any other of these languages.
We're also seeing that the method wars are ending. There seems to be a consensus in emerging OO methods. And lastly, there are a number of signs of growing maturity in this marketplace. It's not like it was three or four years ago or even when we saw a lot of halting starts in the use of OO technology. But, across every domain, virtually every country, virtually every application domain, I have seen successes with regard to the use of OO stuff.
Another observation to make in regard to distributed computing is it's absolutely inescapable, but the challenge is, this leads us to systems that are inherently so complex that most organizations don't know how to cope with them. It's easy to build applications that look stand-alone, or we tie them together. But by no means does that exploit the power of what the distributed architecture can do for us. I urge you to say that it looks like there are a lot of interesting solutions for us on the horizon. But there's an inherent complexity that we simply don't know how to solve yet. There are issues of not just object-oriented distribution, but also of problems of migration and synchronization. These are all fundamentally hard problems, and no amount of technology is going to make them simple for us.
Another element we can point out is this notion that Mary Shaw calls programming without programming. There's an interesting trend that is going to affect some of our projects, which is this idea of what some call componentware. You see this in the trade journals with regard to what Microsoft is doing on things like OLE and COM. In this camp, you see things like Open Doc, which represents a fundamentally different way of how I put applications together. It says I'm going to define these little parts that are domain specific. Now I'll have a mechanism that allows them to communicate. This is exactly what I spoke of earlier when I talked of having domain-specific architectures. In reality, we're seeing a number of domains and organizations that are trying to tackle that. We've seen it for things like trading systems, as I mentioned, the handling of customer support for public utility companies, and the like. All of these are not rocket science, but they represent opportunities for codifying a large domain.
Last thing, and I will wrap it up here. I alluded to this notion that we have tools; we have processes; we have methods. And not one of them alone is sufficient to help us deal with the complexity we must deal with. Software is fundamentally hard, and it's not going to get any easier because of the trends we saw earlier. The good news is that OO stuff seems to help. The other good news is, we in industry seem to have some notions of processes, languages, and methods that help us in this domain as well. Remember, it's a balance. Don't focus on any one of those, but balance them among every one of those.
Thanks very much.
Grady Booch
Rational Software Corporation
2800 San Tomas Expressway
Santa Clara, CA 95051
Voice: 408-496-3600
Internet: egb@rational.com