waterfall (was Re: REPOST: Re: Book "python programming patterns". anybody read this??)

[Alex Martelli]

Peter Milliken wrote:
        ...
> that *all* the people involved in the project can't be brought around to
> the concept/idea that you should sit down and plan the project step by
> step and not proceed to the next until the previous is complete i.e. sit

This is also known as the "waterfall model" of software development.

It just doesn't work, as many decades of industry experience have amply 
shown.  Therefore, if some people "can't be brought around to the 
concept/idea", there may be some excellent reasons for that; e.g., the 
people who resist this horrid idea may have some relevant experience, or 
have read some good book on the subject, or just plain have some common 
sense and on-target intuition.

First of all, development needs to proceed in far smaller increments than 
"a whole application system", as implied by:

> down with the customer, analyise the requires - protype if necessary, but
> get them nailed down and worked out until the customer is happy that he
> knows what he wants!.  

Second, it's imperative to have continuous "feedback loops" from each stage 
back to previous ones.

Domain analysis is fed back from requirement analysis.  Both are fed back 
from architectural design.  The latter absolutely needs feedback from 
subsystem design.  And so on, and so forth.

> Then the programmer should sit down and use the analsysis to
> generate a design (documentation! ugh, how many programmers want to spend
> their life doing documentation! - there is always some excuse not too! :-)

The world is chock full of software projects which have generated far more 
"documentation" than useful code.  Paperwork in out-of-control amounts is 
trivially easy to generate, compared to _working_ code _that does what's 
actually needed_.  The "waterfall model" is responsible for more useless 
reams of empty words being generated than any other software development 
model, I believe.

Being a budding author, as well as an established developer, I wouldn't 
mind "spending my life doing documentation" IF I evaluated that the 
end-product of my efforts would be more useful that way.  When I write, I 
have to carefully weigh the amount of English text I produce versus 
examples, snippets, exercises, for example.  Just as, when I code, I have 
to weigh similar considerations.  The goals are somewhat different, but not 
by all that much (Knuth's theory of "Literate Programming" makes them 
almost coincide: writing a software system is just the same as writing a 
publishable paper about the workings of that software system -- you write 
just one document, with the two things suitably intertwined and fed-back, 
then process it with different post-processors to generate either a, say, 
runnable Pascal program source or a printable TeX markup source).

"No battle plan survives contact with the enemy", and no "pure" software 
design survives contact with the hardware.  Unless you have structured 
mechanisms in place for feeding the lessons of lower-level experiences back 
to the more abstract layers, you're going to lose the battle (not deliver 
good and useful working code) or win it in ways that have really very 
little to do with the written-down "abstractions".  The "central planning" 
pipedreams of unmourned "real-socialist" economies provide another good 
example of this phenomenon.  


> understand it! :-)). But most programmers want to cut code - nothing else

Assuming this assertion of yours, then the only good software development 
methodologies will be those that take it into account and work with it: 
i.e., any methodology that assumes otherwise is a monumental disaster.  
Waterfall, for example.

Analogy: "most workers want to make money -- nothing else".  This isn't 
really true (just as your assertion isn't), but, if it were, what would 
follow?  Obviously, that the only sensible economic systems are those that 
take this into account and work with it.  In other words, from Marx's 
materialistic assumption (that making money is the only motivator), there 
follows that centralized planning (and socialism in general) is unworkable, 
and the only viable economics approaches are those which leverage people's 
greed for money into a constructive system (Smith's "invisible hand" -- 
""It is not from the benevolence of the butcher, the brewer, or the baker, 
that we expect our dinner, but from their regard to their own interest.").

Similarly, then, if it's true that writing code (actually: good, working, 
_useful_ code) is a programmer's key motivator, then the only viable 
methodologies will be those that leverage this into a constructive system.  
"Agile Programming" (particularly in the "Extreme Programming" flavor) does 
give code its rightfully central place among all artifacts of software 
development.  If you don't, then you're pandering to the *bureaucrat*'s 
motivations: the key thing is writing huge formalized reports, producing 
paperwork for its own sake; that anything 'useful' must come out of the 
process is maybe an unfortunate necessity, but the dream system would 
produce nothing but paperwork, paperwork about the paperwork, and so on.  
(The "Yes, Minister" episode "The Compassionate Society" is a good 
caricature of this: most doctors want to cure the sick, nothing else, but 
the real bureaucrat knows that a "well-working" hospital is one with 500 
paper pushers pushing "documentation"... the fact that no funds were left 
to hire any doctors, or admit any patient, is a mere minutia).

Curing people _effectively_ requires producing a small but essential 
paperwork trail (with *second* priority to actually making sick people 
better, please notice); so does coding effectively -- producing code that 
actually does useful stuff -- a small but essential paperwork trail needs 
to be there (with *second* priority).  In both cases, the key use of the 
paperwork is *future* "maintenance".  But the patient had better be alive 
(the software system working), or else perfect paperwork about it can only 
serve to make bureaucrats happy.

So, for example, coding *TESTS* is one of the best ways to express a 
component's specifications: it's concise and unambiguous in a way that 
human language never evolved to be.  Turn the urge-to-code into "test-first 
development" (both design, and coding) and your system's quality is poised 
to take a huge jump upwards.  Involve the customer in the tests, given that 
_their_ motivation is to have a system that works to solve their actual 
problems, and another jump up in usefulness awaits.  Less paperwork, more 
code -- code that is more likely to solve the real problems and ensure that 
they are indeed solidly and reliably solved -- THAT is what's worth 
dreaming of (or, worth doing something about: that's not as hard as all 
that, since _many_ developers share such dreams, and now that Agile 
Programming is finally getting decent exposure, it's not all that hard to 
sell to customers, either!).

If you want waterfalls, try Niagara -- rather pretty place.  Just leave 
them out of software development, PLEASE.


Alex