Is there a "Large Scale Python Software Design" ?

[Peter Hansen]

Alex Martelli wrote (a hell of a lot, as usual, and I do hope
he'll forgive me that I chose to skip/skim some material and
try merely to catch the highlights, doubtless missing some
interesting bits in the process):

> Peter Hansen <peter at engcorp.com> wrote:
>>And you've reemphasized my point.  "Testing" is not test-driven
>>development.  In fact, test-driven development is about *design*,
>>not just about testing.  The two are related, but definitely not
> 
> Hmmm... the way I see it, it's about how one (or, better!!!, two: pair
> programming is a GREAT idea) proceeds to _implement_ a design.  The fact
> that the package (or other kind of component) I'm writing will offer a
> class Foo with a no-parameters constructor, methods A, B, and C with
> parameters thus and thus, etc, has hopefully been determined and agreed

We don'really disagree on this point.  I'd clarify my comments just
by saying that depending on what stage you are looking at, there
is always a preceding decision that could be called "design" and some
subsequent work that implements that design.  If you are figuring out
what requirements your system should have, you are "designing" it for
your eventual users in a sense.  If you are analysing requirements
later on and blocking out the major architectural areas and interfaces,
you are doing design, but then the traditional "designers" might still
have to go to work.  Those designers (being the ones we usually saddle
with the title) then do "detailed design" and specify interfaces and
such as you note above, but they aren't yet doing implementation.  Along
comes the programmer pair and they "design" the implementation in their
heads as they come to a failing acceptance test case, then conceive of
some units tests and some code, designing as they go.

In a nutshell, I was talking about that portion of design that occurs
when a good programmer goes to work figuring out just *how* she will
implement that method A with parameters x and y and a failing test
case that says it should act suchlike...  Certainly TDD is not as
much about the more traditional design, the implementation of which
you refer to above.

> TDD may surely help defining the internal structures and algorithms
> within a single component, of course, if that's what you mean by design.

Yep.. saw this while pruning your text.  Had I read more thoroughly
the first time it would have saved all that typing, which I'm now
loathe to remove. :-(

>>tried TDD, they are unlikely to lump it in with simple "unit testing"
>>as it has other properties that aren't obvious on the surface.
> 
> It sure beats "retrofitting" unit tests post facto.  But I'm not sure
> what properties you have in mind here; care to expand?

You've forgotten them at the moment, but I know you know about those
properties such as how TDD *forces* testability on the design/ 
implementation, and thus improves modularity, how it greatly reduces
the incentive and opportunity to gold-plate, how the most critical
tests are run hundreds or thousands of times during a project instead
of a handful of times just prior to shipping, and so forth.

>>was leading worked on code that, if I recall, was somewhat over
>>100,000 lines of Python code including tests.  I don't recall
>>whether that number was the largest piece, or combining several
>>separate applications which ran together but in a distributed
>>system...  

> I think this qualifies as large, assuming the separate applications had
> to cooperate reasonably closely (i.e. acting as "components", even
> though maybe in separate processes and "only" touching on the same
> database or set of files or whatever).

It was a true distributed system, so yes the components
closely cooperated.  Acceptance tests actually ran both pieces
simultaneously, for the more complex tests, and in some few cases
even involved a simulator of the 16-bit embedded devices so that
the test case spanned four levels (web browser, server, third
piece, and the simulator for smaller gadgets).  The simulator, of
course, was written in Python...

-Peter