Is there a "Large Scale Python Software Design" ?

[Alex Martelli]

Dave Brueck <dave at pythonapocrypha.com> wrote:

> The OP would be well-advised to search the Google archives of c.l.py as
> many (myself included) take the contrarian view - as the project grows in
> size it is harder to justify going with "classic" languages like C++, or
> even Java - the associated costs at each stage of the project are
> relatively larger to begin with, and grow more quickly as well.

I entirely agree with you, Dave.  Moreover, I do have a mass of growing
but as-yet-unorganized notes, based mostly on experiences on large
projects I have consulted for or even been very intimately connected
with, showing why Python is superior to various plausible alternatives
(for various and different reasons in each case) for large-scale
software development, and what principles, practices and patterns best
enable teams in various conditions to actualize those advantages.

That is the book I want to write, the one I have always wanted to write;
the Nutshell and the Cookbook (and now their second editions) keep
delaying that plan, but, in a sense, that's good, because I keep
accumulating useful experiences to enrich those notes, and Python keeps
growing (particularly but not exclusively in terms of third-party
extensions and tools) in ways that refine and sometimes indeed redefine
some key aspects.  To give a simple technical example: I used to have
substantial caveats in those notes cautioning readers to use multiple
inheritance in an extremely sparing, cautious way, due to traps and
pitfalls that made it fragile.  Nowadays, with the advent of 2.3, most
of those traps and pitfalls have gone away (in the newstyle object
model), to the point that the whole issue can be reconsidered.

Anybody who has written serious technical books can gauge the amount of
work it takes to turn "a mass of yet-unorganized notes" into a real
book: it's _staggering_.  I can't seriously undertake the task of making
my copious notes into a book until I can consider devoting at least half
of my time to it for a year -- this means no other books in the making,
_and_ a reduction in the amount of consulting, teaching, mentoring, etc,
that I do.  The biggest general issue is that a book cannot be
_interactive_, _customized_ to the specific skills and interests of a
reader, in the way in which I can customize interactively the kind of
hands-on teaching, mentoring and consulting which I do for a specific
customer.

For a given customer, I can and do find out what kinds of areas they
believe their large projects will cover, what skills their people start
with (and what skills can they expect other people to start with in the
future, depending on expected turnover), the "political" and "social"
dynamics of the team -- is the kind of "customer involvement" that's the
crux of Extreme Programming wrt other kinds of Agile Development
feasible at all, at what cost, etc, for example -- and so on.  I can
avoid spending substantial time and energy on issues which don't matter
to project A even though they may be crucial to most large projects --
believe it or not, SOME projects need no networking, others will never
directly interface to a relational database, etc, etc, even though these
days 9 large projects out of 10 will need to deal with both kinds of
issues; and GUI issues, especially for large projects which mostly deal
with web interfacing vs others which will need traditional GUIs, can be
even more divergent.  And this amount of variety is just for the
_technical_ issues; the political/social/business-plan ones, people's
skills and backgrounds, etc, are even more diverse...

To make a book, I will have to find an organization that works for busy
readers who don't have the time or patience to read through long parts
connected to database issues if they're on one of the few projects that
don't care about databases, and so forth -- structure sections,
chapters, appendices, footnotes, sidebars, ... so that skimming or
skipping the "don't care about it right now" parts can work; find a way
to reach that part of the audience that has never really undertaken a
large project before, or has played in such projects the role of a "cog"
without a clear picture of the whole structure, _as well all_ the lead
architects and tech-savvy project managers.

Lakos did manage, and I admire him immensely for that.  Robert Martin
has also done great work, though his books, while good, are (IMHO) never
_quite_ as excellent as his superb essays (don't get me wrong: I wish I
was half as good as Uncle Bob!-).  Eric Raymond's "Art of Unix
Programming" is one of the most useful books for would-be architects of
software systems that I've ever laid my paws on -- I rate it as close to
the Mythical Man-Month, Design Patterns, Programming Pearls, and a few
of the many recent books on Extreme and other Agile methods (my personal
favorites of the crop are Scott Ambler's and Kent Beck's books).

However, none of these excellent books really addresses the questions
specific to the architecture, design, and development practices that
work best for dynamic VHLLs, and specifically for Python; so, I do
believe the book I dream to write is still needed (even though I might
be a grandfather by the time I'm done with it;-).

Meanwhile, to people and firms which aren't interested in retaining my
professional services, the best advice I can give -- after that of
studying the various books I have mentioned above (as well as good
Python books -- I like my own, but then, of course, I'm biased; I'd also
suggest others, such as Holden's, Pilgrim's, Hetland's, ...) -- is to
try something like:
<http://groups.google.com/groups?safe=images&as_ugroup=*python*&as_uauth
ors=alex%20martelli&lr=&hl=en>
as well as similar searches for the many other authors that contribute
so validly to the Python discussions, of course.

Somewhere or other, in my 8190 posts found by the above Google Groups
search, I have expressed (often more than once, and with different
nuances depending on the exact subject, apparent skills and interests of
other discussants, etc; as well as sometimes based on my changing ideas
on some sub-issue, or changes in Python and other tools and
technologies) a majority of the issues that I touch upon in that "mass
of notes".  Of course, the stuff is yet more disorganized than said
notes; however, it _is_ written to be read and hopefully understood by
others, while most of said notes are written essentially "to myself", to
remind me of the huge variety of things that may need to be covered
regarding the huge variety of facets that make up the subject "Large
Scale System Architecture, Design, and Development Practices with
Python".  Moreover, a majority of the 8190 posts are undoubtedly dealing
with subjects that aren't really related to LSSADDPP.  Hey, there's
_got_ to be some advantage in retaining me, or reading my hopefully
future book, rather than combing through all my posts, no?-)

Seriously: one day do I hope to start putting up some parts of those
notes, mutated into intelligible text and organized into kind of
almost-essays, on my website -- fragments of said future book, but more
accessible and usable than the sheer morass of posts above-mentioned.
But don't hold your breath for _that_, any more than for the book; I've
been meaning to redo my site for _years_, and it just hasn't happened...
there's always something else that looks more interesting, either
intrinsically, and/or because of the little issue of money;-).  Some
stuff (mostly presentations) you can find at www.strakt.com, which also
has important stuff written by Jacob Hallén and others.

People with lot of important and interesting things to teach, who have
managed to do a much better job than me at organizing their stuff on the
web, include for example Fredrik Lundh and Marc-Andre Lemburg.  The
latter gave an hour-long talk this summer at Europython on the subject.
Unfortunately I can't easily find his presentation on
www.europython.org, nor Fredrik's, but I'm sure that an abler searcher
than me will manage, and they do have their own websites as well.  In
any case, I'm sure that either of them could be (and often is, in their
respective professional practices) at least as effective as a teacher,
consultant or mentor, on large-scale software projects in Python, as me;
and the same applies no doubt to many others.  In fact, the Python world
is blessed, in my opinion, with quite a number of excellent people who
might fill such roles -- one more reason to consider Python for
large-scale, mission-critical development, in fact!!!-)


Alex