Is there a "Large Scale Python Software Design" ?

[Alex Martelli]

GerritM <gmuller at worldonline.nl> wrote:

> "Alex Martelli" <aleaxit at yahoo.com> schreef in bericht
> news:1gm1x7t.w8k2hqrv300tN%aleaxit at yahoo.com...
> <...skip...>
> > Lastly, I'm still looking for systematic ways to test system integration
> > that are as effective as unit tests are for each single component or
> > subsystem; but that's an area where type and const checking are of just
> > about negligible help.
> >
> System integration has a completely different nature than unit testing.
> During system integration the "unforeseens" and the "unknowns" pop-up. And
> of course the not-communicated, implicit human assumptions are uncovered.

Exactly -- which is why I'm still looking (doesn't mean I think I'll
find;-).

> All of these problems are not addressed at all by static typing. However,

Essentially not.

> design clarity and compactness does help tremendously. I would expect for
> these reasons that Python is a big plus during system integration of large

Not as much as one might hope, in my experience.  Protocol Adaptation
_would_ help (see PEP 246), but it would need to be widely deployed.

> systems. Of course design attention is required to cope with the
> "non-functional" imapct of Python, such as CPU and memory consumption. on
> top of that (run-time) instrumentation is very helpful. Here again the
> dynamic nature of Python is a big plus.

But the extreme difficulty in keeping track of what amount of memory
goes where in what cases is a big minus.  I recall similar problems with
Java, in my limited experience with it, but for Java I see now there are
commercial tools specifically to hunt down memory problems.  In C++
there were actual _leaks_ which were a terrible problem for us, but
again pricey commercial technology came to the rescue.

With Python, I've found, so far, that tracking where _time_ goes is
quite feasible, with systematic profiling &c (of course profiling is
always a bit invasive, and so on, but no more so in Python than
otherwise), so that in the end CPU consumption is no big deal (it's easy
to find out the tiny hot spot and turn it into an extension iff needed).
But memory is a _big_ problem, in my experience so far, with servers
meant to run a long time and having very large code bases.  I'm sure
there IS a commercial niche for a _good_ general purpose Python tool to
keep track of memory consumption, equivalent to those available for C,
C++ and Java...


Alex