Is there a "Large Scale Python Software Design" ?

[GerritM]

"Alex Martelli" <aleaxit at yahoo.com> schreef in bericht
news:1gm21ye.rego7yswhcmsN%aleaxit at yahoo.com...
> GerritM <gmuller at worldonline.nl> wrote:
<...snip...>
> > design clarity and compactness does help tremendously. I would expect
for
> > these reasons that Python is a big plus during system integration of
large
>
> Not as much as one might hope, in my experience.  Protocol Adaptation
> _would_ help (see PEP 246), but it would need to be widely deployed.
>
I think that I understand how PEP 246 might be an improvement over the
current situation. However, I think that Python 2.3 capabilities already
result in smaller programs and presumably less modules than their equivalent
in Java (or C++). The Objective-C system that we created (360kloc in 1992,
600kloc in 1994) did have a signficant amount of classes that are today
covered by the standard build-ins.I expect that Java and C++ suffer from the
same problem. The packages that I used a long time ago in Java were less
natural than todays Python packages (this might be entirely different today,
I haven't touched Java for centuries, ehh years). My assumption is that
integration problems are at least proportional with the implementation size
(in kloc). So my unproven hypothesis is that since Python programs tend to
be smaller than their Java equivalent that the integration problems are
smaller, purely due to the size.

> > systems. Of course design attention is required to cope with the
> > "non-functional" imapct of Python, such as CPU and memory consumption.
on
> > top of that (run-time) instrumentation is very helpful. Here again the
> > dynamic nature of Python is a big plus.
>
> But the extreme difficulty in keeping track of what amount of memory
> goes where in what cases is a big minus.  I recall similar problems with
> Java, in my limited experience with it, but for Java I see now there are
> commercial tools specifically to hunt down memory problems.  In C++
> there were actual _leaks_ which were a terrible problem for us, but
> again pricey commercial technology came to the rescue.
>
In the same system mentioned above we build our own instrumentation. The
main part was based on insering a small piece of adminstrative code at every
object creation and deletion. This Object Instantation Tracing proved to be
a goldmine of information, including memory use. For instance the memory use
of Lists and Dictionaries could be traced for well defined use cases.
Besides this instrumentation we did the memory management of "bulkdata",
such as images, explicitly. This helps to keep the memory consumption within
specified boundaries and it helps to prevent memory fragmentation problems.
<...snip...>
> But memory is a _big_ problem, in my experience so far, with servers
> meant to run a long time and having very large code bases.  I'm sure
> there IS a commercial niche for a _good_ general purpose Python tool to
> keep track of memory consumption, equivalent to those available for C,
> C++ and Java...
>
The investment in the tools mentioned above were relatively small. However,
this works only if the entire system is based on the same architectural
rules.

The additional challenge of Python relative to Objective-C is its garbage
collection. This provides indeed a poorly predictable memory behavior.

Some of the design aspects mentioned here are described in this chapter of
my PhD thesis:
http://www.extra.research.philips.com/natlab/sysarch/MIconceptualViewPaper.p
df

kind regards, Gerrit