reload() problems (was Re: mutlifile inheritance problem simplified)

[Aahz]

In article <3C9CAFD2.13FE372D at hotmail.com>,
Marc  <maurelius01 at hotmail.com> wrote:
>
>I'm wondering whether or not this is a bug, which I would report, or
>whether I'm doing something incorrectly.  I can accept that others
>don't use reload much, but it seems that identifying bugs in the
>language is good for the language, or identifying known errors in my
>usage is good for me.

No argument here; that's a good attitude.  The problem is that reload()
interacts with Python namespaces in peculiar ways (as you're
discovering), and even if something isn't a bug, it's quite likely to
not function as you'd desire.  (I've seen enough about reload() to say
this with confidence.)

>For myself, I use reload all the time.  Here's an example.  For my
>current task, I'm analysing large data sets (~.01-30 GB), never doing
>the same analysis twice.  This involves many steps, and a complete
>analysis takes alot of time about 20 minutes - 50+ hours.  Most of this
>is done using old classes that I've written in different files and many
>of these inherit from base classes in other files.
>
>The goal is to write an analysis script and let it run, but, of course,
>it rarely works straight away.  I run stuff from IDLE.  If the analysis
>bombs, I can catch the exception and pickup where I left off without
>losing what I've already done:  IDLE just waits for me to notice the
>exception, then using reload, I can modify the classes that caused the
>problem, even a base class that I wrote written 4 years ago and is ten
>layers deep in an inheritance tree, or using numbered DLLs I can even
>modify a C extension on the fly, all without losing anything.  Then
>just rerun, picking up where I left off.  This is great, and I don't
>want to give it up.  I can't see a reanable way to restart python every
>time I have an error.
>
>Anyway, I know this hasn't revived your interest in reload.  I expect
>it's use is very task specific.  I've found it extremely useful.

Seems to me that it would be better/safer to change your architecture so
that you can save checkpoints of intermediate processing.  I mean, what
if your machine just dies (say, power supply failure or a bad disk); do
you really want to lose more than two days of work?  I bet that if you
do this, you'll find many ways of improving your architecture by
streamlining and packaging the information flow.  This also would allow
other opportunities for modifying the processing in the middle of a run
(or restarting a run), quite possibly cutting out many hours of
processing time.

I've changed the Subject: of this thread so that if someone is
interested in reload() zie can find it.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"We should forget about small efficiencies, about 97% of the time.
Premature optimization is the root of all evil."  --Knuth