broken pipes, import errors (LONG, INVOLVED)

[Geoff Gerrietts]

Quoting Bengt Richter (bokr at oz.net):
> On Fri, 8 Mar 2002 16:13:54 -0800, Geoff Gerrietts <geoff at gerrietts.net> wrote:
> >At work, I'm seeing an "interesting" problem in my production
> >environment, or maybe two problems. It's one of those problems that
> 
> I don't have a clue about your situation, but it strikes me that exec and eval both
> use name spaces (local and global dictionaries) that you could specify, but don't.
> 
> I'm wondering if the default (current scope) is ok in your context.

If we were to specify namespaces, globals() and locals() would be the
appropriate namespaces to specify. So we don't.

In this case, exec and eval are both being used as shortcuts, because
the long form in either event is ... longer. A little more code for
the import, a little more code for the eval():

    statement = "import module.submodule.SubSubModule"
    exec statement

where statement consists of translates roughly into:
    
    toplevel = "module"
    modpath  = "module.submodule.SubSubModule"
    locals()[toplevel] = __import__(modpath)

The eval() structure can be similarly replaced with a looped getattr()
on each of the path components. It's just clumsier to do it this way.

Note: while "statement" is assigned directly in the sample code,
that's really just to show everyone what the variable contains.
"statement" is dynamically built, as is "object_nm".

We've tried replacing the eval and exec, to no effect. The variables
do contain the expected values. For two years, they've worked pretty
well. We have no idea what might have changed to make them stop
working pretty well.

> If you are repeatedly reusing the same current scope (e.g., doing
> those exec and eval lines inside a loop) for unrelated things, maybe
> there are name collisions?

When you try to import a module twice, import just does nothing (or at
least very little) after the first effort. In fact, when you first
import, Python caches a copy of the module in the global namespace.
When you subsequently import, it re-uses that cached copy.

The cached copy is also corrupt.

> Maybe logging the output of dir() before and after those lines would
> reveal something?

We've done that, and compared successful runs with runs where the
error erupts. "module" is populated correctly each time. "submodule"
varies fairly drastically in its contents between a successful run and
an unsuccessful run, but it's not clear why.

> Or try specifying local and global dicts and .clear() them after use
> (maybe the default environment is keeping something alive that
> should get garbage-collected and have its destructor executed? Or
> maybe one finally does fire off after a same-name rebinding of a
> reference, and the old destructor is doing inappropriate cleanup for
> the new environment? How does memory usage look?)

This error almost always occurs within the first twenty minutes of a
process (Zope process) that's accustomed to run for a week or so
without being restarted.

> These are just ideas trying to jog your thoughts, hopefully helpfully ;-)
> No specific recommendations -- you'll have to figure what's appropriate for your situation.

I appreciate all the efforts to help. Much of what you've suggested,
we've considered.

I may recommend trying local dictionaries which we then destroy after
exiting the function body, to see if we can elicit any more
information from doing so. I confess I have little hope that it will
solve our problem -- there's nothing happening in that namespace to
remove these objects. They just never show up in the first place; the
import quits halfway but fails to raise an error.

Thanks for the thoughts,
--G.

-- 
Geoff Gerrietts             "Punctuality is the virtue of the bored." 
<geoff at gerrietts net>                               --Evelyn Waugh