need helping tracking down weird bug in cPickle

[Tim Peters]

[Carl J. Van Arsdall]
> Hey everyone, cPickle is raising an ImportError that I just don't quite
> understand.

When that happens, the overwhelmingly most likely cause is that the
set of modules on your PYTHONPATH has changed since the pickle was
first created, in ways such that a module _referenced_ inside the
pickle can no longer be found.

For example, here's file resourcepmdb.py on my box:

def f():
    print "hi"

Here's a bit of Python code that uses it:

import cPickle

import resourcepmdb

pf = open("junk.pik", "wb")
pickler = cPickle.Pickler(pf)
pickler.dump([resourcepmdb.f])
pf.close()

Pickle stores module globals like user-defined classes and module
functions (the case above) not by saving the code they contain, but
_just_ by saving strings recording the module's name and the global's
name.  The unpickling process creates an import out of those strings.

So long as resourcepmdb stays on my PYTHONPATH, unpickling junk.pik
works fine.  For example, this code:

import cPickle

pf = open("junk.pik", "rb")
pickler = cPickle.Unpickler(pf)
whatever = pickler.load()
pf.close()
print whatever

prints something like

    [<function f at 0x009D8970>]

But if resourcepmdb gets off my path (for example, resourcepmdb.py
gets deleted, renamed, or moved -- or PYTHONPATH itself changes), that
same code dies with exactly the error you reported:

Traceback (most recent call last):
  File "pik2.py", line 5, in ?
    whatever = pickler.load()
ImportError: No module named resourcepmdb

> ...
> I've done some research and it looks like cPickle tries to load some
> modules as some kind of test.

That's unlikely ;-)

> From what I can tell the module that cPickle is trying to load is a
> concatenation of the module that it exists in and part of the string
> that was sent in the orignal request.

Sorry, can't tell from here.

> It looks like something where memory is getting overwritten, but I'm not
> really doing anything too fancy, so i'm not sure how to approach this
> problem.  I'm using python2.2.  Does anyone know how I might go about
> trying to troubleshoot something like this?  Is there a good tool or a
> better method of error handling I can use to try and track this down?
> Or is it better to give up troubleshooting and just upgrade to 2.4?

You could at least _install_ a recent Python so you could use the
newer pickletools module.  pickletools.dis() displays a readable
"disassembly" of a pickle, so that there's no need to guess about what
the pickle _thinks_ it's trying to do.  pickletools won't have any
problem working with pickles created by an older Python.

> ...
> Here's the traceback:
>
> Traceback (most recent call last):
>   File "/home/build/bin/resourceManager/resourceQueue.py", line 50, in
> initQueue
>     pickledList = cPickle.load(queueFPtr)
> ImportError: No module named resourcepmdb'

See above for the most obvious way to get an error like that.