Parent module not loaded error

[Graham Dumpleton]

On Jan 14, 9:20 pm, Ståle Undheim <staa... at gmail.com> wrote:
> I have a pretty strange error that I can't figure out the cause off.
> This is in a Django app.
>
> I am using berkelydb, with secondary databases for indexing. The
> secondary databases are associated with a callback that uses cPickle
> to serialize index values. The problem is that cPickle.dumps(value)
> fails when I run it throughmod_wsgior mod_python on the deployment
> server (Ubuntu Linux 8.10), but works fine when when I use Djangos
> runserver on the same computer. Also, on my own computer (Ubuntu Linux
> 8.10), it works fine through mod_python.
>
> Each time I call on cPickle.dumps(value) I get:
> SystemError("Parent module 'd4' not loaded",)
>
> Using sys.exc_info() I get no type or traceback, just that exception.
> The callstack as far as I can figure is:
>
> django
> view
> bsddb.put
> indexer callback
> cPickle.dumps
>
> cPickle.loads works fine within the callback, I can also use
> cPickle.dumps() outside the indexer callback. But inside the callback,
> I need to use normal pickle instead. I just want to know why. I am
> assuming it has something to do with the fact that I go from python to
> C (bsddb) back to python (indexer callback) and back to C (cPickle).
> It is still strange then that cPickle.loads() work, but not
> cPickle.dumps(). I am also only storing integer values when I get this
> error, so no fancy list, dictionaries or objects.

Where is module 'd4'? Is that one of yours defined inside of your
Django project?

One of the problems with Django runserver is that the parent of the
site directory and the site directory are effectively starting points
for searching for modules, even though neither is explicitly listed in
sys.path. The parent is used because runserver does momentarily add
parent to sys.path to import site package root, after which it removes
directory from sys.path. The site directory itself is used because for
runserver that is the current working directory and imports look there
automatically.

End result is that modules inside of site can be imported either via
site package root, or direct. For example, if d4.py was at same level
as settings.py file and site was called mysite, you could import it as
mysite.d4 or just d4. This is bad because means you could end up with
two copies of a module imported under the different name references.

If they way the pickle was done in application was such that reference
was via d4, and then when you deployed to mod_python or mod_wsgi you
only added parent directory of site to Python path, then it will not
be able to find d4, since at that point would only be found as
mysite.d4.

If this is the case, add the site directory to Python path by setting
PythonPath for mod_python or sys.path in WSGI script file for
mod_wsgi. Better still, fix up your code so module references in code,
ie., urls.py, or anywhere else, are always done by site package root
reference and not direct. Ie., such that it uses mysite.d4 and not
just d4.

Would that seem about right?

Graham