pickle, modules, and ImportErrors

[John Ladasky]

I am progressing towards organizing a recent project of mine as a proper Python package.  It is not a huge package, about 700 lines of code so far.  But it breaks into logical pieces, and I don't feel like scrolling back and forth through a 700-line file.

I am running Python 3.4.0 on Ubuntu 14.04, if it matters.  

I want a package which I can use in an iPython session, as well as in programs which share the package directory.  Compatibility within the package directory appears to be easy.  From outside the package, I am getting ImportErrors that I have not been able to fix.

Although I have used pickle often, this is the first time that I have written a package with an __init__.py.  It reads thus:


# __init__.py for my_svr module

from .database import MasterDatabase
from .sampling import Input
from .sampling import MultiSample
from .sampling import Sample
from .sampling import Target
from .sampling import DataPoint
from .model import Prediction
from .model import SVRModel
from .model import comparison
from .training import TrainingSession

__all__ = ("MasterDatabase", "Input", "MultiSample", "Sample", "Target",
           "DataPoint", "Prediction", "SVRModel", "comparison",
           "TrainingSession")


If I execute "import my_svr" in an iPython interpreter, everything works as I think that I should expect:


In [8]: dir(my_svr)
Out[8]: 
['Input',
 'MasterDatabase',
 'MultiSample',
 'Prediction',
 'SVRModel',
 'Sample',
 'Target',
 'DataPoint',
 'TrainingSession',
 '__all__',
 '__builtins__',
 '__cached__',
 '__doc__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__path__',
 '__spec__',
 'comparison',
 'database',
 'model',
 'sampling',
 'training']



My training.py program produces an SVRModel object, and pickles it to a binary file.  The following simple program will unpickle the file saved by training.py, provided that the program resides in the module's own folder:


# reload_test.py

from pickle import load

with open("../models/sample model.pkl", "rb") as f:
    model = load(f)
print(model)


And, I get the str representation of my SVRModel instance.

However, a nearly-identical program in the parent folder fails (note that all I change is the relative path to the file):


# parent_folder_reload_test.py

from pickle import load

with open("models/sample model.pkl", "rb") as f:
    model = load(f)
print(model)


Traceback (most recent call last):
  File "reload test different directory.py", line 6, in <module>
    model = load(f)
ImportError: No module named 'model'


Do I need to "import my_svr.model as model" then?  Adding that line changes nothing. I get the exact same "ImportError: No module named 'model'".  

Likewise for "import my_svr", "from my_svr import *", or even "from my_svr.model import SVRModel".

It is clear that I'm failing to understand something important.

I do not have any circular import dependencies; however, some of the files in my package do need to import definitions from files earlier in my data pipeline.  In order to make everything work inside the module, as well as making a parent-folder "import my_svr" work from a iPython,  I find myself needing to use statements like these inside my training.py program:


try:
    from model import *
    from sampling import *
except ImportError:
    from .model import *
    from .sampling import *


This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.

Any advice is appreciated.  Thanks!