pickle, modules, and ImportErrors

[Devin Jeanpierre]

On Wed, Jan 7, 2015 at 2:12 PM, John Ladasky <john_ladasky at sbcglobal.net> wrote:
> If I execute "import my_svr" in an iPython interpreter, everything works as I think that I should expect:
-snip-
> However, a nearly-identical program in the parent folder fails (note that all I change is the relative path to the file):
> Traceback (most recent call last):
>   File "reload test different directory.py", line 6, in <module>
>     model = load(f)
> ImportError: No module named 'model'
>
>
> Do I need to "import my_svr.model as model" then?  Adding that line changes nothing. I get the exact same "ImportError: No module named 'model'".
>
> Likewise for "import my_svr", "from my_svr import *", or even "from my_svr.model import SVRModel".
>
> It is clear that I'm failing to understand something important.

in the first case, the model module was available as a top-level
module, "model". Pickles referenced that module when serialized. In
the second case, the model module was available as a submodule of the
top level my_svr package. So any pickles serialized from there would
use my_svr.model to refer to the model module. There *is* no model
module in this second case, so deserializing fails.

If you never run model directly, and only ever import it or run it as
my_svr.model, then you will be fine, and pickles will all serialize
and deserialize the same way.

For example, instead of python -i my_svr/model.py, you can use python
-im my_svr.model . (or ipython -im my_svr.model).

P.S. don't use pickle, it is a security vulnerability equivalent in
severity to using exec in your code, and an unversioned opaque
schemaless blob that is very difficult to work with when circumstances
change.

> I do not have any circular import dependencies; however, some of the files in my package do need to import definitions from files earlier in my data pipeline.  In order to make everything work inside the module, as well as making a parent-folder "import my_svr" work from a iPython,  I find myself needing to use statements like these inside my training.py program:
>
>
> try:
>     from model import *
>     from sampling import *
> except ImportError:
>     from .model import *
>     from .sampling import *
>
>
> This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.

This is a symptom of the differing ways you are importing these
modules, as above. If you only ever run them and import them as
my_svr.blahblah, then only the second set of imports are necessary.

P.S. don't use import *, and if you do use import *, don't use more
than one per file -- it makes it really hard to figure out where a
given global came from (was it defined here? was it defined in model?
was it defined in sampling?)

I hope that resolves all your questions!

-- Devin