pickle, modules, and ImportErrors

[Ian Kelly]

On Wed, Jan 7, 2015 at 1:12 PM, John Ladasky <john_ladasky at sbcglobal.net> wrote:
> Do I need to "import my_svr.model as model" then?  Adding that line changes nothing. I get the exact same "ImportError: No module named 'model'".
>
> Likewise for "import my_svr", "from my_svr import *", or even "from my_svr.model import SVRModel".
>
> It is clear that I'm failing to understand something important.
>
> I do not have any circular import dependencies; however, some of the files in my package do need to import definitions from files earlier in my data pipeline.  In order to make everything work inside the module, as well as making a parent-folder "import my_svr" work from a iPython,  I find myself needing to use statements like these inside my training.py program:
>
>
> try:
>     from model import *
>     from sampling import *
> except ImportError:
>     from .model import *
>     from .sampling import *
>
>
> This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.

It sounds like you have import path confusion. The first imports there
are absolute imports. If the absolute paths of the named modules are
just "model" and "sampling", then that would be correct. However, it
sounds like these modules are part of a "my_svr" package, so the
absolute paths of these modules are actually "my_svr.model" and
"my_svr.sampling". In that case, the the absolute path that you're
trying to use for the import is wrong. Either fully specify the module
path, or use the relative import you have below.

Now, why might those absolute imports appear to work even though the
paths are incorrect? Well, it sounds like you may be running a file
inside the package as your main script. This causes some problems.

1) Python doesn't realize that the script it's running is part of a
package. It just calls that module '__main__', and if something else
happens to import the module later by its real name, then you'll end
up loading a second copy of the module by that name, which can lead to
confusing behavior.

2) Python implicitly adds the directory containing the script to the
front of the sys.path list. Since the directory containing the package
is presumably already on the sys.path, this means that the internal
modules now appear in sys.path *twice*, with two different names:
"my_svr.model" and "model" refer to the same source file. Python
doesn't realize this however, and so they don't refer to the same
module. As a result, importing both "my_svr.model" and "model" will
again result in two separate copies of the module being loaded, with
two different names.

It sounds like your pickle file was created from an object from the
"model" module. This works fine when the sys.path is set such that
"model" is the absolute path of a module, i.e. when you run your main
script within the package. When running an external main script,
sys.path doesn't get set that way, and so it can't find the "model"
module.

The solution to this is to avoid executing a module within a package
as your main script. Run a script outside the package and have it
import and call into the package instead. An alternative is to run the
module using the -m flag of the Python executable; this fixes problem
2 above but not problem 1.