pickle, modules, and ImportErrors

[John Ladasky]

On Wednesday, January 7, 2015 12:56:29 PM UTC-8, Devin Jeanpierre wrote:

[snip]

> If you never run model directly, and only ever import it or run it as
> my_svr.model, then you will be fine, and pickles will all serialize
> and deserialize the same way.

Thank you Devin... I re-ran TrainingSession from within ipython, and everything worked the way I hoped.  I obtained an SVRModel object which I could pickle to disk, and unpickle in a subsequent session.  I don't actually need to run the test code that I appended to training.py any more, so I won't.

[snip]

> P.S. don't use pickle, it is a security vulnerability equivalent in
> severity to using exec in your code, and an unversioned opaque
> schemaless blob that is very difficult to work with when circumstances
> change.

For all of its shortcomings, I can't live without pickle.  In this case, I am doing data mining.  My TrainingSession class commandeers seven CPU cores via Multiprocessing.Pool.  Still, even my "toy" TrainingSessions take several minutes to run.  I can't afford to re-run TrainingSession every time I need my models.  I need a persistent object.

Besides, the opportunity for mischief is low.  My code is for my own personal use.  And I trust the third-party libraries that I am using.  My SVRModel object wraps the NuSVR object from scikit-learn, which in turn wraps the libsvm binary.

> > try:
> >     from model import *
> >     from sampling import *
> > except ImportError:
> >     from .model import *
> >     from .sampling import *
> >
> >
> > This bothers me.  I don't know whether it is correct usage.  I don't know whether it is causing my remaining ImportError problem.
> 
> This is a symptom of the differing ways you are importing these
> modules, as above. If you only ever run them and import them as
> my_svr.blahblah, then only the second set of imports are necessary.

OK, I will try refactoring the import code at some point.
 
> I hope that resolves all your questions!

I think so, thanks.