cPickle fails on manually compiled and executed Python function

[dieter]

"Jan Gosmann" <jan at hyper-world.de> writes:

> today I came across some weird behaviour (a bug?) in Python 2.7.13 (on
> Linux) with the cPickle module. The pickle module works and so does
> the pickle module in Python 3.
>
> I have a file fn.py with a minimal function definition:
>
> ```
> def fn():
>     pass
> ```
>
> The actual code that I run is in a separate file (test.py):
>
> ```
> import cPickle
> import pickle
>
> def load_pyfile(filename):
>     source = ''
>     with open(filename, 'r') as f:
>         source += f.read()
>     code = compile(source, filename, 'exec')
>     loaded = {'__file__': filename}
>     exec(code, loaded)
>     return loaded
>
> fn = load_pyfile('fn.py')['fn']
>
> print(pickle.dumps(fn))
> print('----')
> print(cPickle.dumps(fn))
> ```
>
> The first print works fine, but the one with cPickle leads to an
> exception. Here is the output:
>
> ```
> c__main__
> fn
> p0
> .
> ----
> Traceback (most recent call last):
>   File "test.py", line 17, in <module>
>     print(cPickle.dumps(fn))
> TypeError: expected string or Unicode object, NoneType found

"pickle" (and "cpickle") are serializing functions as so called
"global"s, i.e. as a module reference together with a name.
This means, they cannot handle functions computed in a module
(as in your case).

I am quite convinced that "pickle" will not be able to deserialize (i.e. load)
your function (even though it appears to perform the serialization
(i.e. dump).

You are using "pickle/cPickle" for a case not anticipated (computed
functions). This means that this case is not as tested as other
cases. As a consequence, you can see different behavior between
"picke" and "cPickle".