[IPython-dev] [IPython-User] ipyparallel: Can't pickly memoryview objects

[Fernando Perez]

Hi Florian,

sorry, this list is mostly deprecated, with all IPython discussions now
happening on a single list:

https://mail.scipy.org/mailman/listinfo/ipython-dev

The issue is the closure over xs in your lambda; pickle doesn't know how to
serialize closures.  You can make it work if you enable the cloudpickle
serializer instead:

    dview.use_cloudpickle()

as indicated here:
http://ipyparallel.readthedocs.io/en/latest/details.html#closures

An interesting tidbit is that if you move the code in `main` out to the
top-level of the script and manually push `xs` as well, the code works fine
even without the cloudpickle change.  That's because names that require a
lookup into globals() don't require the creation of a closure object, as
explained in the link above.

Cheers,

f


On Thu, Mar 16, 2017 at 2:07 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

> Hello,
>
> I have this small sample program:
>
> import ipyparallel as ipp, numpy as np
>
> class X:
>     def __init__(self, x, xs):
>         self.A = np.full((10, 10), x)
>         np.fill_diagonal(self.A, x*10)
>
>     def norm(self):
>         return np.linalg.norm(self.A)
>
>
> def main():
>     rc = ipp.Client()
>     lview = rc.load_balanced_view()
>     dview = rc[:]
>     dview.push({"X" : X})
>     dview.execute("import numpy as np")
>
>     xs = np.linspace(1, 100, 10)
>     sqxs = []
>
>     sqxs = lview.map(lambda x: X(x, xs), xs)
>     sqxs.wait()
>
>     for i in sqxs:
>         print(i.norm())
>
> if __name__ == "__main__":
>     main()
>
>
> trying it to run gives TypeError: can't pickle memoryview objects.
> Probably because I provide the xs array to X.
>
> How can I work around this situation?
>
> Thanks,
> Florian
>
> Complete traceback:
>
> Traceback (most recent call last):
>   File "ipython_parallel.py", line 30, in <module>
>     main()
>   File "ipython_parallel.py", line 23, in main
>     sqxs = lview.map(lambda x: X(x, xs), xs)
>   File "<decorator-gen-136>", line 2, in map
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 50, in sync_results
>     ret = f(self, *args, **kwargs)
>   File "<decorator-gen-135>", line 2, in map
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 35, in save_ids
>     ret = f(self, *args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 1109, in map
>     return pf.map(*sequences)
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py",
> line 285, in map
>     return self(*sequences, __ipp_mapping=True)
>   File "<decorator-gen-118>", line 2, in __call__
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py",
> line 76, in sync_view_results
>     return f(self, *args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py",
> line 259, in __call__
>     ar = view.apply(f, *args)
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 211, in apply
>     return self._really_apply(f, args, kwargs)
>   File "<decorator-gen-134>", line 2, in _really_apply
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 47, in sync_results
>     return f(self, *args, **kwargs)
>   File "<decorator-gen-133>", line 2, in _really_apply
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 35, in save_ids
>     ret = f(self, *args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py",
> line 1037, in _really_apply
>     metadata=metadata)
>   File "/usr/lib/python3.6/site-packages/ipyparallel/client/client.py",
> line 1395, in send_apply_request
>     item_threshold=self.session.item_threshold,
>   File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py",
> line 166, in pack_apply_message
>     serialize_object(arg, buffer_threshold, item_threshold) for arg in
> args))
>   File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py",
> line 166, in <genexpr>
>     serialize_object(arg, buffer_threshold, item_threshold) for arg in
> args))
>   File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py",
> line 112, in serialize_object
>     buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))
> TypeError: can't pickle memoryview objects
>
> _______________________________________________
> IPython-User mailing list
> IPython-User at scipy.org
> https://mail.scipy.org/mailman/listinfo/ipython-user
>



-- 
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20170320/3e17f429/attachment.html>