[IPython-dev] Using IPython Parallel inside a class

[Rafał Skolasiński]

Hi All,

I found out that the "Hamiltonian" class I will be using is not good for
pickling, therefore I used little different approach to define my solver as
a separate class (that just gets the momentum value) and create object of
it for every run separately.

My solution:
http://www.codesend.com/view/ed72e945804613a914dbddb0d9d23e06/
http://www.codesend.com/view/87df0f95e98e812398e444750784c907/


I even included passing some additional parameters as a default argument to
the function.

If someone would have better solution I will appreciate any feedback.

Cheers,
Rafal

2014-12-04 2:58 GMT+01:00 Michael McKerns <mmckerns at caltech.edu>:

> Try replacing the serialization engine with `dill` or `cloudpickle`.
> That's probably your best chance at serializing and shipping the
> objects across the wire.  IPython.parallel lets you override the
> serializer with either of these.
>
>
> > On Wed, Dec 3, 2014 at 12:37 PM, Rafał Skolasiński
> > <rjtskolasinski at gmail.com
> >> wrote:
> >
> >> Hi,
> >>
> >> It doesn't look like a solution. I still get bunch of errors. And I
> >> would
> >> like to use self.hamiltonian instance, not create more instances of it.
> >> the
> >> hamiltonian object in my real (not just a test) simulation code enters
> >> as
> >> the argument to the Simulation class constructor.
> >>
> >
> > Your original code gave errors trying to serialize the Hamiltonian
> > instance
> > in order to make copies for each machine in the cluster. Some objects
> > can't
> > be serialized like that. In any event, each machine needs to have access
> > to
> > the data that it will process. So, you need to somehow get that data (in
> > whole or in part) to each machine.
> >
> > Hope that helps,
> >
> > -Doug
> >
> >
> >
> >>
> >> And I would also like to execute hamiltonian.get_matrix method in
> >> parallel. The version of that method I use in my real simulation code is
> >> a
> >> little more complex and also take some time to be executed. Not maybe as
> >> much as diagonalization but still.
> >>
> >> Cheers,
> >> Rafal
> >>
> >> 2014-12-03 18:28 GMT+01:00 Doug Blank <doug.blank at gmail.com>:
> >>
> >>> I haven't done much parallel IPython, but it looks like you just need
> >>> to
> >>> define/import everything on the remote clients, something like:
> >>>
> >>> def get_energy(momentum):
> >>>     import scipy.sparse.linalg as sla
> >>>     from scipy.sparse import coo_matrix
> >>>     from scipy.sparse import csc_matrix
> >>>     import numpy as np
> >>>     class Hamiltonian:
> >>>         def get_matrix(self, k):
> >>>             # sleep(3.0)
> >>>             return k**2 * np.diag(range(100))
> >>>
> >>>     hamiltonian = Hamiltonian()
> >>>     mat = hamiltonian.get_matrix(momentum)
> >>>     mat = csc_matrix(mat)
> >>>     ev = sla.eigsh(mat,k=5, sigma = 0.5, which = 'LM')
> >>>     return ev[0]
> >>>
> >>> def compute_parallel(momenta):
> >>>     rc = parallel.Client()
> >>>     dview = rc[:]
> >>>     spectrum = dview.map_sync(get_energy,momenta)
> >>>     return spectrum
> >>>
> >>> -Doug
> >>>
> >>> On Wed, Dec 3, 2014 at 12:21 PM, Rafał Skolasiński <
> >>> rjtskolasinski at gmail.com> wrote:
> >>>
> >>>> Oh, in the file I attached there should ofc be line
> >>>> spectrum = dview.map_sync(self.get_energy,momenta)
> >>>> instead of
> >>>> spectrum = dview.map_sync(self.get_energy_for_parallel,momenta)
> >>>>
> >>>> Sorry for mistake.
> >>>> Raphael
> >>>>
> >>>>
> >>>> 2014-12-03 18:05 GMT+01:00 Rafał Skolasiński
> >>>> <rjtskolasinski at gmail.com>:
> >>>>
> >>>>> Hi Guys,
> >>>>>
> >>>>> I hope this is the right place to ask this kind of question.
> >>>>>
> >>>>> I would like to use IPython parallel in my physical simulation.
> >>>>> In my mind I've got class Simulation, which contain as a field object
> >>>>> hamiltonian of class Hamiltonian.
> >>>>>
> >>>>> Method get_energy should call  hamiltonian.get_matrix(momentum),
> >>>>> diagonalize it with scipy method and return energies.
> >>>>>
> >>>>> I would like to execute it in parallel for a list of different
> >>>>> momenta.
> >>>>>
> >>>>>
> >>>>> I tested it with a standard python 'map' function and it worked so I
> >>>>> thought that easiest way would be to use 'dview.map_sync' in the same
> >>>>> way
> >>>>> but it doesn't looks like a proper way of doing it.
> >>>>>
> >>>>> I am attaching my testing code. I would be very grateful for any help
> >>>>> and ideas how to do it in a nice and clean way.
> >>>>>
> >>>>> Cheers,
> >>>>> Raphael
> >>>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> IPython-dev mailing list
> >>>> IPython-dev at scipy.org
> >>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> IPython-dev mailing list
> >>> IPython-dev at scipy.org
> >>> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >>>
> >>>
> >>
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/ipython-dev
> >>
> >>
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at scipy.org
> > http://mail.scipy.org/mailman/listinfo/ipython-dev
> >
>
>
> ---
>
> Mike McKerns
> California Institute of Technology
> TEL: (626)395-5773 or (626)590-8470
> http://www.its.caltech.edu/~mmckerns
> mmckerns at caltech.edu
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20141204/92ed58b1/attachment.html>