multiprocess passing arguments double asterisks

[ricemom at gmail.com]

On Wednesday, October 26, 2016 at 5:31:18 PM UTC-5, MRAB wrote:
> On 2016-10-26 21:44, pic8690 at gmail.com wrote:
> > On Monday, October 24, 2016 at 12:39:47 PM UTC-5, Thomas Nyberg wrote:
> >> On 10/24/2016 12:45 PM, pic8690 at gmail.com wrote:
> >> > Thanks for the reply.
> >> >
> >> > The code snippet given by Peter is not very clear
> >> >
> >> > I would like to multiprocess a function which is written in python of the form bar(**kwargs) which returns a value.  This example does not return anything
> >> >
> >> > Would you please use my example for the map function?
> >> >
> >> > I appreciate your help,
> >> >
> >> I'm honestly not totally sure what you want to do. However, say you want
> >> to do the following (btw this is basically what Dennis said i nhis last
> >> email, but maybe I can help clarify):
> >>
> >> 	kwargs = {'param1': val1, 'param2': val2})
> >>
> >> Then you'd like to have the following two operations performed in
> >> separate processes:
> >>
> >> 	bar(param1=val1)
> >> 	bar(param2=val2)
> >>
> >> In that case, I guess I would do something like the following. First
> >> define bar_wrapper as follows (*I haven't tested any code here!):
> >>
> >> def bar_wrapper(pair):
> >> 	key, val = pair
> >> 	return bar(**{key: val})
> >>
> >> Then I would probably do something like
> >>
> >> map(bar_wrapper, kwargs.items())
> >>
> >> I.e. basically what I'm doing is taking the key-val pairs and producing
> >> a list of them (as tuples). This is something that you can apply map
> >> too, but not with the original function. So then the wrapper function
> >> converts the tuple back to what you want originally.
> >>
> >> Hopefully I'm understanding correctly and hopefully this helps.
> >>
> >> Cheers,
> >> Thomas
> >
> > Thomas,
> >
> > I have strings & numpy.ndarray as arguments.  The wrapper function works great for strings.
> >
> > Here's what I'm trying to do...
> > **************************
> >
> > I have a serial python fn of the form
> >
> > def bar(**kwargs):
> >   a=kwargs.get("name")
> >   print a
> >   self.a1d=ma.asanyarray(kwargs.get('a1d'), dtype=float)
> >   (****more calculations--takes a long time to compute
> >   returns an object)
> >
> > I am trying to run this function in parallel.
> >
> > Here's my Main program
> >
> > import numpy as np
> > import numpy.ma as ma
> > from delegate import parallelize
> > from hashlib import sha1
> > a1d=np.zeros((32))
> > b1d=np.zeros((32))
> > p=open("/tmp/pdata","rb")
> > pdata=np.load(p)
> > for i in range(0,10):
> >   a1d=pdata['t1d']
> >   b1d=pdata['gz1d']
> >   print a1d,b1d
> >   kwargs={'name':'special','a':a1d,'b':b1d}
> >   val=parallelize(bar,kwargs)
> > ***************************************
> > Error:
> > line 30, in <module>
> >     val=parallelize(bar,kwargs)
> >    delegate.py", line 311, in process
> >     item, items = items[0], items[1:]
> > KeyError: 0
> > **************************************
> > Error: (with the wrapper function)
> > val=parallelize(bar_wrapper,kwargs.items())
> > TypeError: unhashable type: 'numpy.ndarray'
> > ***************************************
> >
> > How do I pass the string & numpy.ndarray to the function bar(**kwargs)?
> >
> > Thank you for suggestions & help,
> >
> >
> > a1d=np.zeros((32))
> > b1d=np.zeros((32))
> > p=open("/tmp/pdata","rb")
> > pdata=np.load(p)
> > for i in range(0,100):
> >   a1d=pdata['t1d']
> >   b1d=pdata['gz1d']
> >   print a1d,b1d
> >   kwargs={'name':'special','a':a1d,'b':b1d}
> >   val=parallelize(bar,kwargs)
> >
> 'parallelize' expects a function and a list of items. It calls the 
> function with each item in the list in parallel, passing the item as the 
> argument of the function.
> 
> That _single_ argument item can be a tuple or a dict.

--if the dict has numpy arrays it fails..

In the above example if
   kwargs={'name':'special','a':2}
   val=parallelize(bar_wrapper,kwargs.items()) it works fine

But if
     kwargs={'name':'special','a':a1d,'b':b1d}
it fails  (a1d & b1d are numpy arrays)