multiprocessing and Array problems

[MRAB]

On 12/04/2012 19:30, Al Niessner wrote:
>
> Here is an update.
>
> def subproc (i):
>      print ("From subprocess " + str(i))
>      print (b[:])
>      return
>
> if __name__ == "__main__":
>      nproc = 3
>      print ("\nBuilding the array for the second computation.")
>      b = multiprocessing.Array (ctypes.c_double, nproc*nproc, lock=False)
>
>      print ("\nSending the task across to see what happens.")
>      pool = multiprocessing.Pool(nproc)
>      for i in range(nproc): pool.apply_async (subproc, kwds={'i':i})
>      pool.close()
>      pool.join()
>      pass
>
> Does work. It is not what I need though. I need to be able to pass b as
> an argument. The reason is that I create some of these in a loop and
> they are not in a global space nor are they know prior to creating the
> pool.
>
Doing this works:

     processes = []
     for i in range(nproc):
         processes.append(multiprocessing.Process(target=subproc, 
kwargs={'a':b, 'i':i}))
     for p in processes:
         p.start()
     for p in processes:
         p.join()

> On Thu, 2012-04-12 at 11:15 -0700, Al Niessner wrote:
>>  I am not subscribed to these lists but I do check them occasionally and
>>  will check them more frequently looking for a response.
>>
>>  I am getting a pickling error that I do not understand. It seems the
>>  shared arrays that I create cannot be pickled. Makes the
>>  multiprocessing.Array fairly useless if it cannot be pickled. Hence, I
>>  am guessing that I am doing something wrong and would like some help
>>  spotting it.
>>
>>  This is basically a cut and paste from the examples in the
>>  documentation:
>>
>>  import ctypes
>>  import multiprocessing
>>
>>  def subproc (a, i):
>>      print ("From subprocess " + str(i))
>>      print (a[:])
>>      return
>>
>>  if __name__ == "__main__":
>>      nproc = 3
>>      print ("\nBuilding the array for the second computation.")
>>      b = multiprocessing.Array (ctypes.c_double, nproc*nproc, lock=False)
>>
>>      print ("\nSending the task across to see what happens.")
>>      pool = multiprocessing.Pool(nproc)
>>      for i in range(nproc): pool.apply_async (subproc, kwds={'a':b,
>>  'i':i})
>>      pool.close()
>>      pool.join()
>>      pass
>>
>>  Whether I run it on 2.6, 2.7, or 3.2 I get the same error.
>>
>>  Exception in thread Thread-1:
>>  Traceback (most recent call last):
>>    File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
>>      self.run()
>>    File "/usr/lib/python2.6/threading.py", line 484, in run
>>      self.__target(*self.__args, **self.__kwargs)
>>    File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
>>  _handle_tasks
>>      put(task)
>>  PicklingError: Can't pickle<class
>>  'multiprocessing.sharedctypes.c_double_Array_9'>: attribute lookup
>>  multiprocessing.sharedctypes.c_double_Array_9 failed
>>
>>
>>  Anyone want to take a stab as to why this error message is being
>>  generated? Again I have tried python 2.6.7, 2.7.2+, and 3.2.2.
>>
>>  Thanks for any and all help in advance.
>>
>