From Al.Niessner at gmx.net  Thu Apr 12 20:15:55 2012
From: Al.Niessner at gmx.net (Al Niessner)
Date: Thu, 12 Apr 2012 11:15:55 -0700
Subject: [Multiprocessing-sig] multiprocessing and Array problems
Message-ID: <1334254555.2395.14.camel@charon>


I am not subscribed to these lists but I do check them occasionally and
will check them more frequently looking for a response.

I am getting a pickling error that I do not understand. It seems the
shared arrays that I create cannot be pickled. Makes the
multiprocessing.Array fairly useless if it cannot be pickled. Hence, I
am guessing that I am doing something wrong and would like some help
spotting it.

This is basically a cut and paste from the examples in the
documentation:

import ctypes
import multiprocessing

def subproc (a, i):
    print ("From subprocess " + str(i))
    print (a[:])
    return

if __name__ == "__main__":
    nproc = 3
    print ("\nBuilding the array for the second computation.")
    b = multiprocessing.Array (ctypes.c_double, nproc*nproc, lock=False)

    print ("\nSending the task across to see what happens.")
    pool = multiprocessing.Pool(nproc)
    for i in range(nproc): pool.apply_async (subproc, kwds={'a':b,
'i':i})
    pool.close()
    pool.join()
    pass

Whether I run it on 2.6, 2.7, or 3.2 I get the same error.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
    put(task)
PicklingError: Can't pickle <class
'multiprocessing.sharedctypes.c_double_Array_9'>: attribute lookup
multiprocessing.sharedctypes.c_double_Array_9 failed


Anyone want to take a stab as to why this error message is being
generated? Again I have tried python 2.6.7, 2.7.2+, and 3.2.2.

Thanks for any and all help in advance.

-- 
Al Niessner

I have never found the companion that was so companionable as solitude.
- From Walden by Henry David Thoreau

The universe is indifferent, and life is brutal; however, it is man's
choice of behavior that makes them malevolent rather than benevolent.

Some will fall in love with life and drink it from a fountain
That is pouring like an avalanche coming down the mountain.
- From the song Pepper by the Butthole Surfers



From Al.Niessner at gmx.net  Thu Apr 12 20:30:37 2012
From: Al.Niessner at gmx.net (Al Niessner)
Date: Thu, 12 Apr 2012 11:30:37 -0700
Subject: [Multiprocessing-sig] multiprocessing and Array problems
In-Reply-To: <1334254555.2395.14.camel@charon>
References: <1334254555.2395.14.camel@charon>
Message-ID: <1334255437.2395.17.camel@charon>


Here is an update.

def subproc (i):
    print ("From subprocess " + str(i))
    print (b[:])
    return

if __name__ == "__main__":
    nproc = 3
    print ("\nBuilding the array for the second computation.")
    b = multiprocessing.Array (ctypes.c_double, nproc*nproc, lock=False)

    print ("\nSending the task across to see what happens.")
    pool = multiprocessing.Pool(nproc)
    for i in range(nproc): pool.apply_async (subproc, kwds={'i':i})
    pool.close()
    pool.join()
    pass

Does work. It is not what I need though. I need to be able to pass b as
an argument. The reason is that I create some of these in a loop and
they are not in a global space nor are they know prior to creating the
pool.

On Thu, 2012-04-12 at 11:15 -0700, Al Niessner wrote:
> I am not subscribed to these lists but I do check them occasionally and
> will check them more frequently looking for a response.
> 
> I am getting a pickling error that I do not understand. It seems the
> shared arrays that I create cannot be pickled. Makes the
> multiprocessing.Array fairly useless if it cannot be pickled. Hence, I
> am guessing that I am doing something wrong and would like some help
> spotting it.
> 
> This is basically a cut and paste from the examples in the
> documentation:
> 
> import ctypes
> import multiprocessing
> 
> def subproc (a, i):
>     print ("From subprocess " + str(i))
>     print (a[:])
>     return
> 
> if __name__ == "__main__":
>     nproc = 3
>     print ("\nBuilding the array for the second computation.")
>     b = multiprocessing.Array (ctypes.c_double, nproc*nproc, lock=False)
> 
>     print ("\nSending the task across to see what happens.")
>     pool = multiprocessing.Pool(nproc)
>     for i in range(nproc): pool.apply_async (subproc, kwds={'a':b,
> 'i':i})
>     pool.close()
>     pool.join()
>     pass
> 
> Whether I run it on 2.6, 2.7, or 3.2 I get the same error.
> 
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
>     self.run()
>   File "/usr/lib/python2.6/threading.py", line 484, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
> _handle_tasks
>     put(task)
> PicklingError: Can't pickle <class
> 'multiprocessing.sharedctypes.c_double_Array_9'>: attribute lookup
> multiprocessing.sharedctypes.c_double_Array_9 failed
> 
> 
> Anyone want to take a stab as to why this error message is being
> generated? Again I have tried python 2.6.7, 2.7.2+, and 3.2.2.
> 
> Thanks for any and all help in advance.
> 

-- 
Al Niessner

I have never found the companion that was so companionable as solitude.
- From Walden by Henry David Thoreau

The universe is indifferent, and life is brutal; however, it is man's
choice of behavior that makes them malevolent rather than benevolent.

Some will fall in love with life and drink it from a fountain
That is pouring like an avalanche coming down the mountain.
- From the song Pepper by the Butthole Surfers