Multiprocessing, shared memory vs. pickled copies

[John Ladasky]

On Apr 7, 6:10 pm, sturlamolden <sturlamol... at yahoo.no> wrote:
> On 8 apr, 02:38, sturlamolden <sturlamol... at yahoo.no> wrote:
>
> > I should probably fix it for 64-bit now. Just recompiliong with 64-bit
> > integers will not work, because I intentionally hardcoded the higher
> > 32 bits to 0.
>
> That was easy, 64-bit support for Windows is done :-)
>
> Now I'll just have to fix the Linux code, and figure out what to do
> with os._exit preventing clean-up on exit... :-(
>
> Sturla

Hi Sturla,

Thanks for finding my discussion!  Yes, it's about passing numpy
arrays to multiple processors.  I'll accomplish that any way that I
can.

AND thanks to the discussion provided here by Philip and Robert, I've
become a bit less confused about pickling.  I have working code which
subclasses ndarray, pickles using __reduce_ex__, and unpickles using
__setstate__.  Just seven lines of additional code do what I want.

I almost understand it, too.  :^)  Seriously, I'm not altogether sure
about the purpose and structure of the tuple generated by
__reduce_ex__.  It looks very easy to abuse and break.  Referencing
tuple elements by number seems rather unPythonic.  Has Python 3 made
any improvements to pickle, I wonder?

At the moment, my arrays are small enough that pickling them should
not be a very expensive process -- provided that I don't have to keep
doing it over and over again!  I'm returning my attention to the
multiprocessing module now.  Processes created by Pool do not appear
to persist.  They seem to disappear after they are called.  So I can't
call them once with the neural net array, and then once again (or even
repeatedly) with input data.  Perhaps I need to look at Queue.

I will retain a copy of YOUR shmarray code (not the Bitbucket code)
for some time in the future.  I anticipate that my arrays might get
really large, and then copying them might not be practical in terms of
time and memory usage.