Sharing objects between processes

[ET]

On Mon, 2009-03-09 at 11:04 -0700, Aaron Brady wrote:
> On Mar 9, 12:47 pm, ET <p... at 2drpg.org> wrote:
> > > Message: 2
> > > Date: Sun, 8 Mar 2009 12:00:40 -0700 (PDT)
> > > From: Aaron Brady <castiro... at gmail.com>
> > > Subject: Re: Sharing objects between processes
> > > To: python-l... at python.org
> > > Message-ID:
> > >    <5514c3df-d74e-47d8-93fc-34dd5119e... at c11g2000yqj.googlegroups.com>
> > > Content-Type: text/plain; charset=ISO-8859-1
> >
> > > On Mar 8, 1:36?pm, ET <p... at 2drpg.org> wrote:
> > > > I have been using the 'threading' library and decided to try swapping it
> > > > out for 'processing'... while it's awesome that processing so closely
> > > > mirrors the threading interface, I've been having trouble getting my
> > > > processes to share an object in a similar way.
> >
> > > > Using the 'with' keyword didn't work, and using normal locks doesn't
> > > > result in the expected behavior (I can get an object to be accessible in
> > > > more than one process, and Python indicates that the instances are
> > > > living at the same address in memory, but changes in one process are not
> > > > reflected in the other[s]). ?I'm sure this is because my expectations
> > > > are incorrect. :)
> >
> > > > The problem, as briefly as possible:
> > > > I have three processes which need to safely read and update two objects.
> >
> > > > I've been working with processing, multiprocessing, and parallel python,
> > > > trying to get this working... I suspect it can be accomplished with
> > > > managers and/or queues, but if there's an elegant way to handle it, I
> > > > have thus far failed to understand it.
> >
> > > > I don't particularly care which library I use; if someone has done this
> > > > or can recommend a good method they're aware of, it'd be incredibly
> > > > helpful.
> >
> > > > Thank you!
> >
> > > There is POSH: Python Object Sharing, which I learned about a while
> > > ago, but never used much.
> >
> > >http://poshmodule.sourceforge.net/
> >
> > > It's UNIX only.
> >
> > Thanks, I'll definitely keep that link handy... unfortunately, this
> > particular project needs to run on Windows as well as Linux-based
> > systems.
> 
> I don't recall whether there was anything in the source that's a deal-
> breaker on Windows.  The source is open, as you could see.
> 
> Other possibilities are 'shelve' and any database... fixed-length
> pickles, a directory of pickles, etc.  Maybe 'multiprocessing' would
> work for your synchronization, while you use a more custom technique
> for data exchange.
> 
> The only other thing I can do is bring to your attention an idea of
> mine for sharing primitives.  It's in the drawing board stage if you
> want to help.
> 
> Of course, it's only after you turned down 'multiprocessing' and
> 'POSH'.  It does things they don't and vice versa.
> --
> http://mail.python.org/mailman/listinfo/python-list

I assumed it wouldn't work in Windows as you mentioned it was UNIX-only;
the readme also states that it's for POSIX systems only.

I'd be more than happy to use multiprocessing; I've attempted to do so.
My question is largely how to implement it, as I have not managed to get
it working despite several attempts from different angles.

Unfortunately, I do need to handle more than primitives, otherwise I'd
have attempted to use the shared ctypes present in at least one of
processing/multiprocessing/parallel python.