Socket logic problem

[John O'Hagan]

On Sun, 25 Oct 2009, Gabriel Genellina wrote:
> En Sat, 24 Oct 2009 06:40:08 -0300, John O'Hagan <research at johnohagan.com>
> 
> escribió:
> > I have several instances of the same generator function running
> > simultaneously, some within the same process, others in separate
> > processes. I
> > want them to be able to share data (the dictionaries passed to them as
> > arguments), in such a way that instances designated as "leaders" send
> > their
> > dictionaries to "follower" instances.
> >
> > I'm trying to use sockets to relay the dicts in pickled form, like this:
> >
> > from socket import socket
> >
> > PORT = 2050
> > RELAY = socket()
> > RELAY.bind(('', PORT))
> > RELAY.listen(5)
> >
> > PICKLEDICT = ''
> > while 1:
> >     INSTANCE = RELAY.accept()[0]
> >     STRING = INSTANCE.recv(1024)
> >     if STRING == "?":
> >         INSTANCE.send(PICKLEDICT)
> >     else:
> >         PICKLEDICT = STRING
> >
> > What I was hoping this would do is allow the leaders to send their dicts
> > to
> > this socket and the followers to read them from it after sending an
> > initial
> > "?", and that the same value would be returned for each such query until
> > it
> > was updated.
> >
> > But clearly I have a fundamental misconception of sockets, as this logic
> > only
> > allows a single query per connection, new connections break the old
> > ones, and
> > a new connection is required to send in a new value.
> 
> You may use sockets directly, but instead of building all infrastructure
> yourself, use a ThreadingTCPServer (or ForkingTCPServer), they allow for
> simultaneous request processing. Even setting up a SimpleXMLRPCServer
> (plus either ThreadingMixIn or ForkingMixIn) is easy enough.
> 
> > Are sockets actually the best way to do this? If so, how to set it up to
> > do
> > what I want? If not, what other approaches could I try?
> 
> See the wiki page on distributed systems:
> http://wiki.python.org/moin/DistributedProgramming
> 
Thanks for that, I didn't realize this was such a complex problem until 
reading the SocketServer docs and the above link. I think I'll aim for using 
something like Pyro when I can get a handle on it; but this quote from the 
socket how-to docs was interesting:

"If you need fast IPC between two processes on one machine, you should look 
into whatever form of shared memory the platform offers. A simple protocol 
based around shared memory and locks or semaphores is by far the fastest 
technique."

So I'll also look into that approach - ICBW but I get the feeling the threaded 
client/server network model may be too much baggage for what is really an IPC 
problem. In fact, while looking into this I've just been using a simple temp 
file to share the data and that's working fine, although it's relatively slow.

Thanks for the pointers,

Regards,

John