Need some IPC pointers

[Devin Jeanpierre]

> I'm thinking sockets, but perhaps there's something simpler/easier.

Sockets are the only thing that will work without threads on all
platforms. You could also use threads and pipes. (I'm not actually
sure how threads+pipes works, but I'm told that it's a viable
approach).

Usually things are simpler and easier because they wrap all the
cross-platform messiness behind a nice API, and then give you a higher
level protocol for IPC on top of that. Unfortunately, the fewer external
dependencies you have, the more you have to rewrite yourself.

For what it's worth, I wrote something potentially similar using Twisted
and AMP. AMP is an Asynchronous Messaging Protocol: this basically
means that clients and servers can send messages to each other at any
time in any order. Twisted makes sure that the right response gets
associated with the right message. This can be very convenient -- you
might request something from another process, and then to compute its
answer it might ask for some additional information from you, and then
you give it that information, and it sends back the final result.

All the communication is done over TCP, usually using Twisted. So this
does involve bringing in a fairly large dependency. For me, the
tradeoff in programmer productivity was worth it, but of course it
really depends on your situation. A more synchronous protocol might be
more useful to you (although my impression was that perhaps you might
be interested in something like AMP), and synchronous protocols
directly on top of sockets are not so difficult to implement (although
making them work across platforms is still too hard for my taste).

Devin

On Wed, Nov 30, 2011 at 4:03 PM, Andrew Berg <bahamutzero8825 at gmail.com> wrote:
> I've done some research, but I'm not sure what's most appropriate for my
> situation. What I want to do is have a long running process that spawns
> processes (that aren't necessarily written in Python) and communicates
> with them. The children can be spawned at any time and communicate at
> any time. Being able to communicate with non-local processes would be
> nice, but is not necessary. The implementation needs to be
> cross-platform, but child processes will use the same OS as the parent
> during runtime.
> I don't think I'll ever need to transfer anything complicated or large -
> just strings or possibly tuples/lists. I'd rather not go outside the
> standard library (but I'd consider it). I don't need to worry about
> compatibility with older Python versions; if it only works with Python
> 3.2, that's not a problem.
> I'm thinking sockets, but perhaps there's something simpler/easier.
>
> --
> CPython 3.2.2 | Windows NT 6.1.7601.17640 | Thunderbird 7.0
> --
> http://mail.python.org/mailman/listinfo/python-list
>