Looking for an IPC solution

[Aaron Brady]

On Friday, August 31, 2012 2:22:00 PM UTC-5, Laszlo Nagy wrote:
> There are just so many IPC modules out there. I'm looking for a solution 
> 
> for developing a new a multi-tier application. The core application will 
> 
> be running on a single computer, so the IPC should be using shared 
> 
> memory (or mmap) and have very short response times. But there will be a 
> 
> tier that will hold application state for clients, and there will be 
> 
> lots of clients. So that tier needs to go to different computers. E.g. 
> 
> the same IPC should also be accessed over TCP/IP. Most messages will be 
> 
> simple data structures, nothing complicated. The ability to run on PyPy 
> 
> would, and also to run on both Windows and Linux would be a plus.
> 
> 
> 
> I have seen a stand alone cross platform IPC server before that could 
> 
> serve "channels", and send/receive messages using these channels. But I 
> 
> don't remember its name and now I cannot find it. Can somebody please help?
> 
> 
> 
> Thanks,
> 
> 
> 
>     Laszlo

Hi Laszlo,

There aren't a lot of ways to create a Python object in an "mmap" buffer.  "mmap" is conducive to arrays of arrays.  For variable-length structures like strings and lists, you need "dynamic allocation".  The C functions "malloc" and "free" allocate memory space, and file creation and deletion routines operate on disk space.  However "malloc" doesn't allow you to allocate memory space within memory that's already allocated.  Operating systems don't provide that capability, and doing it yourself amounts to creating your own file system.  If you did, you still might not be able to use existing libraries like the STL or Python, because one address might refer to different locations in different processes.

One solution is to keep a linked list of free blocks within your "mmap" buffer.  It is prone to slow access times and segment fragmentation.  Another solution is to create many small files with fixed-length names.  The minimum file size on your system might become prohibitive depending on your constraints, since a 4-byte integer could occupy 4096 bytes on disk or more.  Or you can serialize the arguments and return values of your functions, and make requests to a central process.