Scatter/gather on sockets?

[Jean-Paul Calderone]

On 1 Apr 2006 14:56:02 -0500, Roy Smith <roy at panix.com> wrote:
>I've got a bunch of strings in a list:
>
>vector = []
>vector.append ("foo")
>vector.append ("bar")
>vector.append ("baz")
>
>I want to send all of them out a socket in a single send() call, so
>they end up in a single packet (assuming the MTU is large enough).  I
>can do:
>
>mySocket.send ("".join (vector))
>
>but that involves creating an intermediate string.  Is there a more
>efficient way, that doesn't involve that extra data copy?

There is a thing called "scatter-gather I/O", but it has nothing to do with lumping things together in a single packet.  It's all about minimizing memory copying on the way to the network.  The underlying call is writev(), and it is not exposed to Python applications in the standard library.  There are, however, several third-party modules which provide access to it (google for "python writev" - you'll find plenty of material).  You can also call it via the dl module, and in Python 2.5, the ctypes module.

Twisted also exposes this functionality via ITransport.writeSequence (silently falling back to writing "".join(iovec) if a more efficient scatter-gather mechanism is not available (which it isn't in any of the official Twisted reactors (but someday it will be and your program will magically go faster))).

Jean-Paul