High performance IO on non-blocking sockets

[Troels Walsted Hansen]

I'm trying to do IO on non-blocking sockets (within the asyncore framework),
and it seems to me that Python lacks a few primitives that would make this
more efficient.

Let's begin with socket writes. Assume that I have a string called self.data
that I want to send on a non-blocking socket. This string can be anywhere
from 1 to hundreds of megabytes.

This is a classic approach, seen in many Python examples:

  sent = self.socket.send(self.data)
  self.data = self.data[sent:]

This approach is bad because self.data gets reallocated for every socket
send. Worst case, 1 byte is sent each time and the realloc+copy cost goes
through the roof.

  send_size = 32*1024 # for example
  send_buffer_end = self.offset + send_size
  send_buffer = self.data[self.offset:send_buffer_end]
  sent = self.socket.send(send_buffer)
  self.offset += sent

This is the most efficient approach that I've been able to come up with. For
small strings, send_buffer is just a reference to self.data. For large
strings, send_size needs to be tuned to match socket buffers so we don't do
unnecessary copying.

It is still sub-optimal though. I think the optimal approach requires a
small extension of the socket module API, specifically an "offset" parameter
to socket.send().

  sent = self.socket.send(self.data, self.offset)
  self.offset += sent

This would be trivial to implement in socketmodule.c, and eliminate all
unnecessary slicing and copying on socket sends. I'll be glad to submit a
patch for this if the Python gods think this could be added to Python 2.3.

***

Now for recv operations on non-blocking sockets. Assume that I want to read
a known number of bytes (total_recv_size) from a socket and assemble the
result as a Python string called self.data (again, think anywhere from 1 to
hundreds of megabytes of data).

Approach #1 (list+string.join based):

  self.data = []
...
  # following code runs when socket is read-ready
  recv_size = 64*1024 # for example
  data = self.socket.recv(recv_size)
  self.data.append(data)
...
  self.data = ''.join(self.data)

Approach #2 (cStringIO):

  self.data = cStringIO.StringIO()
...
  # following code runs when socket is read-ready
  recv_size = 64*1024 # for example
  data = self.socket.recv(recv_size)
  self.data.write(data)
...
  self.data = self.data.getvalue()

Approach #3 (cStringIO with pre-allocation):

  self.data = cStringIO.StringIO()
  # make cStringIO allocate correctly sized buffer
  self.data.seek(total_recv_size)
  self.data.seek(0)
...
  recv_size = 64*1024 # for example
  # following code runs when socket is read-ready
  data = self.socket.recv(recv_size)
  self.data.write(data)
...
  self.data = self.data.getvalue()

All these three approaches have faults. #1 will cause memory fragmentation
by allocating len(data) strings (in an unlikely worst case, recv() returns 1
byte for each recv()). #2 will reallocate and copy buffer for every X number
of bytes received (cStringIO doubles buffer for every realloc). #3 will
pre-allocate a sufficiently large buffer, but cStringIO needs to initialize
this whole buffer to 0 in order to support random reads.

All approaches will use use at least total_recv_size*2 bytes of memory when
the final Python string is created.

Have I overlooked any better approaches?

I'm not sure what the best solution would be. Perhaps modifying cStringIO to
support an initial size-hint that pre-allocates a buffer without
initializing it until it is needed.

--
Troels Walsted Hansen