High performance IO on non-blocking sockets

[Jeremy Hylton]

On Fri, 2003-03-14 at 05:42, Troels Walsted Hansen wrote:
> Now for recv operations on non-blocking sockets. Assume that I want to read
> a known number of bytes (total_recv_size) from a socket and assemble the
> result as a Python string called self.data (again, think anywhere from 1 to
> hundreds of megabytes of data).
> 
> Approach #1 (list+string.join based):
> 
>   self.data = []
> ...
>   # following code runs when socket is read-ready
>   recv_size = 64*1024 # for example
>   data = self.socket.recv(recv_size)
>   self.data.append(data)
> ...
>   self.data = ''.join(self.data)
> 
> All these three approaches have faults. #1 will cause memory fragmentation
> by allocating len(data) strings (in an unlikely worst case, recv() returns 1
> byte for each recv()).

What's the expected case?  

Jeremy