Sockets: Sending/receiving arbitrary amounts of data

[Neil Schemenauer]

Cary O'Brien wrote:
> In article <mailman.988566942.26667.python-list at python.org>,
> Neil Schemenauer  <nas at python.ca> wrote:
> >    def write_string(s, sock):
> >        sock.send("%lu:" % len(s))
> >        sock.send(s)
> 
> Woops!  Are you *SURE* that the python send will retry in the face
> of short writes?  The docs say send returns the number of bytes
> sent, so you probably need to check the send return value, slice
> off what was sent, and try again.

Not sure.  My man page for send(2) seems to indicate that it does
unless the socket is in non-blocking mode.  I'll have to consult
Stevens when I get home.

> You gotta retry the recv() also if your data is bigger
> than the socket buffering.  But someone else pointed
> this out.

Right.  How about this:

    def _read_size(input):
        size = ""
        while 1:
            c = input.read(1)
            if c == ':':
                break
            elif not c:
                raise IOError, 'short netstring read'
            size = size + c
        return long(size)

    def write_string(s, output):
        output.write('%lu:' % len(s))
        output.write(s)
        output.write(',')

     def read_string(input):
        size = _read_size(input)
        data = ""
        while len(data) < size:
            s = input.read(size)
            if not s:
                raise IOError, 'short netstring read'
            data = data + s
        if input.read(1) != ',':
            raise IOError, 'missing netstring terminator'
        return data

    def write_file(input, output, blocksize=4096):
        size = os.fstat(input.fileno())[6]
        output.write("%lu:" % size)
        while 1:
            data = input.read(blocksize)
            if not data:
                break
            output.write(data)
        output.write(",")

    def read_file(input, output, blocksize=4096):
        size = _read_size(input)
        while size > 0:
            s = input.read(min(blocksize, size))
            if not s:
                raise IOError, 'short netstring read'
            output.write(s)
            size = size - len(s)
        if input.read(1) != ',':
            raise IOError, 'missing netstring terminator'


You have to use .makefile() on your sockets of course.

  Neil