Sockets: Sending/receiving arbitrary amounts of data

[Cary O'Brien]

In article <mailman.988566942.26667.python-list at python.org>,
Neil Schemenauer  <nas at python.ca> wrote:
>Daniel Klein wrote:
>> What if the server doesn't know in advance how much data will
>> be sent from the client?
>
>You need to build this into your protocol.  A lot of existing
>protocols use terminators to signal the end of data like "\r\n"
>or something similar.  I like using Dan Bernstein's netstring
>protocol:
>
>    http://cr.yp.to/proto/netstrings.txt
>
>Just an example, if you want to send an arbitrary string the
>sender would use:
>
>    def write_string(s, sock):
>        sock.send("%lu:" % len(s))
>        sock.send(s)

Woops!  Are you *SURE* that the python send will retry in the face
of short writes?  The docs say send returns the number of bytes
sent, so you probably need to check the send return value, slice
off what was sent, and try again.


>        sock.send(",")
>    
>    data = "Hello World"
>    write_string(data, sock)
>
>reciever:
>
>    def read_string(sock):
>        size = ""
>        while 1:
>            c = sock.recv(1)
>            if c == ":":
>                break
>            elif not c:
>                raise IOError, "short netstring read"
>            size = size + c
>        size = int(size)
>        s = sock.recv(size)


You gotta retry the recv() also if your data is bigger
than the socket buffering.  But someone else pointed
this out.

-- cary


>        if len(s) != size:
>            raise IOError, "short netstring read"
>        if sock.recv(1) != ",":
>            raise IOError, "missing netstring terminator"
>        return s
>
>    data = read_string(sock)
>        
>Obviously the global "sock" must be an open socket connecting the
>two programs.  Some advantages of the netstring protocal is that
>it is efficient, especially for large messages.  It makes it much
>easier to dynamicly manage memory since the reciever knows in
>advance how much data is coming.  It is also 8 bit clean which
>means you can send arbirary data and do things like embed
>netstrings inside of other netstrings.
>
>  Neil
>