Python recv loop

[Roy Smith]

In article <mailman.1612.1360544258.2939.python-list at python.org>,
 Ihsan Junaidi Ibrahim <ihsan at grep.my> wrote:

> I'm implementing a python client connecting to a C-backend server and am 
> currently stuck to as to how to proceed with receiving variable-length byte 
> stream coming in from the server.
> 
> I have coded the first 4 bytes (in hexadecimal) of message coming in from the 
> server to specify the length of the message payload i.e. 0xad{...}

Is this server that you're talking to something that you have control 
over, i.e. are you stuck with this protocol?  Given a choice, I'd go 
with something like JSON, for which pre-existing libraries for every 
language under the sun.

But, let's assume for the moment that you're stuck with this 
length-value encoding.  OK, but it's going to be more complicated than 
you think.  [I assume we're talking TCP here?]

Carefully read the documentation for socket.recv():

> socket.recv(bufsize[, flags]) [...] The maximum amount of data to be received 
> at once is specified by bufsize. 

Linger on the word "maximum", and try to grok the fullness of of how 
annoying that can be.  What it means is that if the other side sent 120 
bytes (octets), recv() might return all 120 at once, or it might return 
them one at a time, or anything in between.

So, what you need to do is call recv() repeatedly in a loop, each time 
passing it a value for bufsize which represents the amount left in the 
message (i.e. the original message length parsed earlier minus all the 
little bits and pieces that have been read so far).

Keep in mind, you also need to do this when you recv() the first 4 
octets, which make up the length field.  What you've got, recv(4), will 
work MOST of the time, but it's perfectly legal for recv() to return a 
short read.  You can't predict how fragmentation and retry timeouts and 
all sorts of low-level crud will cause your message boundaries to get 
scrambled.

> # receive message length
>     print 'receiving data'
>     mlen = sock.recv(4)
>     try:
>         nbuf = int(mlen, 16)
>     except ValueError as e:
>         print 'invalid length type'
>         return -1
> 
>     while True:
>         buf = sock.recv(nbuf)
> 
>         if not buf:
>             break
> 
>     slen = len(buf)
>     str = "{0} bytes received: {1}".format(slen, buf)
>     print str

Do you actually *know* what the value of nbuf is?  Is it possible that 
(somehow) it's 0?  You should print (log, whatever), the value of nbuf, 
just to make sure.

And, once you'e got all this working, tear it all out and convert to 
using something sane like JSON.  Let somebody else worry about all the 
horrible details.