Sockets: Sending/receiving arbitrary amounts of data

[Steve Holden]

"Daniel Klein" <danielk at aracnet.com> wrote in message
news:pneoetskbee729lra63ksqj5o06vkpeo84 at 4ax.com...
> Just starting to get my head around sockets and I'm using PP2E by Mark
Lutz as
> my guide.
>
> The examples all seem to indicate that you need to send/recv a specific
> (maximum) amount of data. The examples all use '1024' bytes to send or
recv.
> Although this can be increased, there is always the possiblility of
needing
> more.
>
Indeed. A big problem in networking generally is knowing what to expect!

> What if the server doesn't know in advance how much data will be sent from
the
> client? The server is waiting for '1024' bytes and the client sends down
10k.

The server's recv() call will return 1024 bytes, and the remainder will be
buffered somewhere in the socket between the client's (sender's) transport
layer and the receiver's (server's) transport layer.

On the other side, asking for 1024 bytes doesn't always mean that's what
you'll get. It's just a maximum.

When the other end closes its socket you should eventually receive a return
from a recv() call with no data. This indicates there will be no more data.
Otherwise your recv() calls will block (i.e., hang waiting for data) untilt
something comes in.

There is no need to have senders sending the same sized chunks of data as
receivers receive. You just keep reading more with recv() until the protocol
tells you this is the end of hte transmission.

> Does this mean I have to first send a "here's how much is coming" message
to
> the server so that the server can receive it in '10' 1k chunks?

Protocols at the application level are expected to make it clear how much
data will be transmitted, either by including byte counts or by using
message formats with unambiguous terminators.

>
And likewise if
> the server sends more than is expected to the client. I don't mind doing
it
> this way, I'd just like to know if there is a 'high-level' way to
implement
> this so that client and server handle the requests regardless of how much
data
> is being sent/recv'd ?
>
It's all pretty much in the hands of the protocol your application uses.

> Thanks in advance for any light you can shed on this,
>
> Daniel Klein
> Portland, OR USA
>
I wrote an FTP stream handler which uses socket calls, it may help you a
little.

    http://www.holdenweb.com/Python/PDCode/ftpStream.py

The test code for the module might give you some ideas, although there I'm
just reading the whole data stream until there's nothing left. To interact,
you'll need to detect the end of the client's requests at the server (and,
of course, vice versa).

regards
 Steve