[newbie] trying socket as a replacement for nc

[Dan Stromberg]

On Fri, Dec 13, 2013 at 8:06 AM, Grant Edwards <invalid at invalid.invalid> wrote:
> On 2013-12-12, Dan Stromberg <drsalists at gmail.com> wrote:
>> On Thu, Dec 12, 2013 at 6:16 AM, Grant Edwards <invalid at invalid.invalid> wrote:
>>
>>>> Sockets reserve the right to split one socket.send() into multiple
>>>> socket.recv()'s on the other end of the communication, or to aggregate
>>>> multiple socket.send()'s into a single socket.recv() - pretty much any way
>>>> the relevant IP stacks and communications equipment feel like for the sake
>>>> of performance or reliability.
>>>
>>> Just to be pedantic: _TCP_ sockets reserver that right.  UDP sockets
>>> do not, and do in fact guarantee that each message is discrete.  [It
>>> appears that the OP is undoubtedly using TCP sockets.]
>>
>> I haven't done a lot of UDP, but are you pretty sure UDP can't at
>> least fragment large packets?  What's a router or switch to do if the
>> Path MTU isn't large enough for an original packet?
>>
>> http://www.gamedev.net/topic/343577-fragmented-udp-packets/
>
> You're conflating IP datagrams and Ethernet packets.  The IP stack can
> fragment an IP datagram into multiple Ethernet packets which are then
> reassembled by the receiving IP stack into a single datagram before
> being passed up to the next layer (in this case, UDP).

As long as you're saying this of UDP, I have no problem with it.

I've seen TCP fragment and not be reassembled though, which suggests
to me that the reassembly's happening in UDP rather than IP.  If it's
done by IP the same way for UDP and TCP, I'd not trust it in UDP
either.

> Did you read the thread you pointed to?  Your question was answerd by
> posting #4 in the thread you cited:
>
>    1) Yes, packets will be fragmented at the network layer (IP), but this
>       is something you do not have to worry about since the network
>       layer will reassemble the fragments before passing them back up
>       to the transport layer (UDP). UDP garentees preserved message
>       boundaries, so you never have to worry about only receiving a
>       packet fragment :~).

Actually, I believe the link I sent (which I skimmed) had people
coming down on both sides of the matter.  Some said that UDP would be
fine for small datagrams, while others said it would be fine,
irrespective of size.

>
> A few other references:
>
> http://tools.ietf.org/html/rfc791
>
>  1.1. Motivation
>
>   [...] The internet protocol provides for transmitting blocks of data
>   called datagrams from sources to destinations, [...] The internet
>   protocol also provides for fragmentation and reassembly of long
>   datagrams, if necessary, for transmission through "small packet"
>   networks.

I've personally seen this fail to occur in TCP - EG, it can cause a
stream of bytes to be written to tape with inconsistent block sizes if
transferred over rsh or ssh.  Usually the block sizes are consistent,
but not always.  Both SunOS 4.1.x and Ultrix had this issue; Ultrix
did it less often than SunOS, but Ultrix did do it.  I don't know for
certain if later *ix have the same issue, because I've been diligently
working around it ever since, but I suspect they do.

I've seen old time socket programmers explain that it cannot be relied
upon in TCP; send() and recv() and (read() and write()) are system
calls that return a length so that you can loop on them until all
relevant data has been transferred.  They don't return that length
just so you can ignore it.

>From the Socket HOWTO
(http://docs.python.org/2/howto/sockets.html#socket-howto) :
Now we come to the major stumbling block of sockets - send and recv
operate on the network buffers. They do not necessarily handle all the
bytes you hand them (or expect from them), because their major focus
is handling the network buffers. In general, they return when the
associated network buffers have been filled (send) or emptied (recv).
They then tell you how many bytes they handled. It is your
responsibility to call them again until your message has been
completely dealt with.

HTH