Lightwight socket IO wrapper

[James Harris]

"Akira Li" <4kir4.1i at gmail.com> wrote in message 
news:mailman.37.1442754893.21674.python-list at python.org...
> "James Harris" <james.harris.1 at gmail.com> writes:
>
>> I guess there have been many attempts to make socket IO easier to
>> handle and a good number of those have been in Python.
>>
>> The trouble with trying to improve something which is already well
>> designed (and conciously left as is) is that the so-called 
>> improvement
>> can become much more complex and overly elaborate. That can apply to
>> the initial idea, for sure, but when writing helper or convenience
>> functions perhaps it applies more to the temptation to keep adding
>> just a little bit extra. The end result can be overly elaborate such
>> as a framework which is fine where such is needed but is overkill for
>> simpler requirements.
>>
>> Do you guys have any recommendations of some *lightweight* additions
>> to Python socket IO before I write any more of my own? Something 
>> built
>> in to Python would be much preferred over any modules which have to 
>> be
>> added. I had in the back of my mind that there was a high-level
>> socket-IO library - much as threading was added as a wrapper to the
>> basic thread module - but I cannot find anything above socket. Is
>> there any?
>
> Does ØMQ qualify as lightweight?

It's certainly interesting. It's puzzling, too. For example,

  http://zguide.zeromq.org/py:hwserver

The Python code there includes

  message = socket.recv()

but given that this is a TCP socket it doesn't look like there is any 
way for the stack to know how many bytes to return. Either ZeroMQ layers 
another end-to-end protocol on top of TCP (which would be no good) or it 
will be guessing (which would not be good either).

There are probably answers to that query but there is a lot of 
documentation, including on reliable communication, and that in itself 
makes ZeroMQ seem overkill, even if it can be persuaded to do what I 
want.

I am impressed that they show code in many languages. I may come back to 
it but for the moment it doesn't seem to be what I was looking for. And 
it is not built in.

>> A current specific to illustrate where basic socket IO is limited: it
>> normally provides no guarantees over how many bytes are transferred 
>> at
>> a time (AFAICS that's true for both streams and datagrams) so the
>> delimiting of messages/records needs to be handled by the sender and
>> receiver. I do already handle some of this myself but I wondered if
>> there was a prebuilt solution that I should be using instead - to 
>> save
>> me adding just a little bit extra. ;-)
>
> There are already convenience functions in stdlib such as
> sock.sendall(), sock.sendfile(), socket.create_connection() in 
> addition
> to BSD Sockets API.
>
> If you want to extend this list and have specific suggestions; see
>  https://docs.python.org/devguide/stdlibchanges.html

That may be a bit overkill just now but it's a good suggestion.

> Or just describe your current specific issue in more detail here.

There are a few things and more crop up as time goes on. For example, 
over TCP it would be helpful to have a function to receive a specific 
number of bytes or one to read bytes until reaching a certain delimiter 
such as newline or zero or space etc. Even better would be to be able to 
use the iteration protocol so you could just code next() and get the 
next such chunk of read in a for loop. When sending it would be good to 
just say to send a bunch of bytes but know that you will get told how 
many were sent (or didn't get sent) if it fails. Sock.sendall() doesn't 
do that.

I thought UDP would deliver (or drop) a whole datagram but cannot find 
anything in the Python documentaiton to guarantee that. In fact 
documentation for the send() call says that apps are responsible for 
checking that all data has been sent. They may mean that to apply to 
stream protocols only but it doesn't state that. (Of course, UDP 
datagrams are limited in size so the call may validly indicate 
incomplete transmission even when the first part of a big message is 
sent successfully.)

Receiving no bytes is taken as indicating the end of the communication. 
That's OK for TCP but not for UDP so there should be a way to 
distinguish between the end of data and receiving an empty datagram.

The recv calls require a buffer size to be supplied which is a technical 
detail. A Python wrapper could save the programmer dealing with that.

Reminder to self: encoding issues.

None of the above is difficult to write and I have written the bits I 
need myself but, basically, there are things that would make socket IO 
easier and yet still compatible with more long-winded code. So I 
wondered if there were already some Python modules which were more 
convenient than what I found in the documentation.

James