Select weirdness

[Donn Cave]

In article <rNOSPAMon-934744.11420922042007 at news.gha.chartermi.net>,
 Ron Garret <rNOSPAMon at flownet.com> wrote:

> The answer is obvious: select is looking only at the underlying socket, 
> and not at the rfile buffers.
> 
> So... is this a bug in select?  Or a bug in my code?

Yes.

I don't see any specific followup to this point, but it is or
at least should be a well known limitation of C library I/O and
select.  select() is an operating system function that has no
way to know the state of your process I/O buffers, nor do the C
stdio buffers don't come with a standard way to inspect that state.
Therefore, if you mix C I/O with select(), you're more or less out
of luck.  This applies directly to Python, because it calls the
operating system select() function and it uses C stdio buffers for
its file object (and entices you to make this mistake by supporting
a file object as an input to select().)

This conflict can be relieved after a fashion by eliminating the
buffer.  The now unbuffered C fgets / Python readline won't store
extra lines that select can't see, but the semantics of the operation
are still a poor fit with the usual application of select.  The
number of system-level I/O calls is significantly greater as you
transfer data one byte at a time, which may add a lot of overhead
if you transfer a lot of data, and your readline() function still
can't return until it gets that '\n', so a half line can block your
application.

It isn't a lot of work to read data with operating system functions
that are compatible with select - os.read(), socket.recv() - and
break it up into lines on your own, and this completely and efficiently
resolves the problem.

I haven't looked at your code.

   Donn Cave, donn at u.washington.edu