[Web-SIG] WSGI: allowing short reads

[Guido van Rossum]

I am taking full responsibility for this inconsistency. The original
read(n) used stdio's fread(), which reads exactly n bytes or until EOF,
whichever comes first. The switch to 3.0 might have been a good time to fix
this, but we didn't, and now it's too late.

If I had to do it over again I would have read(n) return up to n bytes
using at most 1 syscall, and readexactly(n) return n bytes or bust (raising
EOFError if EOF is hit before n bytes are seen). This is what asyncio
streams use and what I recommend for stream-like APIs that don't require
strict compatibility.

Note that read() without parameter and read(-1) are also special (reading
everything until EOF). I think this is unambiguous and doesn't need to be
fixed.

On Sat, Sep 27, 2014 at 5:00 AM, Antoine Pitrou <antoine at python.org> wrote:

>
> Hi,
>
> Robert Collins <robertc at ...> writes:
> >
> > https://github.com/python-web-sig/wsgi-ng/issues/5
> >
> > tl;dr - we don't specify whether read(size) has to return size bytes
> > or just not more than size, today. the IO library is clear that
> > read(n) returns up to n, and also offers read1 that guarantees only
> > one read call.
>
> I think you've got things mixed up. There are two different things in
> "the IO library" (which is really the 3.x IO stack):
>
> * buffered binary I/O has read(n) and read1(n):
>   - read(n) will block until n bytes are received (except for non-blocking
>     fds)
>   - read1(n) will issue at most one system call and can return fewer than
>     n bytes
>
> * raw binary I/O only has read(n):
>   - read(n) will issue at most one system call and can return fewer than
>     n bytes
>
> So, depending on which layer you are placing yourself on, one or the
> other of your statements is wrong.
>
> That said, it would be reasonable for WSGI to expose the raw I/O layer,
> IMHO. "Prettifying" libraries can wrap it inside a BufferedReader if
> they like.
>
> Note that I once proposed generalized prefetching on I/O streams, but
> it was rejected:
> https://mail.python.org/pipermail/python-ideas/2010-September/008179.html
> (skip to the prefetch() proposal)
>
> It was in the context of improving streamed unpickling, which is
> a problem a bit similar - but less horrible - to JSON unserializing;
> since then, the problem was solved in a different way by adding a
> framing layer to pickle protocol 4 :-).
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Web-SIG mailing list
> Web-SIG at python.org
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe:
> https://mail.python.org/mailman/options/web-sig/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/web-sig/attachments/20140927/61f91eed/attachment.html>