Checking for EOF in stream

[kousue at gmail.com]

On Feb 19, 6:58 pm, GiBo <g... at gentlemail.com> wrote:
> Hi!
>
> Classic situation - I have to process an input stream of unknown length
> until a I reach its end (EOF, End Of File). How do I check for EOF? The
> input stream can be anything from opened file through sys.stdin to a
> network socket. And it's binary and potentially huge (gigabytes), thus
> "for line in stream.readlines()" isn't really a way to go.

Could you use xreadlines()? It's a lazily-evaluated stream reader.

> For now I have roughly:
>
> stream = sys.stdin
> while True:
>         data = stream.read(1024)
>         process_data(data)
>         if len(data) < 1024: ## (*)
>                 break
>
> I smell a fragile point at (*) because as far as I know e.g. network
> sockets streams may return less data than requested even when the socket
> is still open.

Well it depends on a lot of things. Is the stream blocking or non-
blocking (on sockets and some other sorts of streams, you can pick
this yourself)? What are the underlying semantics (reliable-and-
blocking TCP or dropping-and-unordered-UDP)? Unfortunately, you really
need to just know what you're working with (and there's really no
better solution; trying to hide the underlying semantics under a
proscribed overlaid set of semantics can only lead to badness in the
long run).

> I'd better like something like:
>
> while not stream.eof():
>         ...
>
> but there is not eof() method :-(
>
> This is probably a trivial problem but I haven't found a decent solution.

For your case, it's not so hard:
http://pyref.infogami.com/EOFError says "read() and readline() methods
of file objects return an empty string when they hit EOF." so you
should assume that if something is claiming to be a file-like object
that it will work this way.

> Any hints?

So:
stream = sys.stdin
while True:
        data = stream.read(1024)
        if data=="":
                break
        process_data(data)