Request for Enhancement

[Kevin Russell]

"Samuel A. Falvo II" wrote:
> 
> I have need to process very large text files in Python, but I don't have any
> idea how long the files are going to be in real-world situations.  It is
> unfortunate that there is no F.eof() function, where F is a Python file.
> 
> Here's what I *want*:
> 
>         while not F.eof():
>                 l = F.readline()
>                 ...process line...
> 
> As it is, I have to do the following:
> 
>         l_list = F.readlines()  #note plural
>         for line in l_list:
>                 ... process line ...
> 
> While this is fine for my test cases, it could consume unacceptable amounts
> of memory when fed large text files.
> 


Try the fileinput module.  It reads lines lazily (only one at a time), 
and it's a whole lot easier to type.

    import fileinput

    for line in fileinput.input("myfile.txt"):
        # do whatever

For further magic, you can give it a list of files, as in:

    for line in fileinput.input(glob.glob("*.txt")):
        # do whatever


For really deep magic, give input() no arguments at all and it'll 
grab the filenames from the command-line (i.e., sys.argv[1:]).

-- Kevin Russell