Reading a text file backwards

[Daniel Yoo]

Rick Holbert <holbertr at dma.org> wrote:
: Jay,

: Try this:

: myfile = open('myfile.txt', 'r')
: mylines = myfile.readlines()
: myfile.close()
: mylines.reverse()


Hi Rick,

But this probably won't work for Jay: he's running into memory issues
because the file's too large to hold in memory at once.  The point is
to avoid readlines().

Here's a generator that tries to iterate backwards across a file.  We
first get the file positions of each newline, and then afterwards
start going through the offsets.

###

def backfileiter(myfile):
    """Iterates the lines of a file, but in reverse order."""
    myfile.seek(0)
    offsets = _getLineOffsets(myfile)
    myfile.seek(0)
    offsets.reverse()
    for i in offsets:
        myfile.seek(i+1)
        yield myfile.readline()

def _getLineOffsets(myfile):
    """Return a list of offsets where newlines are located."""
    offsets = [-1]
    i = 0
    while True:
        byte = myfile.read(1)
        if not byte:
            break
        elif byte == '\n':
            offsets.append(i)
        i += 1
    return offsets
###



For example:

###    
>>> from StringIO import StringIO
>>> f = StringIO("""
... hello world
... this
... is a
... test""")

>>> f.seek(0)
>>> for line in backfileiter(f): print repr(line)
... 
'test'
'is a\n'
'this\n'
'hello world\n'
'\n'
###


Hope this helps!