iterating over a file with two pointers

Joshua Landau joshua at landau.ws
Thu Sep 19 03:04:51 EDT 2013


Although "tee" is most certainly preferable because IO is far slower
than the small amounts of memory "tee" will use, you do have this
option:

    def iterate_file_lines(file):
        """
        Iterate over lines in a file, unlike normal
        iteration this allows seeking.
        """
        while True:
            line = thefile.readline()
            if not line:
                break

            yield line


    thefile = open("/tmp/thefile")
    thelines = iterate_file_lines(thefile)

    for line in thelines:
        print("Outer:", repr(line))

        if is_start(line):
            outer_position = thefile.tell()

            for line in thelines:
                print("Inner:", repr(line))

                if is_end(line):
                    break

            thefile.seek(outer_position)

It's simpler than having two files but probably not faster, "tee" will
almost definitely be way better a choice (unless the subsections can't
fit in memory) and it forfeits being able to change up the order of
these things.

If you want to change up the order to another defined order, you can
think about storing the subsections, but if you want to support
independent iteration you'll need to seek before every "readline"
which is a bit silly.

Basically, read it all into memory like Steven D'Aprano suggested. If
you really don't want to, use "tee". If you can't handle non-constant
memory usage (really? You're reading lines, man) I'd suggest my
method. If you can't handle the inflexibility there, use multiple
files.

There, is that enough choices?



More information about the Python-list mailing list