reading file contents to an array (newbie)

[John Lenton]

On Wed, 07 Jul 2004 17:03:42 -0400, Darren Dale <dd55 at cornell.edu> wrote:
> One more thing, I am reading into arrays that can be 5000 cells wide,
> and arbitrarily long (time-resolved scientific data.) The datafiles are
> reorganized such that only 16 columns are listed on a line, and a '\'
> character indicates that the row continues on the next line of the file.
> Do you have ideas of how to quickly reconstruct these rows? I think mmap
> gets me half the way there, but should I try to avoid testing each
> readline for the presence of a '\' character?

you can either build the string and then split it again,

    line = m.readline()
    while line.endswith("\\\n"):
        line = line[:-2] + m.readline()

Or, you could build the array bit by bit, which gets a bit messy but
might be faster (you'd have to test it)

    more = True
    while more:
        row = array()
        while True:
            line = m.readline()
            if not line:
                more = False  # goto end
            row.extend(map(float, line.split()[:16]))
            if not line.endswith("\\\n"):
                break
        data.append(row)

as usual in this kind of example, the code is very fragile and you'll
want to generalize it a bit before using it "production".

-- 
John Lenton (jlenton at gmail.com) -- Random fortune:
bash: fortune: command not found