reading file contents to an array (newbie)
Darren Dale
dd55 at cornell.edu
Wed Jul 7 17:03:42 EDT 2004
> actually, I find the following more readable, and even faster:
>
> from mmap import mmap, MAP_PRIVATE, PROT_READ
> from os import fstat
>
> f = file('test.dat',mode='rt')
> fd = f.fileno()
> m = mmap(fd, fstat(fd).st_size, MAP_PRIVATE, PROT_READ)
>
> data=[]
> while True:
> line = m.readline()
> if not line: break
> data.extend(map(float, line.split()))
>
I will try this out. I am reading three sets of data from a file with
lengthy headers, so it looks like mmap is a really good solution. Thanks
to Jeff and Chris as well for teaching me something new.
One more thing, I am reading into arrays that can be 5000 cells wide,
and arbitrarily long (time-resolved scientific data.) The datafiles are
reorganized such that only 16 columns are listed on a line, and a '\'
character indicates that the row continues on the next line of the file.
Do you have ideas of how to quickly reconstruct these rows? I think mmap
gets me half the way there, but should I try to avoid testing each
readline for the presence of a '\' character?
More information about the Python-list
mailing list