[SciPy-User] IO of large ASCII table data

[Christopher Barker]

Dan Lussier wrote:
> I am looking to read in large (many million rows) ASCII space
> separated tables into numpy arrays.

If it's all space separated, all one data type, and no comment lines or 
anything like that, then np.fromfile(file, sep=' ', dtype=np.float) is 
about as fast as you can get.

fromfile() is very limited for reading text files, and doesn't handle 
error cases well, but if it works, it's fast.

-Chris

> In consulting the documentation the most likely candidate is
> numpy.genfromtext(...).  Is this function pure python or does it rely
> on a C extension as was the case with Miller's TableIO?

it's python, and not all that fast.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov