[SciPy-User] IO of large ASCII table data

Dan Lussier dan.lussier at sjc.ox.ac.uk
Tue Aug 17 13:41:26 EDT 2010


I am looking to read in large (many million rows) ASCII space
separated tables into numpy arrays.

In the past I have heard of people using Miller's TableIO to do this
but was wondering if a similarly fast method has been more recently
integrated into scipy/numpy?

In consulting the documentation the most likely candidate is
numpy.genfromtext(...).  Is this function pure python or does it rely
on a C extension as was the case with Miller's TableIO?

Any advice here would be great as my application could get seriously
bogged down (both time and memory) in reading these files into arrays
if I get onto the wrong track.

Thanks.



More information about the SciPy-User mailing list