[SciPy-User] IO of large ASCII table data

[Keith Goodman]

On Tue, Aug 17, 2010 at 11:43 AM, Éric Depagne <edepagne at lcogt.net> wrote:
>>
>> Here's the time it takes to read from my home directory (regular HD):
>> >> f = h5py.File('/home/kg/speed.hdf5')
>> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1
>>
>> 0.0879709720612
>>
>> And here's the time it takes to read from my ram disk:
>> >> f = h5py.File('/dev/shm/speed.hdf5')
>> >> t1=time.time(); a = f['arr'][:]; print time.time() - t1
>>
>> 0.086874961853
>>
> That's impressive.
> Any ascii file can be read with h5py?
> And the can we "convert" the data read into numpy arrays ?

h5py can only read hdf5 files, so you'd have to convert the ascii file
to hdf5. Might be worth it if you read the same ascii file many times.
Even better would be to change the program that is writing the ascii
files to write hdf5 files.

To convert from ascii to hdf5:

arr = np.fromfile(...)
f = h5py.File('converted.hdf5')
f['arr'] = arr