[Numpy-discussion] Speeding up loadtxt / savetxt

[Andreas Hilboll]

Hi Stéfan,

> Do you specifically need to store text files?  NumPy's binary storage
> functions (numpy.load and save) are faster.

Yes, I know. But the files I create must be readable by an application
developed in-house at our institude, and that only supports a) ASCII files
or b) some home-grown binary format, which I hate.

> Also, an efficient reader for very simply formatted text is provided
> by numpy.fromfile.

Yes, I heard about it. But the files I have to read have comments in them,
and I didn't find a way to exclude these easily.

Time needed to read a 100M file is ~13 seconds, and to write ~5 seconds.
Which is not too bad, but also still too much ...

Thanks,

Andreas