[SciPy-user] issues while loading scatter data file with load() from pylab...

[Gael Varoquaux]

On Thu, Sep 13, 2007 at 12:40:41PM +0200, fred wrote:
> First question.
> Using load() function from pylab, array returned is a float64.
> Is it possible to directly load it in float32 ?
> I don't need the double precision.
> And I saw nothing with load?

> The issue.

> My scatter data has ~7x1e6 points,
> stored as x, y, z, v per line.

> Using a short C code and fscanf, it takes 12 s and ~240 MB in format 
> double to load it.
> Fine.

> Using load() from pylab to load this file is endless and need more than 
> 1 GB.

Did you try something less "swiss army knife" than pylab.load ? For instance
scipy.io.read_array or something homebaked ? As pylab.load is trying to
accomodate for all sort of weird things, and is very versatile, I bet
something more targetted would be quicker.

Other solution is to store the data in a format better suited for large
data. For instance hdf5 with pytables.

Gaël