[SciPy-user] Re: io.read_array slow
Jordan H. Mantha
mantha at chem.unr.edu
Tue Sep 27 22:52:02 EDT 2005
Christian Kristukat wrote:
>Hi,
> I noticed that io.read_array is really slow compared to a python
>while/readline/split loop (about 5 times slower). I believe to remember
>that at some time it was written in C but when I looked at the source of
>scipy 0.3.2 it seems to be pure python. Is there a evident reason why it
>must be that slow or are there maybe plans to rewrite it in C?
> Regards,
>Christian
I have noticed this too. I was wrote a fitting script that reads in a 3
column data file that has 502,000 rows. I first started with io.read_array
and it took > 10 min. to get the array in. I was reading "Python Scripting
for Computational Science" by Hans Petter Langtangen and used the
following:
data = array([float(x) for x in infile.read().split()], Float)
data.shape = (len(data)/3,3)
and it cut down the time to read in the data to ~ 5 s. This was a dramatic
speed up for me. I know that io.read_array can handle more complicated
cases, but it doesn't always seem to be the best solution for reading in
data.
--Jordan Mantha
More information about the SciPy-User
mailing list