[SciPy-user] Re: io.read_array slow

[Jordan H. Mantha]

Christian Kristukat wrote:

>Hi,
> I noticed that io.read_array is really slow compared to a python 
>while/readline/split loop (about 5 times slower). I believe to remember 
>that at some time it was written in C but when I looked at the source of 
>scipy 0.3.2 it seems to be pure python. Is there a evident reason why it 
>must be that slow or are there maybe plans to rewrite it in C?
> Regards, 
>Christian


I have noticed this too. I was wrote a fitting script that reads in a 3 
column data file that has 502,000 rows. I first started with io.read_array 
and it took > 10 min. to get the array in. I was reading "Python Scripting 
for Computational Science" by Hans Petter Langtangen and used the 
following:

	data = array([float(x) for x in infile.read().split()], Float)
        data.shape = (len(data)/3,3) 

and it cut down the time to read in the data to ~ 5 s. This was a dramatic 
speed up for me. I know that io.read_array can handle more complicated 
cases, but it doesn't always seem to be the best solution for reading in 
data.

--Jordan Mantha