Newbie - converting csv files to arrays in NumPy - Matlab vs. Numpy comparison

[Travis E. Oliphant]

oyekomova wrote:
> Thanks to everyone for their excellent suggestions. I was able to
> acheive the following results with all your suggestions. However, I am
> unable to cross file size of 6 million rows. I would appreciate any
> helpful suggestions on avoiding memory errors. None of the solutions
> posted was able to cross this limit.

Did you try using numpy.fromfile ?

This will not require you to allocate more memory than needed.   If you 
specify a count, it will also not have to re-allocate memory in blocks 
as the array size grows.

It's limitation is that it is not a very sophisticated csv reader (it 
only understands a single separator (plus line-feeds are typically seen 
as a separator).

-Travis