[Matrix-SIG] Reading and handling matrices.

[Konrad Hinsen]

> minimal swapping and even on a 128 Mb machine it was bearable. With Python
> I had to resort to a 256 Mb machine (since, at one point, it has to hold
> the first array, the second array as an array, and also the second array
> as a string while conversion takes place). 

You could have done I/O piecewise:

    chunk_size = 5000 # or whatever you want
    linear = Numeric.ravel(array)
    file = open('binary_data', 'wb')
    i = 0
    while i < len(linear):
       file.write(linear[i:i+chunk_size].tostring())
       i = i + chunk_size
    file.close()

The resulting binary file should be the same, and of course you could
use a similar trick for reading.

> IMHO, binary I/O is a frequent task in data analysis and it would make
> sense to add a tofile() and a fromfile() method for NumPy arrays, even in
> the absence of any efficiency considerations... 

Some routine like the above in Numeric would probably be sufficient,
but I agree it should be there.

On a related issue: somewhat should implement a cPickle interface for
array objects. It's on my to-do list, but unfortunately not very
high up!
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hinsen@cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------